Pawsey Supercomputing Research Centre Status

Resolved: The secondary tape library maintenance was successful on Friday and we've been monitoring it over the weekend.

Identified: Banksia – one of two tape copies unavailable (at risk) The Banksia service is currently in a degraded, “at risk” state as it is operating with only one tape library instead of the standard two. As a result, the secondary copy of files will be unavailable for staging or archiving until Library 2 is restored to service. The primary copy is still available for all data so this should not impact the service. If you experience any issues accessing data please let us know at [email protected]

The issue has been traced to a faulty tape drive in DBA 5. To resolve this, the field engineer will remove the drives from the DBA, replace the faulty unit, and reseat all components. Since DBA 5 is currently blocked, DBA 6 will need to be removed first to allow access for the repair work. This work will take some time but the engineer indicates that in the worst case it will take until Tuesday but they are hoping for resolution today.

Resolved: HPE believe the issue has been resolved and has closed the support case.

They believe the issue was: "Global Flow Control was disabled on the E1000's. Once enabled performance was regained. ClusterStor team working on a fix (CSPROD-18819) to make Global Flow Control enabled all the time moving forward."

Monitoring: HPE made change to the Global Flow Control on scratch and software during January's maintenance, as well as a modification to the configuration of the LAG ports in the Slingshot fabric.

We haven't seen any C_EC_CRIT errors on the login nodes since maintenance, and are continuing to monitor them like hawks.

Monitoring: setonix-04 reported a C_EC_CRIT error yesterday. It is not in the login pool, but HPE are stumped at why this is happening.

Monitoring: HPE rebooted a number of Slingshot switches during maintenance.

We haven't observed any Slingshot errors on the login, data mover or visualisation nodes for 48 hours.

We will continue to monitor.

Investigating: HPE have provided no new information

Investigating: HPE have provided no new information.

Investigating: HPE have provided no new information.

Investigating: HPE have provided no new information.

setonix-08 has slingshot issues. Pawsey is rebooting it.

Investigating: setonix-02 and setonix-03 have been added back to the RR DNS.

Investigating: There appears to be an issue will the Slingshot interfaces in the login nodes in Setonix. We appear to be down to 1 login node in the normal pool of login nodes.

We have had a case open with HPE for weeks, but they appear to be no closer to providing any kind of solution.

Please, please, please, please don't run any computational intensive operations on the login nodes. We have lovely compute nodes for that.

Please be aware that you can log into setonix-workflow.pawsey.org.au and get access to additional "workflow" nodes.