Pawsey Supercomputing Research Centre Status

Resolved: This incident has been resolved.

Monitoring: Automatic QoS changes for accounts that have exceeded allocation are now running again. Staff have identified a few users who may experience 'InvalidQOS' for jobs and will work on a resolution shortly.

Investigating: There was an issue with the quarterly automatic reset overnight of Slurm priorities for projects who had exceeded their allocation in Q2. Staff are working on a resolution and jobs can continue to be submitted and will run as resources are available. Once we have resolved the issue, any jobs which have the incorrect QoS will be automatically updated.

Resolved: Resources have been rebalanced to their optimal configuration. HPE have collected logs from the filesystem and will provide Pawsey a root cause analysis in due course.

Monitoring: Failover has been completed

• Resource has been restored to the nominal High Availability pair
• It will monitored

Investigating: It appears we have a slow Object Storage Server (OSS) serving up part of /scratch. The HPE engineers are going to failover resources to the partner OSS to allow them to reboot the OSS. There will be a brief pause in access to /scratch as clients reconnect.

Please be aware that the /scratch filesystem is becoming increasingly full. We would appreciate the assistance of researchers in removing unused files from the filesystem to ensure it is accessible to all.

Monitoring: Failover has completed by the Engineers

• High availability resources resources has been restored to the original nodes
• System IO load looks nominal
• Logs / Dump is being submitted to Vendor Engineers for analysis

Identified: The issue has been identified and a fix is being implemented.

Investigating: Investigating - An Object Storage Server (OSS) are showing the same symptoms to a previous detected server which is slow in response.

The HPE engineers are going to failover resources to the partner OSS to allow them to reboot the OSS. There will be a brief pause in access to /scratch as clients reconnect.

Please be aware that the /scratch filesystem is becoming increasingly full. We would appreciate the assistance of researchers in removing unused files from the filesystem to ensure it is accessible to all

Resolved: The issue with Staging is resolved. Staging is continuing.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The root cause appears to have been identified by our vendor and we need to take Banksia offline immediately to implement the required fix.

Investigating: Currently there is an issue affecting file staging - only a portion of requests are succeeding. It's is recommended not to use Banks9a at this time.

Resolved: The issue with Staging is resolved. Staging is continuing.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The root cause appears to have been identified by our vendor and we need to take Banksia offline immediately to implement the required fix.

Investigating: Currently there is an issue affecting file staging - only a portion of requests are succeeding. It's is recommended not to use Banks9a at this time.

Resolved: The issue with Staging is resolved. Staging is continuing.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The root cause appears to have been identified by our vendor and we need to take Banksia offline immediately to implement the required fix.

Investigating: Currently there is an issue affecting file staging - only a portion of requests are succeeding. It's is recommended not to use Banks9a at this time.