Pawsey Supercomputing Research Centre Status

Monitoring: Banksia was stable last night under load and thus, it is now returned to full service.

Monitoring: We have implemented a fix on all the Banksia servers and routine or usual load, plus test load - that has now been running for the last three hours has been performing well. We will continue to monitor to verify this is the final fix and that we can return the status back to normal operation. Another update will be provided tomorrow AM. Thankyou for bearing with us.

Identified: We are still working on a fix by way of downgrading the Versity software version even further so we are doing this (our Vendor has recommended this to us). There may be intermittent connectivity issues with individual Banksia servers.

Identified: Banksia has suffered a return of issues under load and we are implementing hotfixes across the cluster in a rolling fashion to attempt to fix these, so please bear with us while we work on this with ur external providers.

Investigating: Banksia has been running again under synthetic staging load for most of today and it appears the workarounds we put in place are working. Suffice to say Banksia is again available for use. If you do happen to strike any issues please log a helpdesk ticket.

Investigating: Further work on Banksia has continued with our external support team. While yesterday’s testing showed progress, Banksia remains unstable under load and is not yet able to be returned to service. Given the ongoing fibre channel issues, we are reaching out to additional experts with specialist experience in fibre channel problems to assist with diagnosis and resolution. Work is continuing and we will provide further updates as information comes to hand. Thank you for your continued patience.

Investigating: Work continues of Banksia today with our external support team, we have noted that there was an unstable fibre channel state which we resolved. We are busy trying to bring up Banksia and ensure it remains stable for use.

Investigating: We have worked to determine there's a block device issue with one of the DDN arrays for the shared filesystem. Work is continuing by our support partner over the weekend to test and find the failed unit parts so we can either implement a workaround or replace them with spare new unit parts and get the Banksia system operating for use. Monday will be when you may expect the update on this item.

Investigating: Banksia update - all of Banksia has been rolled back to the pre-maintenance state, unfortunately shared filesystem problems persist. Exact cause identification (to lead to a fix) work continues with the assistance and guidance of our Support Providers on both the filesystem software side and shared storage hardware side. Another update will be provided today.

Investigating: Good Morning, IBM Support have come back with further information overnight and Pawsey Staff will undertake further work on parts of Banksia that include the DDN block storage and FibreChannel switches and another update will be provided later today after this work is executed. Thankyou for your continued patience.

Investigating: Testing two fixes and rolling back the system to pre-maintenance is underway. We have attended to the banksia servers and they are done, however the issues with the shared filesystem remain. We have attempted to roll back the fibre channel switche OS version also however performing this rollback has hit a snag so a P1 case has been opened with IBM USA to escalate to the SAN Switch specialist team for prompt assistance, diagnosis and rollback.

Investigating: Banksia is again experiencing an issue with the shared filesystem storage on one node again. We are working on this presently - we need to do further work with our external support partner to work on a permanent fix to return Banksia to service.

Monitoring: Banksia was stable last night under load and thus, it is now returned to full service.

Monitoring: We have implemented a fix on all the Banksia servers and routine or usual load, plus test load - that has now been running for the last three hours has been performing well. We will continue to monitor to verify this is the final fix and that we can return the status back to normal operation. Another update will be provided tomorrow AM. Thankyou for bearing with us.

Identified: We are still working on a fix by way of downgrading the Versity software version even further so we are doing this (our Vendor has recommended this to us). There may be intermittent connectivity issues with individual Banksia servers.

Identified: Banksia has suffered a return of issues under load and we are implementing hotfixes across the cluster in a rolling fashion to attempt to fix these, so please bear with us while we work on this with ur external providers.

Investigating: Banksia has been running again under synthetic staging load for most of today and it appears the workarounds we put in place are working. Suffice to say Banksia is again available for use. If you do happen to strike any issues please log a helpdesk ticket.

Investigating: Further work on Banksia has continued with our external support team. While yesterday’s testing showed progress, Banksia remains unstable under load and is not yet able to be returned to service. Given the ongoing fibre channel issues, we are reaching out to additional experts with specialist experience in fibre channel problems to assist with diagnosis and resolution. Work is continuing and we will provide further updates as information comes to hand. Thank you for your continued patience.

Investigating: Work continues of Banksia today with our external support team, we have noted that there was an unstable fibre channel state which we resolved. We are busy trying to bring up Banksia and ensure it remains stable for use.

Investigating: We have worked to determine there's a block device issue with one of the DDN arrays for the shared filesystem. Work is continuing by our support partner over the weekend to test and find the failed unit parts so we can either implement a workaround or replace them with spare new unit parts and get the Banksia system operating for use. Monday will be when you may expect the update on this item.

Investigating: Banksia update - all of Banksia has been rolled back to the pre-maintenance state, unfortunately shared filesystem problems persist. Exact cause identification (to lead to a fix) work continues with the assistance and guidance of our Support Providers on both the filesystem software side and shared storage hardware side. Another update will be provided today.

Investigating: Good Morning, IBM Support have come back with further information overnight and Pawsey Staff will undertake further work on parts of Banksia that include the DDN block storage and FibreChannel switches and another update will be provided later today after this work is executed. Thankyou for your continued patience.

Investigating: Testing two fixes and rolling back the system to pre-maintenance is underway. We have attended to the banksia servers and they are done, however the issues with the shared filesystem remain. We have attempted to roll back the fibre channel switche OS version also however performing this rollback has hit a snag so a P1 case has been opened with IBM USA to escalate to the SAN Switch specialist team for prompt assistance, diagnosis and rollback.

Investigating: Banksia is again experiencing an issue with the shared filesystem storage on one node again. We are working on this presently - we need to do further work with our external support partner to work on a permanent fix to return Banksia to service.

Monitoring: Banksia was stable last night under load and thus, it is now returned to full service.

Monitoring: We have implemented a fix on all the Banksia servers and routine or usual load, plus test load - that has now been running for the last three hours has been performing well. We will continue to monitor to verify this is the final fix and that we can return the status back to normal operation. Another update will be provided tomorrow AM. Thankyou for bearing with us.

Identified: We are still working on a fix by way of downgrading the Versity software version even further so we are doing this (our Vendor has recommended this to us). There may be intermittent connectivity issues with individual Banksia servers.

Identified: Banksia has suffered a return of issues under load and we are implementing hotfixes across the cluster in a rolling fashion to attempt to fix these, so please bear with us while we work on this with ur external providers.

Investigating: Banksia has been running again under synthetic staging load for most of today and it appears the workarounds we put in place are working. Suffice to say Banksia is again available for use. If you do happen to strike any issues please log a helpdesk ticket.

Investigating: Further work on Banksia has continued with our external support team. While yesterday’s testing showed progress, Banksia remains unstable under load and is not yet able to be returned to service. Given the ongoing fibre channel issues, we are reaching out to additional experts with specialist experience in fibre channel problems to assist with diagnosis and resolution. Work is continuing and we will provide further updates as information comes to hand. Thank you for your continued patience.

Investigating: Work continues of Banksia today with our external support team, we have noted that there was an unstable fibre channel state which we resolved. We are busy trying to bring up Banksia and ensure it remains stable for use.

Investigating: We have worked to determine there's a block device issue with one of the DDN arrays for the shared filesystem. Work is continuing by our support partner over the weekend to test and find the failed unit parts so we can either implement a workaround or replace them with spare new unit parts and get the Banksia system operating for use. Monday will be when you may expect the update on this item.

Investigating: Banksia update - all of Banksia has been rolled back to the pre-maintenance state, unfortunately shared filesystem problems persist. Exact cause identification (to lead to a fix) work continues with the assistance and guidance of our Support Providers on both the filesystem software side and shared storage hardware side. Another update will be provided today.

Investigating: Good Morning, IBM Support have come back with further information overnight and Pawsey Staff will undertake further work on parts of Banksia that include the DDN block storage and FibreChannel switches and another update will be provided later today after this work is executed. Thankyou for your continued patience.

Investigating: Testing two fixes and rolling back the system to pre-maintenance is underway. We have attended to the banksia servers and they are done, however the issues with the shared filesystem remain. We have attempted to roll back the fibre channel switche OS version also however performing this rollback has hit a snag so a P1 case has been opened with IBM USA to escalate to the SAN Switch specialist team for prompt assistance, diagnosis and rollback.

Investigating: Banksia is again experiencing an issue with the shared filesystem storage on one node again. We are working on this presently - we need to do further work with our external support partner to work on a permanent fix to return Banksia to service.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: The issue affecting access to the CARTA visualisation service on Setonix has been resolved, and the service is now operational. The underlying cause is still being investigated.We apologise for any inconvenience caused.

Investigating: We are currently investigating an issue affecting access to the CARTA visualisation service on Setonix.

Users may experience a "502 Bad Gateway" error when accessing https://vis.pawsey.org.au/carta/. Further updates will be provided as information becomes available.