Farm HPC cluster Status

Resolved: nas-6-0 has been brought back into service and is correctly serving data again.

Investigating: nas-6-0 is currently having issues and not serving NFS data. Admins are investigating.

Monitoring: The reboot of nas-4-1 seems to have cleared up the issues with the login node. At this point, a reboot of the login node does not appear to be required.

Please open a ticket if you are still having problems.

Identified: Unfortunately, the same problem reoccurred, so nas-4-1 and Farm's login node will be rebooted in a few minutes. Jobs running with sbatch should just pause and resume when the NFS server recovers. srun jobs will be killed when the login node is rebooted.

Admins are still working to determine the combination of client and server kernel versions that is tickling this kernel bug.

Monitoring: Both nas-4-1 and the login node have been downgraded to a kernel version that should not contain this bug. Admins are monitoring.

Identified: System administrators will be restarting NAS-4-1 storage and the Farm login node. Emergency downtime should be approximately one hour.

Identified: The same kernel bug on nas-4-1 has been identified as the root-cause. Admins are investigating the best path forward.

Investigating: System administrators are still seeing hung processes on Farm storage and are continuing to look into the root cause.

Resolved: nas-6-0 has been brought back into service and is correctly serving data again.

Investigating: nas-6-0 is currently having issues and not serving NFS data. Admins are investigating.

Monitoring: The reboot of nas-4-1 seems to have cleared up the issues with the login node. At this point, a reboot of the login node does not appear to be required.

Please open a ticket if you are still having problems.

Identified: Unfortunately, the same problem reoccurred, so nas-4-1 and Farm's login node will be rebooted in a few minutes. Jobs running with sbatch should just pause and resume when the NFS server recovers. srun jobs will be killed when the login node is rebooted.

Admins are still working to determine the combination of client and server kernel versions that is tickling this kernel bug.

Monitoring: Both nas-4-1 and the login node have been downgraded to a kernel version that should not contain this bug. Admins are monitoring.

Identified: System administrators will be restarting NAS-4-1 storage and the Farm login node. Emergency downtime should be approximately one hour.

Identified: The same kernel bug on nas-4-1 has been identified as the root-cause. Admins are investigating the best path forward.

Investigating: System administrators are still seeing hung processes on Farm storage and are continuing to look into the root cause.