Brown University Status

Resolved: This incident has been resolved.

Investigating: We are aware of Listserv Email Groups service (listserv.brown.edu) is experiencing some issues possibly impacting confirmation request emails alongside other mail delivery. OIT is investigating.

Resolved: All services have been restored and are stable.

Additional services may have been degraded or unresponsive during this outage including:

• Academic Coaching
• Advising Sidekick (ASK)
• Computer Science systems
• CourseTools
• Faculty Activity Reporting
• Faculty Hiring
• LibreChat AI Chat
• MyAccount
• MyBrown
• Scan to Ricoh Printers
• Travelsafe
• UFunds

OIT will continue to monitor all affected services. Please report any further problems to OIT.

Monitoring: Some Oscar users have had some failed jobs due to this outage. Please resubmit your Oscar job if it appears to have failed between 7 and 8:45 a.m this morning.

Monitoring: The root problem has been identified and services are returning to stable. If you are unable to access files on network drives or folders on files22.brown.edu, you may need to log out and log back in to your computer to refresh your connections. OIT is continuing to validate all services.

Identified: We have identified the issue and are working with the vendor on a resolution.

Investigating: We are currently experiencing an outage affecting use of files22.brown.edu, including Research and Departmental shares. High performance computing, including Oscar and Globus File Transfers are also experiencing a degradation of service.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently experiencing connectivity issues with files22.brown.edu

OIT is investigating this issue.

Postmortem: The Office of Information Technology (OIT) is sharing this after-action report to provide more details about a major university-scale wireless networking incident on Wednesday, December 3, and Thursday, December 4, 2025. This was an incident that had significant impact on teaching, learning, research, student life, and operations over two workdays. This occurred at a busy time near the close of the academic semester and the calendar year, and created a number of real problems for students, faculty, and staff. Given the critical and widespread impact of these problems, OIT orchestrated a large coordinated response effort at the highest priority to restore services. After stabilizing wireless networking and conducting additional analysis and improvements, we are now sharing the full details about what happened, what we did to resolve it, and what we will be doing going forward to reduce the chance of it happening again.

What services were affected? For how long?

This major incident affected wireless networking services for our Brown, Brown-Guest, and Eduroam networks. Service was degraded or unavailable from approximately 10 a.m. to 2 p.m. on Wednesday, December 3, and again on Thursday, December 4, from approximately 9 a.m. until approximately 5:30 p.m. The worst part of the incident extended from approximately 2 to 5 p.m. on Thursday, when a large number of locations could not use any wireless networks.

Who was affected?

This major incident affected all Brown faculty, staff, students, and guests. Throughout this incident, different people and spaces were affected sporadically and at different times.

What happened?

Before the start of business hours on Wednesday, December 3, OIT completed a scheduled change to replace a central hardware component of our campus wireless networking infrastructure. This upgrade was completed on time, and our change validation tests were successful. However, over the course of the day, the new hardware experienced a series of failures in managing the very high number of concurrent wireless connections that we experience at Brown.‌

As soon as OIT learned about the issue, we initiated an investigation, posted a public alert to our service status page at 10:33 a.m., and immediately reached out to our network hardware vendor to report the problem at the highest possible level of severity, which activates their highest tier of engineering support. They were very responsive and acknowledged the issue right away. We worked with our vendor throughout the entire incident, staying closely engaged as we tested, validated, and applied changes.‌

Some people were able to maintain a network connection by switching to a wired network port or by using a personal wifi hotspot on a mobile device, but we recognize that most members of the Brown community do not have access to these workarounds.

As the problem grew in impact, we made a decision at 2 p.m. to reload the primary switch supporting our wireless network to attempt to restore services. This appeared to resolve the problem, and we updated our public alert. We continued to monitor for further problems. Service remained stable through the business day and evening.‌

The following day, on Thursday, December 4, the problem started reappearing sporadically around 9 a.m., and started growing again in impact. We updated our public alert and reconvened our OIT response team and our vendor. The problems evolved through the course of the day, and at 1:53 p.m., a critical device crashed unexpectedly. This caused the issue to escalate quickly, spreading to affect 900 of the 6,600 wireless access points across many critical areas of campus.‌

We continued through Thursday afternoon to restore services, and determined they were stable again by 5 p.m. We updated our public alert and monitored performance closely through the evening and the following days.

What is OIT doing about it?

After extensive analysis, we learned that these wide-scale service problems ultimately resulted from an unexpected combination of the atypically high number of wireless device connections at Brown and an unanticipated vendor software defect.‌

It is common for students, faculty, and staff to have multiple devices, such as laptops, mobile phones, tablets, and smartwatches all connected to Brown wireless networking. For example, we typically observe an estimated 22,000 devices connecting every day during the academic term. Additionally, the physical density of people in campus spaces can lead to many devices being concentrated in a small area. We took all this into account in planning our scheduled equipment replacement and worked with our network hardware vendor to size the new equipment appropriately in advance. Unfortunately, because of an unexpected software defect, the equipment didn’t perform as expected, resulting in the large number of dropped connections.‌

While the new hardware was initially specified to meet the requirements of our networking environment, we could not predict that these custom configurations would be required to handle our unique volume of devices and traffic at Brown.‌

OIT has completed additional work with our network hardware vendor to resolve the unanticipated software defect and to design and implement custom configurations, which should better serve the needs of our campus environment.

OIT will continue to monitor the wireless networking performance closely to ensure everything remains stable.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we believe the issue to be resolved.

In the event WiFi issues are ongoing, please contact the Service Center at (401) 863-4357.

Investigating: Wifi has been down since 2:05pm in many spaces on College Hill and the Jewelry District.

OIT has escalated for highest-urgency support with our vendor and are investigating further.

Investigating: We experienced a widespread outage from approximately 1:53 to 1:55 and will continue to investigate intermittent problems.

Investigating: OIT is investigating sporadic WiFi connectivity issues affecting multiple areas across College Hill and Jewelry District areas.

We advise the use of a mobile hotspot if available, or connecting to a wired network connection and disabling WiFi on your device where that is available.

Monitoring: The solution that was implemented has been effective and services should be restored. OIT will continue to monitor throughout the day.

Investigating: OIT is currently investigating a fix for the ongoing issue and has implemented a potential solution at 2:00PM. Please be aware that you may still be experiencing issues while we're monitoring network connectivity.

Investigating: We are currently aware of an ongoing Intermittent Network Issue regarding Brown Networks. OIT is investigating.

Resolved: Emery-Wooley experienced a temporary network outage on Wednesday, December 10 from 03:30 p.m. to 03:34 p.m. During this time, users may have experienced network connectivity loss. This issue is now resolved.

Resolved: All services have been restored and are stable.

Additional services may have been degraded or unresponsive during this outage including:

• Academic Coaching
• Advising Sidekick (ASK)
• Computer Science systems
• CourseTools
• Faculty Activity Reporting
• Faculty Hiring
• LibreChat AI Chat
• MyAccount
• MyBrown
• Scan to Ricoh Printers
• Travelsafe
• UFunds

OIT will continue to monitor all affected services. Please report any further problems to OIT.

Monitoring: Some Oscar users have had some failed jobs due to this outage. Please resubmit your Oscar job if it appears to have failed between 7 and 8:45 a.m this morning.

Monitoring: The root problem has been identified and services are returning to stable. If you are unable to access files on network drives or folders on files22.brown.edu, you may need to log out and log back in to your computer to refresh your connections. OIT is continuing to validate all services.

Identified: We have identified the issue and are working with the vendor on a resolution.

Investigating: We are currently experiencing an outage affecting use of files22.brown.edu, including Research and Departmental shares. High performance computing, including Oscar and Globus File Transfers are also experiencing a degradation of service.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently experiencing connectivity issues with files22.brown.edu

OIT is investigating this issue.

Resolved: All services have been restored and are stable.

Additional services may have been degraded or unresponsive during this outage including:

• Academic Coaching
• Advising Sidekick (ASK)
• Computer Science systems
• CourseTools
• Faculty Activity Reporting
• Faculty Hiring
• LibreChat AI Chat
• MyAccount
• MyBrown
• Scan to Ricoh Printers
• Travelsafe
• UFunds

OIT will continue to monitor all affected services. Please report any further problems to OIT.

Monitoring: Some Oscar users have had some failed jobs due to this outage. Please resubmit your Oscar job if it appears to have failed between 7 and 8:45 a.m this morning.

Monitoring: The root problem has been identified and services are returning to stable. If you are unable to access files on network drives or folders on files22.brown.edu, you may need to log out and log back in to your computer to refresh your connections. OIT is continuing to validate all services.

Identified: We have identified the issue and are working with the vendor on a resolution.

Investigating: We are currently experiencing an outage affecting use of files22.brown.edu, including Research and Departmental shares. High performance computing, including Oscar and Globus File Transfers are also experiencing a degradation of service.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently experiencing connectivity issues with files22.brown.edu

OIT is investigating this issue.

Resolved: All services have been restored and are stable.

Additional services may have been degraded or unresponsive during this outage including:

• Academic Coaching
• Advising Sidekick (ASK)
• Computer Science systems
• CourseTools
• Faculty Activity Reporting
• Faculty Hiring
• LibreChat AI Chat
• MyAccount
• MyBrown
• Scan to Ricoh Printers
• Travelsafe
• UFunds

OIT will continue to monitor all affected services. Please report any further problems to OIT.

Monitoring: Some Oscar users have had some failed jobs due to this outage. Please resubmit your Oscar job if it appears to have failed between 7 and 8:45 a.m this morning.

Monitoring: The root problem has been identified and services are returning to stable. If you are unable to access files on network drives or folders on files22.brown.edu, you may need to log out and log back in to your computer to refresh your connections. OIT is continuing to validate all services.

Identified: We have identified the issue and are working with the vendor on a resolution.

Investigating: We are currently experiencing an outage affecting use of files22.brown.edu, including Research and Departmental shares. High performance computing, including Oscar and Globus File Transfers are also experiencing a degradation of service.

Investigating: We are continuing to investigate this issue.

Investigating: We are currently experiencing connectivity issues with files22.brown.edu

OIT is investigating this issue.

Resolved: This incident has been resolved.

Investigating: Rhode Island Energy is experiencing a power outage.

This currently affects the following locations: 1 Davol 110 Elm 70 Ship

Rhode Island Energy reports power is to be expected to be restored around 7:30 AM.