OpenSRS Status

Resolved: The .EU connection issue has been resolved. All services are online.

Start Time: 11/20/2024 09:15 UTC End Time: 11/20/2024 11:15 UTC

Investigating: We are continuing to have connection issues with the .EU Registry. We are waiting for any updates, and will keep you informed as they're made available to us.

Investigating: We are currently investigating this issue.

Identified: We have been notified by the .EU Registry, they are currently experiencing technical issues and are working to resolve them as quickly as possible.

We will keep you informed of any updates as they work to identify and resolve this issue.

Investigating: We are currently experiencing connection issues with the .EU registry. Our engineering team is investigating the issue.

This affects registrations, renewals, and management of .EU domains. Existing domains will continue to resolve normally.

Resolved: This incident has been resolved. Services are fully recovered.

Start Time: 11/14/2024 13:26 UTC End Time: 11/14/2024 17:36 UTC

Monitoring: We are continuing to monitor the results. Services remain stable.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Some users may continue to have intermittent issues accessing Webmail on Cluster A. This may impact users logging into Webmail, and have intermittent sluggish response while using the webmail service.

We will keep you informed as we work on this, and appreciate your patience.

Investigating: We have received reports of some users having intermittent issues accessing webmail, sending and recieving. Teams have been engaged and are investigating.

Resolved: This incident has been resolved.

Monitoring: Our email team has applied a fix for the earlier issues, and all impacted services appear to be restored. We will continue to monitor this incident and provide any further updates. Thank you for your patience.

Investigating: We are currently receiving an increasing number of reports from users experiencing webmail login/loading failures. Our engineering team is actively investigating the issue. We'll provide more updates as they become available.

Resolved: The issue experienced for a small group of email accounts is now resolved. The email service is back to expected levels, with Pop/IMAP and webmail functioning as intended.

Start Time: 11/18/2024 10:39 UTC End Time: 11/18/2024 19:09 UTC

Investigating: We are currently investigating reports of some customers having connectivity issues affecting account access.

We will investigate this and follow up with updates.

Monitoring: Mail services have remained the same for the last hour, but still intermittently slower than our target response times.

We are continuing to work on this and monitor the results, and will post updates as they are available.

Monitoring: The Pop/IMAP connections are currently returning to expected levels for both mail clients and webmail access.

We are monitoring this and will provide updates.

Investigating: This issue is confirmed to be affecting Cluster A mail client connections as well, intermittent issues syncing mail clients for sending and receiving.

We are continuing to look into this, and thank you for your patience.

Investigating: Our efforts to isolate the problem are ongoing. Your patience is greatly appreciated.

Investigating: Our webmail service is intermittently unresponsive. Our Mail Administrators are currently investigating this.

Thank you for your patience.

Postmortem: Wed, Nov 13, 2024 8:00pm - Tue, Nov 19, 4:00pm EST

Description

On Nov 13, 8:00am EST , we had a hardware failure which caused IMAP and Webmail services to fail on one of our mailstores.

The failure and the normal inbound flow of requests resulted in an unexpected increase in the number of in-flight requests, which caused performance issues on the surviving members of the cluster. The net result was a degraded service level that did not self-remedy after the failover as expected.

This resulted in a subset of users unable to access their mailbox during the ongoing maintenance.

The increase in in-flight requests as the peak hours approached further compounded the issue, affecting the load on other platform components. This resulted in issues that affected normal access to IMAP and Webmail services.

Root Causes Found

The cause was due to hardware failure compounded with an increase of workload demands on the surviving network elements.

Solution

In order to recover from the peak backlog, we had to throttle connections and slowly enable service to stabilize the cluster.

Post Mortem

To further address recurrence of issues like this, we have modified the way in which users are assigned to different platform components. We have begun work to transfer user mailboxes to other elements to improve the resource requirements across the board.

Resolved: We are very sorry for the disruption. Thank you again for your patience. Service is once again working as expected.

Start Time: 11/19/2024 17:10 UTC End Time: 11/19/2024 20:42 UTC Duration: 3 hours, 32 minutes

Monitoring: A fix has been implemented for this issue. We're monitoring the results now.

Investigating: We are getting reports that IMAP and POP services are also affected. Please standby while we continue to investigate.

Investigating: We are investigating an issue preventing some users on Cluster A from logging into Webmail. HostedEmail is experiencing issues resulting in delayed access to webmail. IMAP and POP services are unaffected. Our engineering team has been engaged.

We will provide an update once we have additional information.

Resolved: Thank you for your patience. Our team has fixed the issues and tests have been done, all services are functional and working as expected now.

Investigating: Our team is still working on this issue, updates will be provided as they become available. There is currently no ETA at the moment.

Investigating: One of our mailstores on HostedEmail cluster A has been disabled due to maintenance work. Inboxes on this mailstore will be inaccessible until work is completed.

Resolved: This incident has been resolved. Services are fully recovered.

Start Time: 11/14/2024 13:26 UTC End Time: 11/14/2024 17:36 UTC

Monitoring: We are continuing to monitor the results. Services remain stable.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: Some users may continue to have intermittent issues accessing Webmail on Cluster A. This may impact users logging into Webmail, and have intermittent sluggish response while using the webmail service.

We will keep you informed as we work on this, and appreciate your patience.

Investigating: We have received reports of some users having intermittent issues accessing webmail, sending and recieving. Teams have been engaged and are investigating.

Resolved: The issue experienced for a small group of email accounts is now resolved. The email service is back to expected levels, with Pop/IMAP and webmail functioning as intended.

Start Time: 11/18/2024 10:39 UTC End Time: 11/18/2024 19:09 UTC

Investigating: We are currently investigating reports of some customers having connectivity issues affecting account access.

We will investigate this and follow up with updates.

Monitoring: Mail services have remained the same for the last hour, but still intermittently slower than our target response times.

We are continuing to work on this and monitor the results, and will post updates as they are available.

Monitoring: The Pop/IMAP connections are currently returning to expected levels for both mail clients and webmail access.

We are monitoring this and will provide updates.

Investigating: This issue is confirmed to be affecting Cluster A mail client connections as well, intermittent issues syncing mail clients for sending and receiving.

We are continuing to look into this, and thank you for your patience.

Investigating: Our efforts to isolate the problem are ongoing. Your patience is greatly appreciated.

Investigating: Our webmail service is intermittently unresponsive. Our Mail Administrators are currently investigating this.

Thank you for your patience.

Postmortem: Wed, Nov 13, 2024 8:00pm - Tue, Nov 19, 4:00pm EST

Description

On Nov 13, 8:00am EST , we had a hardware failure which caused IMAP and Webmail services to fail on one of our mailstores.

The failure and the normal inbound flow of requests resulted in an unexpected increase in the number of in-flight requests, which caused performance issues on the surviving members of the cluster. The net result was a degraded service level that did not self-remedy after the failover as expected.

This resulted in a subset of users unable to access their mailbox during the ongoing maintenance.

The increase in in-flight requests as the peak hours approached further compounded the issue, affecting the load on other platform components. This resulted in issues that affected normal access to IMAP and Webmail services.

Root Causes Found

The cause was due to hardware failure compounded with an increase of workload demands on the surviving network elements.

Solution

In order to recover from the peak backlog, we had to throttle connections and slowly enable service to stabilize the cluster.

Post Mortem

To further address recurrence of issues like this, we have modified the way in which users are assigned to different platform components. We have begun work to transfer user mailboxes to other elements to improve the resource requirements across the board.

Resolved: We are very sorry for the disruption. Thank you again for your patience. Service is once again working as expected.

Start Time: 11/19/2024 17:10 UTC End Time: 11/19/2024 20:42 UTC Duration: 3 hours, 32 minutes

Monitoring: A fix has been implemented for this issue. We're monitoring the results now.

Investigating: We are getting reports that IMAP and POP services are also affected. Please standby while we continue to investigate.

Investigating: We are investigating an issue preventing some users on Cluster A from logging into Webmail. HostedEmail is experiencing issues resulting in delayed access to webmail. IMAP and POP services are unaffected. Our engineering team has been engaged.

We will provide an update once we have additional information.