Exclaimer Cloud Status

Postmortem: Issue: Degraded Service in both Client and Server-side signatures for UK and EU regions. 

Incident Length: 1 hour and 2 minutes 

Incident Date: 27/03/2025, UTC 11:23 – 12:25 27/03/2025 

Incident Status: Resolved 

Summary 

Customers within the EU and UK experienced failures when trying to retrieve their signatures from the Exclaimer Add-in and would encounter delays in message delivery. 

Root Cause 

Initial implementation work of a new feature within Exclaimer branding system resulted in an unhandled exception whilst the system looked to generate signature templates. The coinciding exception that occurred resulted in the message being queued for retry in applying the signature. This caused a buildup of requests for signatures to be applied to messages. 

Mitigation 

Increased resources were provided to impacted regions to reduce the buildup of requests, and the changes were rolled back to resolve the issue. 

Incident Timeline 

11:17 – Implementation work for a new branding feature was completed and enabled within Exclaimer Cloud. 

11:17 - Initial warnings were generated at this time but investigation indicated none of these were customer impacting. 

11:22 – Additional alerting advised to an increase in wait time for Exclaimer Add-in applied signatures. 

11:30 – SMTP requests to Exclaimer services begin to respond to connection attempts with a deferral due to a lack of signature server processing availability, mail flow alerts are raised due to increased processing time of messages. 

11:37 – Exclaimer engineers begin to investigate the issue. 

11:47 – A recent deployment to the signature template analytic services is isolated as a potential cause of the issue. 

11:50 – The deployment is rolled back within the UK region. Engineers identify that the error rate is unchanged. 11:51 – Implementation work was disabled but left present within Exclaimer systems.  

11:59 – Resourcing within UK and EU regions are increased to process the buildup of signature requests. 

12:12 – Exclaimer Engineers confirm that the service is now recovering. Services are continued to be monitored to ensure that the issue does not reoccur. 

12:17 – Mail flow duration returned to normal levels. 

12:25 – Exclaimer Add-in requests no longer reported any related errors. 

12:49 – Exclaimer Engineers identify the cause of the issue and rollback the feature.

Resolved: After extended monitoring, Exclaimer Engineers have confirmed that signatures are now being applied to messages when sent or whilst composing. A full postmortem will be provided shortly.

If you are continuing to see issues, please raise a ticket with our Support Team who will be able to assist.

Monitoring: We are now seeing signatures being applied through our Add-in and routed through Exclaimer services as expected now. Engineering are continuing to monitor and investigate the cause.

Investigating: We are continuing to investigate this issue.

Investigating: Exclaimer Engineers are currently investigating a possible service alert that may be causing an impact with signature application at this time. Once we have more information from our Engineers this will be provided

Postmortem: Issue: Degraded Service in both Client and Server-side signatures for UK and EU regions. 

Incident Length: 1 hour and 2 minutes 

Incident Date: 27/03/2025, UTC 11:23 – 12:25 27/03/2025 

Incident Status: Resolved 

Summary 

Customers within the EU and UK experienced failures when trying to retrieve their signatures from the Exclaimer Add-in and would encounter delays in message delivery. 

Root Cause 

Initial implementation work of a new feature within Exclaimer branding system resulted in an unhandled exception whilst the system looked to generate signature templates. The coinciding exception that occurred resulted in the message being queued for retry in applying the signature. This caused a buildup of requests for signatures to be applied to messages. 

Mitigation 

Increased resources were provided to impacted regions to reduce the buildup of requests, and the changes were rolled back to resolve the issue. 

Incident Timeline 

11:17 – Implementation work for a new branding feature was completed and enabled within Exclaimer Cloud. 

11:17 - Initial warnings were generated at this time but investigation indicated none of these were customer impacting. 

11:22 – Additional alerting advised to an increase in wait time for Exclaimer Add-in applied signatures. 

11:30 – SMTP requests to Exclaimer services begin to respond to connection attempts with a deferral due to a lack of signature server processing availability, mail flow alerts are raised due to increased processing time of messages. 

11:37 – Exclaimer engineers begin to investigate the issue. 

11:47 – A recent deployment to the signature template analytic services is isolated as a potential cause of the issue. 

11:50 – The deployment is rolled back within the UK region. Engineers identify that the error rate is unchanged. 11:51 – Implementation work was disabled but left present within Exclaimer systems.  

11:59 – Resourcing within UK and EU regions are increased to process the buildup of signature requests. 

12:12 – Exclaimer Engineers confirm that the service is now recovering. Services are continued to be monitored to ensure that the issue does not reoccur. 

12:17 – Mail flow duration returned to normal levels. 

12:25 – Exclaimer Add-in requests no longer reported any related errors. 

12:49 – Exclaimer Engineers identify the cause of the issue and rollback the feature.

Resolved: After extended monitoring, Exclaimer Engineers have confirmed that signatures are now being applied to messages when sent or whilst composing. A full postmortem will be provided shortly.

If you are continuing to see issues, please raise a ticket with our Support Team who will be able to assist.

Monitoring: We are now seeing signatures being applied through our Add-in and routed through Exclaimer services as expected now. Engineering are continuing to monitor and investigate the cause.

Investigating: We are continuing to investigate this issue.

Investigating: Exclaimer Engineers are currently investigating a possible service alert that may be causing an impact with signature application at this time. Once we have more information from our Engineers this will be provided

Postmortem: Issue: Degraded Service in both Client and Server-side signatures for UK and EU regions. 

Incident Length: 1 hour and 2 minutes 

Incident Date: 27/03/2025, UTC 11:23 – 12:25 27/03/2025 

Incident Status: Resolved 

Summary 

Customers within the EU and UK experienced failures when trying to retrieve their signatures from the Exclaimer Add-in and would encounter delays in message delivery. 

Root Cause 

Initial implementation work of a new feature within Exclaimer branding system resulted in an unhandled exception whilst the system looked to generate signature templates. The coinciding exception that occurred resulted in the message being queued for retry in applying the signature. This caused a buildup of requests for signatures to be applied to messages. 

Mitigation 

Increased resources were provided to impacted regions to reduce the buildup of requests, and the changes were rolled back to resolve the issue. 

Incident Timeline 

11:17 – Implementation work for a new branding feature was completed and enabled within Exclaimer Cloud. 

11:17 - Initial warnings were generated at this time but investigation indicated none of these were customer impacting. 

11:22 – Additional alerting advised to an increase in wait time for Exclaimer Add-in applied signatures. 

11:30 – SMTP requests to Exclaimer services begin to respond to connection attempts with a deferral due to a lack of signature server processing availability, mail flow alerts are raised due to increased processing time of messages. 

11:37 – Exclaimer engineers begin to investigate the issue. 

11:47 – A recent deployment to the signature template analytic services is isolated as a potential cause of the issue. 

11:50 – The deployment is rolled back within the UK region. Engineers identify that the error rate is unchanged. 11:51 – Implementation work was disabled but left present within Exclaimer systems.  

11:59 – Resourcing within UK and EU regions are increased to process the buildup of signature requests. 

12:12 – Exclaimer Engineers confirm that the service is now recovering. Services are continued to be monitored to ensure that the issue does not reoccur. 

12:17 – Mail flow duration returned to normal levels. 

12:25 – Exclaimer Add-in requests no longer reported any related errors. 

12:49 – Exclaimer Engineers identify the cause of the issue and rollback the feature.

Resolved: After extended monitoring, Exclaimer Engineers have confirmed that signatures are now being applied to messages when sent or whilst composing. A full postmortem will be provided shortly.

If you are continuing to see issues, please raise a ticket with our Support Team who will be able to assist.

Monitoring: We are now seeing signatures being applied through our Add-in and routed through Exclaimer services as expected now. Engineering are continuing to monitor and investigate the cause.

Investigating: We are continuing to investigate this issue.

Investigating: Exclaimer Engineers are currently investigating a possible service alert that may be causing an impact with signature application at this time. Once we have more information from our Engineers this will be provided