Redox Engine Status

Postmortem: ## Summary

On September 8, an outage at a third-party infrastructure provider we rely on for message processing disrupted our platform for approximately 40 minutes, delaying asynchronous processing and returning errors on synchronous requests. Additionally, this slowed down our processing of transaction artifacts resulting in delayed visibility in our dashboard. Our team executed an emergency failover to backup infrastructure and restored normal processing the same afternoon. No data was lost.

What Happened

In the early afternoon of September 8, a third-party infrastructure provider that Redox relies on for message processing experienced an outage. This caused approximately 40 minutes where our platform was not processing traffic. Our team immediately began an emergency failover, redirecting message processing to backup infrastructure to restore service while the third-party issue was ongoing.

This failover was successful. Processing was restored, and by 4:05 PM Central our team confirmed the platform had caught up and declared the incident resolved.

Impact

Customers experienced approximately 40 minutes of delayed asynchronous processing and errors on synchronous requests while the third-party outage was active. Additionally, our dashboard and log visibility was delayed for approximately 2 and a half hours.

How We Resolved It

Our team redirected message processing to backup infrastructure within minutes of confirming the outage. Processing was fully restored and the dashboard/log visibility was caught up by 4:05 PM Central the same day.

What We're Doing About This

• Re-evaluate failover playbooks, so scaling, monitoring, and cleanup steps are rehearsed and routine
• We are actively migrating away from this third-party provider

We appreciate your patience during this incident. If you have questions or concerns, please reach out to your Redox account team.

‌

The emergency failover used to resolve this incident led to a second, separate issue affecting dashboard and log visibility. See [Incident 2: Dashboard & Log Visibility Delay] for details.

Resolved: The incident has been resolved. Please do not hesitate to reach out to [email protected] if you are continuing to run into anything unexpected.

Monitoring: We applied a fix at 2:23pm Central and processing appears to be operating as normal. We are still seeing delays in log visibility. We will continue to monitor to verify that the issues are fully resolved with our 3rd party vendor.

Investigating: Our automated monitoring tools detected a major outage affecting the API and main site/dashboard at 1:34pm Central. Our team is currently investigating the cause and working on a solution.

Please work with your teams to implement downtime procedures.

If you have any additional questions, please notify us at [email protected].

Postmortem: ## Summary

On September 8, an outage at a third-party infrastructure provider we rely on for message processing disrupted our platform for approximately 40 minutes, delaying asynchronous processing and returning errors on synchronous requests. Additionally, this slowed down our processing of transaction artifacts resulting in delayed visibility in our dashboard. Our team executed an emergency failover to backup infrastructure and restored normal processing the same afternoon. No data was lost.

What Happened

In the early afternoon of September 8, a third-party infrastructure provider that Redox relies on for message processing experienced an outage. This caused approximately 40 minutes where our platform was not processing traffic. Our team immediately began an emergency failover, redirecting message processing to backup infrastructure to restore service while the third-party issue was ongoing.

This failover was successful. Processing was restored, and by 4:05 PM Central our team confirmed the platform had caught up and declared the incident resolved.

Impact

Customers experienced approximately 40 minutes of delayed asynchronous processing and errors on synchronous requests while the third-party outage was active. Additionally, our dashboard and log visibility was delayed for approximately 2 and a half hours.

How We Resolved It

Our team redirected message processing to backup infrastructure within minutes of confirming the outage. Processing was fully restored and the dashboard/log visibility was caught up by 4:05 PM Central the same day.

What We're Doing About This

• Re-evaluate failover playbooks, so scaling, monitoring, and cleanup steps are rehearsed and routine
• We are actively migrating away from this third-party provider

We appreciate your patience during this incident. If you have questions or concerns, please reach out to your Redox account team.

‌

The emergency failover used to resolve this incident led to a second, separate issue affecting dashboard and log visibility. See [Incident 2: Dashboard & Log Visibility Delay] for details.

Resolved: The incident has been resolved. Please do not hesitate to reach out to [email protected] if you are continuing to run into anything unexpected.

Monitoring: We applied a fix at 2:23pm Central and processing appears to be operating as normal. We are still seeing delays in log visibility. We will continue to monitor to verify that the issues are fully resolved with our 3rd party vendor.

Investigating: Our automated monitoring tools detected a major outage affecting the API and main site/dashboard at 1:34pm Central. Our team is currently investigating the cause and working on a solution.

Please work with your teams to implement downtime procedures.

If you have any additional questions, please notify us at [email protected].

Postmortem: ## Summary

Following an emergency failover carried out in response to a third-party outage on September 8, a configuration change made to stabilize that failover infrastructure reduced capacity on the system powering dashboard and log visibility. Message processing itself was not affected — data continued to be sent and received normally throughout this incident. Customers began experiencing delayed visibility into message logs and payload details overnight; we identified the issue by 7:45 AM Central on September 9 and fully resolved it by 1:51 PM Central. No data was lost at any point.

What Happened

As part of stabilizing the backup infrastructure used to restore service during a separate third-party outage earlier that day, our team identified an opportunity to make its scaling configuration more resilient to future failover deployments, and moved quickly to put that improvement in place.

That change was deployed around 4:34 PM Central on September 8. It did not have the intended effect, and capacity on the backup system supporting dashboard and log visibility was reduced as a result. Message processing itself was not affected. Monitoring coverage on that temporary infrastructure was still being extended at the time, so the reduced capacity was not immediately flagged, and a visibility backlog accumulated overnight.

Impact

Customers experienced delayed visibility into message logs and payload details in the Customer Dashboard, beginning overnight and continuing into the morning of September 9. Message processing was not affected — data continued to be sent and received normally. The impact was limited to the dashboard and log visibility layer.

How We Resolved It

Customer reports and internal alerts led our team to begin investigating at approximately 7:30 AM Central on September 9. Within 15 minutes, we identified that the failover infrastructure was underscaled and began remediation.

Between 8:00 AM and 12:00 PM Central, we:

• Increased infrastructure capacity to add processing headroom
• Scaled the services powering dashboard and log visibility
• Distributed traffic across infrastructure to accelerate recovery

Visibility into message logs returned to fully caught-up status by 12:51 PM Central. We returned to normal operating capacity by 1:51 PM Central and confirmed resolution on our status page.

What We're Doing About This

• Re-evaluate failover playbooks, so scaling, monitoring, and cleanup steps are rehearsed and routine
• Require monitoring and alerting to be fully wired up on any temporary or backup infrastructure before a team stands down from an incident
• Add automated deployment tests to catch configuration resets before they reach production, including on temporary or failover infrastructure
• Consider implementing absolute minimum scale limits so services cannot drop below production-safe thresholds, while preserving a standard, well-documented method for intentionally adjusting capacity when needed

We appreciate your patience during this incident. If you have questions or concerns, please reach out to your Redox account team.

‌

This issue arose while our team was stabilizing infrastructure used in an emergency failover for a separate third-party outage. See [Incident 1: Third-Party Service Outage & Processing Delay] for details.

Resolved: This incident has been resolved. Log visibility is restored.

If you have any questions, please reach out to [email protected]

Monitoring: A fix has been implemented and visibility for the logs in the backlog is returning. We are actively monitoring the situation and ensuring that all affected logs are visible again.

Identified: We have pushed a fix that will make new logs visible, however logs received between approximately 5:45PM and 8:40AM will still not be viewable. The backlogs of logs will continue to process through the queue while it catches up.

Identified: We have identified the cause for the log visibility issues and are actively working on a fix. We will update the status of this incident when the fix has been deployed.

Investigating: We are aware of an issue where logs are not visible in the dashboard.

While logs are still processing properly, the dashboard is not displaying them as expected.

If you have questions about specific logs or your connection and whether it has been affected, please contact us at [email protected]

Postmortem: ## Summary

On September 8, an outage at a third-party infrastructure provider we rely on for message processing disrupted our platform for approximately 40 minutes, delaying asynchronous processing and returning errors on synchronous requests. Additionally, this slowed down our processing of transaction artifacts resulting in delayed visibility in our dashboard. Our team executed an emergency failover to backup infrastructure and restored normal processing the same afternoon. No data was lost.

What Happened

In the early afternoon of September 8, a third-party infrastructure provider that Redox relies on for message processing experienced an outage. This caused approximately 40 minutes where our platform was not processing traffic. Our team immediately began an emergency failover, redirecting message processing to backup infrastructure to restore service while the third-party issue was ongoing.

This failover was successful. Processing was restored, and by 4:05 PM Central our team confirmed the platform had caught up and declared the incident resolved.

Impact

Customers experienced approximately 40 minutes of delayed asynchronous processing and errors on synchronous requests while the third-party outage was active. Additionally, our dashboard and log visibility was delayed for approximately 2 and a half hours.

How We Resolved It

Our team redirected message processing to backup infrastructure within minutes of confirming the outage. Processing was fully restored and the dashboard/log visibility was caught up by 4:05 PM Central the same day.

What We're Doing About This

• Re-evaluate failover playbooks, so scaling, monitoring, and cleanup steps are rehearsed and routine
• We are actively migrating away from this third-party provider

We appreciate your patience during this incident. If you have questions or concerns, please reach out to your Redox account team.

‌

The emergency failover used to resolve this incident led to a second, separate issue affecting dashboard and log visibility. See [Incident 2: Dashboard & Log Visibility Delay] for details.

Resolved: The incident has been resolved. Please do not hesitate to reach out to [email protected] if you are continuing to run into anything unexpected.

Monitoring: We applied a fix at 2:23pm Central and processing appears to be operating as normal. We are still seeing delays in log visibility. We will continue to monitor to verify that the issues are fully resolved with our 3rd party vendor.

Investigating: Our automated monitoring tools detected a major outage affecting the API and main site/dashboard at 1:34pm Central. Our team is currently investigating the cause and working on a solution.

Please work with your teams to implement downtime procedures.

If you have any additional questions, please notify us at [email protected].