Redox Engine Status

Resolved: The incident has been resolved and logs delays are at nominal levels. If you have any additional questions please reach out to [email protected]

Monitoring: A fix has been implemented for the delay issues and Redox is actively monitoring our logs. You may continue to experience delays as queues catch up to the present. If you have questions please reach out to [email protected]

Investigating: Redox is currently experiencing a delay in some logs.

About an eighth of logs are delayed by under and hour.

What this means: you may experience a delay before receiving traffic from Redox. However, please be aware that the actual processing of logs is unaffected; message flow is still occurring, albeit at a slower pace. We are actively looking into this issue and will have a fix available as soon as possible.

If you are concerned that your logs have been impacted or have any additional questions, please reach out to [email protected]

Resolved: This issue has been resolved as our third party vendor has resolved the issue and our alerts stayed operational through our monitoring process.

If you have any additional questions please reach out to [email protected]

Monitoring: Our third party vendor has completed their recovery and is monitoring things as are we.

Alerts are working as expected and we closing monitoring the situation to ensure this continues.

If you have any additional questions, please contact [email protected]

Identified: Our third party vendor has completed their recovery and is monitoring things as are we.

Alerts are working as expected and we closing monitoring the situation to ensure this continues.

If you have any additional questions, please contact [email protected]

Identified: Our third-party vendor is in the process of recovering from an incident on their end. We are doing everything within our power on the Redox side to mitigate from our end. We will continue to provide updates as they become available and will have more information available on Monday.

Alerting within the customer dashboard will be inconsistent for now and through the weekend. We recommend reviewing any alerts with scrutiny to ensure they are accurate.

Our own Redox internal alerting and remains unimpacted. This includes automated destination level error retry and failure alerts.

Identified: The fix was deployed but we are still seeing intermittent errors on our end. We have been communicating with our 3rd party vendor to assist further. We will continue to provide updates as they become available.

Identified: The fix was deployed but we are still seeing intermittent errors on our end. We have been communicating with our 3rd party vendor to assist further. We will continue to provide updates as they become available.

Identified: We have identified the cause for the ongoing notification issue and are actively implementing a fix. We will update the status of this incident when this fix has been fully deployed.

Identified: A fix has been put in place for the dashboard so it now reflects current alert statuses as expected. We are continuing to work toward more reliable alert notification services.

Identified: We're continuing to investigate the root cause of the issue. It appears to be related to a third-party vendor. We are seeing degraded performance rather than an outage at this time. We are reaching out to this partner to assist with additional troubleshooting.

Additionally, we are taking steps to put failovers in place to ensure reliability of our dashboard alerts. A fix is also being put in place to ensure the dashboard still displays as expected.

Investigating: Our automated monitoring tools detected a partial outage affecting the dashboard at ~2:00 PM Central. Intermittently, alert rules are unable to be loaded within the dashboard and a subset of alerts are not triggering as expected. Our team is currently investigating the cause and working on a solution.

Please work with your teams to implement downtime procedures. Traffic processing is not impacted.

If you have any additional questions, please notify us at [email protected].