Voyado Status

Postmortem: ## Summary

On 2025-12-12, an issue was introduced in a morning update that caused registration forms to fail silently on submission for multiple customers. The forms appeared to work but did not process user input.

Customer Impact

End-users were unable to submit registration forms or update communication preferences. No error messages were shown, which impacted onboarding and preference management flows.

Root Cause

A bug in the released code affected how registration forms handled certain preference fields, causing submissions to fail without visible errors.

Mitigation

A fix was identified and deployed to resolve the issue.

Next Steps

We are implementing additional testing safeguards and improving our internal processes to detect similar issues earlier.

We apologize for the disruption and appreciate your understanding.

Resolved: Issue has been resolved.

Identified: Fix is being deployed and issue should be resolved in about 45 min.

Identified: The issue has been identified an a hotfix is being prepared.

Investigating: We have reports on issues submitting registration forms - this is being investigated.

Postmortem: ## Summary

On November 7th, an unusually high number of internal events were generated due to a planned feature rollout. This caused delays in processing certain automations.

Customer Impact

Automations triggered by customer interactions or audience targeting were delayed by up to 12 hours. All affected automations were eventually executed. Other parts of the platform operated as normal.

Root Cause and Mitigation

When the new Marketing Groups feature was enabled for all customers, a large volume of events (hundreds of millions) was generated in a short time. The system processing these events could not keep up, which resulted in delays for other events in the same internal queue.

To restore normal flow, we temporarily cleared out the backlog of non-critical events and performed database synchronizations to ensure data accuracy. Normal operation resumed later the same day.

Next Steps

We are improving our monitoring and alerting to detect similar issues earlier, and we are reviewing our event handling capacity and release procedures to prevent future impact from large-scale rollouts.

We apologize for any inconvenience caused and appreciate your patience while we resolved the issue.

Resolved: The issue causing delays in event processing has now been resolved. All systems are operating as expected.

Root cause analysis will be included in Post-Mortem.

Monitoring: A fix has been implemented, and we are now seeing improvements in event processing times. Features such as Interaction and Target Audience triggers in Automations are gradually returning to normal performance.

We continue to monitor the system closely. The underlying root cause is still under investigation, and we’ll provide more information as it becomes available.

Identified: We are currently experiencing a delay in processing events, which impacts features such as Interaction and Target Audience triggers in Automations. Our team is preparing a hotfix to address the issue.

We will continue to provide updates as the situation progresses.