Wyng Status

Postmortem: An issue with event processing streams caused delays in integration processing affected a subset of Wyng clients. Initial behavior was intermittent, with delays occurring sporadically on July 1, that progressed to a backlog of up to 9 hours of transactions for affected clients starting at about 4:00 AM EDT on July 2.

Wyng ops staff took initial action on the issue at 8:30 AM EDT on July 2. By 12:00 EDT the backlog reduced to 2 hours, by 4:20 PM EDT the backlog was 1 hour, and the issue was resolved for all affected clients at 6:38 PM EDT on July 2.

The root cause was interdependence between a set of clients whose transactions were being processed in a shared stream. Significant delays in downstream processing by one client caused a backlog across all clients sharing the stream.

To avoid recurrence of this failure mode, Wyng is prioritizing a infrastructure update to move from shared to dedicated streams for transaction event processing.

Resolved: Delays have been resolved for all impacted clients.

Monitoring: Some clients are experiencing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally.

[UPDATE 08:30 AM EDT] We have taken steps to address the issue and processing delay for impacted clients is recovering now.

Postmortem: An issue with event processing streams caused delays in integration processing affected a subset of Wyng clients. Initial behavior was intermittent, with delays occurring sporadically on July 1, that progressed to a backlog of up to 9 hours of transactions for affected clients starting at about 4:00 AM EDT on July 2.

Wyng ops staff took initial action on the issue at 8:30 AM EDT on July 2. By 12:00 EDT the backlog reduced to 2 hours, by 4:20 PM EDT the backlog was 1 hour, and the issue was resolved for all affected clients at 6:38 PM EDT on July 2.

The root cause was interdependence between a set of clients whose transactions were being processed in a shared stream. Significant delays in downstream processing by one client caused a backlog across all clients sharing the stream.

To avoid recurrence of this failure mode, Wyng is prioritizing a infrastructure update to move from shared to dedicated streams for transaction event processing.

Resolved: Delays have been resolved for all impacted clients.

Monitoring: Some clients are experiencing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally.

[UPDATE 08:30 AM EDT] We have taken steps to address the issue and processing delay for impacted clients is recovering now.

Postmortem: An issue with event processing streams caused delays in integration processing affected a subset of Wyng clients. Initial behavior was intermittent, with delays occurring sporadically on July 1, that progressed to a backlog of up to 9 hours of transactions for affected clients starting at about 4:00 AM EDT on July 2.

Wyng ops staff took initial action on the issue at 8:30 AM EDT on July 2. By 12:00 EDT the backlog reduced to 2 hours, by 4:20 PM EDT the backlog was 1 hour, and the issue was resolved for all affected clients at 6:38 PM EDT on July 2.

The root cause was interdependence between a set of clients whose transactions were being processed in a shared stream. Significant delays in downstream processing by one client caused a backlog across all clients sharing the stream.

To avoid recurrence of this failure mode, Wyng is prioritizing a infrastructure update to move from shared to dedicated streams for transaction event processing.

Resolved: Delays have been resolved for all impacted clients.

Monitoring: Some clients are experiencing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally.

[UPDATE 08:30 AM EDT] We have taken steps to address the issue and processing delay for impacted clients is recovering now.

Postmortem: An issue with event processing streams caused delays in integration processing affected a subset of Wyng clients. Initial behavior was intermittent, with delays occurring sporadically on July 1, that progressed to a backlog of up to 9 hours of transactions for affected clients starting at about 4:00 AM EDT on July 2.

Wyng ops staff took initial action on the issue at 8:30 AM EDT on July 2. By 12:00 EDT the backlog reduced to 2 hours, by 4:20 PM EDT the backlog was 1 hour, and the issue was resolved for all affected clients at 6:38 PM EDT on July 2.

The root cause was interdependence between a set of clients whose transactions were being processed in a shared stream. Significant delays in downstream processing by one client caused a backlog across all clients sharing the stream.

To avoid recurrence of this failure mode, Wyng is prioritizing a infrastructure update to move from shared to dedicated streams for transaction event processing.

Resolved: Delays have been resolved for all impacted clients.

Monitoring: Some clients are experiencing delays in processing integration data flows outbound from Wyng to external systems. Users may notice more integration transactions in a Pending state, and longer elapsed times between data collection by an experience and the delivery of that data to their external system or API. Data collection and storage are not impacted, and are operating normally.

[UPDATE 08:30 AM EDT] We have taken steps to address the issue and processing delay for impacted clients is recovering now.