Stripo Status

Postmortem: At 13:45 UTC the AWS host running the database and cache for our plugin environment failed. Auto-recovery restarted it by 14:00 UTC, but the cache service and several microservices required manual restarts. Full recovery: ~15:00 UTC.

Impact (UTC). Autosave was unavailable 13:48–13:57, then severely slow (~86 s per request vs. under 7 s normally) until ~14:40. Some plugins using external AutoSave callbacks saw saves fail. Co-editing was degraded 14:16–15:00. Editor initialization also failed for some users early on.

Data. No evidence of loss: crash recovery completed correctly, all patches that reached the database were persisted, and duplicate resubmissions were safely rejected. Contact us with a timestamp if you suspect otherwise.

Cause. Hardware failure on the underlying AWS host. It escalated because autosave retries external callbacks synchronously inside the request, exhausting the database connection pool; the cache had no auto-start on reboot; and connection pools did not self-heal. No alarm covered host status or cache availability, so detection came from user reports.

Fixes. Alerting on host status checks and cache availability; cache auto-start with a health check; connection pool recovery; moving external callbacks out of the request path; alerts on pool saturation, error rate and callback latency; a redundancy review for this environment.

We apologize for the disruption. If the impact on your users differed from the above, please contact our support team.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are aware of an issue with the Stripo Plugin: the editor may fail to initialize and does not load for end users. Our team has identified the affected service and is working on a fix. We will post an update as soon as we have more information.

Resolved: The permanent fix has been deployed to all production clusters and all systems are operating normally. During the incident (July 30, 22:48 UTC – July 31, ~04:30 UTC, with residual slowness until ~08:49 UTC), some users experienced login failures, missing team member data, and slow loading of emails. No data was lost. We have added memory-usage alerting to detect this class of issue before it affects users. We apologize for the inconvenience.

Monitoring: Cache capacity has been increased and the system is recovering. Response times are returning to normal as caches warm up. A permanent fix is being deployed.

Identified: We have identified the root cause: an internal caching service (Redis) ran out of memory and began rejecting writes. As a result, permission checks bypassed the cache and put excessive load on the account service, causing slow responses, login failures, and missing account data. We have increased the cache capacity and are preparing a permanent fix.

Investigating: We are investigating reports of users being unable to log in, seeing missing team members, or experiencing slow loading of emails. Our team is working to identify the cause.