Postmortem: Investigation
The incident began on October 17, 2025, at 8:02 AM CET in West Europe (Production 0) and on October 20, 2025, between 8:48 AM CET and 9:15 AM CET across Production 1–5 clusters. It was detected on October 20, 2025, at 1:44 PM CET, after reports indicated that users were able to access Spaces within their own environment without being members of those Spaces. The engineering team initiated the investigation at 3:34 PM CET on October 20, 2025. The root cause was traced to a recent change in the Library App that affected the authorization logic used when fetching Space data.
Mitigation
At 3:58 PM CET, the engineering team began implementing a fix. The first mitigation step involved rolling back the recent changes that introduced the authorization issue. This rollback effectively restored the correct filtering logic, ensuring users could only access Spaces to which they were authorized.
Resolution
By 4:29 PM CET on October 20, 2025, the rollback had been fully deployed across all production clusters. Following verification and monitoring, normal access restrictions were confirmed to be functioning correctly. The issue was considered resolved at this time.
Post-Incident Actions
To prevent similar issues from occurring, the following preventive measures are being implemented:
- Expanding automated testing of the Library feature to ensure access controls work as intended.
- Strengthening how internal APIs are protected by moving to a more restrictive allow-list model.
- Providing additional security training for our engineering teams.
- Improving our development environments to better detect access and permission issues before release.
Impact and Scope
This incident allowed existing platform users from within the same customer environment to view Spaces of which they weren’t members. The issue only affected customers with multiple Spaces. We have directly notified the small number of impacted customers. If you have not been contacted, no assets associated with your account were actively used during the incident window. While certain assets may have been visible to users within your own environment, there is no indication of any interaction, modification or misuse. It is important to note that the visibility issue is strictly confined to users within customers’ own environment — no users from other customers or external third parties had any access. The issue affected all production clusters:
- West Europe (Production 0): October 17, 2025, 8:02 AM CET
- West Europe (Production 1): October 20, 2025, 9:15 AM CET
- East US (Production 2): October 20, 2025, 8:48 AM CET
- Australia East (Production 3): October 20, 2025, 8:48 AM CET
- Canada Central (Production 4): October 20, 2025, 8:48 AM CET
- West Europe (Production 5): October 20, 2025, 9:15 AM CET
All affected clusters were promptly rolled back following the detection of the issue.We sincerely apologize for the disruption caused by this security-related incident. Protecting our customers’ data and maintaining strict access control standards are our highest priorities. We are taking immediate and long-term actions to strengthen safeguards, improve internal validation and reinforce our commitment to delivering a secure and reliable service.