Postmortem: # Summary
On Oct 14, between 12:12 UTC and 12:26 UTC, a service disruption impacted the availability of the Editor component for end-users. The incident occurred during a planned configuration update aimed at decommissioning a legacy service. During execution via Terraform, an undetected state drift between the remote infrastructure and the defined configuration caused an unexpected destruction and recreation of the API Gateway, followed by a listener misconfiguration on the Application Load Balancer (ALB). Total service degradation lasted approximately 14 minutes. Full functionality was restored at 12:26 UTC following targeted manual remediation.
Status
Solved
Timeline (UTC)
Oct 14, 2026
- 12:10 UTC – A routine infrastructure update was initiated via Terraform to remove resources associated with a decommissioned service.
- 12:12 UTC – Upon execution, Terraform identified an implicit dependency drift, triggering an unintended
destroy-and-recreate cycle for the API Gateway instance.
- 12:18 UTC – The API Gateway replacement was provisioned. However, traffic routing failed due to a mismatched configuration state on the Application Load Balancer (ALB) listener bindings.
- 12:20 UTC – Engineering teams identified the routing anomaly on the ALB listener and initiated corrective configuration procedures.
- 12:26 UTC – Listener configuration was fully restored and verified. Traffic flow to the EVC cluster normalized, resolving customer impact.
Incident window: ~14m, all EVC services not reachable
Customer impact
Duration: 14 minutes (12:12 UTC – 12:26 UTC)
Affected Components: EVC Cluster underlying services, Editor interface.
User Experience: End-users attempting to access or perform operations within the Editor experienced temporary service unavailability and request timeouts. No data loss or corruption occurred during this period.
Root Cause
The root cause was an unexpected dynamic resource replacement triggered by Terraform execution, resulting from a latent configuration drift between the active cloud resources and the declarative state file.
During the removal of legacy service declarations, an implicit resource attribute dependency within the Terraform provider schema evaluated the update as a breaking change requiring resource replacement rather than an in-place update. Because the environment had experienced minor manual or unindexed attribute shifts over time (configuration drift), the plan preview did not clearly highlight the cascading effect on the API Gateway's dependent ALB listener configuration. Consequently, when the gateway was recreated, the associated ALB listener bindings failed to map correctly to the target group, cutting off ingress traffic to the EVC cluster.
Lessons Learned & Action Items
To prevent similar issues during future infrastructure changes, the following systemic improvements will be implemented:
Enhanced Pre-Apply Plan Inspection & Safety Gates:
- Implement automated CI/CD checks to block Terraform execution if a plan contains destructive actions (
destroy) on critical network infrastructure (gateways, ALBs) without explicit secondary authorization.
Drift Detection & Reconciliation Automation:
- Introduce periodic scheduled drift-detection pipelines (e.g., automated
terraform plan audits) to identify and reconcile discrepancies between real-world infrastructure and state files prior to active maintenance windows.
Decoupling Dependency Graphs:
- Refactor infrastructure-as-code modules to isolate API Gateway definitions from underlying ALB target group lifecycle management, ensuring gateway updates cannot implicitly impact ingress routing listeners.
