Flatfile Status

Postmortem: # Incident Overview

Date and Time of Incident: June 10, 2025 from approximately 1:50pm to 2:05pm PDT

Nature of Incident: Requests routing to an unavailable frontend pod

Services Affected: Flatfile Platform Dashboard and Spaces

Details of the Incident

At approximately 1:50pm PDT on June 10, the Flatfile Platform Dashboard reported intermittent 503 status codes. This was due to an infrastructure change made earlier in the day. As a consequence 1 in every 4 requests made to the Platform dashboard returned a HTTP 503 status code. The engineering team were alerted to the incident as soon as the 503’s started appearing and manually scaled up the infrastructure resources.

Impact Assessment

The incident was intermittent and affected 1 in 4 requests made by users of the Platform frontend applications for approximately 15 minutes. The API was unaffected.

Root Cause

The root cause was due to an infrastructure scaling configuration made in the early of the morning.

Resolution

Flatfile engineering manually provisioned extra capacity in order for the resources to get allocated appropriately. Following this, further improvements will be made with how we serve traffic to end users.

Security and Data Integrity

Please be assured that this incident did not compromise the security or integrity of your data. Our commitment to data protection remains a top priority.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are investigating an issue where some users are seeing bad gateway errors.

Postmortem: # Incident Overview

A series of compounding events on the US region Platform led to network throttling of a network file system which severely degraded file extractions on the Platform.

Details of the Incident

At approximately 10 AM PDT on June 11, Flatfile experienced several conflating events that resulted in network bandwidth on our network file systems being overwhelmed and throttled by our cloud service provider. The event resulted from a spike in usage on the Platform that led to autoscaling events which dramatically increased read operations against the network file stores, eventually leading to exhaustion of throughput.

This event led to a severe degradation in file extraction performance on the Platform for all customers. The initial response to the incident involved increasing capacity which compounded the problem. At approximately 14:20 PDT, the engineering team provisioned a large increase in throughput which ended the throttling and allowed file extraction jobs to proceed as expected.

Impact Assessment

The incident severely degraded all file extractions and led to increased latency on jobs as the queue became backlogged.

Root Cause

The throttling of throughput on the network file system was the cause of the incident; this was exacerbated by automated scaling operations that attempted to increase queue capacity.

Resolution

Flatfile engineering provisioned increased throughput on the file system to resolve the incident at 14:20 PDT.

Security and Data Integrity

Please be assured that this incident did not compromise the security or integrity of your data. Our commitment to data protection remains a top priority.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are continuing to work on a fix for this issue.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We are seeing an issue where excel files are not being extracted. We have identified a cause and are working on a fix. CSV files are still working.

Postmortem: # Incident Overview

Date and Time of Incident: June 10, 2025 from approximately 1:50pm to 2:05pm PDT

Nature of Incident: Requests routing to an unavailable frontend pod

Services Affected: Flatfile Platform Dashboard and Spaces

Details of the Incident

At approximately 1:50pm PDT on June 10, the Flatfile Platform Dashboard reported intermittent 503 status codes. This was due to an infrastructure change made earlier in the day. As a consequence 1 in every 4 requests made to the Platform dashboard returned a HTTP 503 status code. The engineering team were alerted to the incident as soon as the 503’s started appearing and manually scaled up the infrastructure resources.

Impact Assessment

The incident was intermittent and affected 1 in 4 requests made by users of the Platform frontend applications for approximately 15 minutes. The API was unaffected.

Root Cause

The root cause was due to an infrastructure scaling configuration made in the early of the morning.

Resolution

Flatfile engineering manually provisioned extra capacity in order for the resources to get allocated appropriately. Following this, further improvements will be made with how we serve traffic to end users.

Security and Data Integrity

Please be assured that this incident did not compromise the security or integrity of your data. Our commitment to data protection remains a top priority.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are investigating an issue where some users are seeing bad gateway errors.