Postmortem: # Incident Post-mortem Report
Dashboard exports and imports delayed for all Stores
|
|
| Date |
Oct 25, 2026, 11:25 AM until 2:26 PM (UTC+3, KSA time). Duration: 3 hours 1 minute |
| Type of Incident |
Service degradation: background processing |
| Title |
A very large customer import stalled dashboard exports and imports for all Stores |
| Impact |
132 export requests from 66 Stores were delayed, by 81 minutes on average and up to 3 hours 43 minutes. 3 of them timed out and must be requested again. The customer imports of the Store that started the large import were delayed until 3:12 PM. |
| Total impact on uptime |
No outage. Stores, checkout and the dashboard worked the whole time. Only exports and imports waited. |
Summary
Dashboard exports (orders, customers, wallet, abandoned carts and others) and dashboard imports are built in the background by one shared pool of workers. At 11:23 AM a Merchant started a customer import with about 1.05 million rows. The platform splits every import into parts of 100 rows, so this one became 10,486 parts.
After each part, the platform checks whether the whole import has finished, so it can send the Merchant a summary. That check loaded every part of the import, rows included, into memory: about 200 MB per check for this import, with up to five checks running at once. The workers ran out of memory and restarted again and again. Export requests from every Store waited behind the import parts.
We shipped a code-level change so the check only asks whether any part is still pending and adds up the final totals in the database. It no longer loads the rows, so its memory use no longer grows with the size of the import.
Timeline (KSA time)
| Time |
Event |
| 11:23 AM |
A Merchant starts a customer import of about 1.05 million rows. By 11:27 AM it is split into 10,486 parts. |
| 11:25 AM |
The first export request that has to wait is created. Later requests queue behind the import parts. |
| ~11:34 AM |
The workers start running out of memory and restarting. From here they finish about 5 import parts per restart and almost no exports. |
| 12:26 PM |
A Merchant reports to support that 3 exports requested since 11:30 AM are still processing. |
| ~1:15 PM |
Product Engineering starts the investigation and traces the restarts to the import completion check. |
| 1:54 PM |
Product Engineering prepares the change, with a test that reproduces the memory growth. |
| 2:08 PM |
The workers start clearing the backlog, although they still restart from time to time. |
| 2:26 PM |
The last delayed export finishes. Exports are back to normal. |
| 2:34 PM |
The change is live. |
| ~3:00 PM |
The worker restarts stop. |
| 3:09 PM |
3 export requests that had waited more than 3 hours are marked as timed out. |
| 3:10 PM |
The large import finishes, except one part of 100 rows that the platform retries automatically. |
What was affected
| Area |
State during the incident |
How we know |
| Dashboard exports, all types and all Stores |
Delayed: 132 requests from 66 Stores, 81 minutes on average, up to 3 hours 43 minutes |
Export request records. Orders 72, detailed orders 21, wallet 11, abandoned carts 6, customers 6, reviews 5, order preparation 4, coupons 3, other 4. |
| Exports that timed out |
Failed: 3 requests from 2 Stores |
Export request records. These Merchants need to request the export again. |
| Exports still processing |
1 request |
Export request records. The platform retries it automatically. |
| Customer imports of the Store that started the large import |
Delayed until 3:12 PM: 9 imports |
Import records. All parts finished except one, which is retried automatically. |
| Imports from other Stores |
Not affected |
No other Store started an import during the incident |
| Export file contents |
Not affected |
Each file is built from live data when it runs, so delayed files are complete |
| Stores, checkout and the dashboard |
Not affected |
Only the background workers for exports and imports restarted |
Action taken
- Found the cause: the workers that build exports and imports were restarting because the import completion check ran out of memory.
- Shipped a code-level change: the completion check counts pending parts and adds up the totals in the database instead of loading every part. A test fails on the old behaviour and passes on the new one.
- Confirmed that the delayed exports finished, and identified the 3 timed-out requests so support can ask those Merchants to request them again.
Conclusions
One Merchant's import should not hold up every other Store's exports. Exports and imports share the same workers, so a heavy import delays everyone. The change removes this cause.
Work that runs once per part must not grow with the size of the whole import. The completion check read the entire import after every part. That was fine for typical imports and failed only for a very large one. With the change, an import of this size no longer exhausts the workers' memory.
