Sardine AI Status

Postmortem: ## Overview

On Jan 16 morning US time, Sardine experienced an increase in latency due to increased database usage. /customers, /issuing/risks and /feedbacks API experienced higher latency during 17:57-18:29 UTC.

What happened

Sardine experienced an increase in traffic due to 1) an internal backfill job, 2) large backfill triggered by a client, and 3) overall traffic growth. Resource usage of one of the database system, resulted in long latency.

Impact

*/customers, /issuing/risks and /feedbacks API experienced higher latency during 17:57-18:29 UTC.*

Timeline

• 17:00 UTC: Sardine started to observe higher than usual latency. While latency is higher than usual, it didn’t match the condition for alert this point
• 17:53 UTC: Latency increased drastically
• 17:57 UTC: Page triggered
• 17:58 UTC: Oncall engineer acknowledged the page
• 18:05 UTC: Oncall engineer escalated to DevOps oncall and other senior team members
• 18:19 UTC: Database upsize was triggered
• 18:29 UTC: Latency goes back to normal

What went wrong

• We didn’t have sufficient internal control around internal data processing job that stressed the system
• It took long time until resource upsize was performed
• It took long time before Sardine update the status page

Action items

• Enhance internal process around data load so we can better control the resource usage
• Review and adjust resource configuration
• Improved internal training for oncall engineer

Resolved: 17:00 UTC to 17:52 UTC - moderate latency increase - p90 from stable 500ms to spikes to 2s 17:52 UTC to 18:27 UTC - heavy latency increase (p50 >2s) from 140ms

Identified: The issue has been identified and a fix is being implemented.