Open Exchange Rates Status

Postmortem: This incident was triggered by a routine system maintenance task and escalated due to an edge case in our application caching infrastructure.

At 02:50 UTC, while system maintenance was taking place, a database deadlock was caused by multiple background processes attempting to write data simultaneously. Under normal circumstances, this behaviour would self-correct. However, in this instance one of the processes retried repeatedly, and the database became unresponsive at 03:07 UTC. As a result of this, incomplete data were added to the cache, causing up to 100 currency symbols to be temporarily unavailable in live API responses.

We identified the issue and immediately began working to resolve it, which required several manual interventions in our infrastructure. Service was restored at 03:32 UTC, 25 minutes after the first production issues were reported. The incident was then marked resolved on our Status Page. (The HNL currency symbol however remained unavailable in live API responses, until approximately 11:20 UTC when its data contributors were restored).

A fix for this behaviour is being developed, with deployment planned in the next 2 weeks. In the meantime, there remains a small risk that a similar issue could occur again. We are actively working on reducing this risk ahead of the fix being deployed, and in the unlikely event of a repeat occurrence, our 24/7/365 monitoring team are standing by to resolve it without delay.

Resolved: This incident has been resolved and we are continuing to monitor.

During this partial outage incident, up to 100 currency symbols became temporarily unavailable in our service, beginning at around 03:07 UTC. We were able to identify the issue and began working on a fix immediately, and service was restored by 03:32 UTC. The root cause appears to be related to routine system maintenance. We are continuing to investigate.

Monitoring: A fix has been implemented to restore full service, and we are now monitoring the results. Please stand by for further updates.

Identified: We are aware of issues affecting Open Exchange Rates API and are working to resolve this. Please stand by for updates.