AppZen Status

Resolved: Resolved — Invoice Processing Delays (Autonomous AP)

Between approximately 11:30 AM and 5:00 PM PDT on September 15, some invoices submitted to Autonomous AP remained in a "Processing" state longer than expected.

The cause was an availability issue with a third-party provider in the US-East region, which caused inference requests to time out rather than fail fast. The provider has confirmed the issue on their side.

We mitigated the impact by routing invoice processing to an alternate endpoint, after which new invoices resumed processing normally. All affected invoices have since been reprocessed and have completed successfully. No invoice data was lost.

Processing times have returned to normal, and the service is fully operational.

We apologise for the disruption.

Monitoring: We applied a fix and are monitoring recovery.

Our engineering team has completed the failover to an alternate endpoint. Invoice processing has recovered:

New invoices submitted to Autonomous AP are now processing normally, at expected speeds. Backlogged invoices that were stuck during the incident window are being reprocessed in batches, and the queue is draining steadily.

Customers may continue to see a temporary delay on invoices submitted during the impact window until the backlog fully clears. No customer action is required; these invoices will complete automatically.

We are monitoring system health and backlog drain rate closely and will confirm once the backlog is fully cleared.

Identified: We have identified the cause of the invoice processing delays.

The delays are caused by an availability issue with a third-party service hosted in the US-EAST-1 region. The provider has confirmed the degradation on their side and has identified the root cause as an internal queuing subsystem. Requests from our platform were being accepted but not completed, causing invoices to remain in "Processing".

The impact is isolated to a single endpoint. All other AppZen services are unaffected.

Mitigation in progress: Our engineering team has begun failing over invoice processing to an alternate, healthy endpoint. Once the failover is complete, newly submitted invoices are expected to process normally, and we will begin clearing the accumulated backlog.

No invoice data has been lost. Next update in 30 minutes.

Investigating: We are investigating delays in invoice processing on Autonomous AP.