Nanonets Status

Postmortem: On March 10 between 1:10 PM and 3:30 PM UTC, we experienced a service degradation that impacted Instant Learning Models (ILM) on app.nanonets.com.

Impact:

During this period, our vector database experienced unusually high load due to the query planner selecting a suboptimal execution path for certain queries. This led to elevated database response times and eventually triggered a database restart. As a result, overall system throughput temporarily decreased.

Customers may have observed the following effects during this window:

• Asynchronous uploads for ILM models experiencing delayed file processing.
• Synchronous uploads for ILM models encountering intermittent failures.

Only Instant Learning Models on app.nanonets.com were affected. Other services and model types remained fully operational.

Resolution:

Our engineering team identified the root cause related to query planning behaviour in the vector database and implemented targeted customizations to ensure the optimal execution path is used. This stabilized database performance and restored normal processing capacity.

Next Steps:

System is stable now and we are actively working on additional database optimizations and long-term improvements to further strengthen system stability and prevent similar issues from recurring. These enhancements will be rolled out in the coming days.

We sincerely apologize for the inconvenience caused and appreciate your patience while we worked to resolve the issue. Please feel free to reach out to our support team incase if you have any questions.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue.