Alpaca Status

Postmortem: High usage of database connections was identified around 6:00 PM ET. The team was able to identify an application process that was creating contention due to a lock wait, which persisted even after the request timed out. Both the application and the database were restarted to recover. The issue persisted intermittently but eventually resolved.

Resolved: This incident has been resolved.

Monitoring: We have identified and resolved the issue that was affecting our systems. Since implementing the fix, all systems have been operating normally with no recurrence of the problem. We have been closely monitoring our systems over the past hour, and all indicators show stable performance. Our monitoring infrastructure continues to track system health to detect any potential issues early. Our engineering team remains available to respond immediately if any concerns arise.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: We will continue to monitor the system and take appropriate the action.

Investigating: System is working as expected. We are monitoring the performance

Investigating: We are seeing spike in connections again. Teams are actively working on it

Investigating: After restarting, the database connections are under control. We are monitoring the system.

Investigating: Database is restarted and team is monitoring it

Investigating: Team is still working on the recovery.

Investigating: We are restarting the DB and API calls are expected to fail.

Investigating: We are continuing to investigate this issue.

Investigating: We are still working on identifying the underlying cause of DB connections

Investigating: We are seeing high DB usage causing the API to fail. We are checking internally.

Postmortem: A change in our monitoring system resulted in an alert erroneously firing indicating the database was down. In order to resolve this quickly the database was restarted, resulting in the outage.

Resolved: This incident has been resolved.

Investigating: On October 9, 2025, between approximately 09:48 UTC and 09:50 UTC, we experienced a short service disruption caused by an unexpected restart of production database. This may have resulted in connection errors for some users during this two-minute window. The database systems recovered automatically, and all services have been operating normally since. Our engineering team is actively investigating the root cause to prevent a recurrence.

Postmortem: High usage of database connections was identified around 6:00 PM ET. The team was able to identify an application process that was creating contention due to a lock wait, which persisted even after the request timed out. Both the application and the database were restarted to recover. The issue persisted intermittently but eventually resolved.

Resolved: This incident has been resolved.

Monitoring: We have identified and resolved the issue that was affecting our systems. Since implementing the fix, all systems have been operating normally with no recurrence of the problem. We have been closely monitoring our systems over the past hour, and all indicators show stable performance. Our monitoring infrastructure continues to track system health to detect any potential issues early. Our engineering team remains available to respond immediately if any concerns arise.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: We will continue to monitor the system and take appropriate the action.

Investigating: System is working as expected. We are monitoring the performance

Investigating: We are seeing spike in connections again. Teams are actively working on it

Investigating: After restarting, the database connections are under control. We are monitoring the system.

Investigating: Database is restarted and team is monitoring it

Investigating: Team is still working on the recovery.

Investigating: We are restarting the DB and API calls are expected to fail.

Investigating: We are continuing to investigate this issue.

Investigating: We are still working on identifying the underlying cause of DB connections

Investigating: We are seeing high DB usage causing the API to fail. We are checking internally.

Postmortem: A change in our monitoring system resulted in an alert erroneously firing indicating the database was down. In order to resolve this quickly the database was restarted, resulting in the outage.

Resolved: This incident has been resolved.

Investigating: On October 9, 2025, between approximately 09:48 UTC and 09:50 UTC, we experienced a short service disruption caused by an unexpected restart of production database. This may have resulted in connection errors for some users during this two-minute window. The database systems recovered automatically, and all services have been operating normally since. Our engineering team is actively investigating the root cause to prevent a recurrence.

Postmortem: High usage of database connections was identified around 6:00 PM ET. The team was able to identify an application process that was creating contention due to a lock wait, which persisted even after the request timed out. Both the application and the database were restarted to recover. The issue persisted intermittently but eventually resolved.

Resolved: This incident has been resolved.

Monitoring: We have identified and resolved the issue that was affecting our systems. Since implementing the fix, all systems have been operating normally with no recurrence of the problem. We have been closely monitoring our systems over the past hour, and all indicators show stable performance. Our monitoring infrastructure continues to track system health to detect any potential issues early. Our engineering team remains available to respond immediately if any concerns arise.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: We will continue to monitor the system and take appropriate the action.

Investigating: System is working as expected. We are monitoring the performance

Investigating: We are seeing spike in connections again. Teams are actively working on it

Investigating: After restarting, the database connections are under control. We are monitoring the system.

Investigating: Database is restarted and team is monitoring it

Investigating: Team is still working on the recovery.

Investigating: We are restarting the DB and API calls are expected to fail.

Investigating: We are continuing to investigate this issue.

Investigating: We are still working on identifying the underlying cause of DB connections

Investigating: We are seeing high DB usage causing the API to fail. We are checking internally.

Postmortem: A change in our monitoring system resulted in an alert erroneously firing indicating the database was down. In order to resolve this quickly the database was restarted, resulting in the outage.

Resolved: This incident has been resolved.

Investigating: On October 9, 2025, between approximately 09:48 UTC and 09:50 UTC, we experienced a short service disruption caused by an unexpected restart of production database. This may have resulted in connection errors for some users during this two-minute window. The database systems recovered automatically, and all services have been operating normally since. Our engineering team is actively investigating the root cause to prevent a recurrence.

Postmortem: High usage of database connections was identified around 6:00 PM ET. The team was able to identify an application process that was creating contention due to a lock wait, which persisted even after the request timed out. Both the application and the database were restarted to recover. The issue persisted intermittently but eventually resolved.

Resolved: This incident has been resolved.

Monitoring: We have identified and resolved the issue that was affecting our systems. Since implementing the fix, all systems have been operating normally with no recurrence of the problem. We have been closely monitoring our systems over the past hour, and all indicators show stable performance. Our monitoring infrastructure continues to track system health to detect any potential issues early. Our engineering team remains available to respond immediately if any concerns arise.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: We will continue to monitor the system and take appropriate the action.

Investigating: System is working as expected. We are monitoring the performance

Investigating: We are seeing spike in connections again. Teams are actively working on it

Investigating: After restarting, the database connections are under control. We are monitoring the system.

Investigating: Database is restarted and team is monitoring it

Investigating: Team is still working on the recovery.

Investigating: We are restarting the DB and API calls are expected to fail.

Investigating: We are continuing to investigate this issue.

Investigating: We are still working on identifying the underlying cause of DB connections

Investigating: We are seeing high DB usage causing the API to fail. We are checking internally.

Postmortem: A change in our monitoring system resulted in an alert erroneously firing indicating the database was down. In order to resolve this quickly the database was restarted, resulting in the outage.

Resolved: This incident has been resolved.

Investigating: On October 9, 2025, between approximately 09:48 UTC and 09:50 UTC, we experienced a short service disruption caused by an unexpected restart of production database. This may have resulted in connection errors for some users during this two-minute window. The database systems recovered automatically, and all services have been operating normally since. Our engineering team is actively investigating the root cause to prevent a recurrence.

Postmortem: High usage of database connections was identified around 6:00 PM ET. The team was able to identify an application process that was creating contention due to a lock wait, which persisted even after the request timed out. Both the application and the database were restarted to recover. The issue persisted intermittently but eventually resolved.

Resolved: This incident has been resolved.

Monitoring: We have identified and resolved the issue that was affecting our systems. Since implementing the fix, all systems have been operating normally with no recurrence of the problem. We have been closely monitoring our systems over the past hour, and all indicators show stable performance. Our monitoring infrastructure continues to track system health to detect any potential issues early. Our engineering team remains available to respond immediately if any concerns arise.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: We will continue to monitor the system and take appropriate the action.

Investigating: System is working as expected. We are monitoring the performance

Investigating: We are seeing spike in connections again. Teams are actively working on it

Investigating: After restarting, the database connections are under control. We are monitoring the system.

Investigating: Database is restarted and team is monitoring it

Investigating: Team is still working on the recovery.

Investigating: We are restarting the DB and API calls are expected to fail.

Investigating: We are continuing to investigate this issue.

Investigating: We are still working on identifying the underlying cause of DB connections

Investigating: We are seeing high DB usage causing the API to fail. We are checking internally.

Postmortem: A change in our monitoring system resulted in an alert erroneously firing indicating the database was down. In order to resolve this quickly the database was restarted, resulting in the outage.

Resolved: This incident has been resolved.

Investigating: On October 9, 2025, between approximately 09:48 UTC and 09:50 UTC, we experienced a short service disruption caused by an unexpected restart of production database. This may have resulted in connection errors for some users during this two-minute window. The database systems recovered automatically, and all services have been operating normally since. Our engineering team is actively investigating the root cause to prevent a recurrence.

Postmortem: High usage of database connections was identified around 6:00 PM ET. The team was able to identify an application process that was creating contention due to a lock wait, which persisted even after the request timed out. Both the application and the database were restarted to recover. The issue persisted intermittently but eventually resolved.

Resolved: This incident has been resolved.

Monitoring: We have identified and resolved the issue that was affecting our systems. Since implementing the fix, all systems have been operating normally with no recurrence of the problem. We have been closely monitoring our systems over the past hour, and all indicators show stable performance. Our monitoring infrastructure continues to track system health to detect any potential issues early. Our engineering team remains available to respond immediately if any concerns arise.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: We will continue to monitor the system and take appropriate the action.

Investigating: System is working as expected. We are monitoring the performance

Investigating: We are seeing spike in connections again. Teams are actively working on it

Investigating: After restarting, the database connections are under control. We are monitoring the system.

Investigating: Database is restarted and team is monitoring it

Investigating: Team is still working on the recovery.

Investigating: We are restarting the DB and API calls are expected to fail.

Investigating: We are continuing to investigate this issue.

Investigating: We are still working on identifying the underlying cause of DB connections

Investigating: We are seeing high DB usage causing the API to fail. We are checking internally.

Postmortem: High usage of database connections was identified around 6:00 PM ET. The team was able to identify an application process that was creating contention due to a lock wait, which persisted even after the request timed out. Both the application and the database were restarted to recover. The issue persisted intermittently but eventually resolved.

Resolved: This incident has been resolved.

Monitoring: We have identified and resolved the issue that was affecting our systems. Since implementing the fix, all systems have been operating normally with no recurrence of the problem. We have been closely monitoring our systems over the past hour, and all indicators show stable performance. Our monitoring infrastructure continues to track system health to detect any potential issues early. Our engineering team remains available to respond immediately if any concerns arise.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: We will continue to monitor the system and take appropriate the action.

Investigating: System is working as expected. We are monitoring the performance

Investigating: We are seeing spike in connections again. Teams are actively working on it

Investigating: After restarting, the database connections are under control. We are monitoring the system.

Investigating: Database is restarted and team is monitoring it

Investigating: Team is still working on the recovery.

Investigating: We are restarting the DB and API calls are expected to fail.

Investigating: We are continuing to investigate this issue.

Investigating: We are still working on identifying the underlying cause of DB connections

Investigating: We are seeing high DB usage causing the API to fail. We are checking internally.

Postmortem: A change in our monitoring system resulted in an alert erroneously firing indicating the database was down. In order to resolve this quickly the database was restarted, resulting in the outage.

Resolved: This incident has been resolved.

Investigating: On October 9, 2025, between approximately 09:48 UTC and 09:50 UTC, we experienced a short service disruption caused by an unexpected restart of production database. This may have resulted in connection errors for some users during this two-minute window. The database systems recovered automatically, and all services have been operating normally since. Our engineering team is actively investigating the root cause to prevent a recurrence.

Postmortem: A change in our monitoring system resulted in an alert erroneously firing indicating the database was down. In order to resolve this quickly the database was restarted, resulting in the outage.

Resolved: This incident has been resolved.

Investigating: On October 9, 2025, between approximately 09:48 UTC and 09:50 UTC, we experienced a short service disruption caused by an unexpected restart of production database. This may have resulted in connection errors for some users during this two-minute window. The database systems recovered automatically, and all services have been operating normally since. Our engineering team is actively investigating the root cause to prevent a recurrence.

Postmortem: A change in our monitoring system resulted in an alert erroneously firing indicating the database was down. In order to resolve this quickly the database was restarted, resulting in the outage.

Resolved: This incident has been resolved.

Investigating: On October 9, 2025, between approximately 09:48 UTC and 09:50 UTC, we experienced a short service disruption caused by an unexpected restart of production database. This may have resulted in connection errors for some users during this two-minute window. The database systems recovered automatically, and all services have been operating normally since. Our engineering team is actively investigating the root cause to prevent a recurrence.

Postmortem: A change in our monitoring system resulted in an alert erroneously firing indicating the database was down. In order to resolve this quickly the database was restarted, resulting in the outage.

Resolved: This incident has been resolved.

Investigating: On October 9, 2025, between approximately 09:48 UTC and 09:50 UTC, we experienced a short service disruption caused by an unexpected restart of production database. This may have resulted in connection errors for some users during this two-minute window. The database systems recovered automatically, and all services have been operating normally since. Our engineering team is actively investigating the root cause to prevent a recurrence.