Verkada Command Status

Investigating: We are currently investigating this issue.

Postmortem: ### Post-Incident Report: Camera Search Features Disruption

Between the evening of September 30 and the afternoon of October 2, we experienced a service disruption that impacted several camera search features within our Command platform in the United States region. The issue, which began on the evening of September 30, impacted LPR, AI-powered, Attribute, and Face Search. Importantly, all core camera services, including live video streaming, system alerts, and archives, were unaffected by this incident, and no customer data was lost as a result.

The initial cause is believed to be a routine update that unintentionally reduced system capacity, leading to a service overload. Our team worked in close collaboration with our cloud provider to restore service, and restored impacted features on a rolling basis.

Now that the impacted analytics features have been restored, we are continuing to conduct a thorough joint investigation with our cloud provider in an effort to further support the resilience of our platform. We sincerely apologize for this disruption and thank you for your continued partnership.

Resolved: All camera analytics search features, including AI-powered search, attribute search, face search, and license plate recognition (LPR) search, have been fully restored in the United States region. Indexing is fully caught up, and the system is performing normally. No customer data was lost during this incident, and we appreciate your patience throughout the disruption.

Monitoring: We are continuing to monitor for any further issues. We will post a final update once indexing for AI-powered search and attribute search are fully caught up in the United States region. Thank you for your patience.

Monitoring: All search functionality has been restored and is working. We are currently catching up on indexing for AI-powered search and attribute search, which may cause some results to be not fully up to date. Search results will continue to improve as indexing completes. Our team is monitoring the system closely to ensure normal performance in the United States region. Other search features and regions remain unaffected.

Identified: Face search and attribute search have been partially restored. Searches are working as expected, but indexing has not yet resumed, which may cause some results to appear out of date. Our engineering team continues to restore normal indexing performance in the United States region. Other search features and regions remain unaffected, and we will provide further updates as progress continues.

Identified: We have restored License Plate Recognition (LPR) search functionality. Our engineering team continues to work on fully restoring AI-powered search and attribute search features in the United States region. Other search features and regions remain unaffected, and we will provide further updates as progress continues. Thank you for your patience as we work to resolve this issue.

Identified: We are close to restoring License Plate Recognition (LPR) search functionality and expect it to be available soon. Our engineering team continues to work on fully restoring AI-powered search and attribute search features in the United States region. Other search features and regions remain unaffected, and we will provide further updates as progress continues. Thank you for your patience as we work to resolve this issue.

Identified: Face search has been restored and is now functioning normally. Our engineering team is actively working to restore AI-powered search, attribute search, and license plate recognition (LPR) search features. Other search features remain unaffected, and we will provide further updates as progress continues. Thank you for your patience as we work to fully resolve this issue.

Identified: LPR unlocks are not affected. We are continuing to work on a fix for this issue.

Identified: We are currently experiencing a partial outage affecting our camera analytics in the United States region, including AI-powered search, attribute search, face search, and license plate recognition (LPR) features. Other search features are not impacted at this time. Our engineering team is actively working on a fix and will provide further updates as progress continues.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Resolved: This incident has been resolved.

Investigating: Password logins and Single Sign On still works fine. We are investigating this issue.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently investigating this issue. We recommend granting site-viewer permissions if possible, or use the Verkada pass app as a workaround.