Springboard Health Status

Resolved: After an exhaustive review of error logs, we have verified that the impact was contained to a handful of Lucidworks Search customers, and have already taken action to rotate all affected credentials.

Lucidworks Search implementations include a failsafe mechanism to fall back to lexical-only search in the event that LWAI is unreachable, which would have prevented search errors from any impacted customer environments.  Clients with custom Lucidworks Fusion implementations are encouraged to ensure their particular implementation employs a similar mechanism, following our standard best practice guidance.

Out of an abundance of caution, we will leave the cleanup job that was at the source of this issue turned off over the weekend. The fix for it has already been tested and verified, and will be deployed to production under scrutiny on Monday, August 10.

Any customer still experiencing any issues is encouraged to please follow up with your Lucidworks Customer Excellence representative or Lucidworks Support for further details and assistance.

Monitoring: We have identified the issue to be caused by a background cleanup job that began unintentionally rotating customer application credentials in certain scenarios, beginning around 1700 UTC on August 5.  The job runs periodically, and only in specific edge cases will erroneously identify a given set of client credentials as needing to be replaced.

We have disabled this job in order to mitigate the issue so that it cannot continue to affect any additional customers.  We are producing a report of affected client credentials and Lucidworks Support will be reaching out to each directly via Zendesk to inform them of what needs to be done in order to resolve the issue.

A fix for the underlying bug has been prepared and is currently undergoing testing before being rolled out to our production environment.  We will post another update here as soon as that has been deployed.

Investigating: A subset of Lucidworks Platform, Lucidworks AI, and Connected Search customers are currently experiencing a service disruption related to invalid credentials. End users relying on search and AI capabilities across impacted clients may experience request failures. The root cause of this issue is currently under investigation, but is initially believed to be related to an unintended secret rotation within the Lucidworks Platform backend that only affects certain customer applications. We are actively investigating the situation to fully determine the cause and mitigate the disruption. We will provide further updates as soon as more information becomes available.

Resolved: After an exhaustive review of error logs, we have verified that the impact was contained to a handful of Lucidworks Search customers, and have already taken action to rotate all affected credentials.

Lucidworks Search implementations include a failsafe mechanism to fall back to lexical-only search in the event that LWAI is unreachable, which would have prevented search errors from any impacted customer environments.  Clients with custom Lucidworks Fusion implementations are encouraged to ensure their particular implementation employs a similar mechanism, following our standard best practice guidance.

Out of an abundance of caution, we will leave the cleanup job that was at the source of this issue turned off over the weekend. The fix for it has already been tested and verified, and will be deployed to production under scrutiny on Monday, August 10.

Any customer still experiencing any issues is encouraged to please follow up with your Lucidworks Customer Excellence representative or Lucidworks Support for further details and assistance.

Monitoring: We have identified the issue to be caused by a background cleanup job that began unintentionally rotating customer application credentials in certain scenarios, beginning around 1700 UTC on August 5.  The job runs periodically, and only in specific edge cases will erroneously identify a given set of client credentials as needing to be replaced.

We have disabled this job in order to mitigate the issue so that it cannot continue to affect any additional customers.  We are producing a report of affected client credentials and Lucidworks Support will be reaching out to each directly via Zendesk to inform them of what needs to be done in order to resolve the issue.

A fix for the underlying bug has been prepared and is currently undergoing testing before being rolled out to our production environment.  We will post another update here as soon as that has been deployed.

Investigating: A subset of Lucidworks Platform, Lucidworks AI, and Connected Search customers are currently experiencing a service disruption related to invalid credentials. End users relying on search and AI capabilities across impacted clients may experience request failures. The root cause of this issue is currently under investigation, but is initially believed to be related to an unintended secret rotation within the Lucidworks Platform backend that only affects certain customer applications. We are actively investigating the situation to fully determine the cause and mitigate the disruption. We will provide further updates as soon as more information becomes available.

Resolved: Connected Search and the Lucidworks Platform are fully operational, having experienced no service degradation.  The root cause was confirmed to be isolated to a single customer issuing search requests using invalid credentials. The issue was validated and resolved after confirming all backend systems and platform functionality remained operating normally. Other customers utilizing the Lucidworks Platform are experiencing no impact.

Please contact Lucidworks Support (https://support.lucidworks.com/hc/en-us) for any questions or assistance needed.

Identified: The Connected Search service disruption is limited in scope to a specific client configuration. Affected customers are experiencing 401 Unauthorized response codes related to invalid authorization credentials when attempting to perform search queries through this integration, preventing end users from receiving search results or suggestions.

We have confirmed that this issue is isolated to requests using invalid or expired credentials and that backend services and Connected Search functionality remain fully operational for all other clients. We are actively reproducing the exact credential behavior to resolve the disruption. Further updates will be provided as additional details become available.

Investigating: Lucidworks is currently responding to a report of service disruption affecting Connected Search functionality. While administrative operations within the Lucidworks Platform dashboard remain fully functional, external search operations for a subset of customers utilizing Connected Search are experiencing impact. End users accessing search through affected external applications are currently unable to receive search results or predictive suggestions. We are actively investigating this issue to identify the underlying cause and restore standard operation. Further updates will be provided as additional details become available.

Resolved: After an exhaustive review of error logs, we have verified that the impact was contained to a handful of Lucidworks Search customers, and have already taken action to rotate all affected credentials.

Lucidworks Search implementations include a failsafe mechanism to fall back to lexical-only search in the event that LWAI is unreachable, which would have prevented search errors from any impacted customer environments.  Clients with custom Lucidworks Fusion implementations are encouraged to ensure their particular implementation employs a similar mechanism, following our standard best practice guidance.

Out of an abundance of caution, we will leave the cleanup job that was at the source of this issue turned off over the weekend. The fix for it has already been tested and verified, and will be deployed to production under scrutiny on Monday, August 10.

Any customer still experiencing any issues is encouraged to please follow up with your Lucidworks Customer Excellence representative or Lucidworks Support for further details and assistance.

Monitoring: We have identified the issue to be caused by a background cleanup job that began unintentionally rotating customer application credentials in certain scenarios, beginning around 1700 UTC on August 5.  The job runs periodically, and only in specific edge cases will erroneously identify a given set of client credentials as needing to be replaced.

We have disabled this job in order to mitigate the issue so that it cannot continue to affect any additional customers.  We are producing a report of affected client credentials and Lucidworks Support will be reaching out to each directly via Zendesk to inform them of what needs to be done in order to resolve the issue.

A fix for the underlying bug has been prepared and is currently undergoing testing before being rolled out to our production environment.  We will post another update here as soon as that has been deployed.

Investigating: A subset of Lucidworks Platform, Lucidworks AI, and Connected Search customers are currently experiencing a service disruption related to invalid credentials. End users relying on search and AI capabilities across impacted clients may experience request failures. The root cause of this issue is currently under investigation, but is initially believed to be related to an unintended secret rotation within the Lucidworks Platform backend that only affects certain customer applications. We are actively investigating the situation to fully determine the cause and mitigate the disruption. We will provide further updates as soon as more information becomes available.