Hearsay Systems Status

Postmortem: Timeline:

Start Time: 02/20/25 06:10 AM PT

End Time: 02/20/25 08:15 AM PT

Customer Impact:

Due to a production deployment on February 19th, 2025, Hearsay’s application became temporarily unavailable on February 20th, 2025. Users accessing Hearsay during this incident would see a 500 Internal Server Error message.

The impact on Relate resulted in inbound and outbound text messages being temporarily queued. Once service was restored, all queued inbound and outbound messages were successfully received and delivered with a delay. New inbound and outbound voice calls during this incident were unavailable.

The impact on Social resulted in a delay in scheduled content, with some content now flagged as “Needs Attention” for a manual review.

The impact on our Activity Sync was disrupted, preventing custom workflows for a limited number of customers.

The impact on Actions resulted in unsuccessful API calls through our Public API. All requests during the following period, 6:12 AM PST to 7:52 PST, were not successfully processed. The already queued messages have been sent out. The API returned 500 response codes, which could be gathered on the customer’s side.

All archiving activities have since resumed and may be delayed.

During this incident, there was no data loss.

Resolution:

Hearsay’s engineers rolled back the infrastructure upgrade that caused the problem and redeployed production services to use the earlier stable version.

Future Improvements:

We are exploring alternative methods to achieve the same infrastructural upgrade that was the original goal of the change that caused the outage. In addition, although a significant amount of planning and care was taken to minimize customer impact, any future changes in this area will involve additional scrutiny and testing.

Resolved: Hearsay will mark this incident as resolved, as our platform is functioning as expected. After a thorough investigation has been completed, we will provide a detailed root cause analysis. Kindly log in to the Hearsay platform to ensure everything is working for your users and to confirm that social media posts, text messages, and API calls are all successful.

Monitoring: Hearsay's engineers have identified the issue and implemented a fix to production. We are currently seeing the Hearsay platform load successfully. Please note that if you are still running into errors, please wait a few moments and try again. We will provide another update shortly to confirm this incident is fully resolved.

Investigating: Hearsay's engineers are actively investigating this matter as it remains our top priority. We will continue to provide timely updates on this incident. Thank you for your patience as we work through this matter.

Investigating: Hearsay's engineers are still investigating this incident resulting in the Internal Server Error message. We will provide another update shortly. We appreciate your patience during this time.

Investigating: Hearsay is currently investigating an incident where our platform is resulting in Internal Server Errors. We are treating this with the utmost priority and will share an update soon. We truly apologize for any disruption to your business.

Resolved: Hearsay successfully reverted to a previous version of the platform, and users should no longer experience any reporting issues. After a thorough investigation has been completed, we will provide a post-mortem on this incident.

Investigating: Hearsays engineers are investigating an incident with our reporting where some admins may notice the filter options are temporarily unavailable. Some admins may experience an empty report, but clicking the refresh icon should load the data. At this moment, downloading reports may be temporarily unavailable. Hearsay is treating this as our top priority and will provide an update once we have more information.