PostHog Status

Resolved: This incident has been resolved.

Investigating: We're experiencing an elevated level of API errors and are currently looking into the issue.

Resolved: The email delivery delays have been resolved.

Monitoring: Customer.io incident resolved, we re-enabled verification emails

Monitoring: We're seeing delays with verification and password reset emails due to an ongoing incident with Customer.io: https://status.customerio.com/incidents/x5hwkcw0ddds To unblock logins, we've temporarily disabled verification emails until the issue is resolved.

Investigating: We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

Investigating: We're seeing an unusual spike in tickets about login verification emails and password reset emails not being received. We're currently investigating.

Resolved: Ingestion has caught up and everything is back to normal.

Monitoring: The query load issue is resolved and queries should be loading as normal. We're still working through the backlog of events, so some charts might be showing data that is ~30 minutes old. That should be resolved soon.

No data was lost.

Monitoring: Cluster is looking stable now and we have been able to resume the ingestion of events.

Query performance should be now back to normal, as we are not hitting the limits anymore.

There is still events lag that we are already consuming, so data shown won't be up to date yet. We'll send an update once it's completely recovered.

No data has been lost during this period.

Identified: We are continuing to see queries failing and ingestion lag. We are switching on more capacity, which should hopefully resolve the queries failing (though they'll be showing data that is ~30-60 minutes out of date).

This is not impacting workflows or CDP, and no data has been lost.

We will keep you up to date.

Identified: We are under heavy load, and seeing query outages and delayed ingestion of events. No data is lost.

Identified: Our ClickHouse cluster is going under heavy load right now and the ingestion of events is being delayed.

We have found the root cause and are now working to leave the cluster in a stable state so we can catch up on the ingestion.

Resolved: Ingestion has caught up and everything is back to normal.

Monitoring: The query load issue is resolved and queries should be loading as normal. We're still working through the backlog of events, so some charts might be showing data that is ~30 minutes old. That should be resolved soon.

No data was lost.

Monitoring: Cluster is looking stable now and we have been able to resume the ingestion of events.

Query performance should be now back to normal, as we are not hitting the limits anymore.

There is still events lag that we are already consuming, so data shown won't be up to date yet. We'll send an update once it's completely recovered.

No data has been lost during this period.

Identified: We are continuing to see queries failing and ingestion lag. We are switching on more capacity, which should hopefully resolve the queries failing (though they'll be showing data that is ~30-60 minutes out of date).

This is not impacting workflows or CDP, and no data has been lost.

We will keep you up to date.

Identified: We are under heavy load, and seeing query outages and delayed ingestion of events. No data is lost.

Identified: Our ClickHouse cluster is going under heavy load right now and the ingestion of events is being delayed.

We have found the root cause and are now working to leave the cluster in a stable state so we can catch up on the ingestion.

Resolved: The email delivery delays have been resolved.

Monitoring: Customer.io incident resolved, we re-enabled verification emails

Monitoring: We're seeing delays with verification and password reset emails due to an ongoing incident with Customer.io: https://status.customerio.com/incidents/x5hwkcw0ddds To unblock logins, we've temporarily disabled verification emails until the issue is resolved.

Investigating: We've spotted that something has gone wrong. We're currently investigating the issue, and will provide an update soon.

Investigating: We're seeing an unusual spike in tickets about login verification emails and password reset emails not being received. We're currently investigating.

Resolved: We identified a faulty compute node as root cause of this.

After moving all workloads off this node and removing the node from our infrastructure, error rates are consistently within our expected values.

Thank you for your patience.

Monitoring: We found a few faulty webservers being responsible for this and removed from load balancing.

Seeing signs of recovery and monitoring.

Investigating: We're experiencing an elevated level of API errors in the web portal and parts of the API and are currently looking into the issue.