PostHog Status

Resolved: Lag has recovered and the system is completely functional again.

Sorry for any inconvenience caused by this incident.

Monitoring: The cluster is now responsive and the data ingestion has been resumed. The app is responding better now.

We are still monitoring a couple of fixes we have pushed. We identified a query that was flooding the cluster and which may have been the root cause of this.

Investigating: We have recovered a good part of the cluster, but we are still working to bring it back completely.

The performance may be still degraded. We think some problematic queries may have been the root cause, we are still investigating it.

Investigating: We are trying to bring back the cluster.

The app may be completely unresponsive, and lag is expected during this time, we'll try to provide an update as soon as possible.

Investigating: We have detected a partial outage in our ClickHouse cluster and it's impacting the application response and performance getting insights.

We are investigating the root cause.

Resolved: We rolled out a change that increased load causing queries to be slower. We rolled back that change so performance should be back up.

Investigating: We have spotted that our data infrastructure is under heavy load and it's impacting the time the app takes to load insights or leading to errors when loading them.

We are investigating what could be the root cause.

Resolved: We've not seen a reoccurence of this issue so closing this incident now.

Monitoring: We're still investigating high load for some offline functionality (ie exports) but the vast majority of the app should work fine now

Investigating: Our US app instance is down and pods are unhealthy. We're figuring out why and are working on resolving.

Data ingestion and feature flags are not affected.

Resolved: Lag has recovered and the system is completely functional again.

Sorry for any inconvenience caused by this incident.

Monitoring: The cluster is now responsive and the data ingestion has been resumed. The app is responding better now.

We are still monitoring a couple of fixes we have pushed. We identified a query that was flooding the cluster and which may have been the root cause of this.

Investigating: We have recovered a good part of the cluster, but we are still working to bring it back completely.

The performance may be still degraded. We think some problematic queries may have been the root cause, we are still investigating it.

Investigating: We are trying to bring back the cluster.

The app may be completely unresponsive, and lag is expected during this time, we'll try to provide an update as soon as possible.

Investigating: We have detected a partial outage in our ClickHouse cluster and it's impacting the application response and performance getting insights.

We are investigating the root cause.