PostHog Status

Resolved: We've resolved the incident. This just affected querying and no data was lost

We're still working on finding the root cause for this issue (our clickhouse nodes were segfaulting without warning) and will continue to monitor.

Investigating: We're experiencing failures to load data across the entire app at the moment. We've identified the root cause and are working to resolve this asap.

Investigating: We're experiencing failures to load data across the entire app at the moment. We've identified the root cause and are working to resolve this asap.

Resolved: Parts are back to normal and the cluster is responding normally again.

We'll keep monitoring it and let you know if we find misbehavior again.

Investigating: We are currently investigating increased parts counts on our datastore and we are investigating why these parts are not being merged as they should. This will cause increased query times.

Resolved: We've caught up on our backlog of messages. Ingestion rates look optimal. Parts are being merged as they should. New nodes are fully online. Query latencies are looking great at 100ms avg. Should be smooth sailing from here on out. Enjoy your Friday!

Monitoring: There were some recurring errors in our infrastructure that led us to restart clickhouse nodes.

We are falling behind on events ingestion, as we are replacing some nodes in our ClickHouse cluster. This will increase lag in our ingestion pipeline. Performance may be impacted during this time too. We are still working on this and monitoring it.

Resolved: We've resolved the incident. This just affected querying and no data was lost

We're still working on finding the root cause for this issue (our clickhouse nodes were segfaulting without warning) and will continue to monitor.

Investigating: We're experiencing failures to load data across the entire app at the moment. We've identified the root cause and are working to resolve this asap.

Investigating: We're experiencing failures to load data across the entire app at the moment. We've identified the root cause and are working to resolve this asap.