PostHog Status

Resolved: After monitoring we have seen that all systems are working as normally.

Monitoring: We have been able to identify the root cause and pushed a fix to get event ingestion back to normal and latency is now back to normal.

We'll keep monitoring the infrastructure.

Investigating: Our ingestion infrastructure is processing slowly causing delays for event ingestion. We are investigating what could be the root cause.

Resolved: This incident has been resolved.

Monitoring: We have narrowed down the problem to a very small set of Snowflake batch exports that we have manually cancelled. If you were affected we will be reaching out.

All other batch exports are fully recovered or on the path to recovery. Performance of ongoing batch exports will soon be on pace with real time once again.

Investigating: We were unable to make a full recovery, and the issue seems to persist. We are investigating new potential fixes. In the meantime, batch exports will be delayed.

Monitoring: We are monitoring the backfill process for Snowflake batch exports and any pending large batch exports for other destinations. All backfills are progressing normally.

On-going batch exports are operating normally, but users with pending backfills may see us lag behind real time until all the backfilling is done.

Identified: We have deployed our fixes and have managed to resolve the concurrency issues with Snowflake batch exports.

Most batch exports besides Snowflake should be fully recovered, with the exception of larger batch exports that will still need some time to work through the backlog.

We will shortly begin backfilling any Snowflake batch exports that were cancelled due to this incident.

Identified: We are continuing to work on a fix for this issue.

Identified: We have reasons to believe the cause of the problem is a deadlock happening while connecting to Snowflake. We are attempting to deploy a patch that would deal with the deadlock when it happens, leaving the investigation of what is causing the deadlock for later.

Assuming the patch is successful in addressing the problem we will begin back filling any Snowflake batch exports that were cancelled.

Investigating: We have reason to believe that the problem is related to Snowflake batch exports. In consequence, most Snowflake batch exports are being cancelled to be retried at a later date. We are investigating how to remediate the problem with Snowflake.

Users of other destinations should see batch exports recovering over time. Depending on the size of the data exported, this recovery could take less or more time.

Investigating: We have been making slow progress on batch exports and a backlog has built up, particularly on larger batch exports. It is taking us some time to work through the backlog, so users may see batch exports be delayed in delivering data.

No data loss has happened nor is it expected.

Resolved: This incident has been resolved.

Monitoring: We have narrowed down the problem to a very small set of Snowflake batch exports that we have manually cancelled. If you were affected we will be reaching out.

All other batch exports are fully recovered or on the path to recovery. Performance of ongoing batch exports will soon be on pace with real time once again.

Investigating: We were unable to make a full recovery, and the issue seems to persist. We are investigating new potential fixes. In the meantime, batch exports will be delayed.

Monitoring: We are monitoring the backfill process for Snowflake batch exports and any pending large batch exports for other destinations. All backfills are progressing normally.

On-going batch exports are operating normally, but users with pending backfills may see us lag behind real time until all the backfilling is done.

Identified: We have deployed our fixes and have managed to resolve the concurrency issues with Snowflake batch exports.

Most batch exports besides Snowflake should be fully recovered, with the exception of larger batch exports that will still need some time to work through the backlog.

We will shortly begin backfilling any Snowflake batch exports that were cancelled due to this incident.

Identified: We are continuing to work on a fix for this issue.

Identified: We have reasons to believe the cause of the problem is a deadlock happening while connecting to Snowflake. We are attempting to deploy a patch that would deal with the deadlock when it happens, leaving the investigation of what is causing the deadlock for later.

Assuming the patch is successful in addressing the problem we will begin back filling any Snowflake batch exports that were cancelled.

Investigating: We have reason to believe that the problem is related to Snowflake batch exports. In consequence, most Snowflake batch exports are being cancelled to be retried at a later date. We are investigating how to remediate the problem with Snowflake.

Users of other destinations should see batch exports recovering over time. Depending on the size of the data exported, this recovery could take less or more time.

Investigating: We have been making slow progress on batch exports and a backlog has built up, particularly on larger batch exports. It is taking us some time to work through the backlog, so users may see batch exports be delayed in delivering data.

No data loss has happened nor is it expected.