Treasure Data Status

Investigating: Our utilization dashboards are not updating with up-to-date information. Customers accessing their Treasure Data usage dashboards will see a gap in usage details from early morning Saturday UTC. There is no impact to ongoing usage of Treasure Data, and all usage information is correctly being stored internally. However, the dashboard where customers can view their consumption is not up-to-date. We are working to diagnose the issue, and will provide an update in the next hour.

Monitoring: We have been monitoring the situation for the past 6 hours. At this time, we are unable to provide an updated estimate for full recovery. We are continuing to monitor closely and are also exploring additional measures to improve recovery performance. We will share another update in approximately 6 hours or sooner if there is any significant development. We apologize for the inconvenience.

Monitoring: We have rolled out a fix and are observing our backlog recovering at a safe rate. We are currently processing a backlog of historical data, which is progressing at a controlled rate. Estimated time to a full recover of all data is approximately 32 hours, and we will be monitoring and provide updates through this process.

Identified: We are now attempting to restore the pipeline from a timestamp rather than a checkpoint. There is no risk of user data loss. Downstream systems may observe duplicate events from the affected time window, and this will be limited in scope.

Identified: We have applied a fix and are observing it.

We have identified a window between Thursday 3AM UTC and 11AM Monday UTC where data was not being processed. We have implemented a remediation and new incoming data is not impacted. We are actively working to resume sending of this data, which will arrive out of order along with new incoming data.

Investigating: We are continuing to investigate this issue.

Investigating: Our Ingest API is experiencing a performance issue. We are investigating the cause.

Investigating: Our utilization dashboards are not updating with up-to-date information. Customers accessing their Treasure Data usage dashboards will see a gap in usage details from early morning Saturday UTC. There is no impact to ongoing usage of Treasure Data, and all usage information is correctly being stored internally. However, the dashboard where customers can view their consumption is not up-to-date. We are working to diagnose the issue, and will provide an update in the next hour.

Investigating: Our utilization dashboards are not updating with up-to-date information. Customers accessing their Treasure Data usage dashboards will see a gap in usage details from early morning Saturday UTC. There is no impact to ongoing usage of Treasure Data, and all usage information is correctly being stored internally. However, the dashboard where customers can view their consumption is not up-to-date. We are working to diagnose the issue, and will provide an update in the next hour.

Investigating: Our utilization dashboards are not updating with up-to-date information. Customers accessing their Treasure Data usage dashboards will see a gap in usage details from early morning Saturday UTC. There is no impact to ongoing usage of Treasure Data, and all usage information is correctly being stored internally. However, the dashboard where customers can view their consumption is not up-to-date. We are working to diagnose the issue, and will provide an update in the next hour.

Investigating: Our utilization dashboards are not updating with up-to-date information. Customers accessing their Treasure Data usage dashboards will see a gap in usage details from early morning Saturday UTC. There is no impact to ongoing usage of Treasure Data, and all usage information is correctly being stored internally. However, the dashboard where customers can view their consumption is not up-to-date. We are working to diagnose the issue, and will provide an update in the next hour.