Treasure Data Status

Resolved: Summary: The incident affecting the Personalization Lookup API and related services has been resolved. All systems are now operating normally.

The issue began at approximately 5:47 AM PT, causing elevated error rates, increased latency, and impacting multiple services including CDP KVS Server and Ingest Aggregator. The primary incident was resolved at 6:33 AM PT.

Resolution Timeline:

CDP KVS Server/Personalization API: Resolved ~6:33 AM PT CDP-API: Recovered shortly after Ingest Aggregator: Currently recovering and expected to resolve completely All related alerts have cleared and systems are stable. The engineering team will conduct a post-incident analysis to determine the root cause.

Affected Components:

Personalization Lookup API CDP KVS Server CDP-API Ingest Aggregator

Status: Resolved

Next Steps: Root cause analysis will be conducted by the service team.

Resolved: Services have been restored to normal operations, and all of our systems are reporting healthy once again.

If you observe any lingering issues, please connect with our support team, and we would be happy to help: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support

Monitoring: Service levels continue to improve as upstream services become more stable. Users may still see processing delays, though those should be decreasing as we process the backlog of messages from this event.

We are continuing to monitor our systems and deploy capacity as needed. We will update you with additional information as it develops.

Monitoring: We are seeing continued recovery across our systems, and are currently working to process any buffered or delayed messages. Customers may continue to observe delays in data processing and may notice slower-than-usual processing of configuration changes. We expect these delays to decrease as our systems catch up.

Our team is continuing to track recovery closely, and we'll inform you of any new developments.

Investigating: We are continuing to monitor ongoing recovery efforts, and most of our systems remain operating. As the situation with our upstream provider develops we will react if we discover any new issues.

At this time, we are observing delays in our systems, including Profile API 1.0 and Realtime 2.0. Requests are being processed as they have been received, but users may see a slowdown in results, and may notice delays when updating configurations or journeys.

You may notice continued slower performance or unexpected errors. We are working with our service provider to address these issues and will provide an update with new information soon.

Investigating: Our upstream service provider is gradually recovering its services, and our services are gradually recovering too. Most of our Treasure Data services are back to normal except for Custom Script, autoML and realtime services such as profile API 1.0, and realtime 2.0.

As the upstream service provider incident is still ongoing, you might still encounter unexpected errors or issues. We will closely monitoring our services and work with our upstream service provider.

Investigating: We are currently investigating an issue originating from an upstream service provider used by our infrastructure. This may be affecting the normal operation of some Treasure Data services.

We are working to determine the scope of the impact. Observed symptoms may include, but are not limited to, the failure of various jobs (such as Trino, Hive, Result Export, Import, and Custom Script execution) to start.

Our team is actively monitoring the situation and working with the provider. We will provide further updates as more information becomes available.

Resolved: Services have been restored to normal operations, and all of our systems are reporting healthy once again.

If you observe any lingering issues, please connect with our support team, and we would be happy to help: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support

Monitoring: Service levels continue to improve as upstream services become more stable. Users may still see processing delays, though those should be decreasing as we process the backlog of messages from this event.

We are continuing to monitor our systems and deploy capacity as needed. We will update you with additional information as it develops.

Monitoring: We are seeing continued recovery across our systems, and are currently working to process any buffered or delayed messages. Customers may continue to observe delays in data processing and may notice slower-than-usual processing of configuration changes. We expect these delays to decrease as our systems catch up.

Our team is continuing to track recovery closely, and we'll inform you of any new developments.

Investigating: We are continuing to monitor ongoing recovery efforts, and most of our systems remain operating. As the situation with our upstream provider develops we will react if we discover any new issues.

At this time, we are observing delays in our systems, including Profile API 1.0 and Realtime 2.0. Requests are being processed as they have been received, but users may see a slowdown in results, and may notice delays when updating configurations or journeys.

You may notice continued slower performance or unexpected errors. We are working with our service provider to address these issues and will provide an update with new information soon.

Investigating: Our upstream service provider is gradually recovering its services, and our services are gradually recovering too. Most of our Treasure Data services are back to normal except for Custom Script, autoML and realtime services such as profile API 1.0, and realtime 2.0.

As the upstream service provider incident is still ongoing, you might still encounter unexpected errors or issues. We will closely monitoring our services and work with our upstream service provider.

Investigating: We are currently investigating an issue originating from an upstream service provider used by our infrastructure. This may be affecting the normal operation of some Treasure Data services.

We are working to determine the scope of the impact. Observed symptoms may include, but are not limited to, the failure of various jobs (such as Trino, Hive, Result Export, Import, and Custom Script execution) to start.

Our team is actively monitoring the situation and working with the provider. We will provide further updates as more information becomes available.

Resolved: Services have been restored to normal operations, and all of our systems are reporting healthy once again.

If you observe any lingering issues, please connect with our support team, and we would be happy to help: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support

Monitoring: Service levels continue to improve as upstream services become more stable. Users may still see processing delays, though those should be decreasing as we process the backlog of messages from this event.

We are continuing to monitor our systems and deploy capacity as needed. We will update you with additional information as it develops.

Monitoring: We are seeing continued recovery across our systems, and are currently working to process any buffered or delayed messages. Customers may continue to observe delays in data processing and may notice slower-than-usual processing of configuration changes. We expect these delays to decrease as our systems catch up.

Our team is continuing to track recovery closely, and we'll inform you of any new developments.

Investigating: We are continuing to monitor ongoing recovery efforts, and most of our systems remain operating. As the situation with our upstream provider develops we will react if we discover any new issues.

At this time, we are observing delays in our systems, including Profile API 1.0 and Realtime 2.0. Requests are being processed as they have been received, but users may see a slowdown in results, and may notice delays when updating configurations or journeys.

You may notice continued slower performance or unexpected errors. We are working with our service provider to address these issues and will provide an update with new information soon.

Investigating: Our upstream service provider is gradually recovering its services, and our services are gradually recovering too. Most of our Treasure Data services are back to normal except for Custom Script, autoML and realtime services such as profile API 1.0, and realtime 2.0.

As the upstream service provider incident is still ongoing, you might still encounter unexpected errors or issues. We will closely monitoring our services and work with our upstream service provider.

Investigating: We are currently investigating an issue originating from an upstream service provider used by our infrastructure. This may be affecting the normal operation of some Treasure Data services.

We are working to determine the scope of the impact. Observed symptoms may include, but are not limited to, the failure of various jobs (such as Trino, Hive, Result Export, Import, and Custom Script execution) to start.

Our team is actively monitoring the situation and working with the provider. We will provide further updates as more information becomes available.

Resolved: Services have been restored to normal operations, and all of our systems are reporting healthy once again.

If you observe any lingering issues, please connect with our support team, and we would be happy to help: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support

Monitoring: Service levels continue to improve as upstream services become more stable. Users may still see processing delays, though those should be decreasing as we process the backlog of messages from this event.

We are continuing to monitor our systems and deploy capacity as needed. We will update you with additional information as it develops.

Monitoring: We are seeing continued recovery across our systems, and are currently working to process any buffered or delayed messages. Customers may continue to observe delays in data processing and may notice slower-than-usual processing of configuration changes. We expect these delays to decrease as our systems catch up.

Our team is continuing to track recovery closely, and we'll inform you of any new developments.

Investigating: We are continuing to monitor ongoing recovery efforts, and most of our systems remain operating. As the situation with our upstream provider develops we will react if we discover any new issues.

At this time, we are observing delays in our systems, including Profile API 1.0 and Realtime 2.0. Requests are being processed as they have been received, but users may see a slowdown in results, and may notice delays when updating configurations or journeys.

You may notice continued slower performance or unexpected errors. We are working with our service provider to address these issues and will provide an update with new information soon.

Investigating: Our upstream service provider is gradually recovering its services, and our services are gradually recovering too. Most of our Treasure Data services are back to normal except for Custom Script, autoML and realtime services such as profile API 1.0, and realtime 2.0.

As the upstream service provider incident is still ongoing, you might still encounter unexpected errors or issues. We will closely monitoring our services and work with our upstream service provider.

Investigating: We are currently investigating an issue originating from an upstream service provider used by our infrastructure. This may be affecting the normal operation of some Treasure Data services.

We are working to determine the scope of the impact. Observed symptoms may include, but are not limited to, the failure of various jobs (such as Trino, Hive, Result Export, Import, and Custom Script execution) to start.

Our team is actively monitoring the situation and working with the provider. We will provide further updates as more information becomes available.

Resolved: Services have been restored to normal operations, and all of our systems are reporting healthy once again.

If you observe any lingering issues, please connect with our support team, and we would be happy to help: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support

Monitoring: Service levels continue to improve as upstream services become more stable. Users may still see processing delays, though those should be decreasing as we process the backlog of messages from this event.

We are continuing to monitor our systems and deploy capacity as needed. We will update you with additional information as it develops.

Monitoring: We are seeing continued recovery across our systems, and are currently working to process any buffered or delayed messages. Customers may continue to observe delays in data processing and may notice slower-than-usual processing of configuration changes. We expect these delays to decrease as our systems catch up.

Our team is continuing to track recovery closely, and we'll inform you of any new developments.

Investigating: We are continuing to monitor ongoing recovery efforts, and most of our systems remain operating. As the situation with our upstream provider develops we will react if we discover any new issues.

At this time, we are observing delays in our systems, including Profile API 1.0 and Realtime 2.0. Requests are being processed as they have been received, but users may see a slowdown in results, and may notice delays when updating configurations or journeys.

You may notice continued slower performance or unexpected errors. We are working with our service provider to address these issues and will provide an update with new information soon.

Investigating: Our upstream service provider is gradually recovering its services, and our services are gradually recovering too. Most of our Treasure Data services are back to normal except for Custom Script, autoML and realtime services such as profile API 1.0, and realtime 2.0.

As the upstream service provider incident is still ongoing, you might still encounter unexpected errors or issues. We will closely monitoring our services and work with our upstream service provider.

Investigating: We are currently investigating an issue originating from an upstream service provider used by our infrastructure. This may be affecting the normal operation of some Treasure Data services.

We are working to determine the scope of the impact. Observed symptoms may include, but are not limited to, the failure of various jobs (such as Trino, Hive, Result Export, Import, and Custom Script execution) to start.

Our team is actively monitoring the situation and working with the provider. We will provide further updates as more information becomes available.

Resolved: Services have been restored to normal operations, and all of our systems are reporting healthy once again.

If you observe any lingering issues, please connect with our support team, and we would be happy to help: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support

Monitoring: Service levels continue to improve as upstream services become more stable. Users may still see processing delays, though those should be decreasing as we process the backlog of messages from this event.

We are continuing to monitor our systems and deploy capacity as needed. We will update you with additional information as it develops.

Monitoring: We are seeing continued recovery across our systems, and are currently working to process any buffered or delayed messages. Customers may continue to observe delays in data processing and may notice slower-than-usual processing of configuration changes. We expect these delays to decrease as our systems catch up.

Our team is continuing to track recovery closely, and we'll inform you of any new developments.

Investigating: We are continuing to monitor ongoing recovery efforts, and most of our systems remain operating. As the situation with our upstream provider develops we will react if we discover any new issues.

At this time, we are observing delays in our systems, including Profile API 1.0 and Realtime 2.0. Requests are being processed as they have been received, but users may see a slowdown in results, and may notice delays when updating configurations or journeys.

You may notice continued slower performance or unexpected errors. We are working with our service provider to address these issues and will provide an update with new information soon.

Investigating: Our upstream service provider is gradually recovering its services, and our services are gradually recovering too. Most of our Treasure Data services are back to normal except for Custom Script, autoML and realtime services such as profile API 1.0, and realtime 2.0.

As the upstream service provider incident is still ongoing, you might still encounter unexpected errors or issues. We will closely monitoring our services and work with our upstream service provider.

Investigating: We are currently investigating an issue originating from an upstream service provider used by our infrastructure. This may be affecting the normal operation of some Treasure Data services.

We are working to determine the scope of the impact. Observed symptoms may include, but are not limited to, the failure of various jobs (such as Trino, Hive, Result Export, Import, and Custom Script execution) to start.

Our team is actively monitoring the situation and working with the provider. We will provide further updates as more information becomes available.

Resolved: Services have been restored to normal operations, and all of our systems are reporting healthy once again.

If you observe any lingering issues, please connect with our support team, and we would be happy to help: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support

Monitoring: Service levels continue to improve as upstream services become more stable. Users may still see processing delays, though those should be decreasing as we process the backlog of messages from this event.

We are continuing to monitor our systems and deploy capacity as needed. We will update you with additional information as it develops.

Monitoring: We are seeing continued recovery across our systems, and are currently working to process any buffered or delayed messages. Customers may continue to observe delays in data processing and may notice slower-than-usual processing of configuration changes. We expect these delays to decrease as our systems catch up.

Our team is continuing to track recovery closely, and we'll inform you of any new developments.

Investigating: We are continuing to monitor ongoing recovery efforts, and most of our systems remain operating. As the situation with our upstream provider develops we will react if we discover any new issues.

At this time, we are observing delays in our systems, including Profile API 1.0 and Realtime 2.0. Requests are being processed as they have been received, but users may see a slowdown in results, and may notice delays when updating configurations or journeys.

You may notice continued slower performance or unexpected errors. We are working with our service provider to address these issues and will provide an update with new information soon.

Investigating: Our upstream service provider is gradually recovering its services, and our services are gradually recovering too. Most of our Treasure Data services are back to normal except for Custom Script, autoML and realtime services such as profile API 1.0, and realtime 2.0.

As the upstream service provider incident is still ongoing, you might still encounter unexpected errors or issues. We will closely monitoring our services and work with our upstream service provider.

Investigating: We are currently investigating an issue originating from an upstream service provider used by our infrastructure. This may be affecting the normal operation of some Treasure Data services.

We are working to determine the scope of the impact. Observed symptoms may include, but are not limited to, the failure of various jobs (such as Trino, Hive, Result Export, Import, and Custom Script execution) to start.

Our team is actively monitoring the situation and working with the provider. We will provide further updates as more information becomes available.

Resolved: Services have been restored to normal operations, and all of our systems are reporting healthy once again.

If you observe any lingering issues, please connect with our support team, and we would be happy to help: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support

Monitoring: Service levels continue to improve as upstream services become more stable. Users may still see processing delays, though those should be decreasing as we process the backlog of messages from this event.

We are continuing to monitor our systems and deploy capacity as needed. We will update you with additional information as it develops.

Monitoring: We are seeing continued recovery across our systems, and are currently working to process any buffered or delayed messages. Customers may continue to observe delays in data processing and may notice slower-than-usual processing of configuration changes. We expect these delays to decrease as our systems catch up.

Our team is continuing to track recovery closely, and we'll inform you of any new developments.

Investigating: We are continuing to monitor ongoing recovery efforts, and most of our systems remain operating. As the situation with our upstream provider develops we will react if we discover any new issues.

At this time, we are observing delays in our systems, including Profile API 1.0 and Realtime 2.0. Requests are being processed as they have been received, but users may see a slowdown in results, and may notice delays when updating configurations or journeys.

You may notice continued slower performance or unexpected errors. We are working with our service provider to address these issues and will provide an update with new information soon.

Investigating: Our upstream service provider is gradually recovering its services, and our services are gradually recovering too. Most of our Treasure Data services are back to normal except for Custom Script, autoML and realtime services such as profile API 1.0, and realtime 2.0.

As the upstream service provider incident is still ongoing, you might still encounter unexpected errors or issues. We will closely monitoring our services and work with our upstream service provider.

Investigating: We are currently investigating an issue originating from an upstream service provider used by our infrastructure. This may be affecting the normal operation of some Treasure Data services.

We are working to determine the scope of the impact. Observed symptoms may include, but are not limited to, the failure of various jobs (such as Trino, Hive, Result Export, Import, and Custom Script execution) to start.

Our team is actively monitoring the situation and working with the provider. We will provide further updates as more information becomes available.

Resolved: Services have been restored to normal operations, and all of our systems are reporting healthy once again.

If you observe any lingering issues, please connect with our support team, and we would be happy to help: https://docs.treasuredata.com/articles/#!pd/contacting-treasure-data-support

Monitoring: Service levels continue to improve as upstream services become more stable. Users may still see processing delays, though those should be decreasing as we process the backlog of messages from this event.

We are continuing to monitor our systems and deploy capacity as needed. We will update you with additional information as it develops.

Monitoring: We are seeing continued recovery across our systems, and are currently working to process any buffered or delayed messages. Customers may continue to observe delays in data processing and may notice slower-than-usual processing of configuration changes. We expect these delays to decrease as our systems catch up.

Our team is continuing to track recovery closely, and we'll inform you of any new developments.

Investigating: We are continuing to monitor ongoing recovery efforts, and most of our systems remain operating. As the situation with our upstream provider develops we will react if we discover any new issues.

At this time, we are observing delays in our systems, including Profile API 1.0 and Realtime 2.0. Requests are being processed as they have been received, but users may see a slowdown in results, and may notice delays when updating configurations or journeys.

You may notice continued slower performance or unexpected errors. We are working with our service provider to address these issues and will provide an update with new information soon.

Investigating: Our upstream service provider is gradually recovering its services, and our services are gradually recovering too. Most of our Treasure Data services are back to normal except for Custom Script, autoML and realtime services such as profile API 1.0, and realtime 2.0.

As the upstream service provider incident is still ongoing, you might still encounter unexpected errors or issues. We will closely monitoring our services and work with our upstream service provider.

Investigating: We are currently investigating an issue originating from an upstream service provider used by our infrastructure. This may be affecting the normal operation of some Treasure Data services.

We are working to determine the scope of the impact. Observed symptoms may include, but are not limited to, the failure of various jobs (such as Trino, Hive, Result Export, Import, and Custom Script execution) to start.

Our team is actively monitoring the situation and working with the provider. We will provide further updates as more information becomes available.