Zid Status

Postmortem: The team released an incident fix for an issue where the count of products under categories was not calculated correctly, and was instead showing 0 for parent categories whose products are all placed inside child categories.

As a result of this fix, there was an increase in DB queries, leading to a spike in CPU utilization of our database. This happened mainly due to cache warming jobs used in the old catalog system. While the fix was intended for the Dashboard, an inadvertent consequence happened on “Old Catalog” – not Vitrin (stores on Vitrin were browsable safely).

During that period, some users who did not have the store cached on their end may have experienced downtime with some stores. Others experienced degraded performance only on cart actions.

‌

Action Taken:

1. We immediately reverted the troublesome deployment.
2. The congested queues for store cashing were purged to avoid having blocked jobs
3. Assembled a rapid response team to decouple this dependency and prevent such incidents in the future.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: Some stores were experiencing slower than usual response times.

Postmortem: The team released an incident fix for an issue where the count of products under categories was not calculated correctly, and was instead showing 0 for parent categories whose products are all placed inside child categories.

As a result of this fix, there was an increase in DB queries, leading to a spike in CPU utilization of our database. This happened mainly due to cache warming jobs used in the old catalog system. While the fix was intended for the Dashboard, an inadvertent consequence happened on “Old Catalog” – not Vitrin (stores on Vitrin were browsable safely).

During that period, some users who did not have the store cached on their end may have experienced downtime with some stores. Others experienced degraded performance only on cart actions.

‌

Action Taken:

1. We immediately reverted the troublesome deployment.
2. The congested queues for store cashing were purged to avoid having blocked jobs
3. Assembled a rapid response team to decouple this dependency and prevent such incidents in the future.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: Some stores were experiencing slower than usual response times.