Rebuy Status

Postmortem: What happened

On July 14, a routine update to how one of our systems retrieves product information caused a rendering issue for storefront widgets on a subset of merchant sites. If your account uses widgets with tag-based rules, such as cross-sell recommendations or Bundle Builder, you may have noticed widgets not loading, or products disappearing after the page loaded.

This issue did not affect checkout, admin, or any non-widget parts of your storefront.

Root cause

Product tag data is stored in two different formats behind the scenes, a legacy format and a newer one, depending on when and how a shop's product catalog was originally set up. The update changed the path our system used to retrieve that data for widgets, and that path did not convert the newer format into the format our widget code expects.

On accounts where product tags were stored in the newer format, the widget code could not process the data correctly, which caused the rendering issues described above.

Resolution

We reverted the update, which restored normal widget rendering about 30 minutes after the issue began. Because some widget data is temporarily cached for performance, a small number of accounts continued to see stale results until that cache cleared. We proactively refreshed the cache across all affected accounts.

What we're doing to prevent this from happening again

• Restoring the missing data conversion step and adding an automated test that checks both retrieval paths always return matching, correctly formatted data before any future release can go out.
• Building a faster internal tool to refresh cached widget data across accounts, so if a similar issue ever occurs again, we can resolve the residual impact more quickly.
• Reconciling the two underlying tag data formats into a single consistent format, to remove this class of issue at the source rather than only guarding against it downstream.

Resolved: All issues have been resolved at this time. Widgets and bundle builders across affected sites are rendering normally. Our team is investigating the root cause and will post an update as soon as it's available.

Monitoring: A fix has been implemented for the widget loading issue. The sites that were previously reporting errors are no longer seeing them, and widgets and bundle builders are rendering as expected. Our team continues to test to verify stability.

Investigating: We're still actively investigating the widget loading issue affecting some merchant sites. Our team has identified a likely contributing factor and are focusing on that now.

Investigating: We're aware some merchants are experiencing an issue where merchandising widgets and bundle builders aren't rendering on their storefronts. Our team is actively investigating and working toward a fix. This issue is not affecting all merchants, and we'll share an update as soon as we have more information.

Postmortem: What happened

On July 14, a routine update to how one of our systems retrieves product information caused a rendering issue for storefront widgets on a subset of merchant sites. If your account uses widgets with tag-based rules, such as cross-sell recommendations or Bundle Builder, you may have noticed widgets not loading, or products disappearing after the page loaded.

This issue did not affect checkout, admin, or any non-widget parts of your storefront.

Root cause

Product tag data is stored in two different formats behind the scenes, a legacy format and a newer one, depending on when and how a shop's product catalog was originally set up. The update changed the path our system used to retrieve that data for widgets, and that path did not convert the newer format into the format our widget code expects.

On accounts where product tags were stored in the newer format, the widget code could not process the data correctly, which caused the rendering issues described above.

Resolution

We reverted the update, which restored normal widget rendering about 30 minutes after the issue began. Because some widget data is temporarily cached for performance, a small number of accounts continued to see stale results until that cache cleared. We proactively refreshed the cache across all affected accounts.

What we're doing to prevent this from happening again

• Restoring the missing data conversion step and adding an automated test that checks both retrieval paths always return matching, correctly formatted data before any future release can go out.
• Building a faster internal tool to refresh cached widget data across accounts, so if a similar issue ever occurs again, we can resolve the residual impact more quickly.
• Reconciling the two underlying tag data formats into a single consistent format, to remove this class of issue at the source rather than only guarding against it downstream.

Resolved: All issues have been resolved at this time. Widgets and bundle builders across affected sites are rendering normally. Our team is investigating the root cause and will post an update as soon as it's available.

Monitoring: A fix has been implemented for the widget loading issue. The sites that were previously reporting errors are no longer seeing them, and widgets and bundle builders are rendering as expected. Our team continues to test to verify stability.

Investigating: We're still actively investigating the widget loading issue affecting some merchant sites. Our team has identified a likely contributing factor and are focusing on that now.

Investigating: We're aware some merchants are experiencing an issue where merchandising widgets and bundle builders aren't rendering on their storefronts. Our team is actively investigating and working toward a fix. This issue is not affecting all merchants, and we'll share an update as soon as we have more information.