Firstup Status

Monitoring: Impact has now been fully mitigated and we do not expect a recurrence. This incident will remain in monitoring state while we evaluate the longer term mechanism required to completely resolve the issue.

Identified: Root cause is identified and we are actively working to mitigate the issue.

Investigating: Moving incident back to active state. Mitigation efforts have been insufficient to fully isolate Customers from impact. Symptoms remain similar where users are being signed out, but other connection errors may be intermittently displayed as well.

Investigating: We have seen a recurrence of this issue and are currently investigating. Same authentication errors are being displayed for some programs. Updates to follow asap.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: Immediate impact has been mitigated. Errors are no longer appearing and successful log in checks have been done. Stability has been restored and we are continuing to monitor. Root cause to follow.

Investigating: Cause of issue has been identified and we are currently taking action to mitigate which will resolve the authentication errors being displayed.

Investigating: We are currently investigating a platform incident impacting the web experience channel for some users. A 'You've been signed out or your access has been expired' is appearing on some communities. Updates to follow asap.

Postmortem: Summary:
On August 4th, 2025 we began receiving Customer reports of the following error being displayed on the Web Experience for impacted users: "Error: Minified React error #300; visit https://reactjs.org/docs/error-decoder.html?invariant=300 for the full message or use the non-minified dev environment for full errors and additional helpful warnings." This behavior was attributed to a change that was made as a part of the hot fix deployed earlier in the day that adjusted the retention period for local (web browser) storage of specific files used to render the Web Experience. An unexpected side effect from that change resulted in content attempting to be rendered prematurely--and the low level react error being displayed to all users during the time the platform incident was active. The initial attempt to rollback the hot fix was insufficient, forcing a more invasive removal of the entire local storage management functionality This action fully restored service for the Web Experience channel.

Impact:
A system error message was returned to any end-user attempting to refresh the Web Experience between 11:51 AM and 13:03 PM PT (1 hour, 24 minutes).
"Error: Minified React error #300; visit https://reactjs.org/docs/error-decoder.html?invariant=300 for the full message or use the non-minified dev environment for full errors and additional helpful warnings."

Other services and endpoints including Creator Studio and the Mobile Experience remained fully functional.

Root Cause:
Root cause was determined to be due to a software change made earlier in the day on August 4th as a part of a hot fix which altered the retention period for local (web browser) storage of specific files used to render the Web Experience.

A few months ago, we completely changed the behavior for how the Firstup Web Experience loads to make it more resilient to backend failures, planned maintenance activities, and hiccups in client network connectivity. Short of a full offline mode, focus was made around custom homepage content, quicklinks, and other features and functionality that comprise the Firstup Employee Intranet offering. This work changed how the application itself loads--at a high level it fires up with all of the content it has stored locally, and then checks to see if it can refresh the content (if it is new, accessible to the client…etc).

The hot fix itself that triggered this platform incident unexpectedly broke how the Web Experience application initialization worked that is used by this new local caching mechanism, such that the app attempted to render content before everything it needed to render it successfully had been loaded from the local web browser storage (cache)--resulting in the low level react error from the underlying application framework.

Mitigation:
A platform incident was declared at 11:51 am PT on Monday, August 4th and the incident team determined the root cause at 12:10 pm PT. Issue was attempted to be mitigated earlier and faster by rolling back the hot fix which proved to be insufficient, so a decision was made to remove the entire local storage management functionality that had been added several months back to improve the graceful degradation of Custom Homepage, Quicklinks, and other products that comprise the Firstup Employee Intranet offering. By not utilizing local storage (caching) at all, the early attempt to render content before the app was ready was corrected indirectly, thus mitigating the impact to the Web Experience channel. 

Recurrence Prevention:
The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA (Root Cause Assessment) process:

• Testing and Analysis - More rigorous testing on UAT for client side changes and analysis to detect any errors/irregularities before a change goes live.
• Additional real-time monitoring to be implemented that will alert the Engineering team of client-side only errors (on authenticated pages, such as the Web Experience homepage).
• Improvement has been identified for the rollback scripts which will reduce rollback time to seconds in the event of an incident.
• Add mechanism to clear client side cache more easily without the need for a platform wide emergency change for cases such as this.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: Immediate impact has been mitigated. Work continues to determine root cause. The Custom Homepage Degradation feature has been disabled in the interim. A follow-up communication will be provided once additional information is available and regularly timed updates will cease so long as the impact remains mitigated.

Investigating: Rolling back most recent change has not fixed the underlying problem. Focus shifting to local browser cache management given the impact is intermittent. No ETA yet for full restoration of service. Scope remains limited to the Web Experience. Other service endpoints remain fully operational.

Investigating: Investigation is continuing in to possible root cause. An incident bridge has been initiated and primary focus is on the source of the error message currently being displayed for impacted users: "Error: Minified React error #300; visit https://reactjs.org/docs/error-decoder.html?invariant=300 for the full message or use the non-minified dev environment for full errors and additional helpful warnings."

A change rollback is underway for the most recent change based on time correlation.

Investigating: We are currently investigating a new platform incident impacting the web experience channel for some users.

Resolved: This incident has been resolved. All previously errored user sync files have now been successfully reprocessed, and summary emails were sent as expected.

We have been monitoring the user sync service overnight and no further issues have been observed. Any new errors that appear will be valid user sync processing errors and can be addressed using the guidance in our Help Centre article: https://support.firstup.io/hc/en-us/articles/13072635044503-SFTP-File-Upload-Errors-and-User-Sync-Results

If you have any questions or encounter further issues, please reach out to our Support team at support.firstup.io.

Monitoring: We have applied a fix for the issue affecting user sync processing. We are reprocessing errored jobs and closely monitoring the results.

Investigating: We are investigating an issue affecting User Sync (User File via SFTP processing). This is resulting in user sync jobs encountering elevated error rates. You may see the user sync email showing that many records were not processed due to errors.

Our next update will be in 30 minutes.