Viafoura Status

Postmortem: ### What Happened

On Saturday, November 8th, our authentication service experienced a complete outage that impacted all Viafoura services. During routine database maintenance earlier in the week, we initiated a repair process to ensure data consistency. This repair process unexpectedly created sustained high load on our database infrastructure, which ultimately prevented the authentication service from responding to requests.

All customer-facing Viafoura services were unavailable during this period, with the exception of Quicksight reporting dashboards. Users were unable to log in, comment, or interact with any community features on sites.

We understand the significant disruption this caused to your audiences and sincerely apologize for the impact to your operations.

Resolution

Our engineering team immediately began migrating our authentication database to more robust infrastructure with higher-performance storage. This migration took approximately 7 hours to complete safely while preserving all data integrity. Once the migration was complete, all services were restored to full functionality.

Prevention

Immediate Actions Completed:

• Migrated authentication infrastructure to high-performance storage that can handle maintenance operations without service impact
• Implemented enhanced monitoring for database performance during maintenance windows

Short-Term Actions In Progress:

• We are completing a planned migration to a more resilient managed database infrastructure (AWS RDS Aurora)
• This migration is already in advanced testing stages and is now our highest priority
• Expected completion: End of this week

This architectural change will provide significantly improved reliability and eliminate the type of infrastructure constraints that caused this outage.

Our team is committed to ensuring the stability and reliability you expect from Viafoura.

Thank you for your patience and partnership.

Resolved: Services Restored - All Components Operational All services have been fully restored. Community experiences, moderation console, and administrative tools are now functioning normally. What happened: A database issue caused a platform-wide service disruption affecting all components with the exception of the analytics dashboard. Our engineering team identified and resolved the root cause.

Next steps: We are conducting a thorough post-incident analysis and will share a detailed retrospective report on Monday, November 10th, including root cause analysis and preventative measures being implemented.

We sincerely appreciate your patience during this disruption.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard).

Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard). Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard).

Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: Our engineering team continues to actively work on resolution with full leadership oversight. We will continue to provide updates as we work toward restoring service.

Thank you for your patience.

Identified: We are currently experiencing a service disruption affecting all Viafoura components across our platform. This includes community experiences, moderation console, and administrative tools.

We've identified a database issue that is preventing our services from loading properly. Our engineering team has isolated the root cause and is actively working to restore full functionality.

Current status: Issue identified: Database connectivity All customer sites affected Engineering team engaged and implementing fix

We understand the impact this has on your communities and operations. We're prioritizing resolution and will continue to provide updates until services are restored.

Identified: The issue has been identified and a fix is being implemented.

Postmortem: ### What Happened

On Saturday, November 8th, our authentication service experienced a complete outage that impacted all Viafoura services. During routine database maintenance earlier in the week, we initiated a repair process to ensure data consistency. This repair process unexpectedly created sustained high load on our database infrastructure, which ultimately prevented the authentication service from responding to requests.

All customer-facing Viafoura services were unavailable during this period, with the exception of Quicksight reporting dashboards. Users were unable to log in, comment, or interact with any community features on sites.

We understand the significant disruption this caused to your audiences and sincerely apologize for the impact to your operations.

Resolution

Our engineering team immediately began migrating our authentication database to more robust infrastructure with higher-performance storage. This migration took approximately 7 hours to complete safely while preserving all data integrity. Once the migration was complete, all services were restored to full functionality.

Prevention

Immediate Actions Completed:

• Migrated authentication infrastructure to high-performance storage that can handle maintenance operations without service impact
• Implemented enhanced monitoring for database performance during maintenance windows

Short-Term Actions In Progress:

• We are completing a planned migration to a more resilient managed database infrastructure (AWS RDS Aurora)
• This migration is already in advanced testing stages and is now our highest priority
• Expected completion: End of this week

This architectural change will provide significantly improved reliability and eliminate the type of infrastructure constraints that caused this outage.

Our team is committed to ensuring the stability and reliability you expect from Viafoura.

Thank you for your patience and partnership.

Resolved: Services Restored - All Components Operational All services have been fully restored. Community experiences, moderation console, and administrative tools are now functioning normally. What happened: A database issue caused a platform-wide service disruption affecting all components with the exception of the analytics dashboard. Our engineering team identified and resolved the root cause.

Next steps: We are conducting a thorough post-incident analysis and will share a detailed retrospective report on Monday, November 10th, including root cause analysis and preventative measures being implemented.

We sincerely appreciate your patience during this disruption.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard).

Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard). Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard).

Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: Our engineering team continues to actively work on resolution with full leadership oversight. We will continue to provide updates as we work toward restoring service.

Thank you for your patience.

Identified: We are currently experiencing a service disruption affecting all Viafoura components across our platform. This includes community experiences, moderation console, and administrative tools.

We've identified a database issue that is preventing our services from loading properly. Our engineering team has isolated the root cause and is actively working to restore full functionality.

Current status: Issue identified: Database connectivity All customer sites affected Engineering team engaged and implementing fix

We understand the impact this has on your communities and operations. We're prioritizing resolution and will continue to provide updates until services are restored.

Identified: The issue has been identified and a fix is being implemented.

Postmortem: ### What Happened

On Saturday, November 8th, our authentication service experienced a complete outage that impacted all Viafoura services. During routine database maintenance earlier in the week, we initiated a repair process to ensure data consistency. This repair process unexpectedly created sustained high load on our database infrastructure, which ultimately prevented the authentication service from responding to requests.

All customer-facing Viafoura services were unavailable during this period, with the exception of Quicksight reporting dashboards. Users were unable to log in, comment, or interact with any community features on sites.

We understand the significant disruption this caused to your audiences and sincerely apologize for the impact to your operations.

Resolution

Our engineering team immediately began migrating our authentication database to more robust infrastructure with higher-performance storage. This migration took approximately 7 hours to complete safely while preserving all data integrity. Once the migration was complete, all services were restored to full functionality.

Prevention

Immediate Actions Completed:

• Migrated authentication infrastructure to high-performance storage that can handle maintenance operations without service impact
• Implemented enhanced monitoring for database performance during maintenance windows

Short-Term Actions In Progress:

• We are completing a planned migration to a more resilient managed database infrastructure (AWS RDS Aurora)
• This migration is already in advanced testing stages and is now our highest priority
• Expected completion: End of this week

This architectural change will provide significantly improved reliability and eliminate the type of infrastructure constraints that caused this outage.

Our team is committed to ensuring the stability and reliability you expect from Viafoura.

Thank you for your patience and partnership.

Resolved: Services Restored - All Components Operational All services have been fully restored. Community experiences, moderation console, and administrative tools are now functioning normally. What happened: A database issue caused a platform-wide service disruption affecting all components with the exception of the analytics dashboard. Our engineering team identified and resolved the root cause.

Next steps: We are conducting a thorough post-incident analysis and will share a detailed retrospective report on Monday, November 10th, including root cause analysis and preventative measures being implemented.

We sincerely appreciate your patience during this disruption.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard).

Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard). Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard).

Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: Our engineering team continues to actively work on resolution with full leadership oversight. We will continue to provide updates as we work toward restoring service.

Thank you for your patience.

Identified: We are currently experiencing a service disruption affecting all Viafoura components across our platform. This includes community experiences, moderation console, and administrative tools.

We've identified a database issue that is preventing our services from loading properly. Our engineering team has isolated the root cause and is actively working to restore full functionality.

Current status: Issue identified: Database connectivity All customer sites affected Engineering team engaged and implementing fix

We understand the impact this has on your communities and operations. We're prioritizing resolution and will continue to provide updates until services are restored.

Identified: The issue has been identified and a fix is being implemented.

Postmortem: ### What Happened

On Saturday, November 8th, our authentication service experienced a complete outage that impacted all Viafoura services. During routine database maintenance earlier in the week, we initiated a repair process to ensure data consistency. This repair process unexpectedly created sustained high load on our database infrastructure, which ultimately prevented the authentication service from responding to requests.

All customer-facing Viafoura services were unavailable during this period, with the exception of Quicksight reporting dashboards. Users were unable to log in, comment, or interact with any community features on sites.

We understand the significant disruption this caused to your audiences and sincerely apologize for the impact to your operations.

Resolution

Our engineering team immediately began migrating our authentication database to more robust infrastructure with higher-performance storage. This migration took approximately 7 hours to complete safely while preserving all data integrity. Once the migration was complete, all services were restored to full functionality.

Prevention

Immediate Actions Completed:

• Migrated authentication infrastructure to high-performance storage that can handle maintenance operations without service impact
• Implemented enhanced monitoring for database performance during maintenance windows

Short-Term Actions In Progress:

• We are completing a planned migration to a more resilient managed database infrastructure (AWS RDS Aurora)
• This migration is already in advanced testing stages and is now our highest priority
• Expected completion: End of this week

This architectural change will provide significantly improved reliability and eliminate the type of infrastructure constraints that caused this outage.

Our team is committed to ensuring the stability and reliability you expect from Viafoura.

Thank you for your patience and partnership.

Resolved: Services Restored - All Components Operational All services have been fully restored. Community experiences, moderation console, and administrative tools are now functioning normally. What happened: A database issue caused a platform-wide service disruption affecting all components with the exception of the analytics dashboard. Our engineering team identified and resolved the root cause.

Next steps: We are conducting a thorough post-incident analysis and will share a detailed retrospective report on Monday, November 10th, including root cause analysis and preventative measures being implemented.

We sincerely appreciate your patience during this disruption.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard).

Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard). Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard).

Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: Our engineering team continues to actively work on resolution with full leadership oversight. We will continue to provide updates as we work toward restoring service.

Thank you for your patience.

Identified: We are currently experiencing a service disruption affecting all Viafoura components across our platform. This includes community experiences, moderation console, and administrative tools.

We've identified a database issue that is preventing our services from loading properly. Our engineering team has isolated the root cause and is actively working to restore full functionality.

Current status: Issue identified: Database connectivity All customer sites affected Engineering team engaged and implementing fix

We understand the impact this has on your communities and operations. We're prioritizing resolution and will continue to provide updates until services are restored.

Identified: The issue has been identified and a fix is being implemented.

Postmortem: ### What Happened

On Saturday, November 8th, our authentication service experienced a complete outage that impacted all Viafoura services. During routine database maintenance earlier in the week, we initiated a repair process to ensure data consistency. This repair process unexpectedly created sustained high load on our database infrastructure, which ultimately prevented the authentication service from responding to requests.

All customer-facing Viafoura services were unavailable during this period, with the exception of Quicksight reporting dashboards. Users were unable to log in, comment, or interact with any community features on sites.

We understand the significant disruption this caused to your audiences and sincerely apologize for the impact to your operations.

Resolution

Our engineering team immediately began migrating our authentication database to more robust infrastructure with higher-performance storage. This migration took approximately 7 hours to complete safely while preserving all data integrity. Once the migration was complete, all services were restored to full functionality.

Prevention

Immediate Actions Completed:

• Migrated authentication infrastructure to high-performance storage that can handle maintenance operations without service impact
• Implemented enhanced monitoring for database performance during maintenance windows

Short-Term Actions In Progress:

• We are completing a planned migration to a more resilient managed database infrastructure (AWS RDS Aurora)
• This migration is already in advanced testing stages and is now our highest priority
• Expected completion: End of this week

This architectural change will provide significantly improved reliability and eliminate the type of infrastructure constraints that caused this outage.

Our team is committed to ensuring the stability and reliability you expect from Viafoura.

Thank you for your patience and partnership.

Resolved: Services Restored - All Components Operational All services have been fully restored. Community experiences, moderation console, and administrative tools are now functioning normally. What happened: A database issue caused a platform-wide service disruption affecting all components with the exception of the analytics dashboard. Our engineering team identified and resolved the root cause.

Next steps: We are conducting a thorough post-incident analysis and will share a detailed retrospective report on Monday, November 10th, including root cause analysis and preventative measures being implemented.

We sincerely appreciate your patience during this disruption.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard).

Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard). Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard).

Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: Our engineering team continues to actively work on resolution with full leadership oversight. We will continue to provide updates as we work toward restoring service.

Thank you for your patience.

Identified: We are currently experiencing a service disruption affecting all Viafoura components across our platform. This includes community experiences, moderation console, and administrative tools.

We've identified a database issue that is preventing our services from loading properly. Our engineering team has isolated the root cause and is actively working to restore full functionality.

Current status: Issue identified: Database connectivity All customer sites affected Engineering team engaged and implementing fix

We understand the impact this has on your communities and operations. We're prioritizing resolution and will continue to provide updates until services are restored.

Identified: The issue has been identified and a fix is being implemented.

Postmortem: ### What Happened

On Saturday, November 8th, our authentication service experienced a complete outage that impacted all Viafoura services. During routine database maintenance earlier in the week, we initiated a repair process to ensure data consistency. This repair process unexpectedly created sustained high load on our database infrastructure, which ultimately prevented the authentication service from responding to requests.

All customer-facing Viafoura services were unavailable during this period, with the exception of Quicksight reporting dashboards. Users were unable to log in, comment, or interact with any community features on sites.

We understand the significant disruption this caused to your audiences and sincerely apologize for the impact to your operations.

Resolution

Our engineering team immediately began migrating our authentication database to more robust infrastructure with higher-performance storage. This migration took approximately 7 hours to complete safely while preserving all data integrity. Once the migration was complete, all services were restored to full functionality.

Prevention

Immediate Actions Completed:

• Migrated authentication infrastructure to high-performance storage that can handle maintenance operations without service impact
• Implemented enhanced monitoring for database performance during maintenance windows

Short-Term Actions In Progress:

• We are completing a planned migration to a more resilient managed database infrastructure (AWS RDS Aurora)
• This migration is already in advanced testing stages and is now our highest priority
• Expected completion: End of this week

This architectural change will provide significantly improved reliability and eliminate the type of infrastructure constraints that caused this outage.

Our team is committed to ensuring the stability and reliability you expect from Viafoura.

Thank you for your patience and partnership.

Resolved: Services Restored - All Components Operational All services have been fully restored. Community experiences, moderation console, and administrative tools are now functioning normally. What happened: A database issue caused a platform-wide service disruption affecting all components with the exception of the analytics dashboard. Our engineering team identified and resolved the root cause.

Next steps: We are conducting a thorough post-incident analysis and will share a detailed retrospective report on Monday, November 10th, including root cause analysis and preventative measures being implemented.

We sincerely appreciate your patience during this disruption.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard).

Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard). Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: We are continuing to work on resolving the database issue affecting all services (excluding analytics dashboard).

Our team remains fully engaged in restoring service. We will provide further updates as progress is made. Thank you for your continued patience.

Identified: Our engineering team continues to actively work on resolution with full leadership oversight. We will continue to provide updates as we work toward restoring service.

Thank you for your patience.

Identified: We are currently experiencing a service disruption affecting all Viafoura components across our platform. This includes community experiences, moderation console, and administrative tools.

We've identified a database issue that is preventing our services from loading properly. Our engineering team has isolated the root cause and is actively working to restore full functionality.

Current status: Issue identified: Database connectivity All customer sites affected Engineering team engaged and implementing fix

We understand the impact this has on your communities and operations. We're prioritizing resolution and will continue to provide updates until services are restored.

Identified: The issue has been identified and a fix is being implemented.