Healthie Status

Resolved: All scheduled payments from 10/20 have been attempted.

Investigating: Due to the AWS incident, automated payments were not run this morning. We will process all scheduled payments tomorrow (10/21). Please do not manually charge for the payments in the interim as that may lead to double charges.

Resolved: Webhooks are fully caught up, and response times are back to normal. This incident is resolved.

Investigating: We have been able to scale up and beyond our typical mid-day capacity. Response times should be back to normal, and background processes (specifically webhooks) have begun to catch up. We continue to monitor.

Investigating: We are seeing some (smaller) successes amongst failing scaling attempts and have been able to add more capacity which should help with response times. We're still far below normal response times and capacity and continue to work to get everything back to normal here.

Investigating: AWS has shared a positive update- "the internal subsystems of EC2 are now showing early signs of recovering in a few Availability Zones (AZs) in the US-EAST-1 Region. We are applying mitigations to the remaining AZs at which point we expect launch errors and network connectivity issues to subside.".

Once we are able to launch new EC2 instances, we would expect response times to return to normal and this incident to be resolved. We're continuing to monitor.

Investigating: AWS is "in the process of validating a fix and will deploy to the first AZ as soon as they have confidence they can do so safely."

Investigating: AWS "have identified and are applying next steps to mitigate throttling of new EC2 instance launches.". This is what needs to be restored for Healthie's response times to return to normal, so this is a positive step. We continue to monitor.

Investigating: AWS has "narrowed down the source of the network connectivity issues" and identified the root cause. They are actively working on mitigations but are still "throttling requests for new EC2 instances" which blocks us from provisioning needed capacity. We continue to monitor the situation closely.

Investigating: AWS continues to experience severe networking issues (as they are updating at https://health.aws.amazon.com/health/status). We continue to monitor and re-attempt to provision additional capacity.

Investigating: We've confirmed we're seeing slower response times. We continue to work to provision the needed capacity. We have paused sending webhooks to help lessen server traffic as we work to scale capacity. All will be sent once we're able to successfully scale.

Investigating: AWS continues to have issues with EC2 instances which is preventing us from automatically scaling to our normal server capacity (https://health.aws.amazon.com/). This is leading to slower than normal response times. Our team is monitoring and working with our host to scale up to the needed capacity.

Resolved: AWS has recovered most services and Healthie became available again at ~5 AM eastern. We are continuing to monitor.

Investigating: Our hosting provider, AWS, is currently experiencing a substantial incident with multiple services. This is causing the Healthie platform to become unavailable. The AWS incident can be followed here: https://health.aws.amazon.com/health/status

The most recent update from AWS is:

"Oct 20 12:11 AM PDT We are investigating increased error rates and latencies for multiple AWS services in the US-EAST-1 Region"

Resolved: This is now resolved.

Investigating: We are continuing to investigate this issue.

Investigating: We are investigating possible delays in pulling results from ChangeHealth. This is specific to Change Health. eLabs Direct by Healthie, Rupa, etc are not impacted.

Resolved: AWS has recovered most services and Healthie became available again at ~5 AM eastern. We are continuing to monitor.

Investigating: Our hosting provider, AWS, is currently experiencing a substantial incident with multiple services. This is causing the Healthie platform to become unavailable. The AWS incident can be followed here: https://health.aws.amazon.com/health/status

The most recent update from AWS is:

"Oct 20 12:11 AM PDT We are investigating increased error rates and latencies for multiple AWS services in the US-EAST-1 Region"