Splunk On-Call Status

Resolved: We are no longer experiencing any issues involving delayed or failed SMS or Phone call notification delivery. Please contact Splunk On-Call support if you encounter any further problems.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Monitoring: We are continuing to monitor for any further issues.

Monitoring: Call/Voice queues have now cleared and we are seeing voice notifications being delivered in real time again. Currently, all notification types are operational and working as expected. We are continuing to monitor notification rates and will provide further updates on this Status Page Incident as the ongoing AWS incident progresses.

Thank you for your continued patience while we work to resolve this issue.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Monitoring: We are continuing to montior the situation. Currently SMS notifications are delivering as normal. However, Voice/Call notification are deeply backlogged and you may experience repeated calls for incidents that have already been actioned in your On-Call instance.

We are working directly with our notification provider to help migitate this isse as soon as possible.

Where possible, to avoid single points of failure, please take action to diversify your On-Call personal paging policies to include all forms of notification: SMS, Voice, Push, WhatsApp

Thank you for your contnued patience while we work to resolve this issue.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Monitoring: AWS fix has been implemented.

We are seeing the successful delivery of SMS and Voice through our notification provider. However, there still may be a backlog of notifications to be processed.

Moving incident status to Monitoring.

Where possible, to avoid single points of failure, please take action to diversify your On-Call personal paging policies to include all forms of notification: SMS, Voice, Push, WhatsApp

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Identified: Our cloud telephony partner (SMS and Voice) is seeing partial recovery and is scaling their systems to handle the influx of traffic and processing of notifications.

AWS update:

Increased Error Rates and Latencies

Oct 20 3:35 AM PDT The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now. Some requests may be throttled as they work toward full resolution.

Identified: The issue has been identified and is related to an ongoing AWS incident affecting multiple services: https://health.aws.amazon.com/health/status

Please continue to diversify your paging policies to incorporate Push and email.

Current AWS status:

Increased Error Rates and Latencies Oct 20 2:27 AM PDT We are seeing significant signs of recovery. Most requests should now be succeeding. We continue to work through a backlog of queued requests. We will continue to provide additional information.

Oct 20 2:22 AM PDT We have applied initial mitigations and we are observing early signs of recovery for some impacted AWS Services. During this time, requests may continue to fail as we work toward full resolution. We recommend customers retry failed requests. While requests begin succeeding, there may be additional latency and some services will have a backlog of work to work through, which may take additional time to fully process. We will continue to provide updates as we have more information to share, or by 3:15 AM.

Investigating: Please bear with us while we troubleshoot this issue — updates to follow.

Where possible, please adjust your personal paging policies to Push notification and email.

If you are affected by this reporting feature, please reach out to On-Call Support.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Resolved: We are no longer experiencing any issues involving delayed or failed SMS or Phone call notification delivery. Please contact Splunk On-Call support if you encounter any further problems.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Monitoring: We are continuing to monitor for any further issues.

Monitoring: Call/Voice queues have now cleared and we are seeing voice notifications being delivered in real time again. Currently, all notification types are operational and working as expected. We are continuing to monitor notification rates and will provide further updates on this Status Page Incident as the ongoing AWS incident progresses.

Thank you for your continued patience while we work to resolve this issue.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Monitoring: We are continuing to montior the situation. Currently SMS notifications are delivering as normal. However, Voice/Call notification are deeply backlogged and you may experience repeated calls for incidents that have already been actioned in your On-Call instance.

We are working directly with our notification provider to help migitate this isse as soon as possible.

Where possible, to avoid single points of failure, please take action to diversify your On-Call personal paging policies to include all forms of notification: SMS, Voice, Push, WhatsApp

Thank you for your contnued patience while we work to resolve this issue.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Monitoring: AWS fix has been implemented.

We are seeing the successful delivery of SMS and Voice through our notification provider. However, there still may be a backlog of notifications to be processed.

Moving incident status to Monitoring.

Where possible, to avoid single points of failure, please take action to diversify your On-Call personal paging policies to include all forms of notification: SMS, Voice, Push, WhatsApp

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Identified: Our cloud telephony partner (SMS and Voice) is seeing partial recovery and is scaling their systems to handle the influx of traffic and processing of notifications.

AWS update:

Increased Error Rates and Latencies

Oct 20 3:35 AM PDT The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now. Some requests may be throttled as they work toward full resolution.

Identified: The issue has been identified and is related to an ongoing AWS incident affecting multiple services: https://health.aws.amazon.com/health/status

Please continue to diversify your paging policies to incorporate Push and email.

Current AWS status:

Increased Error Rates and Latencies Oct 20 2:27 AM PDT We are seeing significant signs of recovery. Most requests should now be succeeding. We continue to work through a backlog of queued requests. We will continue to provide additional information.

Oct 20 2:22 AM PDT We have applied initial mitigations and we are observing early signs of recovery for some impacted AWS Services. During this time, requests may continue to fail as we work toward full resolution. We recommend customers retry failed requests. While requests begin succeeding, there may be additional latency and some services will have a backlog of work to work through, which may take additional time to fully process. We will continue to provide updates as we have more information to share, or by 3:15 AM.

Investigating: Please bear with us while we troubleshoot this issue — updates to follow.

Where possible, please adjust your personal paging policies to Push notification and email.

If you are affected by this reporting feature, please reach out to On-Call Support.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Resolved: We are no longer experiencing any issues involving delayed or failed SMS or Phone call notification delivery. Please contact Splunk On-Call support if you encounter any further problems.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Monitoring: We are continuing to monitor for any further issues.

Monitoring: Call/Voice queues have now cleared and we are seeing voice notifications being delivered in real time again. Currently, all notification types are operational and working as expected. We are continuing to monitor notification rates and will provide further updates on this Status Page Incident as the ongoing AWS incident progresses.

Thank you for your continued patience while we work to resolve this issue.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Monitoring: We are continuing to montior the situation. Currently SMS notifications are delivering as normal. However, Voice/Call notification are deeply backlogged and you may experience repeated calls for incidents that have already been actioned in your On-Call instance.

We are working directly with our notification provider to help migitate this isse as soon as possible.

Where possible, to avoid single points of failure, please take action to diversify your On-Call personal paging policies to include all forms of notification: SMS, Voice, Push, WhatsApp

Thank you for your contnued patience while we work to resolve this issue.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Monitoring: AWS fix has been implemented.

We are seeing the successful delivery of SMS and Voice through our notification provider. However, there still may be a backlog of notifications to be processed.

Moving incident status to Monitoring.

Where possible, to avoid single points of failure, please take action to diversify your On-Call personal paging policies to include all forms of notification: SMS, Voice, Push, WhatsApp

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Identified: Our cloud telephony partner (SMS and Voice) is seeing partial recovery and is scaling their systems to handle the influx of traffic and processing of notifications.

AWS update:

Increased Error Rates and Latencies

Oct 20 3:35 AM PDT The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now. Some requests may be throttled as they work toward full resolution.

Identified: The issue has been identified and is related to an ongoing AWS incident affecting multiple services: https://health.aws.amazon.com/health/status

Please continue to diversify your paging policies to incorporate Push and email.

Current AWS status:

Increased Error Rates and Latencies Oct 20 2:27 AM PDT We are seeing significant signs of recovery. Most requests should now be succeeding. We continue to work through a backlog of queued requests. We will continue to provide additional information.

Oct 20 2:22 AM PDT We have applied initial mitigations and we are observing early signs of recovery for some impacted AWS Services. During this time, requests may continue to fail as we work toward full resolution. We recommend customers retry failed requests. While requests begin succeeding, there may be additional latency and some services will have a backlog of work to work through, which may take additional time to fully process. We will continue to provide updates as we have more information to share, or by 3:15 AM.

Investigating: Please bear with us while we troubleshoot this issue — updates to follow.

Where possible, please adjust your personal paging policies to Push notification and email.

If you are affected by this reporting feature, please reach out to On-Call Support.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Monitoring: We want to update you on the status of the On-Call Review report issue. Our team has implemented several fixes to address the root cause, and as a result, we are reducing the incident severity to "Minor" and updating the status to "Monitoring" while we complete final data validation.

We recognize that some issues with how On-Call Reporting data appears in the UI are still present. Please know that our engineering team is working diligently to fully resolve these remaining problems.

If this issue is impacting your access to On-Call Review reporting data and you need information urgently, please contact our Support team by submitting a ticket or using the in-product live chat. Upon request, we’re happy to provide a manually generated CSV containing the accurate data you need from the On-Call Review report.

Thank you for your understanding and patience as we work to fully resolve this incident.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Identified: Splunk On-Call Engineering and Support teams continue to give our full attention to resolving this issue as soon as possible.

** Please note that this is a complex problem. Our teams are working urgently and diligently towards providing our customers with accurate On-Call-Review data. It is our top priority.

If you are affected by this reporting feature, please reach out to On-Call Support.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Thank you again for your patience while we work to resolve this incident and the accuracy of the On-Call Review report.

Identified: Splunk On-Call Engineering and Support teams continue to give our full attention to resolving this issue as soon as possible.

** Please note that this is a complex problem. Our teams are working urgently and diligently towards providing our customers with accurate On-Call-Review data. It is our top priority.

If you are affected by this reporting feature, please reach out to On-Call Support.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Thank you again for your patience while we work to resolve this incident and the accuracy of the On-Call Review report.

Identified: Update: Our Engineering and Support teams continue to give our full attention to resolving this issue as soon as possible.

** Please note that this is a complex problem involving data from different sources. Our teams are working urgently and diligently, with accurate On-Call-Review data as our top priority.

If you are affected by this reporting feature, please reach out to On-Call Support.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Thank you again for your patience while we work to resolve this incident and the accuracy of the On-Call Review report.

Identified: Update: Please bear with us as we continue to work on a fix for the On-Call Review reporting feature. The Splunk On-Call Engineering team and Senior Support leadership continue to work towards providing you with accurate On-Call Review reporting.

** Please note that this is a complex problem involving data from different sources. Our teams are working urgently and diligently, with accurate On-Call-Review data as our top priority.

If you are affected by this reporting feature, please reach out to On-Call Support.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Thank you again for your patience while we work to resolve this incident and the accuracy of the On-Call Review report.

Identified: Update: Please bear with us as we continue to work on a fix for the On-Call Review reporting feature. The Splunk On-Call Engineering team and Senior Support leadership continue to work towards providing you with accurate On-Call Review reporting.

** Please note that this is a complex problem involving data from different sources. Our teams are working urgently and diligently, with accurate On-Call-Review data as our top priority.

If you are affected by this reporting feature, please reach out to On-Call Support.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Thank you again for your patience while we work to resolve this incident and the accuracy of the On-Call Review report.

Identified: Update: We continue to work on a fix for the On-Call Review reporting feature. The Splunk On-Call Engineering team and senior Support leadership continue to work towards providing you with accurate On-Call Review reporting.

Please note that this is a complex problem involving data from many sources. Our teams are working urgently and diligently, with accurate On-Call-Review data as our top priority.

If you are affected by this reporting feature, please reach out to On-Call Support.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Thank you again for your patience while we work to resolve this incident and the accuracy of the On-Call Review report.

Identified: We continue to work on a fix for this issue. If you are affected and require immediate attention, please reach out to Support via a support ticket or through in-product live chat. We've upgraded this incident to Major, as we are currently unable to provide an accurate CSV for the On-Call Review at this time.

We've prioritized this issue with our Engineering teams. They are addressing the issue with urgency.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Our apologies again for this issue! Please stay tuned to this status page for updates! Thank you for your patience.

Identified: Update: Please bear with us as we continue to work on a fix for this issue. If you are affected and require immediate On-Call Review reporting data, please reach out to Support via a support ticket or through in-product live chat. We'll gladly provide an accurate, manually generated CSV of the data you are searching for in the On-Call Review report upon request.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Again, our apologies for this issue! Please stay tuned to this status page for further updates! Thank you for your patience while we work towards a resolution.

Identified: We are currently experiencing inaccurate reporting data in the On-Call Review report (OCR) within the web client.

This is a known issue, and we're working on a fix as soon as possible. If you are affected by this reporting feature, please reach out to On-Call Support.

Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support

Our apologies for this issue! Please stay tuned to this status page for updates! Thank you for your patience while we work towards a resolution.