Ivanti Cloud Status

Monitoring: We are continuing to investigate and monitor the environment. An emergency maintenance is scheduled shortly to complete works that aims to help improve the current situation and increase our ability to troubleshoot the workflow and email issues some customers are currently experiencing (https://status.ivanticloud.com/incidents/rh7nj7grcf2c).

Monitoring: We are monitoring the workflow and AppServer pods in IRW PRD.

Identified: We are continuing to work on a fix for this issue.

Identified: IRW PRD Workflow Issues – Workflow timeout errors and delayed workflow instances.

Resolved: This incident has been resolved.

Monitoring: We are continuing to investigate and monitor the environment. An emergency maintenance is scheduled shortly to complete works that aims to help improve the current situation and increase our ability to troubleshoot the workflow and email issues some customers are currently experiencing (https://status.ivanticloud.com/incidents/rh7nj7grcf2c).

We will be closing this particular incident imminently with the plan to continue all tracking and updates via a single landscape incident which can be found here: https://status.ivanticloud.com/incidents/94tqq9t0ffrx

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: We have identified an issue causing a performance impact to a number to customers in the IRW PRD environment. We have identified the underlying cause and we are working to restore service to normal operation as soon as possible.

Identified: The issue has been identified and we are currently working to implement a fix.

Investigating: A number of customers in IRW are reporting significant delays in workflow and email processing in their production environments.

Resolved: Service have been failed back over to PRD. All services are healthy and available. If any issues are noticed please engage customer support.

Monitoring: We have successfully failed over to the DR environment, and tenant services are continuing to come back online. We are actively monitoring system stability and performance.

Our DBA team is currently working on the failback process to the primary production environment. This process is taking longer than initially expected. Currently we are looking at completing the needed work and being ready to move services back to the primary datacenter overnight during the early morning hours of March 19th AEDT time.

Please note the following are currently impaired: Reporting services will remain unavailable until production database services are fully restored. Response times are likely to be slower in automation and web UI

We will continue to provide updates as more information becomes available.

Note: The failback process to the original environment will be seamless, and no action is required from customers during or after this process.

Once we are confident in the timings of rolling back we will update both Status Page and Cloud Calendar with the specific date/time.

Monitoring: We have successfully failed over to the DR environment, and tenant services are continuing to come back online. We are actively monitoring system stability and performance. Some performance degradation may be experienced as services stabilize in the DR environment.

Our DBA team is currently working on the failback process to the primary production environment. This process is taking longer than initially expected, and at this time we do not have an estimated timeline for completion. Reporting services will remain unavailable until production database services are fully restored. We will continue to provide updates as more information becomes available.

Note: The failback process to the original environment will be seamless, and no action is required from customers during or after this process.

Monitoring: We have successfully failed over to the DR environment and tenant services are continuing to come back online. We are actively monitoring system stability and performance. Some performance degradation may be experienced as services stabilize in the DR environment.

Our DBA team is currently working on failover environment fixes. This failover process is taking longer than initially expected, and at this time we do not have an estimated timeline for completion.

Reporting services will remain unavailable until production database services are fully restored. We will continue to provide updates as more information becomes available.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: We have successfully failed over to the DR environment and tenant services are continuing to come back online. We are actively monitoring system stability and performance. Some performance degradation may be experienced as services stabilize in the DR environment. Our DBA team is currently working on failover environment fixes. This failover process is taking longer than initially expected, and at this time we do not have an estimated timeline for completion. Reporting services will remain unavailable until production database services are fully restored. We will continue to provide updates as more information becomes available.

Identified: We encountered unexpected errors in the restoration of the data from the corrupted disks. As a result we will be pivoting to our Disaster Recovery site. This will result in roughly 15 minutes of data loss (data loss would be between 7:47 AM and 8:02 AM AEDT Mar 17, the 15 minutes leading up to the outage). Once we have completed the rerouting to the DR site we will update here. As a result of pivoting to the DR site reporting services will remain unavailable.

Identified: Working with Microsoft we have identified a fix and are in the process of implementing. Expected recovery is approximately 1 hour.

Identified: Microsoft Senior Engineers have engaged and are investigating. The work on reverting to the DR site is temporarily on hold while the Senior Engineer investigates as we have determined that there will be 15 minutes of lost data as part of the move to our DR site.

Identified: We are preparing to enable our DR location. The work to reroute traffic, enable data services, and restore tenant access is expected to take 1-1.5 hours to complete. There may be some period of degraded performance immediately following the rerouting of traffic. While the DR work is taking place the teams are continuing to work with our vendor in hopes they can resolve the issue faster without the need to invoke DR.

Identified: We have escalated and are working with our Service Provider to restore service.

Identified: Ivanti Engineering teams are engaged and working towards a resolution.

Identified: The issue has been identified and a fix is being implemented.

Investigating: Tenants are temporarily not accessible. We are actively investigating this and will provide an update once we have more information.

Resolved: This incident has been resolved.

Monitoring: A fix has been implemented and we are monitoring the results.

Identified: The issue has been identified and a fix is being implemented.

Investigating: We are currently experiencing delays with device discovery in NVU. Newly installed agent devices or connector-imported devices may not appear as expected.