Pipefy Status

Postmortem: Root Cause

The issue was caused by exceeding the execution limit in integration queues, leading to a loop in automation that triggered numerous webhooks. This resulted in an overload of a specific flow, creating a large processing queue. Consequently, clients experienced delays in their process executions.

Resolution

To resolve the issue, the customer service account was deactivated, which helped alleviate the problem.

Action Plan

To prevent future occurrences, a rate limit will be activated on the system, and the timeout for execution flows will be reduced.

Resolved: This incident has been resolved.

Monitoring: We are continuing to monitor for any further issues.

Monitoring: A fix has been implemented and we are monitoring the results.

Investigating: We are currently investigating this issue.

Resolved: This incident has been resolved.

Monitoring: We are currently monitoring the services to ensure full stability as the backlog of requests clears up. Please note that some intermittent slowness might still be experienced until all queued tasks are fully processed

Identified: The root cause of the queue processing delays has been identified as an ongoing infrastructure issue on the side of our third-party provider, Oracle Cloud (OCI)

Investigating: We are currently investigating an issue causing degraded performance and slowness in services that require background queues. Customers may notice that certain requests are taking longer than usual to process.