Summary

Between Tuesday, March 26, 2024 02:40 UTC to 07:00 UTC, all users of the Infinity Portal in UAE region couldn’t login and using all the cloud applications hosted in the portal.

The incident started after adding an environment variable. The introduction of the new variable in another service led to unexpected behavior.

Once the issue was identified, the team move the variable to a specific service and solve the issue.

‌

Incident Timeline

• Wednesday, March 26, 2024, 06:00 UTC – reports start coming in from users experiencing login issues
• Wednesday, March 26, 2024, 06:15 UTC – the team started investigating the issue
• Wednesday, March 26, 2024, 07:10 UTC – discovered a variable that was deployed a day before which cause the issue
• Wednesday, March 26, 2024, 07:40 UTC – complete the moving of the variable to a specifc service and restore the system to stable state

Root Cause Analysis

During a planned change to the Infinity Portal system variable, intended to enhance our system's capabilities, an unforeseen chain of events occurred, leading to service outage.

We sincerely apologize for the recent system outage and appreciate your patience.

We have successfully tested this procedure on the staging environment without encountering any issues.
Next Steps: we will promptly implement an alert system to detect and address such issues faster in the future.