Summary

Between Wednesday, February 14, 2024 07:07 UTC to 08:54 UTC, all users of the Infinity Portal (All regions) experienced degraded performance of the login and using all the cloud applications hosted in the portal.
Due to sporadic failures, some users were able to log in while others couldn’t. Those who were able to log in were seeing random errors in various Infinity Portal pages.

The incident started after a specific deployment, intended to enhance our system's capabilities and security. An unforeseen chain of events was triggered, leading to multiple service degradations.

Once the issue was identified, the team performed quick maintenance actions to restore the system to a stable state.

Incident Timeline

• Wednesday, February 14, 2024, 07:07 UTC – A deployment triggered in the system
• Wednesday, February 14, 2024, 07:35 UTC – Reports start coming in from users experiencing login and operational issues on all regions
• Wednesday, February 14, 2024, 07:40 UTC – The team started investigating the issue and monitoring different system components
• Wednesday, February 14, 2024, 08:09 UTC – Maintenance actions were taken to restore the system to stable state
• Wednesday, February 14, 2024, 08:54 UTC – System was back to normal on all regions. Incident closed

Root Cause Analysis

During a planned rolling update to the Infinity Portal system, intended to enhance our system's capabilities and security, an unforeseen chain of events was triggered, leading to multiple service degradations.
Next Steps
We sincerely apologize for the recent system outage and appreciate your patience. To prevent future incidents, we plan to:

1. Define tasks for a major effort of wide stability and operability improvements to be implemented in the next weeks
2. Fix alerting of critical places in our system