Between Tuesday, February 27, 2024, 21:30 UTC to Wednesday, February 28, 2024, 02:30 UTC, all users of the Infinity Portal (EU and US regions) couldn’t log in and use the portal.
Tuesday, February 27, 2024, 21:30 UTC – Our servers started to fail
Tuesday, February 27, 2024, 22:10 UTC – Reports and alerts that users were not able to log in to the portal
Tuesday, February 27, 2024, 22:10 UTC – The team started investigating and discovered the load on the system
Wednesday, February 28, 2024, 02:00 UTC – After implementing all necessary measures, we decided to apply rate limits on the pathway responsible for the load
Wednesday, February 28, 2024, 02:30 UTC – The system has been fully recovered and is back to normal
We encountered server issues as a result of a database query generating a large response. The problem escalated because the init process of this specific server added more load on the system.
By Implementing rate limits on the pathway responsible for generating the load, we successfully resolved the issue, while assuring no degradation took place in the portal.
Subsequently, after completing our investigation and validating a solution, we implemented the fix to address the root cause of this specific process, allowing us to remove the applied rate limits, without any impact on the portal.
We apologize for the recent system outage and any inconvenience it may have caused. We understand the importance of our service availability and appreciate your patience during this time. Additionally, we already implemented a fix and are taking further steps to prevent future disruptions.
We are planning to transition to a more secure and isolated initialization process for our services to avoid unintentional system overloading during startup.