Post Mortem: SSO partial outage for DoiT Console App Date: June 6, 2023

Incident Summary: On June 6, 2023, the DoiT Console App experienced intermittent 503 errors for users trying to sign in with SAML/OIDC preventing access to DoiT Console

Timeline (EST UTC-5:00):

• May 31, 2023, 3:46 AM: Initial report of the issue, single customer reported they were unable to use SSO to sign in
• June 1, 2023, 12:55 AM: Analysis of HAR provided by the customer showed a 503 response when calling Google Identity Toolkit API signInWithIdp method
• June 5, 2023, 7:59 AM: After internal investigation without any findings a P2 support ticket was opened with Google Support
• June 6, 2023, 10:17 AM: Following more reports of the issue from additional customers the support ticket was escalated and raised to P1. The incident was also reported in DoiT support portal
• June 6, 2023, 12:53 PM: Following a meeting with Google Support an internal P1 bug was opened for Google Identity Toolkit engineering team
• June 7, 2023, 1:15 PM: Google support suggest we check whether the blocking functions might cause the issue
• June 7, 2023, 2:30 PM: The issue was identified and a fix was deployed
• June 8, 2023, 1:00 AM: The fix was validated and the incident was closed

Root Cause: The root cause of the issue was a change rolled out to blocking function code. In some cases the code would fail and prevented the successful sign in process.

Resolution and Recovery: We deployed a fix that will handle the cases that caused the error without failing the sign in process to allow successful login flow.

Going forward, our team and Google Support team are still investigating in order to fully understand when this cases occur to ensure optimal user experience for the DoiT Console