What happened

At 11:39 GMT on 21st December 2022, users started being presented with errors when signing in. This error relates to our sign-in system communicating with Microsoft Azure, which hosts the majority of our accounts. The secret key that we were using to identify ourselves to Azure had expired at this moment so all attempts to fetch user information started to fail. The fix was simple: generate a fresh secret key and apply it to our system.

We’re sorry for the inconvenience caused while people were trying to sign in to get work done, and hope it didn’t impact too badly on your day.

How we’ll avoid it happening again

We’ll do a couple of things:

• We’ll ensure that the expiry date for this new secret is well known and that reminders are set up to renew it well ahead of time. We may also investigate alternative authentication methods that don’t require an expiring key.
• Increase the amount of monitoring in the sign-in system that specifically looks at Azure, to minimise the total time to fix if it ever did expire again.