On August 14 2019, from approximately 11pm EST until 7:30AM EST August 15th, our SSO service became unavailable for some customers. Any already signed in users or customers using Brightidea Login (Non SSO) remained unaffected but a limited amount of users that attempted to sign in using our SSO service were impacted.
Root cause Analysis
During our scheduled weekly deploy on August 14th, we deployed a change to our SSO service for capabilities to capture more detailed SSO transaction logs. This change resulted in an unexpected fatal error.
At 11:07pm & 11:24pm EST, log files for some of our servers were rotated, the newly created log file did not have correct write permissions and threw a fatal write permission error. This resulted in our SSO service failing.
We rolled back the changes made to our SSO service for additional logs. We will also be expanding
Apologies for any inconvenience.
The Brightidea Team.