*RESOLVED*
Summary
On March 20, 2023, a system outage occurred from 16:09 to 16:24 EST, impacting all Brightidea systems hosted in the us-east server region. The cause of the outage was a security patch applied outside of the standard maintenance window. The patch process disconnected the web servers, resulting in the unavailability of the Brightidea systems. The issue was detected when both internal Brightidea monitoring and customer users reported the issue. The issue was resolved by restoring connections to the web servers at 16:24 eastern time.
Root cause Analysis
The root cause of the system outage was the application of a security patch outside of the standard
maintenance window, which disconnected the web servers, leading to the outage. The incident occurred due to a failure to follow established maintenance protocols.
Resolution
To prevent a similar incident from occurring in the future, the following corrective and preventive actions
will take place:
- Immediate training will be conducted to ensure team members are up-to-date on the
maintenance protocol - Review and update maintenance documentation regularly to ensure its accuracy and
completeness. - Improve communication and collaboration between teams responsible for maintenance and
deployment
Apologies for any inconvenience.
Thank you.
The Brightidea Team.
Comments