On March 20, 2023, a system outage occurred from 16:09 to 16:24 EST, impacting all Brightidea systems hosted in the us-east server region. The cause of the outage was a security patch applied outside of the standard maintenance window. The patch process disconnected the web servers, resulting in the unavailability of the Brightidea systems. The issue was detected when both internal Brightidea monitoring and customer users reported the issue. The issue was resolved by restoring connections to the web servers at 16:24 eastern time.
Root cause Analysis
The root cause of the system outage was the application of a security patch outside of the standard
maintenance window, which disconnected the web servers, leading to the outage. The incident occurred due to a failure to follow established maintenance protocols.
To prevent a similar incident from occurring in the future, the following corrective and preventive actions
will take place:
- Immediate training will be conducted to ensure team members are up-to-date on the
- Review and update maintenance documentation regularly to ensure its accuracy and
- Improve communication and collaboration between teams responsible for maintenance and
Apologies for any inconvenience.
The Brightidea Team.