Individual systems recovered rapidly, but the updates rolled through several systems affecting different portions of the service until the process was stopped and the last system was brought back online around 12:18 AM on 9/17/24.
We understand that our customers depend on our services for normal business operations, and are working to continuously improve our processes. Thank you for your patience during this issue. The processes for categorizing the safety of updates for application beyond standard maintenance windows are being updated to prevent future recurrence of this issue.
Posted Sep 17, 2024 - 00:15 EDT
Identified
At 10:08 PM ET on 9/16/2024 engineers monitoring operating system patches noted that some systems that were scheduled to apply non-reboot updates were rebooting and causing an unplanned disruption in service. The underlying cause has been identified as a mis-categorization of updates that were expected to not require reboot actually triggering a reboot unexpectedly.
Posted Sep 16, 2024 - 22:00 EDT
This incident affected: CarbonHub, UtilityManagement, EnergyCAP Enterprise, EnergyCAP Professional, EnergyCAP Express, EnergyCAP Implementation, EnergyCAP Utility Platform, Bill CAPture, Weather Services, and Utility Platform Services.