This incident has been resolved. The underlying issue was traced to an operating system problem with a particular host virtual machine. All relevant data was migrated from the damaged infrastructure while it was repaired over night and the repair was monitored for effectiveness. At this time it has been stable for several hours so the issue is being marked as resolved.
Posted Oct 15, 2024 - 08:46 EDT
Update
We are continuing to investigate this issue. Automated recovery is proving effective at mitigating the problem and minimizing the impact, but the root cause is still unknown.
Posted Oct 14, 2024 - 17:29 EDT
Update
We are able to observe the problem happening and are able to recover from it relatively quickly, but have not yet found the underlying cause. We have temporarily put automated recovery processes in place to minimize disruption while we investigate the issue further. This limits the impact when the disruption occurs but does not eliminate it completely. So far only a subset of our infrastructure is experiencing the issue and therefore only a subset of customers are seeing the intermittent disruptions.
Posted Oct 14, 2024 - 16:20 EDT
Investigating
We're experiencing an elevated level of API errors and are currently looking into the issue.
Posted Oct 14, 2024 - 14:20 EDT
This incident affected: CarbonHub, UtilityManagement, EnergyCAP Enterprise, EnergyCAP Optimum, and Bill CAPture.