BMC sensor read errors on L4 nodes in AMS

时间线

1. 已解决2023年2月12日 09:46 UTC

   Sensor data is complete for every node. Duration: 1 h 4 min. The fixed firmware is being rolled out during regular maintenance; no host reboot is required.

2. 观察中2023年2月12日 09:36 UTC

   All 9 BMCs have been reset and report cleanly. Watching for recurrence.

3. 已确定原因2023年2月12日 09:09 UTC

   A BMC firmware version on 9 nodes has a bug in its sensor polling that surfaces after a long uptime. Resetting the BMC clears it without touching the host; we are doing that node by node.

4. 调查中2023年2月12日 08:47 UTC

   The BMCs on some L4 nodes in AMS are returning sensor read errors since 08:42 UTC. Power and temperature readings in the customer area are missing for those machines. The machines themselves are unaffected.