BMC sensor read errors on H100 PCIe nodes in YUL

时间线

1. 已解决2025年1月11日 10:42 UTC

   Sensor data is complete for every node. Duration: 44 minutes. The fixed firmware is being rolled out during regular maintenance; no host reboot is required.

2. 观察中2025年1月11日 10:32 UTC

   All 6 BMCs have been reset and report cleanly. Watching for recurrence.

3. 已确定原因2025年1月11日 10:08 UTC

   A BMC firmware version on 6 nodes has a bug in its sensor polling that surfaces after a long uptime. Resetting the BMC clears it without touching the host; we are doing that node by node.

4. 调查中2025年1月11日 10:00 UTC

   The BMCs on some H100 PCIe nodes in YUL are returning sensor read errors since 09:58 UTC. Power and temperature readings in the customer area are missing for those machines. The machines themselves are unaffected.