BMC sensor read errors on H100 PCIe nodes in YUL

타임라인

1. 해결됨2025. 1. 11. 오전 10:42 UTC

   Sensor data is complete for every node. Duration: 44 minutes. The fixed firmware is being rolled out during regular maintenance; no host reboot is required.

2. 모니터링 중2025. 1. 11. 오전 10:32 UTC

   All 6 BMCs have been reset and report cleanly. Watching for recurrence.

3. 원인 확인2025. 1. 11. 오전 10:08 UTC

   A BMC firmware version on 6 nodes has a bug in its sensor polling that surfaces after a long uptime. Resetting the BMC clears it without touching the host; we are doing that node by node.

4. 조사 중2025. 1. 11. 오전 10:00 UTC

   The BMCs on some H100 PCIe nodes in YUL are returning sensor read errors since 09:58 UTC. Power and temperature readings in the customer area are missing for those machines. The machines themselves are unaffected.