Cooling alarm in YUL, row D; some nodes thermally throttled

타임라인

1. 해결됨2025. 5. 12. 오후 8:48 UTC

   Row D is back at target temperature and no GPU is throttling. Duration: 36 minutes. Affected workloads ran slower during the event but were not interrupted.

2. 모니터링 중2025. 5. 12. 오후 8:39 UTC

   The tripped unit is back in service and inlet temperatures are falling. GPU clocks are returning to normal.

3. 원인 확인2025. 5. 12. 오후 8:20 UTC

   One of the two CRAC units serving row D tripped on a pressure alarm. The remaining unit is holding the row a few degrees above target. The facility has an engineer on the unit.

4. 조사 중2025. 5. 12. 오후 8:16 UTC

   Inlet temperatures in row D of YUL rose above the alert threshold at 20:12 UTC. GPUs on 17 machines in that row have reduced their clocks to stay within limits. Machines remain reachable and workloads continue at lower performance.