Cooling alarm in YUL, row D; some nodes thermally throttled

タイムライン

1. 解決済み2025/05/12 20:48 UTC

   Row D is back at target temperature and no GPU is throttling. Duration: 36 minutes. Affected workloads ran slower during the event but were not interrupted.

2. 監視中2025/05/12 20:39 UTC

   The tripped unit is back in service and inlet temperatures are falling. GPU clocks are returning to normal.

3. 原因特定2025/05/12 20:20 UTC

   One of the two CRAC units serving row D tripped on a pressure alarm. The remaining unit is holding the row a few degrees above target. The facility has an engineer on the unit.

4. 調査中2025/05/12 20:16 UTC

   Inlet temperatures in row D of YUL rose above the alert threshold at 20:12 UTC. GPUs on 17 machines in that row have reduced their clocks to stay within limits. Machines remain reachable and workloads continue at lower performance.