Cooling alarm in YUL, row D; some nodes thermally throttled

时间线

1. 已解决2025年5月12日 20:48 UTC

   Row D is back at target temperature and no GPU is throttling. Duration: 36 minutes. Affected workloads ran slower during the event but were not interrupted.

2. 观察中2025年5月12日 20:39 UTC

   The tripped unit is back in service and inlet temperatures are falling. GPU clocks are returning to normal.

3. 已确定原因2025年5月12日 20:20 UTC

   One of the two CRAC units serving row D tripped on a pressure alarm. The remaining unit is holding the row a few degrees above target. The facility has an engineer on the unit.

4. 调查中2025年5月12日 20:16 UTC

   Inlet temperatures in row D of YUL rose above the alert threshold at 20:12 UTC. GPUs on 17 machines in that row have reduced their clocks to stay within limits. Machines remain reachable and workloads continue at lower performance.