중대 해결됨 검토 게시됨

PDU fault in rack AMS-B06

전력 및 냉각 — AMS · 대역 외 관리 — AMS

타임라인

해결됨2023. 2. 15. 오전 2:52 UTC

Every machine in rack B06 has been up since 02:52 UTC. Duration: 16 minutes. The PDU is being replaced; the 8 machines that lost power receive the SLA extension automatically. Data on local disks is intact; the machines simply rebooted.
원인 확인2023. 2. 15. 오전 2:43 UTC

The A-side PDU in rack B06 failed with a breaker fault. The 8 machines that rebooted are the ones whose B-side supply was not seated correctly after a recent hardware change. All are powered from the B-side now and are booting.
조사 중2023. 2. 15. 오전 2:37 UTC

8 machines in rack B06 of AMS lost power at 02:36 UTC. Machines in the rack are dual-fed, but these 8 rebooted when one PDU failed. Engineers are at the rack.

사후 검토 · 2023. 2. 18. 오후 2:40 UTC

발생한 일과 변경 사항

원인

The A-side PDU in AMS-B06 suffered a breaker fault. Every machine in the rack has two power supplies on separate feeds, but on 8 of them the B-side cord had been left partially seated during a hardware change the previous week, so they lost power when the A side went.

영향도

8 machines rebooted and were unavailable for 16 minutes. No data was lost; the machines came back to the state they were installed in, with local disks intact.

변경 사항

Every hardware change now ends with a power-redundancy test: each feed is dropped in turn and the machine must stay up.
The faulty PDU model is being phased out across the fleet.
Term extensions were applied to the 8 machines the same day.

PDU fault in rack AMS-B06

타임라인

발생한 일과 변경 사항

원인

영향도

변경 사항

관련 페이지