PDU fault in rack AMS-B06
Energia e refrigeração — AMS · Gerenciamento fora de banda — AMS
Linha do tempo
-
Resolvido
Every machine in rack B06 has been up since 02:52 UTC. Duration: 16 minutes. The PDU is being replaced; the 8 machines that lost power receive the SLA extension automatically. Data on local disks is intact; the machines simply rebooted.
-
Causa identificada
The A-side PDU in rack B06 failed with a breaker fault. The 8 machines that rebooted are the ones whose B-side supply was not seated correctly after a recent hardware change. All are powered from the B-side now and are booting.
-
Investigando
8 machines in rack B06 of AMS lost power at 02:36 UTC. Machines in the rack are dual-fed, but these 8 rebooted when one PDU failed. Engineers are at the rack.
Revisão pós-incidente ·
O que aconteceu, e o que mudou
Causa
The A-side PDU in AMS-B06 suffered a breaker fault. Every machine in the rack has two power supplies on separate feeds, but on 8 of them the B-side cord had been left partially seated during a hardware change the previous week, so they lost power when the A side went.
Impacto
8 machines rebooted and were unavailable for 16 minutes. No data was lost; the machines came back to the state they were installed in, with local disks intact.
O que mudamos
- Every hardware change now ends with a power-redundancy test: each feed is dropped in turn and the machine must stay up.
- The faulty PDU model is being phased out across the fleet.
- Term extensions were applied to the 8 machines the same day.
Os horários são mostrados no seu fuso horário ().
Páginas relacionadas
- Status do serviçoStatus de cada componente e centro de dados do gpuserver.io, sondados a cada 60s em cinco cidades, com disponibilidade de 90 dias e log de incidentes desde 2022.
- Histórico de incidentesTodos os incidentes e janelas de manutenção em gpuserver.io desde setembro de 2022, mês a mês, com linha do tempo e revisão pós-incidente.
- Acordo de nível de serviçoO compromisso de 99,9%: o que conta como indisponibilidade, como é medido de fora, cinco minutos de prazo devolvidos por minuto perdido, e as exclusões por completo.
- RedePortas sem contador até 25 Gbit/s, duas operadoras e um IX por site, filtragem DDoS sempre ativa, IPv6 roteado — e as quatro coisas que não oferecemos, já ditas.