Out-of-band network partition in ZRH
Gestión fuera de banda — ZRH
Cronología
-
Resuelto
Out-of-band access in ZRH has been stable since 08:49 UTC. Duration: 1 h 17 min. Any reinstall or power action queued during the window has been retried and completed.
-
En observación
Consoles are reachable again. We are restoring the state sync between the two firewall members.
-
Causa identificada
The firewall pair in front of the ZRH management network lost its state synchronisation and both members went passive. We have forced one member active.
-
Investigando
No IPMI or KVM console in ZRH is reachable since 07:32 UTC. Production traffic is unaffected. Reinstalls and power actions from the customer area will fail for machines in ZRH until this is fixed.
Revisión posterior al incidente ·
Qué ocurrió y qué cambió
Causa
A firmware update on the management-network firewall pair in ZRH reset the state-synchronisation key on one member. When the heartbeat failed, both members concluded the other was active and went passive.
Impacto
IPMI, KVM, reinstalls and remote power actions were unavailable in ZRH for 1 h 17 min. Production connectivity, workloads and data were unaffected.
Qué cambiamos
- Firewall firmware updates are now applied one member at a time with an explicit failover test between the two.
- The split-brain guard was changed so that a member with a healthy upstream link stays active rather than going passive.
Las horas se muestran en su zona horaria ().
Páginas relacionadas
- Estado del servicioEstado en vivo de gpuserver.io: componentes y centros de datos sondeados cada 60 s desde 5 ciudades, con disponibilidad de 90 días e incidentes desde 2022.
- Historial de incidentesTodos los incidentes y ventanas de mantenimiento de gpuserver.io desde septiembre de 2022, mes a mes, con cronología y revisión posterior al incidente.
- Acuerdo de nivel de servicioEl compromiso del 99,9 %: qué cuenta como inactividad, cómo se mide desde fuera, cinco minutos de plazo por cada minuto perdido, y las exclusiones completas.
- RedPuertos sin contador hasta 25 Gbit/s, dos operadores y un IX por sede, filtrado DDoS permanente, IPv6 enrutado — y las cuatro cosas que no ofrecemos, dichas ya.