Out-of-band network partition in STO
Gestione fuori banda — STO
Cronologia
-
Risolto
Out-of-band access in STO has been stable since 20:33 UTC. Duration: 25 minutes. Any reinstall or power action queued during the window has been retried and completed.
-
In osservazione
Consoles are reachable again. We are restoring the state sync between the two firewall members.
-
Causa individuata
The firewall pair in front of the STO management network lost its state synchronisation and both members went passive. We have forced one member active.
-
In analisi
No IPMI or KVM console in STO is reachable since 20:08 UTC. Production traffic is unaffected. Reinstalls and power actions from the customer area will fail for machines in STO until this is fixed.
Analisi post-incidente ·
Cosa è successo e cosa è cambiato
Causa
A firmware update on the management-network firewall pair in STO reset the state-synchronisation key on one member. When the heartbeat failed, both members concluded the other was active and went passive.
Impatto
IPMI, KVM, reinstalls and remote power actions were unavailable in STO for 25 minutes. Production connectivity, workloads and data were unaffected.
Cosa abbiamo cambiato
- Firewall firmware updates are now applied one member at a time with an explicit failover test between the two.
- The split-brain guard was changed so that a member with a healthy upstream link stays active rather than going passive.
Gli orari sono mostrati nel fuso orario locale ().
Pagine correlate
- Stato del servizioStato di ogni componente e data center gpuserver.io, sondati ogni 60 secondi da cinque città: disponibilità su 90 giorni e registro incidenti dal 2022.
- Cronologia incidentiOgni incidente e finestra di manutenzione su gpuserver.io dall’inizio del monitoraggio, a settembre 2022, mese per mese, con cronologia e analisi post-incidente.
- Accordo sul livello di servizioL’impegno del 99,9%: cosa conta come downtime, come viene misurato dall’esterno, cinque minuti di durata in più per ogni minuto perso, e le esclusioni per intero.
- RetePorte senza contatore fino a 25 Gbit/s, due carrier e un IX per sede, DDoS sempre filtrato, IPv6 instradato — e quattro cose che non offriamo, dette subito.