Out-of-band network partition in STO
Out-of-band management — STO
Timeline
-
Resolved
Out-of-band access in STO has been stable since 20:33 UTC. Duration: 25 minutes. Any reinstall or power action queued during the window has been retried and completed.
-
Monitoring
Consoles are reachable again. We are restoring the state sync between the two firewall members.
-
Identified
The firewall pair in front of the STO management network lost its state synchronisation and both members went passive. We have forced one member active.
-
Investigating
No IPMI or KVM console in STO is reachable since 20:08 UTC. Production traffic is unaffected. Reinstalls and power actions from the customer area will fail for machines in STO until this is fixed.
Post-incident review ·
What happened, and what changed
Cause
A firmware update on the management-network firewall pair in STO reset the state-synchronisation key on one member. When the heartbeat failed, both members concluded the other was active and went passive.
Impact
IPMI, KVM, reinstalls and remote power actions were unavailable in STO for 25 minutes. Production connectivity, workloads and data were unaffected.
What we changed
- Firewall firmware updates are now applied one member at a time with an explicit failover test between the two.
- The split-brain guard was changed so that a member with a healthy upstream link stays active rather than going passive.
Times are shown in your time zone ().
Related pages
- Service statusLive status of every gpuserver.io component and data centre, probed every 60 seconds from five cities, with 90-day uptime and the incident log since 2022.
- Incident historyEvery incident and maintenance window on gpuserver.io since monitoring began in September 2022, month by month, with its timeline and post-incident review.
- Service level agreementThe 99.9% commitment: what counts as downtime, how it is measured from outside, five minutes of term back per minute lost, and the exclusions in full.
- NetworkUnmetered ports up to 25 Gbit/s, two carriers and an IX per site, always-on DDoS filtering, routed IPv6 — and the four things we do not offer, stated up front.