Out-of-band network partition in STO
带外管理——STO
时间线
-
已解决
Out-of-band access in STO has been stable since 20:33 UTC. Duration: 25 minutes. Any reinstall or power action queued during the window has been retried and completed.
-
观察中
Consoles are reachable again. We are restoring the state sync between the two firewall members.
-
已确定原因
The firewall pair in front of the STO management network lost its state synchronisation and both members went passive. We have forced one member active.
-
调查中
No IPMI or KVM console in STO is reachable since 20:08 UTC. Production traffic is unaffected. Reinstalls and power actions from the customer area will fail for machines in STO until this is fixed.
事故复盘 ·
发生了什么，改变了什么
原因
A firmware update on the management-network firewall pair in STO reset the state-synchronisation key on one member. When the heartbeat failed, both members concluded the other was active and went passive.
影响
IPMI, KVM, reinstalls and remote power actions were unavailable in STO for 25 minutes. Production connectivity, workloads and data were unaffected.
我们改变了什么
- Firewall firmware updates are now applied one member at a time with an explicit failover test between the two.
- The split-brain guard was changed so that a member with a healthy upstream link stays active rather than going passive.
时间以您所在时区（）显示。