Loss of external connectivity in REY
网络——转接与对等互联——REY · 带外管理——REY
时间线
-
已解决
REY has been fully reachable since 14:48 UTC. Duration: 5 minutes. Workloads kept running; every machine in the site receives the SLA extension automatically.
-
调查中
All machines in REY became unreachable from outside at 14:43 UTC. Both transit carriers and the exchange are down simultaneously, which points at something inside the site rather than at a carrier.
事故复盘 ·
发生了什么，改变了什么
原因
A configuration change intended to be applied to one core switch at a time was pushed to both at once by the automation, because the change set was tagged with the site rather than with a single device. The change removed and re-added the external VLAN; with both switches doing it simultaneously there was no path out of the site for 5 minutes.
影响
Every machine in REY was unreachable from the internet for 5 minutes. Out-of-band access went down with it because the management network shares the same uplinks. No workload was interrupted and no data was lost.
我们改变了什么
- The automation now refuses to target more than one core device per site in a single change, without exception.
- The out-of-band network in every site now has an independent uplink so that we retain console access during an event of this kind.
- Term extensions were applied to every machine in REY.
时间以您所在时区（）显示。