Loss of external connectivity in REY

사후 검토 · 2026. 6. 19. 오전 4:21 UTC

발생한 일과 변경 사항

원인

A configuration change intended to be applied to one core switch at a time was pushed to both at once by the automation, because the change set was tagged with the site rather than with a single device. The change removed and re-added the external VLAN; with both switches doing it simultaneously there was no path out of the site for 5 minutes.

영향도

Every machine in REY was unreachable from the internet for 5 minutes. Out-of-band access went down with it because the management network shares the same uplinks. No workload was interrupted and no data was lost.

변경 사항

• The automation now refuses to target more than one core device per site in a single change, without exception.
• The out-of-band network in every site now has an independent uplink so that we retain console access during an event of this kind.
• Term extensions were applied to every machine in REY.