Критический Устранено Разбор опубликован

Loss of external connectivity in REY

Сеть — транзит и пиринг — REY · Внеполосное управление — REY

Статус: Устранено
Влияние: Критический
Начато: 16 июн. 2026 г., 14:43 UTC
Устранено: 16 июн. 2026 г., 14:48 UTC
Длительность: 5 минут
Затронуто: Серьёзный сбойСеть — транзит и пиринг — REY

Серьёзный сбойВнеполосное управление — REY
Референс: b56rdfq97j

Текущий статус

Обновления по инциденту пишутся на английском языке дежурным инженером и не переводятся.

Хронология

Устранено16 июн. 2026 г., 14:48 UTC

REY has been fully reachable since 14:48 UTC. Duration: 5 minutes. Workloads kept running; every machine in the site receives the SLA extension automatically.
Идёт расследование16 июн. 2026 г., 14:45 UTC

All machines in REY became unreachable from outside at 14:43 UTC. Both transit carriers and the exchange are down simultaneously, which points at something inside the site rather than at a carrier.

Разбор инцидента · 19 июн. 2026 г., 04:21 UTC

Что произошло и что изменилось

Причина

A configuration change intended to be applied to one core switch at a time was pushed to both at once by the automation, because the change set was tagged with the site rather than with a single device. The change removed and re-added the external VLAN; with both switches doing it simultaneously there was no path out of the site for 5 minutes.

Влияние

Every machine in REY was unreachable from the internet for 5 minutes. Out-of-band access went down with it because the management network shares the same uplinks. No workload was interrupted and no data was lost.

Что мы изменили

The automation now refuses to target more than one core device per site in a single change, without exception.
The out-of-band network in every site now has an independent uplink so that we retain console access during an event of this kind.
Term extensions were applied to every machine in REY.