Loss of external connectivity in REY
Red — tránsito y peering — REY · Gestión fuera de banda — REY
Cronología
-
Resuelto
REY has been fully reachable since 14:48 UTC. Duration: 5 minutes. Workloads kept running; every machine in the site receives the SLA extension automatically.
-
Investigando
All machines in REY became unreachable from outside at 14:43 UTC. Both transit carriers and the exchange are down simultaneously, which points at something inside the site rather than at a carrier.
Revisión posterior al incidente ·
Qué ocurrió y qué cambió
Causa
A configuration change intended to be applied to one core switch at a time was pushed to both at once by the automation, because the change set was tagged with the site rather than with a single device. The change removed and re-added the external VLAN; with both switches doing it simultaneously there was no path out of the site for 5 minutes.
Impacto
Every machine in REY was unreachable from the internet for 5 minutes. Out-of-band access went down with it because the management network shares the same uplinks. No workload was interrupted and no data was lost.
Qué cambiamos
- The automation now refuses to target more than one core device per site in a single change, without exception.
- The out-of-band network in every site now has an independent uplink so that we retain console access during an event of this kind.
- Term extensions were applied to every machine in REY.
Las horas se muestran en su zona horaria ().
Páginas relacionadas
- Estado del servicioEstado en vivo de gpuserver.io: componentes y centros de datos sondeados cada 60 s desde 5 ciudades, con disponibilidad de 90 días e incidentes desde 2022.
- Historial de incidentesTodos los incidentes y ventanas de mantenimiento de gpuserver.io desde septiembre de 2022, mes a mes, con cronología y revisión posterior al incidente.
- Acuerdo de nivel de servicioEl compromiso del 99,9 %: qué cuenta como inactividad, cómo se mide desde fuera, cinco minutos de plazo por cada minuto perdido, y las exclusiones completas.
- RedPuertos sin contador hasta 25 Gbit/s, dos operadores y un IX por sede, filtrado DDoS permanente, IPv6 enrutado — y las cuatro cosas que no ofrecemos, dichas ya.