Loss of external connectivity in REY
Rede — trânsito e peering — REY · Gerenciamento fora de banda — REY
Linha do tempo
-
Resolvido
REY has been fully reachable since 14:48 UTC. Duration: 5 minutes. Workloads kept running; every machine in the site receives the SLA extension automatically.
-
Investigando
All machines in REY became unreachable from outside at 14:43 UTC. Both transit carriers and the exchange are down simultaneously, which points at something inside the site rather than at a carrier.
Revisão pós-incidente ·
O que aconteceu, e o que mudou
Causa
A configuration change intended to be applied to one core switch at a time was pushed to both at once by the automation, because the change set was tagged with the site rather than with a single device. The change removed and re-added the external VLAN; with both switches doing it simultaneously there was no path out of the site for 5 minutes.
Impacto
Every machine in REY was unreachable from the internet for 5 minutes. Out-of-band access went down with it because the management network shares the same uplinks. No workload was interrupted and no data was lost.
O que mudamos
- The automation now refuses to target more than one core device per site in a single change, without exception.
- The out-of-band network in every site now has an independent uplink so that we retain console access during an event of this kind.
- Term extensions were applied to every machine in REY.
Os horários são mostrados no seu fuso horário ().
Páginas relacionadas
- Status do serviçoStatus de cada componente e centro de dados do gpuserver.io, sondados a cada 60s em cinco cidades, com disponibilidade de 90 dias e log de incidentes desde 2022.
- Histórico de incidentesTodos os incidentes e janelas de manutenção em gpuserver.io desde setembro de 2022, mês a mês, com linha do tempo e revisão pós-incidente.
- Acordo de nível de serviçoO compromisso de 99,9%: o que conta como indisponibilidade, como é medido de fora, cinco minutos de prazo devolvidos por minuto perdido, e as exclusões por completo.
- RedePortas sem contador até 25 Gbit/s, duas operadoras e um IX por site, filtragem DDoS sempre ativa, IPv6 roteado — e as quatro coisas que não oferecemos, já ditas.