Loss of external connectivity in AMS
Rede — trânsito e peering — AMS · Gerenciamento fora de banda — AMS
Linha do tempo
-
Resolvido
AMS has been fully reachable since 18:00 UTC. Duration: 10 minutes. Workloads kept running; every machine in the site receives the SLA extension automatically.
-
Causa identificada
A configuration push to the two core switches in AMS was applied in the wrong order and removed the external VLAN from both before re-adding it. The push has been rolled back.
-
Investigando
All machines in AMS became unreachable from outside at 17:50 UTC. Both transit carriers and the exchange are down simultaneously, which points at something inside the site rather than at a carrier.
Revisão pós-incidente ·
O que aconteceu, e o que mudou
Causa
A configuration change intended to be applied to one core switch at a time was pushed to both at once by the automation, because the change set was tagged with the site rather than with a single device. The change removed and re-added the external VLAN; with both switches doing it simultaneously there was no path out of the site for 10 minutes.
Impacto
Every machine in AMS was unreachable from the internet for 10 minutes. Out-of-band access went down with it because the management network shares the same uplinks. No workload was interrupted and no data was lost.
O que mudamos
- The automation now refuses to target more than one core device per site in a single change, without exception.
- The out-of-band network in every site now has an independent uplink so that we retain console access during an event of this kind.
- Term extensions were applied to every machine in AMS.
Os horários são mostrados no seu fuso horário ().
Páginas relacionadas
- Status do serviçoStatus de cada componente e centro de dados do gpuserver.io, sondados a cada 60s em cinco cidades, com disponibilidade de 90 dias e log de incidentes desde 2022.
- Histórico de incidentesTodos os incidentes e janelas de manutenção em gpuserver.io desde setembro de 2022, mês a mês, com linha do tempo e revisão pós-incidente.
- Acordo de nível de serviçoO compromisso de 99,9%: o que conta como indisponibilidade, como é medido de fora, cinco minutos de prazo devolvidos por minuto perdido, e as exclusões por completo.
- RedePortas sem contador até 25 Gbit/s, duas operadoras e um IX por site, filtragem DDoS sempre ativa, IPv6 roteado — e as quatro coisas que não oferecemos, já ditas.