Loss of external connectivity in AMS
Netzwerk — Transit & Peering — AMS · Out-of-Band-Management — AMS
Zeitleiste
-
Behoben
AMS has been fully reachable since 18:00 UTC. Duration: 10 minutes. Workloads kept running; every machine in the site receives the SLA extension automatically.
-
Ursache erkannt
A configuration push to the two core switches in AMS was applied in the wrong order and removed the external VLAN from both before re-adding it. The push has been rolled back.
-
Untersuchung läuft
All machines in AMS became unreachable from outside at 17:50 UTC. Both transit carriers and the exchange are down simultaneously, which points at something inside the site rather than at a carrier.
Nachbesprechung des Vorfalls ·
Was passiert ist und was sich geändert hat
Ursache
A configuration change intended to be applied to one core switch at a time was pushed to both at once by the automation, because the change set was tagged with the site rather than with a single device. The change removed and re-added the external VLAN; with both switches doing it simultaneously there was no path out of the site for 10 minutes.
Auswirkung
Every machine in AMS was unreachable from the internet for 10 minutes. Out-of-band access went down with it because the management network shares the same uplinks. No workload was interrupted and no data was lost.
Was wir geändert haben
- The automation now refuses to target more than one core device per site in a single change, without exception.
- The out-of-band network in every site now has an independent uplink so that we retain console access during an event of this kind.
- Term extensions were applied to every machine in AMS.
Zeiten werden in Ihrer Zeitzone angezeigt ().
Verwandte Seiten
- ServicestatusLive-Status jeder gpuserver.io-Komponente und jedes Rechenzentrums, alle 60 Sekunden aus fünf Städten gemessen, mit 90-Tage-Verfügbarkeit und Protokoll seit 2022.
- VorfallshistorieJeder Vorfall und jedes Wartungsfenster auf gpuserver.io seit September 2022, mit Zeitleiste und Nachbesprechung des Vorfalls.
- Vereinbarung zum ServicelevelDie 99,9-%-Zusage: was als Ausfallzeit zählt, wie sie von außen gemessen wird, fünf Minuten Laufzeit zurück je verlorener Minute, und die Ausschlüsse vollständig.
- NetzwerkPorts ohne Volumenbegrenzung bis 25 Gbit/s, zwei Carrier und ein IX pro Standort, ständige DDoS-Filterung, geroutetes IPv6 — und vier Dinge, die wir nicht anbieten.