Provisioning paused: driver package failed validation

时间线

1. 已解决2023年3月6日 13:38 UTC

   Resolved at 13:38 UTC. Duration: 1 h 25 min. 7 orders were delivered late; each customer has been told, and the delay is added to their term.

2. 观察中2023年3月6日 13:26 UTC

   Provisioning has resumed and the queue is clearing. Each affected node is being re-provisioned from scratch.

3. 已确定原因2023年3月6日 12:49 UTC

   The package was corrupted on our mirror during a sync. We have re-fetched it from the upstream source and its checksum matches.

4. 调查中2023年3月6日 12:18 UTC

   New provisioning is paused since 12:13 UTC: the NVIDIA driver package (530.30.02) failed its checksum validation on every node it was applied to. We stop rather than ship a machine with an unverified driver.

事故复盘会在解决后 5 天内发布于此。