Term activation delayed after confirmed payments
Payments & invoicing · Provisioning pipeline
Timeline
-
Resolved
Resolved at 18:11 UTC. Duration: 40 minutes. Terms were activated up to 27 minutes late; every term runs its full length from the actual activation time.
-
Monitoring
The queue has drained; every confirmed payment has an active term and an order in provisioning.
-
Identified
The activation worker crashed on a malformed order record and did not restart. It has been restarted and the record corrected.
-
Investigating
Payments are being confirmed but the step that activates the term and hands the order to provisioning is not running since 17:31 UTC. Confirmed payments are queued, not lost.
Post-incident review ·
What happened, and what changed
Cause
The activation worker stopped on an order record whose payment amount field held two decimals more than the schema allowed, and the supervisor did not restart it because it exited with a code that was classified as intentional.
Impact
6 confirmed payments waited up to 27 minutes for activation. No payment was lost and every term received its full duration.
What we changed
- The worker now skips and reports a malformed record instead of stopping.
- Every exit code of the worker now triggers a restart; the classification that suppressed it was removed.
- The queue depth is now an alert in its own right, independent of the worker's health.
Times are shown in your time zone ().
Related pages
- Service statusLive status of every gpuserver.io component and data centre, probed every 60 seconds from five cities, with 90-day uptime and the incident log since 2022.
- Incident historyEvery incident and maintenance window on gpuserver.io since monitoring began in September 2022, month by month, with its timeline and post-incident review.
- Service level agreementThe 99.9% commitment: what counts as downtime, how it is measured from outside, five minutes of term back per minute lost, and the exclusions in full.
- NetworkUnmetered ports up to 25 Gbit/s, two carriers and an IX per site, always-on DDoS filtering, routed IPv6 — and the four things we do not offer, stated up front.