- What happened
- During a second round of live failover drills on Nov 9, two drills failed because no replacement machine was ordered. The system refused to reserve a standby slot because its daily standby spending total was wrong.
- Root cause
- The daily total kept adding running cost for replacement machines that had already taken over their jobs. By early morning it counted roughly three times the real amount and passed the daily limit, so every later replacement was refused without a clear log entry.
- What was fixed
- The total now counts a machine that took over a job at its reserved amount only. Refused reservations are now logged with a clear reason, the leftover search records are closed, and the drill note now says the search ended without a machine. Every limit value is unchanged. The fix is published.
- Current status
- Until the fix, a customer job needing a replacement while spend protection was on could have been refused the same way. Job contracts are limited to Kilawatt Cloud's own test account, so no customer job was affected. A fresh series of 20 drills is running to confirm the fix, and results will be published here, including any failures.
Platform status
Kilawatt Cloud system status
No active faults. Not enough customer traffic to rate every component
Derived from real payment, provisioning, execution and health records over the last 24 hours. Payments, Provisioning and API include our own live verification runs through the production API; open health incidents count customer machines only.
Checked Mon, 16 Nov 2026 01:33:17 GMT
Components
Payments (x402)
No payments in the last 24 hours, so there is nothing to rate yet.
Basis: Real payments through the production API in the last 24 hours. A rating needs at least 20 attempts.
Provisioning
No provisioning requests in the last 24 hours, so there is nothing to rate yet.
Basis: Real provisioning requests through the production API in the last 24 hours. A rating needs at least 20 attempts.
API
No API executions in the last 24 hours, so there is nothing to rate yet.
Basis: Real API executions through the production API in the last 24 hours. A rating needs at least 20 attempts.
GPU Health Monitoring
No open health incidents. Last health sample recorded 9157 minutes ago.
Basis: Open incident windows on customer machines and the most recent health sample.
Incident history
Resolved entries, newest first. These are incidents and additions from this platform, including incidents we caused ourselves. Each entry states what happened, why, what changed, and the current status.
55 entries, scroll to see older ones
- What happened
- Between Nov 8 and Nov 9, Kilawatt Cloud ran 11 automated failover drills on its own test account. Each drill starts a small test job, stops its machine on purpose after at least two checkpoints are saved, and measures whether a replacement machine takes over from the saved checkpoint. In the 10 drills after the first fix, 5 resumed from a checkpoint, between 5 minutes 1 second and 6 minutes 5 seconds after the machine was stopped. The other 5 did not resume. Every drill is counted here, including the failures.
- Root cause
- Three defects explain every failed drill. In the first drill, a price safety check blocked a test replacement and ended the search instead of trying the next supplier. In one later drill, a replacement order that was never confirmed stalled the search. In the other four failed drills, a startup-script error on one supplier's replacement machines meant the machine came up and reported ready, but never restarted the job.
- What was fixed
- All three are fixed and published. The search now moves to the next supplier after a price refusal. An unconfirmed order is checked against the supplier's own machine list before its slot is released. Every startup script is now checked for errors before any machine is ordered. Customer pricing and margin rules were not changed. Test drills use a capped, relaxed replacement price rule, which is recorded on each drill.
- Current status
- 20 more drills are running to confirm the fixes, and their results will be published here when the series ends, including any failures. These drills use small GPU test jobs on Kilawatt Cloud's own test account, not customer workloads. Job contracts and automatic resume are limited to that test account, so no customer job was affected.
- What happened
- Kilawatt Cloud completed the build of automatic checkpoint restart.
- Root cause
- After a host is lost and a replacement machine is ready, the replacement fetches the job's latest saved checkpoint, restores it, starts the job and confirms the restart on its own, with no manual step.
- What was fixed
- Billing stays paused during the gap.
- Current status
- Switched on for Kilawatt Cloud's own test jobs since Nov 8 and exercised in live drills. See the Nov 9 entry for measured results.
- What happened
- Kilawatt Cloud completed safeguards for replacement and standby machines:
- Root cause
- limits on standby spending per job and per day, a permanent record written before any machine is requested, an append-only standby log, and an automatic cleanup sweep that alerts if a machine cannot be confirmed removed.
- What was fixed
- These safeguards are switched off pending live verification.
- Current status
- Switched on since Nov 8 and exercised in the same live drills. See the Nov 9 entry.
- What happened
- A payment that settled when no supplier had stock kept the money as account credit, but the reply did not say so.
- Root cause
- Ours. The reply for a job that could not be placed was shared with API key launches and did not describe what happened to an x402 payment.
- What was fixed
- The reply now says the payment is held as credit on the paying account and is not lost, with the transaction hash and the credited amount for that payment.
- Current status
- Verified by tests. A live check is pending until this change is published.
- What happened
- Three rare failure cases after settlement could leave a settled payment without a credit and without an alert.
- Root cause
- Ours. These cases were logged on the server only, and nobody was told about them.
- What was fixed
- Each case now raises an owner alert and marks the payment so it is listed for review. Money is never moved automatically.
- Current status
- Verified by tests. No live case has been seen since the change.
- What happened
- Before today an agent paying over x402 was not held to per-payer limits we had recorded as gaps.
- Root cause
- Ours. Per-wallet spend, rate, concurrency and duration limits had not been built; only account-level and platform-wide safeguards applied to these payments.
- What was fixed
- Every paying wallet now gets default limits of $20 per rolling 24 hours, 6 requests per minute, 2 concurrent jobs and 3,600 seconds per job, adjustable per wallet. A request that would break a limit is refused before payment settlement, so no money moves on a refused request.
- Current status
- Verified: a real paid job (17 cents) went through the live path on Oct 29 and the applied policy was recorded.
- What happened
- Some payment header checks relied on fixed values or failed open, and a replayed payment lookup was not scoped to the paying wallet.
- Root cause
- Ours. The checks had been built for the paying path, not for malformed or replayed input.
- What was fixed
- Malformed headers, a wrong signer and replayed payments are rejected before settlement. The limits behind these checks are now configurable and fail closed, and a replay lookup returns only a job that belongs to the same payer.
- Current status
- Verified by automated tests (651 at publish time of the build) and the live paid run on Oct 29.
- What happened
- Pay and decline decisions on x402 requests were not recorded with the policy that applied.
- Root cause
- Ours. Requests were logged, but not the basis for each decision.
- What was fixed
- Every pay and decline decision is now recorded with the policy that applied and the limit that was hit, if any. Job commands, environment values and job output are never included.
- Current status
- Verified: first live row recorded Oct 29.
- What happened
- Internal payment tables carried unused browser access permissions.
- Root cause
- Not a failure. Row security was already on with no browser policies, so no data was exposed.
- What was fixed
- The unused browser permissions were removed, and a test now guards new tables against the same mistake.
- Current status
- No customer data was accessed or exposed.
- What happened
- Our own review of incident response and detection on Oct 27 found that alerts only went to server logs nobody reads, that an admin could not revoke any API key, suspend an account or stop all jobs for one account in a single action, that key creation and revocation left no history, and that nothing checked for a key used from a new address. The review was internal. Nothing was exploited and no customer was affected.
- Root cause
- Ours. Detection and admin controls had been built per machine and per request, not for an owner who needs to be told quickly and needs a fast way to stop something.
- What was fixed
- Published Oct 27. First, owner email alerts for supplier balance check failures, orphan machines, reconciler findings, failover drill results, admin actions, API key creation and revocation, supplier auto pauses, spend cap hits, launch guard trips, payment settlement failures and missed scheduled runs. One email per alert type per 15 minutes, no secrets or account identifiers in the body, and every alert recorded with its delivery result. Second, key history: a record of key created, revoked, first use and use from a new address (addresses stored only as a keyed hash), plus hourly shadow checks for a signup spike and for one key spending far above its normal rate. These alert only and never block a request. Third, admin controls: revoke any API key, suspend or unsuspend an account or payer, stop all running jobs for one account, key or payer in a single audited action, and an address block list.
- Current status
- Checked live on Oct 27: the sender domain is verified and a test alert was delivered to the owner inbox. Not yet checked on real events: the key history records, the hourly checks and the admin buttons. Account suspension and the address block list are built but switched off until we test them and approve them. A written incident response plan, a customer notice template, an access review and a backup restore test are not done yet. The switched off weekly failover drill no longer sends an alert.
- What happened
- Our scheduled live run on Oct 27 used real paid jobs, one supplier at a time. Two suppliers passed (ready at about 176 and 183 seconds and ran their full booking). One supplier's machine never became ready on two tries (a 2-minute and a 10-minute booking), a second stayed queued for the whole 5-minute booking, and at a third the launch failed at payment time because no machine was left. A retry at that third supplier was blocked by our own limit of three never-ready launches in 24 hours, tripped by the other two suppliers, so its capacity was not re-tested.
- Root cause
- Not confirmed. Two suppliers accepted work and never produced a usable machine, in the same run. The platform reports ready only after a real health signal, so none of these was counted as running.
- What was fixed
- None yet. Every affected launch was refunded or credited automatically. No temporary keys were created and no machine from the run is marked live in our records. We did not re-check each supplier's own console after the run, so deletion rests on the automatic 15-minute orphan check.
- Current status
- Under watch. The readiness polling for the two stalled suppliers will be reviewed. The third supplier will be re-tested after the 24-hour limit clears. This was our own test run, so no customer was affected.
- What happened
- Two gaps in the x402 endpoint. First, when a payment authorization was replayed, the reply named the job by guessing from the account and the time, so two overlapping payments from one account could have returned each other's job. Second, when payment settlement failed, the reply carried no machine-readable failure receipt in the response header.
- Root cause
- Ours. The job was never stored against its payment, and the failure path never built the header.
- What was fixed
- The exact job is now stored on its payment record and a replay returns only that job (or none if it is not linked). A failed settlement now carries a failure receipt header with a fixed reason. Published Oct 27.
- Current status
- Checked on live data on Oct 27: of six payments in that day's verification run, five were linked to their job and the sixth was a launch refused at payment time, which correctly has no job. A live replay and a live settlement failure have not been tested yet. Payments made before Oct 27 have no stored job link, so a replay of one of those returns no job.
- What happened
- We added, but did not turn on: signed job receipts (needs a signing key we have not set), one-time recovery of a lost API key by a signed proof of ownership, a stock check before quoting at the supplier that lacked one, a measured reliability score that only records what it would have chosen, a weekly automatic failover drill, a hard limit that stops a machine at its spend cap, and job contracts available to the owner test account only.
- Root cause
- Not a failure. These answer gaps listed on this page (lost API key, price quoted before stock is checked).
- What was fixed
- Each ships off by default and goes live only after a live paid test and our approval. The score never changes routing.
- Current status
- Not live. The earlier entries on the key recovery and stock gaps stay open until the checks confirm them.
- What happened
- Our review of the public x402 endpoint against the protocol found three gaps: browsers could not send or read the idempotency headers across origins, the wrong-method (405) reply was missing the cross-origin headers, and two of the version 1 payment error replies did not carry the payment-required header.
- Root cause
- Ours. The endpoint had been tested for the paying path, not for how browser and protocol clients read these replies.
- What was fixed
- The headers were added. Reply bodies and status codes are unchanged. Published Oct 26.
- Current status
- Checked live on Oct 27: the OPTIONS preflight answers 204 with the idempotency headers listed, a wrong-method request returns a JSON 405, and an unpaid request returns the standard 402. The payment-required header on the two version 1 error replies has not been checked live because it needs a paid request.
- What happened
- Requests for a price without paying share one platform-wide allowance. A small number of addresses could have used it all and blocked others from getting a price quote.
- Root cause
- Ours. The per-address limit was larger than a fair share of the platform-wide limit.
- What was fixed
- Each address now gets a smaller share of the platform-wide allowance, and the unpaid price reply is cached for 30 seconds per server for identical requests. Paid requests, forced placements and test accounts never use the cache, so a payment always gets a fresh quote.
- Current status
- Published Oct 26. Checked with free requests only: an unpaid request still returns the standard 402. We have not load-tested it.
- What happened
- The Capacity Index could be read as a page and as JSON, but not downloaded as a table.
- Root cause
- Not an incident.
- What was fixed
- A CSV download was added with the same figures and the same definitions. Supplier labels are fixed anonymous labels.
- Current status
- Checked live on Oct 27: the download returns 200 with a header row and the definition in the response headers.
- What happened
- Between Oct 20 and Oct 21 UTC we destroyed eight running machines at one supplier, by hand, during real jobs bought through an agent client, to see what the platform does. Each job was detected as lost and a replacement was claimed at a second supplier. Six other paid jobs in the same batch were left out of the count (four never booted on one supplier host, one booking window was too short, one was not a drill).
- Root cause
- Not a failure. This is a verification run. It also found problems of our own, listed under current.
- What was fixed
- Detection took between about 2 and 4 minutes after each machine was destroyed (122 to 243 seconds; the destroy time was recorded to the minute, so this is a range). The platform claimed the replacement within one second of detection in every drill (0.5 to 0.7 seconds). Replacement-ready times were not recorded, so readiness of each replacement is confirmed only by later health checks. These jobs had no checkpoints, so the drills measure detection and replacement, not resuming work. A supplier host that never booted four times was blocked in code the same day.
- Current status
- Verified from our database. Open items found by the drills: a launch limit was reported to the buyer as "rate limiting" instead of naming the real limit; a concurrency slot stayed held until the paid window ended even after the machine was destroyed; two never-ready refunds posted later than they should have; and the time a replacement becomes ready is not yet recorded.
- What happened
- We added a check that records when a supplier's recent jobs fail to start at an unusually high rate. For now it only records and does not change where jobs are sent.
- Root cause
- Not a failure. This is a new safeguard.
- What was fixed
- Observation mode only, with a minimum sample size and protection against ever removing the last available supplier.
- Current status
- Running in observation mode. Real pausing is off.
- What happened
- We added an internal report that shows which individual GPU sources completed jobs and which did not, so we can measure real coverage.
- Root cause
- Not a failure. This is a new internal tool.
- What was fixed
- Built on existing source tracking and visible to staff only.
- Current status
- Available to staff. Results are internal and not published.
- What happened
- Public pages did not show launch and readiness counts for each supplier.
- Root cause
- Not an incident. This is an addition.
- What was fixed
- The Capacity Index now shows, for each of five suppliers, jobs launched, became ready, never became ready, the never-ready share and the median time to first ready, with sample sizes and the date range. Counts start on Sep 21, 2026. The figures combine our own test runs with other traffic and are not a customer success rate.
- Current status
- Resolved. The figures update from stored job records.
- What happened
- Reliability Report 1 and the new scorecard count jobs differently, so their totals differ.
- Root cause
- Different definitions, not a data error. Reliability Report 1 counted every order placed in its window, including 80 orders where no machine started. The scorecard counts only jobs where the supplier returned a machine, and starts on Sep 21, 2026.
- What was fixed
- The scorecard page now states its definition and start date, and notes that Reliability Report 1 also counted orders where no machine started.
- Current status
- Resolved. The report's figures are unchanged and reproduce under its own definition.
- What happened
- Our read-only capacity census covered five GPU types.
- Root cause
- Not an incident. This is an addition.
- What was fixed
- It now also covers lower-cost GPU types where a supplier lists them, and records 'not listed' separately from zero. Results are internal. Public figures are unchanged.
- Current status
- Resolved.
- What happened
- The Oct 16 verification run could not start because the stored owner test key was an address-shaped value, not a usable key. Three scheduled runs were blocked by it (Oct 16, Oct 19 and the morning of Oct 20).
- Root cause
- Ours. The wrong value was stored as the test key and was mixed up again twice while being replaced.
- What was fixed
- A new owner test key was generated and stored in the secrets form, and the full five-supplier run was repeated on Oct 20.
- Current status
- Four of five suppliers passed with real paid jobs and reached ready in about 16, 112, 175 and 189 seconds. At one supplier the job stayed loading for the whole 10-minute booking and never became ready; it was refunded automatically. Supplier-side checks found no machine left running. No customer was affected: these were our own test runs.
- What happened
- An audit of our stored job records found that 45 of 109 jobs at one supplier were counted as never ready. 40 of them came from one batch of 5-minute test bookings on Oct 8. None of the 45 had a supplier error or an in-machine agent report recorded.
- Root cause
- Most likely ours. Readiness at this supplier depended on the first report from our in-machine agent, which took about 100 to 375 seconds on machines that became ready. Short bookings could end before that report arrived. The platform also did not read this supplier's own machine status, so we cannot tell whether those machines had booted.
- What was fixed
- This supplier's machine status is now read and stored as evidence. A running status means waiting for the agent, not ready, and a machine with no agent report after 15 minutes is treated as stuck in boot. Owner test jobs pinned to this supplier now have a 10-minute minimum booking. The machine type recorded at launch is kept, and the time from start to the first agent report is now recorded for every machine.
- Current status
- Under watch. This is the most likely cause, not a confirmed one. Since the changes, verification launches at this supplier have become ready: about 175 seconds on Oct 20 and 183 seconds on Oct 27, both on 10-minute bookings, confirmed by in-machine health reports. That is two launches, too few to call the problem closed. A third launch on Oct 23 was cut off before it could be followed to the end. A job counts as ready only when a real agent report arrives.
- What happened
- A review found that the Privacy Policy did not describe data the platform stores for pay-per-job (x402) requests.
- Root cause
- Ours. The policy had not been updated to cover pay-per-job (x402) data.
- What was fixed
- The policy now describes the paying wallet address, network, transaction hash and signed payment authorization, sanctions screening of paying addresses, settlement on a public blockchain, 90-day retention of API request logs, and that IP addresses are stored masked or as a one-way hash.
- Current status
- Resolved. The updated policy is effective October 16, 2026.
- What happened
- Four published report summaries had unclear phrasing, for example about who paid for test runs.
- Root cause
- Ours. The wording was unclear.
- What was fixed
- The wording of the four summaries was changed. The evidence, figures and reports themselves are unchanged.
- Current status
- Resolved.
- What happened
- Public pages still said four suppliers after a fifth supplier was connected.
- Root cause
- Ours. The page text was not updated when the fifth supplier was added.
- What was fixed
- Public pages now say five suppliers. The About page adds the figure "70+ capacity sources seen available", taken from the read-only listing check on Oct 15, 2026. Dated reports keep their original supplier counts.
- Current status
- Resolved.
- What happened
- Our job records stored the supplier and a region of 'auto', but not which host, data center or region ran a job. After the Oct 14 test batch we could not say how many distinct machines it had used.
- Root cause
- Ours. The platform did not capture location details while machines were running, and supplier systems return no details for machines after they are deleted, so the information could not be recovered afterwards.
- What was fixed
- From Oct 15 each job records the host, data center or region its supplier reports while the machine is running. Only values the supplier returns are stored and nothing is inferred. A read-only listing check also counts the distinct hosts, data centers and regions each supplier lists as available, every six hours. It never rents or changes a machine.
- Current status
- Resolved for jobs from Oct 15 onward. Details for jobs before Oct 15 cannot be recovered. A count of distinct sources that have run real jobs will be published after the first verified batch.
- What happened
- During our own test batch on Oct 14, ten machines at one supplier were still running after their jobs had ended. The supplier's account has a maximum number of machines, so eight later launches were refused for reaching that limit and three more were refused for no capacity. No outside customer was routed to this supplier, which was limited to internal test traffic.
- Root cause
- Ours, not the supplier's. Our stop step sent the delete request but did not check that the machine had actually disappeared from the supplier's list, so some machines stayed running and counted against the limit.
- What was fixed
- The delete path now reads the supplier's machine list after every delete. If the machine is still listed, an alert is raised, the job is marked as a failed stop instead of stopped, and the stop is retried. The result is stored on the job record. The fix was published on Oct 15.
- Current status
- Resolved for the deletion defect. On Oct 15 we created one real machine at that supplier, deleted it with the production stop path, and the supplier's own list showed zero machines afterwards. The supplier remains limited to internal test traffic until more batches have run.
- What happened
- During our own test batch on Oct 14, two jobs started on one supplier and were moved to another after the first machine was judged unhealthy. The move replaced the first machine's identifier in our job record with the new one, so the first machine no longer appeared in the record, although the supplier's billing listed it.
- Root cause
- Ours. Failover overwrote the machine identifier instead of keeping the earlier one, and it ignored the result of the shutdown request for the first machine, so we could not confirm from our side that it had been deleted.
- What was fixed
- Failover now keeps every earlier machine on the job record. A shutdown that is not confirmed is logged as an alert and as an attempt record. A rent request that is confirmed without a machine identifier no longer triggers a second rent. A scheduled check now lists the supplier's live machines and flags any that have no record in our database; it flags and does not delete.
- Current status
- Resolved for new jobs. The supplier's billing page showed no running charges afterwards. On Oct 25 the operator checked the supplier consoles and found no machines running, and the scheduled supplier-machine check, running since Oct 25, has flagged no orphaned machine. The shutdown result for these two machines was not kept at the time, so deletion is confirmed by these later checks, not by a record from Oct 14.
- What happened
- On Oct 10, the first outside tester ran the public x402 endpoint with real signed payments and reported nine findings. Two could have caused a buyer to overpay or pay twice: a request body that was not valid JSON was quoted as a default, most expensive job, and a replayed payment authorization returned an unclear error instead of a clear already-used response. The others were integration issues: unclear errors for an underfunded wallet, a missing machine-readable price-changed code, unknown GPU names returning a capacity error instead of a validation error, no machine-readable response to the wrong HTTP method or to discovery requests, missing timing fields on job receipts, and an internal payment-processor id returned to buyers.
- Root cause
- The endpoint had been verified for the paying path but not for malformed input, replayed authorizations, error clarity and discovery. The default job values were applied to any body that failed to parse, which is how a broken client could sign for a job it did not mean to buy.
- What was fixed
- Eight changes are in the published code: a body that is not valid JSON is rejected with a clear 400 and no quote; a wrong content type is rejected with a 415; a replayed payment authorization returns a clear 409 with the original transaction and job where known; the price-changed response carries a machine-readable code and the new amount; an underfunded wallet or invalid signature returns a stable error code with a hint; a mistyped GPU name returns a 400 with the list of valid names; the wrong HTTP method returns a JSON 405 and a discovery file is served at /.well-known/x402; job receipts include ready, ended and runtime fields; and the payment-processor id is no longer returned to buyers. The real cost field is still returned, rounded into buckets.
- Current status
- Resolved for customers. The fixes are in the published code. On Oct 25 we re-tested the free checks live against the production endpoint: invalid JSON returned a 400 with no quote, a wrong content type returned a 415, an unknown GPU name returned a 400 with the list of valid names, the wrong HTTP method returned a JSON 405, the discovery file was served at /.well-known/x402, and an unpaid request returned a standard 402 with no payment-processor id. The paid checks (replayed authorization, price-changed code, underfunded wallet, receipt fields, refund and overpay credit) still need a live paid re-test and will be added when done. A code review on Oct 25 also found that the one-time API key cannot be recovered if the response that carries it is lost; a free recovery by signed proof of ownership was built on Oct 26 and is switched off until a live paid test passes. A 60 second test job showed an end time about two minutes after creation, so job receipts may show a runtime longer than the paid window; this is being investigated. The replayed-job and failed-settlement gaps found in our follow-up review were fixed on Oct 27 (see the entry of that date).
- What happened
- On Oct 2, during our own live verification runs, two launches did not deliver. A paid request for an 8-card machine at one supplier was refused because no machine of that size was available, and one 8-card machine at another supplier never passed a health check. The other jobs in the same round completed normally.
- Root cause
- The refused request has the same cause as the Sep 30 capacity entry: the price quote is produced before the platform confirms that a machine of that full size is in stock. The second machine did not reach a usable state at the supplier, and the platform only reports a machine as ready after a genuine health signal.
- What was fixed
- Built on Oct 26 and switched off while we measure it (it covers the one supplier that lacked a stock check): check stock for the full card count in one machine before giving a quote. No fix is needed for the second case; the automatic never-ready refund worked as designed.
- Current status
- Customer impact resolved. The refused request failed within about three seconds of payment and nothing was charged for compute. The never-ready machine stopped at the end of its paid window and was refunded in full automatically. In both cases the amount was returned as account credit on the paying wallet, not paid back on-chain. No machine was left running and no funds were lost. A stock check before quoting was added on Oct 26 and is switched off while we measure it in observation mode. This entry stays open until that data confirms the fix.
- What happened
- On Sep 30, during our own live verification runs, two paid requests for single-card machines at one supplier were refused because the supplier had no machine of that type available. No machine was placed in either case.
- Root cause
- The price quote is produced before the platform confirms that a matching machine is in stock, so a request can be paid for capacity the supplier does not actually have.
- What was fixed
- Built on Oct 26 and switched off while we measure it (it covers the one supplier that lacked a stock check): check stock for the full card count in one machine before giving a quote. The automatic refund worked as designed.
- Current status
- Customer impact resolved. Both requests failed within about three seconds of payment, nothing was charged for compute, and the full amount was returned as account credit. No machine was left running and no funds were lost. A stock check before quoting was added on Oct 26 and is switched off while we measure it in observation mode. This entry stays open until that data confirms the fix.
- What happened
- On Sep 30, during our own live verification runs, one launch at one supplier never passed a health check and was never reported as running. The supplier reported the machine as exited before it ever became healthy.
- Root cause
- The supplier's machine did not reach a usable state. The platform only reports a machine as ready after a genuine health signal, so it was never billed as running.
- What was fixed
- None needed. The automatic never-ready refund worked as designed.
- Current status
- Customer impact resolved. The machine was stopped and refunded in full automatically. No machine was left running and no funds were lost. Boot reliability at this supplier remains under watch.
- What happened
- On Sep 29, during our own verification runs, four launches at one supplier never passed a health check and were never reported as running. For one of these four machines, a health incident was also recorded that evening because the machine stayed unreachable.
- Root cause
- The supplier's machines did not reach a usable state. The platform only reports a machine as ready after a genuine health signal, so none of these was ever billed as running. The cause at the supplier is still being watched; this supplier has had boot reliability problems before.
- What was fixed
- None needed for customer impact. The automatic never-ready refund returned the charge for three launches, and for the fourth the automatic downtime credit from its health incident (3 cents) was posted instead of a refund; that launch was charged 2 cents, so it received 1 cent more than it was charged. Routing and failover protections remain in place.
- Current status
- Customer impact resolved. Every affected launch had its charge returned automatically and no machine was left running. Confirmed from our own job, refund and incident records. Reliability at this supplier remains under watch.
- What happened
- Failover had been verified against test logic but never against a genuine, unplanned supplier failure. Separately, per-supplier provisioning had never been checked one supplier at a time, in isolation, to rule out one supplier's failure masking another's.
- Root cause
- No incident had yet occurred, under the tightened two-signal failover rule, in which a real (not simulated) machine died mid-job. And prior multi-supplier tests ran suppliers concurrently, which could hide a single supplier's problem behind a successful failover to a different one.
- What was fixed
- None needed. This was a verification exercise, not a bug fix. A real production job (f3c8c3c9) was launched on Kilawatt infrastructure; the underlying machine failed on its own. The platform's two-signal rule confirmed the outage at 23:22:01 UTC, dispatched a capped-loss replacement to Kilawatt infrastructure immediately, and the job completed successfully rather than failing. Separately, each supplier tested (Kilawatt infrastructure, Kilawatt infrastructure, Kilawatt infrastructure, Kilawatt infrastructure) was dispatched and observed in isolation, one at a time, with no other supplier running concurrently.
- Current status
- Verified live and documented in a standalone report ("MCP Agent Layer — Live Multi-Supplier Dispatch & Capped-Loss Failover Verification", published Sep 27, 2026): Kilawatt infrastructure, Kilawatt infrastructure and Kilawatt infrastructureeach reached a healthy state in 7-8 minutes when tested alone; Kilawatt infrastructure did not reach healthy within the test window when isolated, and its capped-loss failover during a live outage is the proof case above. This closes the previously open "live natural-failure confirmation" item on the failover fix and gives the first isolated (non-concurrent) per-supplier comparison.
- What happened
- A customer's launch response included their machine's health-report token. With it, a customer could report GPU faults that never happened, and the system would mark the machine as down and pay automatic downtime (SLA) credits.
- Root cause
- The health-report endpoint trusted the token alone, and the token was given to the customer instead of only to the machine. Credits for faults seen only inside the machine didn't need any confirmation from the compute supplier.
- What was fixed
- The token is no longer returned to customers and uses its own dedicated secret. Stopped machines can't send reports. Faults reported only from inside a machine now earn credit only when the supplier's own monitoring also saw the machine unhealthy at that time, and the report endpoint is rate-limited.
- Current status
- Resolved and verified live on production on Sep 27, 2026. A fake fault report sent with a real machine token was accepted but paid no credit, while a fault confirmed by the supplier still paid correctly. We found no sign it was used by any customer; the only credits ever paid through it were from our own tests, and these were reversed.
- What happened
- After the never-ready full refund was introduced, an account could repeatedly launch jobs that never became usable and be refunded each time, while the platform still paid its compute suppliers for each provisioning attempt.
- Root cause
- Launch checks covered balance, spend caps, request rate and concurrency, but nothing tracked how many of an account's recent launches ended without ever becoming ready, and there was no platform-wide cost breaker.
- What was fixed
- Each account (including pay-per-request wallets) is now limited to 3 never-ready launches per rolling 24 hours, after which new launches are refused before any funds are held. New launches also pause platform-wide if supplier cost on never-ready machines exceeds a set hourly threshold.
- Current status
- Resolved and verified live on production. The test launches all became ready, so the account was given test never-ready history; a real launch from that account was then refused before any funds were held, while a second account was still allowed.
- What happened
- When a job paid via on-chain pay-per-request (x402) never became ready, the automatic full refund was credited to an internal account balance tied to the paying wallet, with no way to withdraw it back on-chain.
- Root cause
- x402 revenue settles to fiat, so there was no company-held USDC wallet or payout path for the internal balance a refund lands in.
- What was fixed
- Internal credit on an x402 wallet, including never-ready refunds, can now be returned as real USDC on Base to the wallet that paid. Payouts are approved by an operator, not automatic: they are sanctions-screened, capped at $50 each, sent from a company-held wallet, and logged.
- Current status
- The first live payout (2.81 USDC) is on-chain in Base transaction 0x900dc485…c937. Self-service withdrawals are not offered yet.
- What happened
- A job that received the new never-ready full refund could also separately receive a downtime/SLA credit for the same period, paying the customer twice for one failure.
- Root cause
- The two credit systems did not check each other's state: the downtime-credit path had no knowledge of whether the never-ready refund had already been paid for the same job.
- What was fixed
- The downtime-credit system now skips crediting any job that already received the never-ready refund.
- Current status
- Resolved. Caught and fixed during live testing of the never-ready refund the same day, and re-verified live: the refund posts exactly once and no second credit follows.
- What happened
- If a job ended — timed out, failed, or was stopped — without ever reaching a confirmed-healthy state, the customer was still charged in full with no refund.
- Root cause
- No automated check existed linking a job's outcome to refund eligibility; ending a job simply kept whatever had been charged.
- What was fixed
- An automatic full refund now triggers whenever a job ends without ever confirming readiness, across every end-of-life path (completion, stop, failure, reaper).
- Current status
- Resolved and verified live on production with a real refund posted to a real test account's wallet.
- What happened
- A single failed status check from a provider — for example a transient rate-limit or server error — could have triggered failover of a genuinely healthy running job.
- Root cause
- The failover threshold did not require sustained or repeated negative evidence; one bad response was enough.
- What was fixed
- Failover now requires two consecutive negative signals with a minimum time gap, or a sustained failure window. A single blip no longer triggers it.
- Current status
- Resolved and now confirmed under a real failure: on Sep 27 a live Kilawatt infrastructure machine died unexpectedly mid-job (job f3c8c3c9); the two-signal rule caught it, failover replaced the job on Kilawatt infrastructure, and the job completed successfully rather than failing. See the capped-loss failover verification entry below for the full timeline.
- What happened
- Jobs were reported as running and billed the moment a GPU provider accepted the request, before any health check confirmed the machine was actually usable.
- Root cause
- No readiness gate existed between provider acceptance and marking a job live; provider acceptance was treated as proof of a working machine.
- What was fixed
- Jobs now start in a provisioning state and only transition to healthy/running once a genuine readiness signal is confirmed — a provider status check or an in-machine health report.
- Current status
- Resolved and verified live on production.
- What happened
- Signed-in customers had direct database write access to their own machine records — including billing rate and status fields — instead of those changes being restricted to server-side logic only.
- Root cause
- The table's access rules granted broad write permissions to any signed-in user for their own rows. An earlier change had limited what customers could read on those records but never touched the write permissions.
- What was fixed
- Customer write access to machine records was removed entirely; only server-side logic can create or change them now. The one console flow that previously wrote with the customer's own access was moved server-side, with ownership pinned to the signed-in customer.
- Current status
- Resolved and verified live against production with a real signed-in customer account: attempts to change the billing rate, change the status, delete a record, and insert a new one were all refused, and the customer's normal machine list still loads.
- What happened
- The public API masks which underlying GPU supplier handles a job, but signed-in customers could bypass that masking by reading supplier-identifying fields — supplier name, supplier machine ID, and raw supplier error text — directly from the database for their own jobs and machines.
- Root cause
- Masking was applied in the API layer, but the underlying database rows were also directly readable by the owning customer, and those rows carried the unmasked fields.
- What was fixed
- Column-level access restrictions now block customers from reading supplier-identifying fields on their own records, while normal customer-visible fields (status, GPU, cost, credits) remain available. Admin and server-side access is unchanged.
- Current status
- Resolved and independently re-verified live against production with a real signed-in customer account: every attempt to read the restricted fields was refused, and the API's masked responses were re-confirmed to contain no supplier names or supplier machine IDs.
- What happened
- After the sanctions refresh was moved to an hourly background job, a silently stalled refresh could keep serving the cached list for up to seven days and still allow a payment through. Cold start was safe (no list blocks), but a list that had gone stale between refreshes was trusted for far too long.
- Root cause
- MAX_STALENESS_MS was set to 7 days. That ceiling was meant as a backstop, but because the hourly job could fail repeatedly without surfacing, a week-old list could still read as usable and let a payment pass. The payment path fails closed on a missing or unusable list, but the definition of 'unusable' was too permissive.
- What was fixed
- Found by a technically precise public comment on the x402 postmortem that asked specifically whether a cold start or a stalled refresh fails open or closed. The ceiling was tightened from 7 days to 48 hours — a healthy hourly cache is never more than an hour old, so 48 hours tolerates a full day of missed refreshes and then blocks. A real OFAC-listed address was re-confirmed still blocked, and a clean address still passes.
- Current status
- Resolved and verified live. Before the fix, backdating the cache to 3 days old returned 'allowed'; after the fix the same stale cache returns 'blocked'. The sanctioned address stays blocked and the clean address still passes.
- What happened
- Three documentation defects found in one review. The docs told developers to send requests to api.kilawattcloud.dev, a host that never existed and does not resolve. The GPU 'count' parameter was documented as accepting 1–512. And the homepage named the A100 as available while the rate card carried no A100 price at all.
- Root cause
- The public documentation and rate card were written ahead of the API and were never reconciled against what the gateway actually enforces. No customer request was affected by a system fault — the failure was that anyone following the docs literally would have hit an unresolvable host or a rejected request.
- What was fixed
- The docs, API explorer, quick-start and console snippets now point at the real host, www.kilawattcloud.dev/api/public/v1/, with the four endpoints that actually exist. The documented GPU count was corrected to the limits the gateway really enforces: 1–8 for H100, H200, A100 and L40S, 1–4 for B200, and up to 16 for workstation-class cards. The A100 was confirmed genuinely quotable across the providers checked by a live quote and added to the rate card as a live-quoted card with a conservative floor price, since real cost ranges from about $0.54 to $15.92 per GPU-hour depending on provider.
- Current status
- Resolved, verified live and published the same day. A real API call to the corrected endpoint returned a priced, pre-authorized response; a request for 512 GPUs was rejected by the live gateway, as were 64 and 9, while 8 succeeded; and a live A100 quote returned real cost from the providers checked.
- What happened
- Until this change, a GPU that degraded or died mid-job was only caught by the customer. Hardware was verified at creation time and never again for the rest of the run.
- Root cause
- No continuous in-flight health checking existed, and any credit for lost time would have required the customer to notice and ask for it.
- What was fixed
- Every running instance is now probed on a short interval. Degraded and down windows are recorded with exact start and end times, and the proportional cost is credited back to the wallet automatically on recovery — no claim, no support ticket. The credit and the exact window appear on the job receipt.
- Current status
- Resolved (with caveat) — SLA-linked health monitoring and automatic credits. The in-machine health reporting is proven live end-to-end. On Sep 27 a simulated fault run went through the real system on a production machine that had been confirmed ready by its own health reports: a GPU fault report (Xid + ECC) was sent from outside the machine to the real reporting endpoint, using the machine's own credentials. The fault was simulated, and no real hardware fault occurred. The real system opened incidents, marked the machine down, logged the report, closed the incidents on the next clean report, and posted automatic SLA credits. The test found a bug where a milder incident could hide a more serious one; it was fixed the same day. It also found that overlapping incidents could each post their own credit; that was fixed the same day too. Not yet proven: catching a real hardware fault as it happens, which can't be forced without damaging rented equipment.
- What happened
- Jobs submitted for an A40 or A10 were provisioned on an RTX 4090 instead, without the request failing and without the substitution being disclosed.
- Root cause
- Ours, not the provider's. Our own GPU mapping carried a fallback allow-list that included the RTX 4090 and was missing proper A40 and A10 entries, so an unmatched request quietly fell through to whatever the fallback listed.
- What was fixed
- The fallback was removed. Each GPU now maps only to true equivalents, an unavailable card is refused outright instead of substituted, and the hardware actually provisioned is verified against the request before the charge is taken. A mismatch is torn down and failed over rather than billed. Receipts state the GPU that was actually provisioned.
- Current status
- Resolved and verified with real paid test jobs: an A40 request provisioned as an A40, and an A10 request that could not be honoured was refused with the funds returned to wallet credit instead of being silently swapped.
- What happened
- Under simultaneous load, six concurrent execution requests could settle six payments while only three jobs were actually created — money taken with no compute behind it.
- Root cause
- The balance check and the reservation were separate steps, so concurrent requests each read the same balance before any of them wrote. Settlement also happened before the limit and profitability checks that could still reject the job.
- What was fixed
- Reservations are now atomic in the database: check and hold happen in one operation, so concurrent requests cannot both win the same funds. Rate, concurrency, spend and margin checks all run before any payment is settled, and a reservation that does not become a job is released back to the wallet.
- Current status
- Resolved and verified with a live six-request burst: exactly three jobs were created, three requests were rejected, and the rejected ones left the balance untouched.
- What happened
- On the x402 execution route, a payment could be settled before the rate-limit, concurrency and spend-cap checks ran at all — so a request that was going to be rejected on limits could still take the money first and refund it after.
- Root cause
- The route settled the payment early in the request flow and only afterwards evaluated whether the job was allowed to run under the caller's limits.
- What was fixed
- The full pre-authorization dry-run (balance, rate limit, concurrency, spend cap) was moved ahead of payment settlement, so nothing is charged for a request the checks would reject. Verified against a real settled transaction.
- Current status
- Resolved. Later the same day the reservation itself was also made atomic under concurrency — see the separate entry above; that was the deeper fix for the same class of problem, found and fixed at a different time.
- What happened
- An actual registered agent wallet ran 18 real completed jobs through the public x402 endpoint, settling real USDC across 22 payments, with zero jobs dropped. During the run, Kilawatt infrastructure failed 100% of the attempts made to it that day (11 of 11).
- Root cause
- Nothing to fix on our side — this was a live, unplanned provider outage, not a simulation. It was the first real-world test of the failover system under load.
- What was fixed
- No code change was needed. The routing layer automatically failed every failed Kilawatt infrastructure attempt over to Kilawatt infrastructure — a provider that had never previously appeared as a live failover destination — and to Kilawatt infrastructure, and every job still completed and settled correctly.
- Current status
- Resolved and proven. Our records for that evening show the job burst and the Kilawatt infrastructure-to-Kilawatt infrastructure/Kilawatt infrastructure failovers with real settled payments; none of the 18 jobs was lost.
- What happened
- A launch-safety review found seven real gaps at once: launch paths only checked the first hour of cost instead of the full job, there were no per-key rate or concurrency limits, customers had no spend cap of their own, running instances were never metered or reaped, Kilawatt infrastructure provisioning was hitting the wrong endpoint and returning 404s, provider failover did not actually fall through from Kilawatt infrastructure to Kilawatt infrastructure on failure, and gateway-launched instances were not recorded or auto-terminated at their paid-through time.
- Root cause
- The platform had been built feature-first: jobs launched and providers were called, but the safety rails around money and lifecycle had never been systematically checked.
- What was fixed
- All seven were fixed in one pass: full-cost pre-authorization on every launch path, default per-key limits of 3 concurrent jobs and 30 requests/minute (locked against self-elevation), an optional customer-set monthly spend cap, a 15-minute metering and auto-reap scheduler, the corrected Kilawatt infrastructure endpoint, a real Kilawatt infrastructure-to-Kilawatt infrastructure failover path, and recording plus paid-through termination for gateway-launched instances.
- Current status
- Resolved and verified live with real instance creation and destruction on both Kilawatt infrastructure and Kilawatt infrastructure.
Includes our own live verification runs through the production API.
Component states are computed from our control-plane records at page load; they are not a synthetic external probe of every provider. Questions: hello@kilawattcloud.dev

