Pushing a Config Change to a Laptop That's Asleep

An admin opens a dashboard, flips a switch, and expects every laptop in the company to obey. Some of those laptops are open. Some are shut. Some belong to someone on leave for three weeks. One is a cloned VM image that thinks it is a machine that was decommissioned in March.

This is fleet control, and it looks trivial until you write it down. I spent a good chunk of this year on the problem. What follows is the shape of the answer, minus anything specific to where I work — the design generalises to any system that pushes state to devices it does not own.

The obvious design, and why it breaks#

The first instinct is a command queue. The admin's action becomes a message. Each device has an inbox. Devices drain their inbox when they connect.

admin flips switch
  -> enqueue { device_id, "block extension X" }
  -> device connects, drains inbox, applies command

It is easy to build and it fails in a specific, nasty way.

A laptop goes offline for three weeks. During those three weeks, the admin blocks an extension, unblocks it, blocks it again, then removes the policy entirely. Four commands land in that inbox. The laptop wakes up and replays all four. For a few seconds it blocks something that is no longer policy. If any of those commands had side effects — killing a process, quarantining a file — it just did four things nobody asked for, in an order that reflects history rather than intent.

The deeper problem: a queue stores what happened, and what you actually want to converge on is what is true now. Those are different objects. Every stale-replay bug I have seen traces back to conflating them.

Desired state, not commands#

Replace the inbox with a desired state per device, plus a revision number:

device 7f3a:
  revision: 42
  desired:
    - extension X: blocked
    - extension Y: allowed

The admin's switch does not enqueue anything. It rewrites the desired state in one transaction — delete the old rows, insert the new ones, bump the revision — and that revision becomes the single thing a device needs to compare against.

Now the three-week-old laptop wakes up, reports revision 12, gets told the current revision is 42, pulls the whole state, and applies the difference between what it has and what it should have. It never learns that the extension was blocked and unblocked twice in between, because that history is not something it needs.

This also makes the reconcile path idempotent for free. Applying desired state twice is a no-op. Applying a command twice is a bug.

Two different routing problems#

Once state lives in the database, the network layer has exactly one job: tell a device that something changed. That job splits in two, and conflating them is the second trap.

Request/response — routing by connection. A device asks for its state. Some server answers. Any server can answer, because the answer comes from the database. This is a plain HTTP request and it needs no cleverness at all.

Push — routing by identity. The backend needs to reach one specific device, which is connected to one specific gateway pod, which the backend does not know. This is the hard half.

The naive fix is sticky routing: track which pod holds which device, look it up, forward. Now you have a distributed registry to keep consistent, and every pod restart invalidates a chunk of it.

The better fix is to stop tracking. Put a message bus in the middle and let subscription do the routing:

backend                     gateway pods                 devices
   |                                                        |
   |  publish signal.<org>.<route_id>                       |
   +-------------> [ NATS ] ---> pod holding that device ---+
                        |
                        +--> other pods: no subscriber, message dropped

The gateway pod that holds a device's socket subscribes to that device's subject. The backend publishes to the subject without knowing or caring which pod is listening. If no pod is listening, the device is offline and the message is dropped — which is correct, because the device will reconcile from the database when it reconnects anyway.

The push carries no payload. It is a doorbell, not a delivery. It says "your state changed, go look." Everything authoritative comes from the database over a normal request. That one decision removes an entire category of bug, because a message that is dropped, duplicated, or delivered out of order costs you nothing but a redundant fetch.

The routing ID is a security boundary#

Here is the mistake I nearly shipped.

The subject contains a device identifier. The obvious move is to let the device tell the gateway which subject to bind to at connection time. It knows its own ID, after all.

That is not routing — that is letting the client choose its own address. A device inside the org can bind to another device's subject and receive its pushes. There is no cross-org leak, so it looks safe in a demo, and it is still wrong.

The routing ID has to be something the device cannot choose:

• It is minted by the backend, not the client.
• It is a random opaque value, distinct from the device's credential and from any ID a human sees.
• It arrives inside a signed token. The gateway verifies the signature offline and takes the routing ID from the verified claims, never from anything the client sent alongside it.

Three separate identifiers, three separate jobs: a credential proves who you are, a device ID is what humans and audit logs refer to, a route ID is where messages go. Collapsing any two of them is how you get an authorization hole that reads like a naming convention.

Clocks lie, and they lie in your favour#

Two bugs, same root cause, both found the hard way.

Expiry is computed on the wrong clock. The backend mints a token with an expiry stamped from its clock. The device checks "am I still valid?" against its own clock. If the device is thirty seconds slow, it believes a dead token is alive, gets a 401, retries — and if the retry path does not drop the cached token, it retries forever with the same dead credential. The fix is small: treat a 401 as proof the token is dead, regardless of what your clock thinks.

Timestamps come from the device. If "last seen" is taken from the device's payload, a laptop with a wrong clock falls outside your seven-day activity window and silently disappears from inventory. Nothing errors. The device is simply not there any more. Server time, or GREATEST(device_time, now()) — but never the device alone.

The pattern in both: the endpoint's clock is input, not truth.

Rolling out to devices that push back#

The last piece is the one people skip. Devices are not passive.

• A device can be cloned. Two machines from the same VM image present the same identity and fight over the same connection slot, each superseding the other in a loop. You need to detect duplicate identity and alarm on it, not just accept the newest.
• Connections need caps. Unbounded accept means one bad org, or one retry storm, exhausts file descriptors on a pod and takes down every device on it.
• The push is a hint, so keep the poll. A slow periodic reconcile catches everything the doorbell missed. If your system only converges when a push lands, you have built a delivery guarantee you cannot honour.

What I would tell myself at the start#

Three things.

Model the world, not the events. Desired state converges. Command queues replay. If you find yourself writing logic to skip stale commands, the queue was the wrong primitive.

Make the push worthless. If losing a notification costs nothing because the next reconcile fixes it, you have removed your dependency on delivery guarantees that distributed messaging cannot give you anyway.

Decide what is authoritative, once, and never take it from the client. Not the routing ID, not the timestamp, not the expiry check. Every one of those bugs looked like a small oversight and every one of them was the same oversight.

None of this is novel — it is the reconciliation model Kubernetes uses, applied to machines that are asleep half the time and that you do not control. But the failure modes only become obvious once a laptop that has been shut for three weeks opens up and does four things nobody asked for.