How Nixpacks, buildctl, and BuildKit Actually Fit Together

How Nixpacks, buildctl, and BuildKit Actually Fit Together#

I had three names in my head and no clean mental model.

Nixpacks showed up in docs as "build your app into a container." BuildKit was "the next-gen builder Docker uses under the hood." buildctl was a CLI I installed because a Rust worker needed to call something.

I kept asking the wrong question: which one builds my Next.js app?

Answer: none of them alone. They are three layers in a pipeline. Nixpacks writes the recipe. buildctl submits the job. BuildKit runs the job and pushes the image.

I learned this while wiring a self-hosted deploy platform (vercel-clone on GitHub). The concepts generalize anywhere you want "git push → container in a registry → run it." This post is that mental model — not a project changelog.

Start Here: What Is a Container Image?#

A container image is not a VM disk. It is:

1. A config — default command, env, exposed ports, user
2. An ordered stack of layers — filesystem diffs, each immutable once built
3. A manifest — a list of layer digests + pointer to the config

Something has to:

• Run each build step (RUN bun install, COPY . ., …)
• Turn each step into a layer
• Optionally push those layers to a registry (an HTTP store for manifests and blobs)

docker build does all of that inside Docker Engine (dockerd). That is fine for local work. For a deployment platform you often want:

That table is why many platforms split planning (what to build) from execution (build and push).

The Three Tools, One Sentence Each#

Chain:

git clone  →  nixpacks build  →  buildctl build  →  image in registry  →  docker run
   │              │                    │
   │         writes Dockerfile    buildkitd runs it
   │         + install/build      + push layers

If you remember one diagram:

┌─────────────┐     ┌─────────────┐     gRPC/socket     ┌─────────────┐
│  Nixpacks   │     │  buildctl   │ ──────────────────► │  buildkitd  │
│  (planner)  │     │  (client)   │ ◄── logs/status ─── │  (builder)  │
└─────────────┘     └─────────────┘                     └──────┬──────┘
       │                    │                                   │
       │ .nixpacks/         │ reads context + dockerfile        │ push
       │ Dockerfile         │                                   ▼
       └────────────────────┴──────────────────────────► registry:5000

Nixpacks: The Planner#

Nixpacks (Railway's open-source builder) answers: given this repo, what base image, install command, build command, and start command should we use?

It is the spiritual successor to Heroku buildpacks and competitors like Cloud Native Buildpacks — opinionated detection instead of you hand-writing a Dockerfile for every framework.

What it does#

nixpacks build -o . --install-cmd "bun install" --env NIXPACKS_NODE_VERSION=22 .

Typical output under your project root:

.nixpacks/
  Dockerfile       ← this is what BuildKit will execute
  (plan / metadata)

In my worker, install command is chosen from lockfiles:

• pnpm-lock.yaml → pnpm install
• yarn.lock → yarn install
• bun.lock / bun.lockb → bun install
• else → npm install

Project-specific build_command from the database is passed as --build-cmd when set.

What it does not do#

• Pull base images at scale (BuildKit does)
• Execute RUN steps in production isolation (BuildKit does)
• Push to a registry (BuildKit does)
• Know about your Postgres or NATS (it only sees the cloned repo)

Think of Nixpacks as a Dockerfile generator with good defaults for Node, Python, Rust, etc. You could replace it with a hand-written Dockerfile and the rest of the pipeline would still work.

Why use it at all?#

Hand-written Dockerfiles do not scale across "connect any GitHub repo" products. You need detection:

• Next.js vs Vite vs plain Node API
• Bun vs pnpm vs npm
• Monorepo root_dir

Nixpacks trades flexibility for speed of integration. The failure mode is subtle plans (wrong Node version, wrong install) — which is why you log the full plan to the user during deploy.

BuildKit and buildkitd: The Builder#

BuildKit is Moby's build engine — what powers docker buildx under the hood. The long-running process is buildkitd.

In compose it is often a dedicated container:

buildkit:
  image: moby/buildkit
  privileged: true
  command: ["--addr", "unix:///run/buildkit/buildkitd.sock", "--config", "/etc/buildkit/buildkitd.toml"]
  volumes:
    - /var/run/buildkit:/var/run/buildkit

What buildkitd does when a build starts#

1. Receives a solve request (frontend + local folders + output spec)
2. Parses the Dockerfile into a DAG — each RUN, COPY, FROM is a node
3. Executes steps — usually each RUN in a short-lived container (OCI worker)
4. Caches layers when inputs match a previous build (CACHED in logs)
5. Exports the result — here, push to registry, not back to the client

Config snippet that matters for local dev:

[grpc]
address = ["unix:///run/buildkit/buildkitd.sock"]
socketMode = "0666"
 
[worker.oci]
enabled = true
snapshotter = "overlayfs"
max-parallelism = 8
 
[registry."registry:5000"]
http = true
insecure = true

• Unix socket — how clients talk to the daemon on the same machine
• registry:5000 insecure — local registry speaks HTTP, not HTTPS; BuildKit must be told explicitly or push fails
• privileged — nested containers and mounts for build steps

DAG and cache (why BuildKit feels "smart")#

Each Dockerfile instruction becomes a node. Independent branches can run in parallel (max-parallelism). Cache keys include parent layer, instruction text, and file hashes for COPY.

Log lines like:

#5 [stage-0 2/6] RUN bun install
#5 DONE 12.3s

are BuildKit progress output — not Nixpacks and not your application printing to stdout.

buildctl: The Client#

buildctl is the CLI shipped with BuildKit. Same relationship as docker CLI ↔ dockerd.

The worker does not embed BuildKit as a library. It spawns a subprocess:

let mut buildctl = Command::new("buildctl");
buildctl.args([
    "build",
    "--frontend", "dockerfile.v0",
    "--local", "context=.",
    "--local", "dockerfile=.nixpacks",
    "--opt", &format!("network={build_network}"),
    "--output", &format!("type=image,name={},push=true", build_image_ref),
]);
buildctl.env("BUILDKIT_HOST", "unix:///var/run/buildkit/buildkitd.sock");

buildctl:

• Reads local directories (context, dockerfile)
• Opens a gRPC session to buildkitd via BUILDKIT_HOST
• Streams human-readable progress to stderr (forwarded to deploy logs)

It does not compile TypeScript. It does not push blobs itself when push=true — buildkitd pushes after the solve completes.

The Unix socket (not a remote server)#

Worker and buildkit containers both mount:

volumes:
  - /var/run/buildkit:/var/run/buildkit

So they share buildkitd.sock on the host. This is two containers on one Docker host, not BuildKit running on another machine.

The Full buildctl Command, Flag by Flag#

After Nixpacks, the worker runs (conceptually):

cd /tmp/builds/{deployment_id}
 
buildctl build \
  --frontend dockerfile.v0 \
  --local context=. \
  --local dockerfile=.nixpacks \
  --opt network=vercel-clone_build-net \
  --output type=image,name=registry:5000/deployment-{uuid}:latest,push=true

Why push=true matters#

Without it, the built image can be sent back over the socket to the client as a huge tarball. On a real Next.js image that is slow and memory-heavy. The worker comment in code says it plainly: avoid sending the image tarball back to the client (which hangs on large images).

The worker's job ends with a string tag in Postgres, not with bytes of the image in RAM.

Registry: Two Hostnames, One Registry#

This confused me longer than it should have.

Same registry process. Two DNS names depending on who connects:

let build_image_ref = image_tag("registry:5000", deployment_id);
// → buildctl --output name=...
 
let serve_image_ref = image_tag("localhost:5000", deployment_id);
// → stored in DB, used by docker run on the host

Ready in the database does not mean the image is on the host. BuildKit pushed to the in-network hostname. The API must docker run --pull always so the host daemon fetches layers from localhost:5000.

Same shape as any CI system: build agent pushes to registry.internal; production nodes pull from a URL they can resolve.

Build Network vs Serve Network#

Three networks in the stack:

default     → db, nats, api, minio (platform internals)
build-net   → worker, buildkit, registry (build RUN steps see this)
serve-net   → traefik, api, preview containers (user traffic)

--opt network=build-net means npm install inside the build cannot hit postgres:5432 on default. It can still reach registry:5000 for push.

Preview containers later run on serve-net with Traefik labels — unrelated to the build network opt.

What You See in Deploy Logs#

Order in a typical deploy:

1. cloning repository
2. running nixpacks plan — framework detection, generated Dockerfile preview
3. running buildctl build — thousands of lines:
   • load build definition
   • load metadata for node:22 (or whatever Nixpacks chose)
   • RUN bun install, RUN next build, …
   • exporting to image, pushing layers

Those lines flow: worker stdout/stderr → NATS → API → Postgres + SSE → dashboard.

BuildKit is verbose. That is normal. Batching log lines in the UI (tens of ms) keeps the tab from freezing on 2000+ lines.

Permissions (briefly)#

Non-root workers need read/write access to buildkitd.sock (and a sane HOME); the API still needs docker.sock to run previews — different sockets, different containers. I hit three separate permission failures wiring that up on macOS; the full stack traces and fixes are in Phase 2, not repeated here.

End-to-End Sequence#

1. Deploy created → job on NATS
2. Worker: git clone → /tmp/builds/{id}
3. Worker: nixpacks build → .nixpacks/Dockerfile
4. Worker: buildctl build
      → buildkitd pulls bases, runs RUN on build-net
      → push registry:5000/deployment-{id}:latest
5. Worker: publish Ready + image_ref localhost:5000/...
6. API: docker run --pull always --network serve-net ...
7. Traefik: Host(`{hash}-preview.localhost`) → container

Nixpacks owns step 3. buildctl owns step 4's request. buildkitd owns step 4's execution and push.

One thing I'd tell past me#

When a deploy fails at "building", open .nixpacks/Dockerfile before you touch Rust or NATS. That file is the contract between Nixpacks and BuildKit — wrong Node version, wrong install command, wrong build step. No buildctl flag fixes a bad Dockerfile. I burned hours on socket permissions and queue ordering while the generated plan was wrong under my nose.

If You Want to Go Deeper#

• Repo with the full pipeline: github.com/Avik-creator/vercel-clone
• In-repo reference doc: BUILDKIT-AND-BUILDCTL.md
• Upstream: Moby BuildKit, Nixpacks

I also wrote about swapping MinIO artifacts for this image pipeline (and the permission gauntlet that followed) in Phase 2 of the Rust deploy experiment — same machinery, more war stories.

Questions or corrections — GitHub or open an issue on the repo.