Skip to content

feat(netd): carry a VM's host port mappings to whoever configures the host - #1155

Closed
kvinwang wants to merge 1 commit into
feat/netd-ingressfrom
feat/netd-ingress-ports
Closed

feat(netd): carry a VM's host port mappings to whoever configures the host#1155
kvinwang wants to merge 1 commit into
feat/netd-ingressfrom
feat/netd-ingress-ports

Conversation

@kvinwang

Copy link
Copy Markdown
Collaborator

Stacked on #1154, which is itself stacked on #1145. Review bottom-up.

The bug

port_map is implemented as QEMU hostfwd= entries on a user-mode netdev:

let hostfwd_index = self.prepared.networks.iter()
    .position(|n| n.mode == NetworkingMode::User);

A bridge NIC has no user-mode netdev, so hostfwd_index is None and not one hostfwd= is emitted. There is no warning, no validation, and GetInfo keeps reporting the ports as though they worked. A VM moved from user mode to a bridge silently loses every published port.

I hit this while planning a production gateway's move off slirp: five port mappings, including the WireGuard endpoint 100+ CVMs depend on, would have evaporated on the first boot with nothing in the log to say so.

Why the VMM cannot just fix it

It runs without CAP_NET_ADMIN by design — that is the entire reason netd exists. And the userspace forwarder that used to cover this was removed in e2e607f, correctly: proxying on the host hands back the per-packet cost that leaving user mode was meant to escape.

netd is privileged, and it is the only component that sees every VMM instance on a host — so it is also the only one that can arbitrate a host port between them. This host runs three dstack deployments against one netd.

What this does

Carries the requirement instead of implementing it. ingress on prepare_bridge states protocol, host address, host port, guest port; the response reports what was established plus the guest_ip netd assigned.

Every field is caller-named. That is what bridge, mac and queues already get, and the opposite of filtered. The line is not "policy vs. mechanism" — it is whether netd can check what it is handed. An nwfilter name cannot be checked for whether it filters anything (allow-arp satisfies "some filter" and drops nothing), so naming one is excluded. A host port is a closed, enumerable space a policy can be stated over. Naming is not deciding: which ports may be handed out stays netd's configuration, exactly as allowed_bridges governs the bridge a caller names.

host_address is not decoration — an admin port on loopback and a published one differ only there.

The netd in this repo does not forward. It does not name ingress in hello, and it refuses a request carrying ports rather than building the interface without them — refused alongside the other checks about the request, before the host is inspected, since it is a fact about the request. Accepting ports and forwarding nothing is the exact failure being ended.

The VMM asks hello first and does not send ports to a netd that will not forward them; it warns instead. A VM deployed before any of this has been running with its ports dropped, and refusing to launch it now would turn a silent misconfiguration into an outage on upgrade. macvtap and custom get the same warning permanently — the first bypasses the host bridge, the second hands the netdev string to the operator.

GetInfo grows guest_ip and ingress per interface. What the node did, against the ports on the configuration, which is what was asked for. An empty ingress against a non-empty request is now a visible difference instead of silence.

Reporting guest_ip also closes what ReportDhcpLease was for, without reinstating a DHCP hook: a netd that forwards is necessarily also the authority on the address, because a DNAT rule needs one at install time and a lease does not exist until the guest has booted.

Compatibility

Additive. ingress defaults empty, so an older caller asks for nothing and an existing bridge VM behaves exactly as before apart from a new warning. Proto fields are new tags; ports on the configuration is untouched.

Testing

cargo test -p dstack-vmm — 190 pass, 3 new. cargo clippy -- -D warnings clean.

New cases cover the request shape and the empty default, the refusal firing before host inspection (and not firing when nothing is asked for), and hello not advertising a feature this netd lacks.

One of these caught a bad assertion of mine: I had asserted on a bridge named br0 not existing, and the dev host has a br0. The test now uses a name nothing will have.

Not covered: an actual forwarding netd, which by construction is not in this repository.

… host

`port_map` is implemented as QEMU `hostfwd=` entries on a user-mode
netdev. A bridge NIC has none, so `configure_networking` finds no
user-mode interface to hang them on and emits none -- with no warning, no
validation, and `GetInfo` still reporting the ports as though they worked.
A VM moved from user mode to a bridge silently loses every published port.

The VMM cannot fix that itself. It runs without `CAP_NET_ADMIN` by design,
and the userspace forwarder that used to cover this case was removed in
e2e607f because proxying on the host gives back the per-packet cost that
leaving user mode was meant to escape. netd is privileged, and it is the
only component that sees every VMM instance on a host, so it is also the
only one that can arbitrate a host port between them.

So carry the requirement rather than implement it. `ingress` on
`prepare_bridge` states protocol, host address, host port and guest port;
the response reports what was actually established, along with the
`guest_ip` netd assigned. The host address is not decoration -- an admin
port bound to loopback and a published one differ only there.

Every field is caller-named, which is the treatment `bridge`, `mac` and
`queues` already get, and the opposite of `filtered`. The distinction is
whether netd can check what it is handed: an nwfilter name cannot be
checked for whether it filters anything, so naming one is excluded, while
a host port is a closed space a policy can be stated over. Naming is not
deciding; which ports may be handed out stays netd's own configuration,
exactly as `allowed_bridges` governs the bridge a caller names.

The netd here does not forward, does not name `ingress` in `hello`, and
refuses a request carrying ports rather than building the interface
without them -- refused with the other checks about the request, before
the host is inspected. Accepting them and forwarding nothing is the
failure being ended.

The VMM asks `hello` first and simply does not send ports to a netd that
will not forward them, warning instead. A VM deployed before any of this
existed has been running with its ports dropped; refusing to launch it now
would turn a silent misconfiguration into an outage on upgrade. `macvtap`
and `custom` get the same warning permanently, since neither has anywhere
to forward to.

`GetInfo` grows `guest_ip` and `ingress` per interface: what the node did,
against the `ports` on the configuration, which is what was asked for. An
empty `ingress` against a non-empty request is now a visible difference
rather than silence. Reporting `guest_ip` also closes what `ReportDhcpLease`
was for, without a DHCP hook -- a netd that forwards is necessarily also
the authority on the address, because a DNAT rule needs one at install
time and a lease does not exist until the guest has booted.
Copilot AI lite review requested due to automatic review settings August 28, 2026 14:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@kvinwang

Copy link
Copy Markdown
Collaborator Author

Folding into #1154 — the two together grew well past what was asked for (host port info plus workdir on the netd API). #1154 is being rewritten to just that.

@kvinwang kvinwang closed this Aug 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants