feat(netd): carry a VM's host port mappings to whoever configures the host - #1155
Closed
kvinwang wants to merge 1 commit into
Closed
feat(netd): carry a VM's host port mappings to whoever configures the host#1155kvinwang wants to merge 1 commit into
kvinwang wants to merge 1 commit into
Conversation
… host `port_map` is implemented as QEMU `hostfwd=` entries on a user-mode netdev. A bridge NIC has none, so `configure_networking` finds no user-mode interface to hang them on and emits none -- with no warning, no validation, and `GetInfo` still reporting the ports as though they worked. A VM moved from user mode to a bridge silently loses every published port. The VMM cannot fix that itself. It runs without `CAP_NET_ADMIN` by design, and the userspace forwarder that used to cover this case was removed in e2e607f because proxying on the host gives back the per-packet cost that leaving user mode was meant to escape. netd is privileged, and it is the only component that sees every VMM instance on a host, so it is also the only one that can arbitrate a host port between them. So carry the requirement rather than implement it. `ingress` on `prepare_bridge` states protocol, host address, host port and guest port; the response reports what was actually established, along with the `guest_ip` netd assigned. The host address is not decoration -- an admin port bound to loopback and a published one differ only there. Every field is caller-named, which is the treatment `bridge`, `mac` and `queues` already get, and the opposite of `filtered`. The distinction is whether netd can check what it is handed: an nwfilter name cannot be checked for whether it filters anything, so naming one is excluded, while a host port is a closed space a policy can be stated over. Naming is not deciding; which ports may be handed out stays netd's own configuration, exactly as `allowed_bridges` governs the bridge a caller names. The netd here does not forward, does not name `ingress` in `hello`, and refuses a request carrying ports rather than building the interface without them -- refused with the other checks about the request, before the host is inspected. Accepting them and forwarding nothing is the failure being ended. The VMM asks `hello` first and simply does not send ports to a netd that will not forward them, warning instead. A VM deployed before any of this existed has been running with its ports dropped; refusing to launch it now would turn a silent misconfiguration into an outage on upgrade. `macvtap` and `custom` get the same warning permanently, since neither has anywhere to forward to. `GetInfo` grows `guest_ip` and `ingress` per interface: what the node did, against the `ports` on the configuration, which is what was asked for. An empty `ingress` against a non-empty request is now a visible difference rather than silence. Reporting `guest_ip` also closes what `ReportDhcpLease` was for, without a DHCP hook -- a netd that forwards is necessarily also the authority on the address, because a DNAT rule needs one at install time and a lease does not exist until the guest has booted.
Collaborator
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #1154, which is itself stacked on #1145. Review bottom-up.
The bug
port_mapis implemented as QEMUhostfwd=entries on a user-mode netdev:A bridge NIC has no user-mode netdev, so
hostfwd_indexisNoneand not onehostfwd=is emitted. There is no warning, no validation, andGetInfokeeps reporting the ports as though they worked. A VM moved from user mode to a bridge silently loses every published port.I hit this while planning a production gateway's move off slirp: five port mappings, including the WireGuard endpoint 100+ CVMs depend on, would have evaporated on the first boot with nothing in the log to say so.
Why the VMM cannot just fix it
It runs without
CAP_NET_ADMINby design — that is the entire reasonnetdexists. And the userspace forwarder that used to cover this was removed in e2e607f, correctly: proxying on the host hands back the per-packet cost that leaving user mode was meant to escape.netdis privileged, and it is the only component that sees every VMM instance on a host — so it is also the only one that can arbitrate a host port between them. This host runs three dstack deployments against one netd.What this does
Carries the requirement instead of implementing it.
ingressonprepare_bridgestates protocol, host address, host port, guest port; the response reports what was established plus theguest_ipnetd assigned.Every field is caller-named. That is what
bridge,macandqueuesalready get, and the opposite offiltered. The line is not "policy vs. mechanism" — it is whether netd can check what it is handed. An nwfilter name cannot be checked for whether it filters anything (allow-arpsatisfies "some filter" and drops nothing), so naming one is excluded. A host port is a closed, enumerable space a policy can be stated over. Naming is not deciding: which ports may be handed out stays netd's configuration, exactly asallowed_bridgesgoverns the bridge a caller names.host_addressis not decoration — an admin port on loopback and a published one differ only there.The netd in this repo does not forward. It does not name
ingressinhello, and it refuses a request carrying ports rather than building the interface without them — refused alongside the other checks about the request, before the host is inspected, since it is a fact about the request. Accepting ports and forwarding nothing is the exact failure being ended.The VMM asks
hellofirst and does not send ports to a netd that will not forward them; it warns instead. A VM deployed before any of this has been running with its ports dropped, and refusing to launch it now would turn a silent misconfiguration into an outage on upgrade.macvtapandcustomget the same warning permanently — the first bypasses the host bridge, the second hands the netdev string to the operator.GetInfogrowsguest_ipandingressper interface. What the node did, against theportson the configuration, which is what was asked for. An emptyingressagainst a non-empty request is now a visible difference instead of silence.Reporting
guest_ipalso closes whatReportDhcpLeasewas for, without reinstating a DHCP hook: a netd that forwards is necessarily also the authority on the address, because a DNAT rule needs one at install time and a lease does not exist until the guest has booted.Compatibility
Additive.
ingressdefaults empty, so an older caller asks for nothing and an existing bridge VM behaves exactly as before apart from a new warning. Proto fields are new tags;portson the configuration is untouched.Testing
cargo test -p dstack-vmm— 190 pass, 3 new.cargo clippy -- -D warningsclean.New cases cover the request shape and the empty default, the refusal firing before host inspection (and not firing when nothing is asked for), and
hellonot advertising a feature this netd lacks.One of these caught a bad assertion of mine: I had asserted on a bridge named
br0not existing, and the dev host has abr0. The test now uses a name nothing will have.Not covered: an actual forwarding netd, which by construction is not in this repository.