# Network and egress

**Spinbox is experimental: APIs change and data may be lost.** What is built and what proves it is on [Status](https://spin-stack.github.io/website/reference/status.md); what Spinbox does not do is on [Limitations](https://spin-stack.github.io/website/overview/limitations.md).

> How network policies control where a workspace connects, what is always blocked, how DNS works, and how to see what it reached.

A workspace reaches only what its **network policies** allow. Policies are allowlists with
names, such as `web` or `registry-only`, and a workspace runs under one or more of them, called
its **tags**. They are enforced on the runner, outside the virtual machine, so nothing done
inside the workspace — even as root — can change them.

## The default is closed

[Egress is closed by default](https://spin-stack.github.io/website/reference/status.md#network). A workspace created without `--network-tag` runs under the installation's
`default_network_tags`. If the installation has not set any, it runs under **`offline`**, which
every runner has and which allows nothing.

No setting means "open". An installation that wants workspaces to reach the internet by
default declares a policy that says so, for example `allow: ["0.0.0.0/0"]`, and names it in
`default_network_tags`. What a workspace can reach always has a name somebody chose.

*Diagram: How a packet leaves a workspace: the guest, its machine's network namespace, the host's egress namespace, and the node.* Its labels: How a packet leaves a workspace · the guest - the tenant is root in here · netns per machine - /var/run/netns/<workspace uuid> · netns per host - spin-egress · the host's own namespace · a process · src 169.254.68.6 · eth0 · MTU from host_mtu, which the · kernel takes as dev->max_mtu · tap0 · 169.254.68.5/30 · the gateway, and the resolver · virtio-net / vhost-net · the descriptor QEMU inherited at exec · masquerade · oifname veth-vm-N · forwarded · veth-vm-N · 10.255.N.2/30 · src becomes 10.255.N.2 - unique on this host · veth-eg-N · 10.255.N.1/30 · veth pair · forward chain · policy drop · wan0 · 10.255.255.2/30 · accept, and record where it went · spin-eg0 · 10.255.255.1/30 · the only interface this project puts here, · and it belongs to no machine · veth pair · masquerade · saddr 10.255.0.0/16 · oifname the node's NIC · the node's NIC · never moved, never shared, · never given a second address · src becomes the node's · the network · Two translations, each doing one thing. · The first turns the address every guest shares into one · that is unique here - 169.254.68.6 is identical on every · machine on every host, which is what lets a checkpoint · resume anywhere, and an address that repeats cannot cross · anything shared. · The second turns that into the node's, which is what a · node's neighbours expect to see. · The egress namespace in between translates nothing, so · every packet crossing it still says which machine sent it. · That is what a per-machine rule is written against.. ([source](https://github.com/spin-stack/spin/blob/8eee32b9ddc8d535b3300d0da3927a30b33f22c4/docs/network/datapath))

## Always blocked

Whatever its policies say, a workspace never reaches:

- private, loopback, link-local and reserved addresses, including the cloud's instance
  metadata service,
- any address of the runner it runs on,
- anything over IPv6.

These drops come before any policy, so `0.0.0.0/0` means "the internet", not "the runner's
network".

## What a policy allows

```yaml
network_policies:
  - name: agent
    allowNames: [github.com, "*.githubusercontent.com", registry.npmjs.org]
    allow: [140.82.112.0/20]
  - name: web
    allowPorts: [80, 443]
```

- **`allow`**: IPv4 prefixes.
- **`allowNames`**: host names, exact or with one leading `*.`. When the workspace looks up an
  allowed name, the runner opens the address it answered with, for that workspace only, for
  as long as the answer lives (between two minutes and an hour). A connection that started in
  that time keeps going after it.
- **`allowPorts`**: destination ports at any address. A number is TCP. UDP is asked for by name
  (`"53/udp"`), and `icmp` allows ping.

A name is exactly as precise as DNS: if one address serves many names, allowing one of them
allows them all.

**Tags add up.** A workspace with two tags reaches everything either one allows.

**DNS always works.** A workspace's resolver is the runner's, which answers under every policy,
so no policy needs to open port 53. It removes from its answers any address of the runner's own
networks, and every IPv6 answer.

*Diagram: What the host's forward chain decides for a packet from a workspace, in order.* Its labels: What the forward chain decides, in order · a packet from a machine · iifname veth-eg-* · 1 · ip daddr in the node's own space · 10/8 · 172.16/12 · 192.168/16 · 127/8 · 169.254/16 · 100.64/10 · 224/4 · 240/4 · + every address the node answers on · 2 · ip saddr @tag_<name> ip daddr <allowed> · what a tag permits · no · dropped, and counted · match · 3 · ip saddr @tag_<name> · and nothing else, when the tag says so · no · accepted - and recorded in @observed · machine . destination, with a counter, 6 h · match · 4 · oifname wan0 · the default: open beyond the node · no · match · 5 · iifname wan0 ct state established,related · the way back, and only for a connection · a machine opened · no · match · 6 · veth-eg-* to veth-eg-* · one machine to another, said out loud · no · match · match · no rule claimed it · policy drop · no · Why this order. · 1 is first because 4 is what would otherwise carry it: the · node's LAN is reached through the uplink like everything else, · so a drop written after the accept never runs. It matches the · destination alone and not the pair of interfaces - what is kept · away is a place, not a path, and a rule naming wan0 would stop · applying the day there is a second way out. · 2 and 3 are between them, so a tag cannot open the node and a · tag that closes a machine is reached by what its own rules did · not claim. · 4 is open on purpose. There is no allowlist to write yet, so · the accept records where it went instead - see @observed, and · `SIGUSR1` to the runner.. ([source](https://github.com/spin-stack/spin/blob/8eee32b9ddc8d535b3300d0da3927a30b33f22c4/docs/network/egress-policy))

*Diagram: Two workspaces with the same address on different segments: why one cannot reach the other.* Its labels: Same address, different segment · machine A · machine B · spin-egress - one per host · tap0 · 169.254.68.5 · 52:54:00:00:00:01 · veth-vm-0 · 10.255.0.2 · NAT · veth-eg-0 · 10.255.0.1 · tap0 · 169.254.68.5 · 52:54:00:00:00:01 · veth-vm-1 · 10.255.1.2 · NAT · veth-eg-1 · 10.255.1.1 · forward chain · policy drop · wan0 · iifname veth-eg-* oifname wan0 · dropped, and counted · iifname veth-eg-* oifname veth-eg-* · - one machine to another · ip daddr in the node's ranges · - the host, its LAN, the IMDS · the internet · Both machines have the same address and the same MAC, and · neither can see the other's: the two /30s meet only in the · egress namespace. · Linux would happily route between them - it holds a connected · route to each - so the ruleset is what stops it, and the · ruleset is fail-closed. The machine-to-machine drop is written · out even though the policy already covers it, because a rule · that somebody decided is different from a default, and it · counts the attempts. · The tenant is root on the far side of tap0. Everything that · enforces anything is on this side of it.. ([source](https://github.com/spin-stack/spin/blob/8eee32b9ddc8d535b3300d0da3927a30b33f22c4/docs/network/isolation))

## Changing a workspace's policies

```bash
spin workspace policies                          # the policies you can choose from
spin workspace retag api --network-tag web --network-tag agent
```

The owner can choose among the policies the installation declares. Removing a policy from a
running workspace closes the connections it had open, not only new ones.

Administrators declare policies in the [installation settings](https://spin-stack.github.io/website/admin/configuration.md) or in the
dashboard (Admin → Network), which shows for each one whether every runner enforces the
current version.

## Seeing what happened

```bash
spin workspace egress api
```

shows what the workspace reached in the last six hours, with the most traffic first, the names
it looked up that failed, and what was refused in the last hour, with the name it was looked up
by. The workspace's **Network** tab in the dashboard shows the same. When something does not
work, the refused list tells you which policy is too narrow.

## Inbound

Nothing on the network can open a connection into a workspace. SSH, the terminal, `spin exec`
and published ports are all **sessions**: the runner opens the machine's vsock and dials out to
the relay; you dial in to the relay too. See [Ports and sharing](https://spin-stack.github.io/website/concepts/ports.md).

The programs Spinbox runs inside the workspace reach the control plane the same way, over vsock
through the runner. A workspace is never told the control plane's address, and `offline` does
not cut its terminal off.