A cleanroom is a clone on a bridge with no uplink

A cleanroom is a clone of the snapshot you are inspecting, started as a new domain whose only NIC is on a bridge with no uplink. The production domain stays on the production dataset and the production bridge. When the look is done, shut the test domain off and destroy the clone. The origin snapshot stays.

The clone is not a second copy. A snapshot is not a backup. It sits on the same pool as the snapshot. If that pool is what failed, receive the replica onto other disks and clone there. Instant recovery is a clone you can promote. Do not promote for a look. Test a virtual machine restore is the boot on other storage.

What keeps the test off the production network?

The NIC, before the first start. If the domain still names the production bridge, the clone answers for the production address as soon as it runs. Taking the NIC off after boot is late. A client writes to whichever address answers.

The bridge has to have no path out. A VLAN called isolated still routes when the switch has an uplink or a gateway. A Linux bridge with no cable still reaches production when the host has an address on that bridge and forwarding is on.

A second machine on this same bridge is fine if the guest needs something to talk to. That machine gets one NIC, on this bridge. A helper with a second NIC on the production bridge is a router. The test guest leaves through it.

Which network has no uplink?

On libvirt, no forward element isolates the network from any other network. The exception in the network XML docs is a guest on that network acting as a router. That is why this domain gets one NIC.

Isolation from other networks is not isolation from the host. Libvirt’s isolated example omits forward and still sets 192.168.152.1 on the bridge. Guests there can talk to the host. If the host forwards, they can reach what the host can route. Omit ip as well. The example with no addresses is the one those docs describe as unable to reach the host through this network.

Define a network named cleanroom, on a bridge named virbr-clean, with ipv6='yes', and with no forward and no ip. ipv6='yes' lets two guests on this bridge use IPv6 with each other when no IPv6 gateway is set. It does not add a gateway. stp='on' and delay='0' are bridge timers, not an uplink.

An empty forward element is not isolated. If the element is present and mode is omitted, libvirt assumes mode='nat'. nat, route, and open forward through the host. mode='bridge' attaches the network to a bridge that already exists, which is how a production uplink comes back. port isolated='yes' stops guests on one bridge from talking to each other. It does not remove an uplink. Do not use it in place of omitting forward.

After virsh net-define and virsh net-start, dump the active network, not the file you saved. virsh net-dumpxml cleanroom is the bridge that is up. virsh net-dumpxml --inactive is the config for the next start. If those two differ, do not start the guest. Fix the definition, restart the network only when no guest is on it, and dump the active XML again. It has to show no forward and no ip.

Then ip -4 addr show dev virbr-clean should print no address, and ip -6 addr show dev virbr-clean should show no global address. Linux can add a link-local fe80 address when the bridge comes up. Do not SSH to it, and do not add a route to it. Before the guest starts, ip link show master virbr-clean should list nothing. After start, a vnet or tap device is the guest. A physical NIC (eno, eth, ens) is an uplink. If one is enslaved, the name cleanroom does not make this bridge isolated.

On Proxmox, bridge-ports none only means no physical port. The routed and NAT examples in the Proxmox network docs use bridge-ports none and still put an address on the bridge, then turn forwarding on. That is a path. The cleanroom bridge is inet manual, has bridge-ports none, has no address, and has no post-up that enables forwarding or masquerade. Do not use vmbr0 if that is the management bridge. A VLAN interface on the physical NIC is an uplink even when the guest tag looks private. A Proxmox bridge name is at most 10 characters. vmbr9 fits.

On VMware, create a standard switch with no physical adapter and put the test port group on that switch. A port group named Isolated on the switch that already has the uplinks is still on those uplinks. Do not add a VMkernel port on that switch. Guests can reach a VMkernel that shares it.

On Hyper-V, use a private switch. It has no external adapter and no host virtual adapter. An internal switch gives the host an adapter. An external switch has the uplink.

How do you attach the clone?

Clone the snapshot you mean to inspect, not the newest name on the dataset. Read origin on the clone and clones on the snapshot before you define the domain. If they do not name each other, do not start.

Copy the domain XML saved with that snapshot. The catalog is the domain XML saved with the snapshot. Change the name, the UUID, the MAC, the nvram path, and the guest-agent socket. Point every disk source at the clone. Leave the disk targets, the bus, and the boot order as they were.

Attach the NIC with type='network' and source network='cleanroom', and stop at one interface. That follows the network you dumped. A type='bridge' source that names virbr-clean is right only while that device is still the bridge this network created. A second interface on br0, vmbr0, or the production port group makes this guest the router in the libvirt exception.

Do not stop the production domain so the clone can reuse its name. Do not point this XML at the production zvol. Two opens of one zvol can corrupt the guest. Read the disk path in virsh dumpxml before virsh start.

The snapshot is crash-consistent unless you froze the guest first. Make application-consistent virtual machine backups. If the database will not open, write that down. Promoting the clone does not repair a torn write.

Clone every disk from that same moment, including EFI if it is its own dataset. One dataset per VM is the job, the quota, and the key.

What still reaches production?

You added an address on virbr-clean so you could SSH. Remove it. Use virsh console, or a display you open on the host. The display is not an address on this bridge. Forwarding on the host is normal for the other networks. It becomes a path out of this bridge only when the bridge has an address, a forward element, or a second NIC.

A static route or a DNS server inside the guest can name a production address. On a bridge with no path, that packet dies here. If the address answers, shut the guest off. Pulling the NIC later does not undo a write that already landed.

If you enslaved a physical NIC to virbr-clean so a VM could get out, that NIC is the uplink. Remove it. If you leave the test domain defined, someone can move its NIC back to the production bridge and start it. Undefine it when the notes are written. A clone on this pool still does not tell you the replica boots. Keep an off-site virtual machine backup. The recovery point objective is the time since the last successful send.

How do you prove the bridge has no path out?

Before virsh start, the active dump has no forward and no ip. The bridge has no physical NIC as master and no global address. The domain has one interface on cleanroom, disk sources on the clone, and a name, UUID, MAC, nvram path, and guest-agent socket that are not production’s.

After start, from inside the guest, a production address does not answer. If it does, shut the guest off and fix the bridge before you treat the boot as a pass.

A record written before the snapshot’s creation time is present. A record written on production after that time is absent. If the later record is there, you started production, or you cloned a later snapshot.

Write down the snapshot name and guid, the clone name, the network name, that the live dump had no forward and no ip, whether a production address answered, the record you found, and that the domain was undefined and the clone destroyed.

The script builds its own fixtures. It does not run virsh, create a bridge, or start a guest. It rejects a network with a forward or ip element, a Proxmox stanza with an address or a physical port, and a domain that still has a production disk, a production bridge, or a second NIC. It accepts the fixture that has none of those. It does not prove libvirt started the network, that a physical NIC is absent on this host, or that a guest failed to reach a production address.

#!/usr/bin/env bash
set -euo pipefail

root="$(mktemp -d)"
trap 'rm -rf "$root"' EXIT

python3 - "$root" <<'PY'
import copy
import sys
import uuid
import xml.etree.ElementTree as ET
from pathlib import Path

root = Path(sys.argv[1])
prod_name = "db"
prod_uuid = "6f1e5c2a-1111-4222-8333-444455556666"
prod_mac = "52:54:00:11:22:33"
prod_bridge = "br0"
prod_nvram = "/var/lib/libvirt/qemu/nvram/db_VARS.fd"
prod_channel = (
    "/var/lib/libvirt/qemu/channel/target/domain-1-db/org.qemu.guest_agent.0"
)
prod_prefix = "/dev/zvol/tank/vm-db/"
clone_prefix = "/dev/zvol/tank/vm-db-inspect/"
clean_net = "cleanroom"
clean_bridge = "virbr-clean"
new_mac = "52:54:00:aa:bb:cc"
new_nvram = "/var/lib/libvirt/qemu/nvram/db-inspect_VARS.fd"
new_channel = (
    "/var/lib/libvirt/qemu/channel/target/domain-db-inspect/org.qemu.guest_agent.0"
)


def write(name, text):
    path = root / name
    path.write_text(text.strip() + "\n", encoding="utf-8")
    if not path.is_file() or path.stat().st_size == 0:
        raise SystemExit(f"fixture missing: {name}")
    return path


def network_faults(path):
    net = ET.parse(path).getroot()
    if net.tag != "network":
        raise SystemExit(f"{path.name}: root is {net.tag}")
    faults = []
    fwd = net.find("forward")
    if fwd is not None:
        faults.append("forward:" + (fwd.get("mode") or "nat"))
    if net.find("ip") is not None:
        faults.append("ip")
    port = net.find("port")
    if port is not None and port.get("isolated") == "yes":
        faults.append("port-isolated")
    return faults


def expect_net(name, text, expected):
    got = network_faults(write(name, text))
    if got != expected:
        raise SystemExit(f"{name}: got {got}, expected {expected}")


clean = write(
    "cleanroom.xml",
    """
<network ipv6="yes">
  <name>cleanroom</name>
  <bridge name="virbr-clean" stp="on" delay="0"/>
</network>
""",
)
if network_faults(clean) != []:
    raise SystemExit("cleanroom network was rejected")
net = ET.parse(clean).getroot()
if net.get("ipv6") != "yes" or net.findtext("name") != clean_net:
    raise SystemExit("cleanroom name or ipv6 flag")
if net.find("bridge") is None or net.find("bridge").get("name") != clean_bridge:
    raise SystemExit("cleanroom bridge name")

expect_net(
    "host-ip.xml",
    """
<network>
  <name>private</name>
  <bridge name="virbr2"/>
  <ip address="192.168.152.1" netmask="255.255.255.0"/>
</network>
""",
    ["ip"],
)
expect_net(
    "empty-forward.xml",
    """
<network>
  <name>copied-default</name>
  <bridge name="virbr0"/>
  <forward/>
</network>
""",
    ["forward:nat"],
)
expect_net(
    "nat.xml",
    """
<network>
  <name>default</name>
  <forward mode="nat"/>
  <bridge name="virbr0"/>
  <ip address="192.168.122.1" netmask="255.255.255.0"/>
</network>
""",
    ["forward:nat", "ip"],
)
expect_net(
    "route.xml",
    """
<network>
  <name>routed</name>
  <forward mode="route"/>
</network>
""",
    ["forward:route"],
)
expect_net(
    "open.xml",
    """
<network>
  <name>open</name>
  <forward mode="open"/>
</network>
""",
    ["forward:open"],
)
expect_net(
    "bridge-mode.xml",
    """
<network>
  <name>uplink</name>
  <forward mode="bridge"/>
  <bridge name="br0"/>
</network>
""",
    ["forward:bridge"],
)
expect_net(
    "port-isolated.xml",
    """
<network>
  <name>isolated-ports</name>
  <forward mode="bridge"/>
  <bridge name="br0"/>
  <port isolated="yes"/>
</network>
""",
    ["forward:bridge", "port-isolated"],
)


def proxmox_faults(text):
    faults = []
    ports = []
    for line in text.splitlines():
        stripped = line.strip()
        if stripped.startswith("#") or stripped == "":
            continue
        if stripped.startswith("bridge-ports "):
            ports.append(stripped.split(None, 1)[1])
        if stripped.startswith("address ") or stripped.startswith("gateway "):
            faults.append("address")
        if (
            "ip_forward" in stripped or "MASQUERADE" in stripped
        ) and "forward-hook" not in faults:
            faults.append("forward-hook")
    if ports != ["none"]:
        faults.append("ports")
    if "inet manual" not in text:
        faults.append("not-manual")
    return faults


def expect_px(name, text, expected):
    got = proxmox_faults(write(name, text).read_text(encoding="utf-8"))
    if sorted(got) != sorted(expected):
        raise SystemExit(f"{name}: got {got}, expected {expected}")


expect_px(
    "vmbr9",
    """
auto vmbr9
iface vmbr9 inet manual
    bridge-ports none
    bridge-stp off
    bridge-fd 0
""",
    [],
)
expect_px(
    "vmbr-nat",
    """
auto vmbr0
iface vmbr0 inet static
    address 10.10.10.1/24
    bridge-ports none
    bridge-stp off
    bridge-fd 0
    post-up echo 1 > /proc/sys/net/ipv4/ip_forward
    post-up iptables -t nat -A POSTROUTING -s 10.10.10.0/24 -o eno1 -j MASQUERADE
""",
    ["address", "forward-hook", "not-manual"],
)
expect_px(
    "vmbr-uplink",
    """
auto vmbr0
iface vmbr0 inet manual
    bridge-ports eno1
    bridge-stp off
    bridge-fd 0
""",
    ["ports"],
)


def nic_ok(iface):
    src = iface.find("source")
    if src is None:
        return False
    if iface.get("type") == "network" and src.get("network") == clean_net:
        return src.get("bridge") is None
    if iface.get("type") == "bridge" and src.get("bridge") == clean_bridge:
        return src.get("network") is None
    return False


def domain_faults(domain):
    faults = []
    if domain.findtext("name") == prod_name:
        faults.append("name")
    if domain.findtext("uuid") == prod_uuid:
        faults.append("uuid")
    if domain.findtext("./os/nvram") == prod_nvram:
        faults.append("nvram")
    ifaces = domain.findall("./devices/interface")
    if len(ifaces) != 1:
        faults.append("nics:" + str(len(ifaces)))
    for iface in ifaces:
        mac = iface.find("mac")
        if mac is not None and mac.get("address") == prod_mac:
            faults.append("mac")
        if not nic_ok(iface):
            faults.append("nic")
    for src in domain.findall("./devices/disk/source"):
        path = src.get("dev") or src.get("file") or ""
        if (not path.startswith(clone_prefix)) or path.startswith(prod_prefix):
            faults.append("disk")
    if prod_channel in ET.tostring(domain, encoding="unicode"):
        faults.append("channel")
    return faults


def build_domain():
    domain = ET.Element("domain", {"type": "kvm"})
    ET.SubElement(domain, "name").text = prod_name
    ET.SubElement(domain, "uuid").text = prod_uuid
    os_el = ET.SubElement(domain, "os")
    ET.SubElement(os_el, "nvram").text = prod_nvram
    devices = ET.SubElement(domain, "devices")
    disk = ET.SubElement(devices, "disk", {"type": "block", "device": "disk"})
    ET.SubElement(disk, "source", {"dev": prod_prefix + "disk0"})
    ET.SubElement(disk, "target", {"dev": "vda", "bus": "virtio"})
    iface = ET.SubElement(devices, "interface", {"type": "bridge"})
    ET.SubElement(iface, "mac", {"address": prod_mac})
    ET.SubElement(iface, "source", {"bridge": prod_bridge})
    channel = ET.SubElement(devices, "channel", {"type": "unix"})
    ET.SubElement(channel, "source", {"mode": "bind", "path": prod_channel})
    ET.SubElement(
        channel, "target", {"type": "virtio", "name": "org.qemu.guest_agent.0"}
    )
    return domain


def rewrite(domain):
    target = domain.find("./devices/disk/target")
    kept = (target.get("dev"), target.get("bus"))
    domain.find("name").text = "db-inspect"
    domain.find("uuid").text = str(uuid.uuid4())
    domain.find("./os/nvram").text = new_nvram
    iface = domain.find("./devices/interface")
    iface.set("type", "network")
    src = iface.find("source")
    if "bridge" in src.attrib:
        del src.attrib["bridge"]
    src.set("network", clean_net)
    iface.find("mac").set("address", new_mac)
    disk_src = domain.find("./devices/disk/source")
    prod = disk_src.get("dev")
    if prod is None or not prod.startswith(prod_prefix):
        raise SystemExit("unexpected disk " + str(prod))
    disk_src.set("dev", clone_prefix + prod[len(prod_prefix) :])
    domain.find("./devices/channel/source").set("path", new_channel)
    if (target.get("dev"), target.get("bus")) != kept:
        raise SystemExit("rewrite changed the disk target")
    return domain


prod = build_domain()
before = domain_faults(prod)
for item in ("name", "uuid", "nvram", "mac", "nic", "disk", "channel"):
    if item not in before:
        raise SystemExit("production fixture missing fault " + item)

rewritten = rewrite(build_domain())
defined = root / "defined.xml"
defined.write_bytes(ET.tostring(rewritten, encoding="utf-8"))
again = ET.parse(defined).getroot()
if domain_faults(again) != []:
    raise SystemExit("rewritten domain still has faults: " + str(domain_faults(again)))
if again.findtext("name") == prod_name or again.findtext("uuid") == prod_uuid:
    raise SystemExit("production name or uuid remains")
if not again.find("./devices/disk/source").get("dev").startswith(clone_prefix):
    raise SystemExit("disk is not on the clone")

second = copy.deepcopy(again)
extra = ET.SubElement(second.find("devices"), "interface", {"type": "bridge"})
ET.SubElement(extra, "mac", {"address": "52:54:00:dd:ee:ff"})
ET.SubElement(extra, "source", {"bridge": prod_bridge})
second_faults = domain_faults(second)
if "nics:2" not in second_faults or "nic" not in second_faults:
    raise SystemExit("second nic accepted: " + str(second_faults))

kept_disk = copy.deepcopy(again)
kept_disk.find("./devices/disk/source").set("dev", prod_prefix + "disk0")
if "disk" not in domain_faults(kept_disk):
    raise SystemExit("production disk accepted")

print("cleanroom=virbr-clean")
print("ok")
PY

What do you destroy?

Shut the domain off. Undefine it, and delete the test nvram file if you copied one for this look and you are finished with it. Destroy the clone, not the origin snapshot. Do not promote. Promote is for the dataset you will keep. This was a look.

Leave the cleanroom network defined if you will use it again. Do not add an address to it between tests. A network that has grown an ip element is not the network you proved.