Creating a bhyve VM within a VPN connection.

While I was developing while doing Home Office with my laptop running FreeBSD 13.0-CURRENT, I realized a couple of things:

1. Vagrant has no support for bhyve.
2. Virtualbox crashes sporadically on non stable or released versions of FreeBSD.

My colleagues insist on me running something that works with vagrant and Virtualbox, but I also use this laptop to develop things for FreeBSD, so I need it to run "head".

I discovered that Virtualbox creates a very good configured NAT interface for the virtual machines, allowing traffic to go through them without any problem. Although, the ping command does not get through FreeBSD's hosts.

I needed something that is "stable" in the development branch of FreeBSD and, at the same time, gives me the possibility to do Home Office without breaking my working flow.

Setup for bhyve

I am using openconnect to create the VPN and it creates the interfaces tun0 in FreeBSD.

As I am doing Home Office and my network configuration doesn't allow me to just plug an Ethernet cable that easily, I'll be using my wireless interface wlan0.

I usually run several virtual machines in the same network and I am going to create two bridges, one for the VPN connection and a second one for the wild, wild internet.

First of all, let's create the bridges:

# ifconfig bridge create name inetnat up
# ifconfig inetnat inet 10.0.0.1/24

# ifconfig bridge create name vpnnat up
# ifconfig vpnnat inet 11.0.0.1/24

After that, the tap interfaces for the bhyve virtual machines:

# ifconfig tap0 create
# ifconfig tap1 create

Having all the network interfaces for the virtual machine, we have to add them to the matching bridge:

# ifconfig inetnat addm tap0
# ifconfig vpnnat addm tap1

Another setup on the host to get this working, will be to allow the kernel to forward packages and enable the tap interfaces when the machine boots, this is done by setting the following sysctl variables:

# sysctl net.inet.ip.forwarding=1
# sysctl net.link.tap.up_on_open=1

Additionally, we have to declare some routing rules in /etc/pf.conf and start pf:

ext_if="wlan0"
vpn_if="tun0"

virt_inet_net="10.0.0.0/24"
virt_vpn_net="11.0.0.0/24"

scrub all

nat on $ext_if from $virt_inet_net to any -> ($ext_if)
nat on $vpn_if from $virt_vpn_net to any -> ($vpn_if)

pass log all


And now, start pf:

# service pf onestart 

Booting the bhyve virtual machine

Booting a virtual machine using bhyve is not difficult, you can use the command under /usr/share/examples/bhyve or use the following commands:

# bhyveload -c /dev/nmdm0A -d /dev/zvol/zroot/vm0 \
   -m 2048M -S vm0
# bhyve -A -c 2 -m 2048M -l com1,/dev/nmdm0A -S \
   -s 0:0,hostbridge \
   -s 1:0,lpc \
   -s 2:0,virtio-blk,/dev/zvol/zroot/vm0 \
   -s 3:0,virtio-net,tap0 \
   -s 4:0,virtio-net,tap1 \
   vm0

The machine is using a ZFS block device and nullmodem terminal driver, this way our terminal is not disturbed by the control characters sent by bhyve.

The machine has the two created tap interfaces attached to it, as expected.
We are configuring the machine to have two cores and 2048 MB of RAM, the rest is standard.

Configuring the virtual machine's static routes

Lastly, we have to configure the network for the machine. We will use the following commands:

# sysrc ifconfig_vtnet0="10.0.0.2/24"
# sysrc ifconfig_vtnet1="11.0.0.2/24"
# sysrc defaultrouter="10.0.0.1"
# sysrc static_routes="vpn_route"
# sysrc vpn_route="-net <vpn-network> -gateway 11.0.0.1"

Of course, the network configuration of your company's network may (most likely) differ from mines, so replace <vpm-network> for the network configuration of your company.

Restart the vm and you'll be ready to work without a problem (as long as you remember to copy over your /etc/resolv.conf)