~drscream

Some hardware is not supported in illumos yet, but luckily there is bhyve which supports pci passthrough to any guest operating system. To continue with my OmniOS desktop on “modern” hardware I would love wifi support, so why not using a bhyve guest as router zone which provide the required drivers?

First of all it requires an up-to-date OmniOS version which provide bhyve support. Including the following IPS packages:

$ pkg install system/bhyve
$ pkg install system/zones/brand/bhyve

You can also found additional system requirements at the OmniOS bhyve hypervisor page.

Simple overview

                                   __   _
                                 _(  )_( )_
                                (_  inet  _)
                                  (_) (__)
                                     |
                                     |
                                     |
           o-- global zone ----------|--------------------------o
           |        o-- bhyve zone --|----o                     |
           |        |                |    |                     |
pci-wifi --|- passthrough -> wifi-device  |                     |
           |        |         |           |                     |
           |        |       {NAT}         |                     |
           |        |         |           |                     |
           |        |        vnic <--> [ etherstub ] <---> vnic |
           |        o---------------------o                     |
           |                                                    |
           o----------------------------------------------------o

Global zone network configuration

An etherstub is required which provides a virtual switch between the virtual nic in the global zone and the one in the bhyve zone.

$ dladm create-etherstub switch0

Now it’s required to assign to virtual nics on this etherstub.

$ dladm create-vnic -l switch0 uplink0
$ dladm create-vnic -l switch0 bhyve0

You can verify your configuration with dladm show-vnic. It’s also required to assign an IP address to the uplink0 vnic which allow the communication into the bhyve zone.

$ ipadm create-addr -T static -a 10.1.1.2/30 uplink0/v4

Choose whatever network you like, but because of the communication from only two nics I choose a /30 network.

Configure passthrough device via ppt

Lookup the PCI information for the device you like to passthrough to your bhyve zone. On this example I’m looking for the wireless network card.

$ prtconf -dD
...
   pci8086,1311 (pciex8086,85) [Intel Corporation Centrino Adv....] (...)

Look for the correct pciex entry, I didn’t got it working with the pci entry only. The prtpicl -v tool might also provide you additional information about your PCI devices.

The pciex information need to be stored in the following two files:

The /etc/ppt_aliases should look similar to the /etc/driver_aliases file, which will overwrite or assign the required ppt driver on the device:

ppt "pciex8086,85"

The /etc/ppt_matches file contains only the PCI information:

pciex8086,85

An reboot is recommended to attach the driver on the device. A verification can be done with the following commands:

Look for the attached driver on the device:

$ prtconf -dD | grep ppt

Use pptadm to list dev and path:

$ pptadm list -a
DEV        VENDOR DEVICE PATH
/dev/ppt0  8086   85     /pci@0,0/pci8086,1e12@1c,1/pci8086,1311@0

The /dev/ppt0 will be used later to passthrough it to the bhyve zone.

Use zonecfg to configure guest zone

For the bhyve zone it’s required to create a dataset for the zonepath and a zvol which contains the guest itself. My zone will be called gw because it’s the gateway to the internet.

# Create the zvol
$ zfs create -V 10G rpool/bhyve0

# Create the zonepath
$ zfs create rpool/zones/gw

# Create some dataset for isos
$ zfs create rpool/iso

Download the latest FreeBSD (or any operating system you like) ISO and store it in /rpool/iso. For me it looks like:

$ cd /rpool/iso
$ wget -O freebsd.iso https://download.freebsd.org/...-bootonly.iso

To create a zone via zonecfg, I always recommend create one file which contains the relevant data for zonecfg to work correctly, for example: /tmp/gw.zone:

create -b
set brand=bhyve
set zonepath=/zones/gw
add net
  set physical=bhyve0
  set global-nic=switch0
end
add device
  set match=/dev/zvol/rdsk/rpool/bhyve0
end
add device
  set match=/dev/ppt0
end

add fs
  set dir=/rpool/iso/freebsd.iso
  set special=/rpool/iso/freebsd.iso
  set type=lofs
  add options ro
  add options nodevices
end
add attr
  set name=cdrom
  set type=string
  set value=/rpool/iso/freebsd.iso
end

add attr
  set name=bootrom
  set type=string
  set value=BHYVE_RELEASE
end
add attr
  set name=bootdisk
  set type=string
  set value=rpool/bhyve0
end
add attr
  set name=extra
  set type=string
  set value="-S -s 8:0,passthru,/dev/ppt0"
end

Everything related to the freebsd.iso can be removed later if you do not need the install medium anymore. The BHYVE_RELEASE bootrom is required for FreeBSD to work correctly, it might depends on your operating system which one you need to use.

The following two settings are required to get passthrough working correctly:

This will add the ppt0 device into the zone and allow the required access.

add device
  set match=/dev/ppt0
end

Will add extra parameters to the bhyve command to passthrough /dev/ppt0 correctly to the PCI ID 8:0 in the guest. The PCI ID can be chosen by yourself as long as it’s not used as an parameter already.

add attr
  set name=extra
  set type=string
  set value="-S -s 8:0,passthru,/dev/ppt0"
end

The -S is required to disallow the guest memory to be swapped. Otherwise passthrough isn’t supported!

Finish the configuration and creating the zone

$ zonecfg -z gw -f /tmp/gw.zone

Use zoneadm to install and boot the zone:

$ zoneadm -z gw install
$ zoneadm -z gw boot

If something is going wrong and the zone doesn’t boot correctly check the bhyve log file in zonepath/root/tmp/init.log (example: /zones/gw/root/tmp/init.log).

Configure FreeBSD bhyve guest

You should be able to log into the zone via zlogin:

$ zlogin -C gw

Follow the operating system installation based on you needs. For me it’s an awesome FreeBSD installation which is straightforward anyway.

After the installation some network configuration is required to support the wireless card and setup NAT.

Modify the /boot/loader.conf to load the correct firmware:

if_iwn_load="YES"
iwn6000fw_load="YES"
iwn6000g2afw_load="YES"

wlan_wep="YES"
wlan_ccmp_load="YES"
wlan_tkip_load="YES"

Setup the network interfaces via /etc/rc.conf:

# Check for the guest "ethernet" interface
ifconfig_vtnet0="inet 10.1.1.1 netmask 255.255.255.252"

# Wireless network interface
wlans_iwn0="wlan0"
ifconfig_wlan0="WPA up"
create_args_wlan0="country DE"

# Enable PF and IP forwarding
gateway_enable="YES"
pf_enable="YES"

Add the NAT entry for /etc/pf.conf:

nat on wlan0 from {10.1.1.0/30} to any -> (wlan0)

To configure the wireless network (as required) I use the wpa_supplicant and store the information in /etc/wpa_supplicant.conf:

wpa_passphrase "SSID" "PWD" >> /etc/wpa_supplicant.conf

A reboot might be required to load the required firmware and configure the network as described.

Test network configuration

The global zone should be able to ping the bhyve guest via:

$ ping 10.1.1.1

If you haven’t configured a default gateway yet, it might be good time to do that:

$ route add default 10.1.1.1

After that you should be able to reach other networks and the always loved internet via your awesome bhyve router :-)