I have been replicated this same setup a couple of times during last year, using FreeBSD jails and on illumos SmartOS zones, as I’m trying to learn more about illumos zones and FreeBSD jails.
This post is to document for my future self on how I did it using FreeBSD jails, on a next blog post I’ll document how to set the same on illumos using SmartOS.
Here is a rough diagram of my current setup :
┌────────────────────────┐
┌───────▶ │Jail: Jellyfin │
│ └────────────────────────┘
│ ┌────────────────────────┐
├───────▶ │Jail: Backups │
│ └────────────────────────┘
│ ┌────────────────────────┐
├───────▶ │Jail: Neverwinter │
┌───────────────┐ │ └────────────────────────┘
│ │ │ ┌────────────────────────┐
┌────────┐ ┌────────┐ │ │ ┌────────────┐ ├───────▶ │Jail: Minecraft Bedrock │
│ │ ◀───▶│ igb2 │◀─────▶ │ │ ◀─────▶ │Jail : nat_a│ ◀─────┘ └────────────────────────┘
│ ISP │ └────────┘ │ │ └────────────┘
│ Router │ ┌────────┐ │ FreeBSD 13 │
│ │ ◀───▶│ igb3 │◀─────▶ │ │ ┌────────────┐
└────────┘ └────────┘ │ │ ◀─────▶ │Jail : nat_b│ ◀──────┐ ┌────────────────────────┐
▲ ┌────────┐ │ │ └────────────┘ ├───────▶│Jail: Gitea │
│ │ vtnet0 │◀─────▶ │ │ │ └────────────────────────┘
│ └────────┘ └───────────────┘ │ ┌────────────────────────┐
▼ ▲ ├───────▶│Jail: Grocy │
┌────────┐ │ │ └────────────────────────┘
│ Wifi │ │ │ ┌────────────────────────┐
│ router │◀──────────┘ └───────▶│Jail: znc │
└────────┘ └────────────────────────┘
Nat jails
For my usecase the most interesting jails are the nat_a and nat_b jails, which are vnet jails that perform routing and nat for the other jails. To isolate the network from the host(jail 0) I use vnet to move the igb interfaces to the nat jails, the effect is that those nics won’t be available on the host (jail 0) My ISP router is on bridge mode, so each igb interface connected to it will get an ip address by dhcp, so each nat jail will have two interfaces :
- A physical network interfac (igb2 on nat_a)
- A netgraph node of type eiface connected to a bridge node, and the bridge is connected to the vtnet0 interface.
Note:
Kernel modules ng_ether and pf must be loaded before nat_(a|b) jails starts.
The ng interface will be used to communicate to the internal network and serve as a gateway for the other jails. Here is the nat_a jail configuration:
nat_a {
persist;
vnet=new;
vnet.interface = igb2, ng0_$name;
host.hostname = "$name";
path=/jails/$name;
exec.system_user = "root";
exec.jail_user = "root";
exec.prestart += "jng bridge $name vtnet0";
exec.start += "/sbin/ifconfig ng0_$name 192.168.1.201 netmask 255.255.255.0 up";
exec.start += "/sbin/ifconfig lo0 127.0.0.1 up";
exec.start +="dhclient igb2";
exec.start += "/bin/sh /etc/rc";
exec.created += "rctl -a jail:$name:memoryuse:deny=512m";
exec.stop = "/bin/sh /etc/rc.shutdown";
exec.prestop = "ifconfig igb2 -vnet $name";
exec.poststop = "rctl -r jail:$name:";
exec.poststop += "jng shutdown $name";
devfs_ruleset="11";
mount.devfs;
sysvmsg = new;
sysvshm = new;
sysvsem = new;
}
The relevant parts here are the ones related to vnet and the jng script which creates a virtual nic.
Here is step by step:
vnet.interface = igb2, ng0_$name
This simply will move interfaces igb2 and ng0_$name ($name is replaced with the jail name) from the host to the jail, at this point ng0_$name does not exists yet, but will be created on the prestart step:
exec.prestart += "jng bridge $name vtnet0";
-
First the jng script is located in /usr/share/examples/jails/jng, so we need to copy it to /usr/local/bin.
-
The jng script will create a netgraph node of type NG_EIFACE(4) and a node of type NG_BRIDGE(4) with vtnet0 and then connect them.
So before starting the jail, the interface ng0_$name is created and then passed to the jail along the igb2 nic.
Note:
By default jng script will create ng_eiface interfaces with the prefix ng0,ng1,..and so on.
jng is able to output a graph of your connections, but to be able to do it we need to install first graphviz:
$ sudo pkg install graphviz
$ sudo jng graph -o jails.svg
The other important part of this jail configuration is :
exec.prestop = "ifconfig igb2 -vnet $name";
exec.poststop += "jng shutdown $name";
This will take out igb2 from the jail and back into the host (jail 0) and jng shutdown will remove the ng0_$name interface as it is not needed anymore.
exec.start +="dhclient igb2";
devfs_ruleset="11";
It just starts dhclient on the igb2 interface, and sets the jail to use devfs_rulset=11.
This ruleset unhides /dev/bpf and /dev/pf to use dhcp and pf, so we are able to run the pf firewall and dhclient inside a vnet jail.
[devfsrules_jail=11]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path 'bpf*' unhide
add path 'pf*' unhide
add path 'pflog' unhide
add path 'pfsynv' unhide
Now the /etc/pf.conf rules for nat_a are as follows :
set block-policy drop
set fail-policy drop
set state-policy if-bound
ext_if="igb2"
int_if="ng0_nat_a"
set skip on lo0
scrub in on $ext_if
nat log on $ext_if inet from !($ext_if) to any -> ($ext_if)
# minecraft bedrock
#UDP: 19132-19133, 25565
rdr pass log on $ext_if proto { tcp } from any to $ext_if port 25565 -> 192.168.1.235 port 25565
rdr pass log on $ext_if proto { udp } from any to $ext_if port 25565 -> 192.168.1.235 port 25565
rdr pass log on $ext_if proto { udp } from any to $ext_if port 19132:19133 -> 192.168.1.235 port 19132:19133
pass in on $ext_if proto udp to port { 67 }
pass out log (all) quick on $ext_if from $int_if to any
For nat_b jail the difference the ext_if will be igb3 and int_if is ng0_nat_b also redirects point to other jails.
Also /etc/rc.conf for nat_a and nat_b should have pf and gateway enabled.
gateway_enable="YES"
pf_enable="YES"
pflog_enable="YES"
syslogd_enable="NO"
Services Jails
Neverwinter Nights
This is just a Linux jail as there is no FreeBSD version released, but the server runs perfectly using Linux emulation. For creating this jail, we just debootstrap ubuntu into a zfs dataset. this jail configuration is the following :
nwn {
persist;
vnet=new;
linux=new;
host.hostname = $name;
vnet.interface = "ng0_$name";
path = /jails/$name/root;
exec.prestart += "jng bridge $name vtnet0";
exec.start += "env LD_LIBRARY_PATH=/native/lib /native/libexec/ld-elf.so.1 /native/sbin/ifconfig ng0_$name 192.168.1.109
1 netmask 255.255.255.0 up";
exec.start += "env LD_LIBRARY_PATH=/native/lib /native/libexec/ld-elf.so.1 /native/sbin/route add default 192.168.1.202
";
exec.start += "sh /etc/rc.local";
exec.created += "rctl -a jail:$name:pcpu:deny=200";
exec.created += "rctl -a jail:$name:memoryuse:deny=2g";
exec.poststop += "jng shutdown nwn";
exec.poststop = "rctl -r jail:$name:";
mount.fstab = /jails/$name/fstab;
mount.devfs;
allow.mount;
allow.mount.devfs;
}
Here are the interesting parts :
exec.start += "env LD_LIBRARY_PATH=/native/lib /native/libexec/ld-elf.so.1 /native/sbin/ifconfig ng0_$name 192.168.1.109
We cannot configure networking using the Linux binaries, so we need to use FreeBSD native tools.
To do this I just copied what’s needed to execute ifconfig into the native directory on the jail path, so it’s able to
execute when the jail starts.
neirac@cl-west-prod-002:/jails/nwn/root/native $ tree
.
├── lib
│ ├── lib80211.so.1
│ ├── libbsdxml.so.4
│ ├── libc.so.7
│ ├── libjail.so.1
│ ├── libm.so.5
│ ├── libnv.so.0
│ ├── libsbuf.so.6
│ └── libutil.so.9
├── libexec
│ ├── ld-elf.so.1
│ └── ld-elf32.so.1
└── sbin
├── ifconfig
└── route
jails/nwn/fstab has the following contents:
devfs /jails/nwn/root/dev devfs rw 0 0
tmpfs /jails/nwn/root/dev/shm tmpfs rw,size=1g,mode=1777 0 0
fdescfs /jails/nwn/root/dev/fd fdescfs rw,linrdlnk 0 0
linprocfs /jails/nwn/root/proc linprocfs rw 0 0
linsysfs /jails/nwn/root/sys linsysfs rw 0 0
The following lines restrict the jail to use only 2 cpus and a maximum of 2gb of ram.
exec.created += "rctl -a jail:$name:pcpu:deny=200";
exec.created += "rctl -a jail:$name:memoryuse:deny=2g";
And this just removes the rctl when the jail stops.
exec.poststop = "rctl -r jail:$name:";
And finally, on start the jail will execute /etc/rc.local that is just a script to start neverwinter nights EE server
exec.start += "sh /etc/rc.local";
Minecraft Bedrock
This approach is better, is a native FreeBSD jail running Linux emulation to run the minecraft bedrock server. In this case I just bootstrap Ubuntu into the /compat/linux/ubuntu.
minecraft {
persist;
vnet=new;
vnet.interface = "ng0_$name";
host.hostname = "$name";
path=/jails/mcbserver;
mount.fstab="/jails/mcbserver/etc/fstab";
exec.system_user = "root";
exec.jail_user = "root";
exec.prestart += "jng bridge $name vtnet0";
exec.start += "/sbin/ifconfig ng0_$name 192.168.1.235 netmask 255.255.255.0 up";
exec.start += "/sbin/ifconfig lo0 127.0.0.1 up";
exec.start += "/sbin/route add default 192.168.1.201";
exec.start += "/usr/local/etc/rc.d/minecraft_server";
exec.stop += "/bin/sh /etc/rc.shutdown";
exec.created += "rctl -a jail:$name:pcpu:deny=50";
exec.created += "rctl -a jail:$name:memoryuse:deny=1g";
exec.poststop += "jng shutdown $name";
exec.poststop += "rctl -r jail:$name:";
allow.set_hostname;
allow.mount;
allow.mount.devfs;
mount.devfs;
}
/jails/mcbserver/etc/fstab
Device Mountpoint FStype Options Dump Pass#
devfs /jails/mcbserver/compat/ubuntu/dev devfs rw,late 0 0
tmpfs /jails/mcbserver/compat/ubuntu/dev/shm tmpfs rw,late,size=1g,mode=1777 0 0
fdescfs /jails/mcbserver/compat/ubuntu/dev/fd fdescfs rw,late,linrdlnk 0 0
linprocfs /jails/mcbserver/compat/ubuntu/proc linprocfs rw,late 0 0
linsysfs /jails/mcbserver/compat/ubuntu/sys linsysfs rw,late 0 0
~
Finally, this scripts starts the minecraft bedrock server, chrooting into the Linux environment.
The following is assumed:
- The mcbserver account is created
- The bedrock server has been downloaded and installed into mcbserver’s home.
exec.start += "/usr/local/etc/rc.d/minecraft_server";
This is the ugly script, but runs:
#!/bin/sh
chroot /compat/ubuntu /bin/bash << "EOT"
su - mcbserver
screen -d -m ./bedrock_server
EOT
~
WIP …
References
https://people.freebsd.org/~julian/netgraph.html
https://wiki.freebsd.org/TomMarcoen/JailNetworking
https://wiki.freebsd.org/VIMAGE/VNETSamples
https://wiki.freebsd.org/LinuxJails