Network Booting FreeBSD


Installing an operating system is easy for a computer or two.  What about installing for multiples of them?  In some situations, one wants to save money and time from buying that many boot devices and maintain the disk images separately.  Network boot comes handy.  It is relatively easy with FreeBSD, especially since we do not need whatever RAM disk image just to load whatever drivers.

The idea is, we have one computer holding the disk image, and let other computers obtain the image from there and start their journeys.  We first let these computers obtain their network information.  Together with the network configuration, these computers also get the location to download the network boot loader.  With the network boot loader, they will able to mount a network file system.

I tried to avoid GPL software.  Otherwise I would have tried something more integrated.  Anyway, that software piece got discovered huge vulnerabilities recently…


I will assume you have your cloud environment, with a network isolated from elsewhere.  For example, in Virtual Box, create a “host-only” network and plug in your computers (vtnet1,  To make dynamic IP address allocation possible, one will want to enable promiscuous mode of the interfaces.  Of course, in addition, one will need a out-going “NAT” network for internet access (vtnet0).  I leave other details as readers’ exercise.

Operating System Files

Similar to setting up a full-blown jail or installing without an installer, one needs to deploy a directory to hold the operating system files.  In this exercise, I randomly picked a directory “/compute”.

# mkdir /compute
# tar Jxf base.txz -C /compute
# tar Jxf kernel.txz -C /compute

Dynamic Host Configuration

Typical computers nowadays can be configured to boot from the network in the BIOS.  The first thing such computers do is to of course obtain network configuration.  Therefore, the dynamic host configuration server is the first thing we need to set.  In this practice, I used the OpenBSD DHCP Daemon.  The usage is similar to that described in the handbook.

# pkg install dhcpd

The file dhcpd.conf(5) “/usr/local/etc/dhcpd.conf” is as follows.  In short, it allows IP addresses be allocated for 7200 seconds per lease.  And for the subnet, the addresses to can be allocated to the computers dynamically.  The default gateway will be  More importantly, when the computer boots, it should obtain the file “/boot/pxeboot” (from the DHCP server, by default), and mount the NFS location “” as root file system.

default lease-time 7200;
max-lease-time 7200;
subnet netmask {
  option routers;
  option rootpath ""
  filename "/boot/pxeboot"

And the rc.conf(5) file “/etc/rc.conf” appended as follows.  As you guess it right, just one statement to enable the DHCP server.


Trivial File Transfer

By no means I say the file transfer protocol is trivial.  The so-called “trivial file transfer” is yet another protocol for transferring file without complicated handshakes.  Here I used the package “tftp-hpa”.

# pkg install tftp-hpa

By default, it uses the directory “/usr/local/tftp”.  But we can act lazy.  The rc.conf(5) gets appended as follows:

tftpd_flags="-s /compute"

If you concern the security, you should make a single directory (like the default /usr/local/tftp) and copy the file “/compute/boot/pxeboot” there.  Right, one file will do.

Network File System

To let the system boot, we prepare a network file system, and make the /compute accessible.  The exports(5) file “/etc/exports” look as follows:

/compute -network -alldirs -maproot=root

And the rc.conf(5) is appended as follows.  You may want to refer here if you want to run it behind a firewall.


Starting the Services

As a friendly reminder, you will need to enable and start the services – the DHCP daemon, the TFTP daemon, and the network file system.  You will need proper firewall rules to allow network traffic.  In particular, you will need UDP port 67, 68, and 69 for DHCP and TFTP to work.

How it Works

DHCP Requests: At the beginning, the computer tries to obtain network configuration by broadcasting requests.  If you are stuck here, try to fix the DHCP daemon.

螢幕快照 2017-10-20 上午12.21.16

TFTP for Boot Loader: After obtaining the network configuration, the computer tries to obtain the boot loader with TFTP.  If you are stuck here, fix the TFTP configurations.

螢幕快照 2017-10-20 上午12.23.35

Network File System: With the first stage boot loader ready, it tries to mount the network file system.  If you are stuck here, check the network file system configurations.

螢幕快照 2017-10-20 上午12.22.14

The Loader: When you see this screen, most of the services, DHCP, FTFP, and NFS, are already used.  If you are stuck beyond this point, read the error message, and good luck…

螢幕快照 2017-10-20 上午12.23.52


Simple Experiment with Jails and Resource Control


While it has been a good practise to have one computer doing one role, it has also been  asked if a computer can be split for multiple functions as if it is a group of computers.  One would immediately answer virtual machines, but it can be too heavy and complicated, say like having virtual machines inside the virtual machines you ordered in a public cloud.  Another would say container and cgroup in L…, em, what?  In other operating systems, we have other more battle-proven stuffs like zones in Solaris and Jails in FreeBSD.  Jails in FreeBSD had not been too attractive to me because it obviously lacked the resource control so that out-of-control processes can make resource starvation for others.  Thankfully, since FreeBSD 9.0, the rctl(8) has made the world a fairer place.

How simple is it to make a jail and try resource control, and gain confidence it works?  The suggested method in the FreeBSD handbook is to make a directory and reinstall FreeBSD there with the appropriate distribution files.  It is a generic method but it can be more inspiring.  Let’s begin.  In this article, we will demonstrate writing a dummy program, instantiate it in a jail, and fiddle the resource control.

System Setup

First of all, in order to use the resource control, add the following line to the loader.conf(5) /boot/loader.conf.  The file could be missing on a fresh installation, you can create a new one if it is the case.  After this, reboot to make it effective.


The Program and Compilation

Let us go to do some programming.  We want a dummy program that consumes some memory and processing power.  Here is my randomly crafted C program.  It is deliberately running some nonsense loops and leaking some memory.  Once every 10 seconds, it allocates 16 MB (malloc) of memory and uses 1 MB (memset) of it.  (The remaining 15 MB is lazily allocated and forgotten.)  By watching the lines printed, you get a sense how fast the process runs.  The longer it runs, the more memory it consumes.

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

int main(int argc, char** argv) {
  time_t then, now;
  int i, j, allocated;
  void* region;

  i = j = allocated = 0;
  while (1) {
    while(now - then < 10) {
      for (i = 0; i < (1 << 25); ++i)
        j *= i;
      printf("now: %ld\n", now);
    then = now;
    region = malloc(1 << 24);
    memset(region, j % 23, (1 << 20));
    allocated += 16;
    printf("allocated: %d\n", allocated);
  return 0;

Usually, the program can be compiled simply with clang(1) command.  In other to compile for the jail in the most lazy method, call clang with the “-static” option.  The executable is significantly larger.  Mark down the absolute path of the directory so you can do the next step.

# ls
# clang resource.c -o resource -static -Wall
# ls
resource resource.c
# ldd resource
ldd: resource: not a dynamic ELF executable
# pwd

Running inside a Jail

Then, it’s time to start the jail.  The simplest way is to create a whatever jail(1), encapsulate at the path at the ssching’s home, and execute the executable we compiled.

# jail -c -u ssching path=/home/ssching \
  exec.start='./resource' name=myjail

Open another terminal and issue the top(1) command.  Press key “j” once and you see a process with non-zero jail number.  We see the process is consuming 100% CPU power and the memory usage is increasing.

2648    14  ... 8288K 872K CPU1  1 0:09 100.67% resource

Stopping a Jail

Stopping a jail, attached or not, can be as simple as “-r” command.  Upon execution, all the processes in the jail will be killed.

# jail -r myjail

Processor Resource Control

Open yet another terminal and issue the following commands.

# rctl -a jail:myjail:pcpu:deny=50

The usage becomes a bit tamed but it fluctuates a lot from time to time.

2648    14 ...  392M 25544K CPU3  3 3:52  50.83% resource

Memory Resource Control

Similarly, the virtual memory usage, or physical memory usage can be constrained as follows.  The virtual memory counts the total memory (SIZE in the top display) allocated, the physical memory counts the actual memory used (RES in the top display).

# rctl -a jail:myjail:vmemoryuse=240M
# rctl -a jail:myjail:memoryuse=120M

If memory usage is beyond the allowed, the system refuses to allocate more memory.  Given the simple design of the program code above, it breaks with signal 11.

now: 1505838082
now: 1505838083
jail: ./resource: exited on signal 11

Resource Control Rules

To check what resource control rules are applied, simply a “rctl” command will do.

# rctl 

Clearing Resource Rules

Clearing the existing rules can be as simple as using the “-r” command, for example:

# rctl -r jail:myjail:memoryuse
# rctl -r jail:myjail
# rctl -r jail:

Actual Resource Used

Last but not the last, usage of a jail can be listed.

# rctl -u jail:myjail

Other Thoughts

In fact, the resource control can be also applied per process, per user, per group, etc.  The command is mostly similar, just replace “jail” with “process”, “user”, “group”, etc.  Say, a computer is being shared among a few friends.  Even if they do not want to be in jails (who wants?  Pun intended…), their resource usage can be constrained and accounted by the command.

In addition to the processor and memory usage, the resource control can also put limitations on other items like disk read / write (hint: use the word throttle instead of deny), which is useful when the IOPs is counted in a public cloud…

In the next article, we will explore how to put a complicated program, with dynamic linked libraries, into a jail.

OpenStack with FreeBSD


Recently, I come across NFV Express through the FreeBSDNews article.  The installation code is already openly available on Github.  At the first glance, the code is so amazing to be clean!  (You definitely should look at them if you know programming.)  When there is time, I will try them with my stack of FreeBSD computers…

Openstack is used to be Linux-centric, yet it is mostly written in Python scripts.  Theoretically, it is possible to make it running on any capable platforms… if you do not hate working with Python scripts.

Sorry for this short article.  This is indeed just a reminder for me to follow up.  🙂

Update on 4 Apr 2017: the tool requires Xen to be installed.  I tried installing it on the new computer but UEFI is not supported.  I am going to schedule another older computer to try.