Getting FreeBSD Jail to Run


In this short article, I discuss the way to run a full-blown FreeBSD jail, with some common tunings to get it more “normal”.


I assume the host has two network interfaces.  The “vtnet0” is Internet-facing and “vtnet1” is the intranet, where the jail will be running on.  This way, the jail does not have direct access to the Internet.  I also assume we set up the jail at “/root/myjail”.  You can define other locations you want.

Installing a Jail

To run an operating system, one need to install it on some media.  In context of a jail, we install it to a directory.  The FreeBSD handbook tells you to install from source and it is not necessary since BSD system files are mostly tar balls.  Download what you require like this.  For a minimal system, you will need the “base.txz”.  Hmm, no, “kernel.txz” is not required for the jail.

# mkdir /root/myjail
# tar Jxvf base.txz -C /root/myjail

Running a Jail

To run a jail, in particular, with networking, try the following command.

# jail -c path=/root/myjail name=myjail \
  interface=vtnet1 ip4.addr= \
  exec.start="/bin/sh /etc/rc"

Where are We?螢幕快照 2017-10-15 下午11.00.02

Here is what it looks like.  No, we are not inside the jail.  The so-called resource configuration script completed execution and we get back to the shell of the host.

Attaching to Jail

Attaching to a jail is simple.  We can simply start one more process in the jail.  (There is no such requirement all processes originate from the same process tree.)

# jexec myjail /bin/tcsh

When finished, you can get out by hanging up the shell (Ctrl-D).  That’s right, you won’t kill the jail by doing this.

Intranet Connection

Inside the jail, try ping other places, like the host IP…

# ping
ping: ssend socket: Operation not permitted

To allow ping, you need to change the following, in the host.  If you want to allow this from the beginning, the “allow.raw_sockets=1” option can be passed when the jail is initiated next time.

# jail -m name=myjail allow.raw_sockets=1

Internet Connection

The Intranet now works.  What about going to the Internet?

# ping
PING ( 56 data bytes

Seems there is something wrong with the routing.  Modify the PF rules in the host.  Put this line before the first block / pass statement.  (I assume you have a PF firewall installed.). After you reload the rules (“service pf reload”), it should work.

nat pass vtnet0 from vtnet1:network to any -> (vtnet0)

Domain Name Resolve

Let’s randomly pick a domain name to ping.

# ping
ping: cannot resolve Host name lookup failure

Ask your network administrators if there are suggested name servers.  If you have nothing in concern, use anything that works.  For my laziness, I use the shortest IP addresses I can recite.  Do this inside the guest:

# cat >> /etc/resolv.conf << EOF

Or, in the host:

# cat >> /root/myjail/etc/resolv.conf << EOF

Listing Processes

System tools like “ps” does not work by default:

# ps
ps: empty file: invalid argument

I think there could be better ways.  But the simplest way is to mount the device file system:

# mount -t devfs devfs /root/myjail/dev

Please note it is not necessary a good idea to people you don’t trust.


Network File System with Firewall in FreeBSD


Network file system nfs(8) in FreeBSD is built on top of rpc(3) infrastructure where rpcbind(8) daemon is responsible binding the services for the clients.  Together with the companion services rpc.lockd(8)rpc.statd(8), and mountd(8), providing the total service with a firewall is tricky since the ports are different all the time.  In addition, sometimes the services are better to be available to the intranet only, not the internet.

Server Flags

Here is how to lock these services to the non-changing ports and non-wildcard addresses in the rc.conf(5).  In the example, I assumed the IP address to provide the service as “”:

rpc_lockd_flags="-h -p 2632"
rpc_statd_flags="-h -p 2633"
mountd_flags="-h -p 2634"
nfs_server_flags="-h -t -u"

Network Ports to Open

Here are the ports to open for the aforementioned services, both UDP and TCP.  The firewall can be adjusted accordingly:

Installing FreeBSD without the Installer


Here are the steps to install FreeBSD without a installer (here is the one using installer, and here is the one appending packages later on).  First of all, one will need a minimum boot media to boot (maybe a USB flash drive) and get into a shell environment.

Creation of the volumes, boot sector and boot code:

Assume you have a fresh SAS drive named /dev/da0 and you want FreeBSD be there.  You first create a GPT partition scheme and then three partitions.  The first is a boot code.  The second is swap.  The third is the root file system.  The latter two file systems are aligned to 1-megabyte boundaries.

Usually, people put the root file system as the second and the swap as the third.  I insist doing the reverse since this allows me to expand the root file system when the disk expands.

gpart create -s gpt /dev/da0
gpart add -t freebsd-boot -s 512K /dev/da0
gpart add -t freebsd-swap -s 2047M -a 1M /dev/da0
gpart add -t freebsd-ufs -a 1M /dev/da0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 /dev/da0
newfs -U /dev/da0p3

Network connection:

Making a network connection outside is easy, if your cables are already plugged.  Assume your network card is vtnet0…  (The numbers need to be replaced.)

ifconfig vtnet0 inet net mask

Extract the minimum installation files:

Here, I assume you have your “.txz” files ready.  If not, you may want to find it in the same subnet (an existing computer, for example).  Afterwards, we change root into the destination as if we already booted the computer with it.

mount /dev/da0p3 /mnt
cat kernel.txz | tar -Jxvf - -C /mnt
cat base.txz | tar -Jxvf - -C /mnt
chroot /mnt

File system table:

Even if you want to forget all the upcoming steps, this current one is the last one important.  A proper file system table is crucial for FreeBSD to boot.  (This is unlike some other operating system the whereabout of the root file system is hardcoded in some strange place.)

cat > /etc/fstab << EOF
/dev/da0p2 none swap sw 0 0
/dev/da0p3 /    ufs  rw 1 1

Other configuration files:

Up to your taste, modify as many as you can…

cat > /etc/rc.conf << EOF
ifconfig_vtnet1=inet net mask

cat > /boot/loader.conf << EOF

cat > /etc/sysctl.conf << EOF


Switch off.  Remove the boot media.  Boot again.

Simple Experiment with Jails and Resource Control


While it has been a good practise to have one computer doing one role, it has also been  asked if a computer can be split for multiple functions as if it is a group of computers.  One would immediately answer virtual machines, but it can be too heavy and complicated, say like having virtual machines inside the virtual machines you ordered in a public cloud.  Another would say container and cgroup in L…, em, what?  In other operating systems, we have other more battle-proven stuffs like zones in Solaris and Jails in FreeBSD.  Jails in FreeBSD had not been too attractive to me because it obviously lacked the resource control so that out-of-control processes can make resource starvation for others.  Thankfully, since FreeBSD 9.0, the rctl(8) has made the world a fairer place.

How simple is it to make a jail and try resource control, and gain confidence it works?  The suggested method in the FreeBSD handbook is to make a directory and reinstall FreeBSD there with the appropriate distribution files.  It is a generic method but it can be more inspiring.  Let’s begin.  In this article, we will demonstrate writing a dummy program, instantiate it in a jail, and fiddle the resource control.

System Setup

First of all, in order to use the resource control, add the following line to the loader.conf(5) /boot/loader.conf.  The file could be missing on a fresh installation, you can create a new one if it is the case.  After this, reboot to make it effective.


The Program and Compilation

Let us go to do some programming.  We want a dummy program that consumes some memory and processing power.  Here is my randomly crafted C program.  It is deliberately running some nonsense loops and leaking some memory.  Once every 10 seconds, it allocates 16 MB (malloc) of memory and uses 1 MB (memset) of it.  (The remaining 15 MB is lazily allocated and forgotten.)  By watching the lines printed, you get a sense how fast the process runs.  The longer it runs, the more memory it consumes.

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

int main(int argc, char** argv) {
  time_t then, now;
  int i, j, allocated;
  void* region;

  i = j = allocated = 0;
  while (1) {
    while(now - then < 10) {
      for (i = 0; i < (1 << 25); ++i)
        j *= i;
      printf("now: %ld\n", now);
    then = now;
    region = malloc(1 << 24);
    memset(region, j % 23, (1 << 20));
    allocated += 16;
    printf("allocated: %d\n", allocated);
  return 0;

Usually, the program can be compiled simply with clang(1) command.  In other to compile for the jail in the most lazy method, call clang with the “-static” option.  The executable is significantly larger.  Mark down the absolute path of the directory so you can do the next step.

# ls
# clang resource.c -o resource -static -Wall
# ls
resource resource.c
# ldd resource
ldd: resource: not a dynamic ELF executable
# pwd

Running inside a Jail

Then, it’s time to start the jail.  The simplest way is to create a whatever jail(1), encapsulate at the path at the ssching’s home, and execute the executable we compiled.

# jail -c -u ssching path=/home/ssching \
  exec.start='./resource' name=myjail

Open another terminal and issue the top(1) command.  Press key “j” once and you see a process with non-zero jail number.  We see the process is consuming 100% CPU power and the memory usage is increasing.

2648    14  ... 8288K 872K CPU1  1 0:09 100.67% resource

Stopping a Jail

Stopping a jail, attached or not, can be as simple as “-r” command.  Upon execution, all the processes in the jail will be killed.

# jail -r myjail

Processor Resource Control

Open yet another terminal and issue the following commands.

# rctl -a jail:myjail:pcpu:deny=50

The usage becomes a bit tamed but it fluctuates a lot from time to time.

2648    14 ...  392M 25544K CPU3  3 3:52  50.83% resource

Memory Resource Control

Similarly, the virtual memory usage, or physical memory usage can be constrained as follows.  The virtual memory counts the total memory (SIZE in the top display) allocated, the physical memory counts the actual memory used (RES in the top display).

# rctl -a jail:myjail:vmemoryuse=240M
# rctl -a jail:myjail:memoryuse=120M

If memory usage is beyond the allowed, the system refuses to allocate more memory.  Given the simple design of the program code above, it breaks with signal 11.

now: 1505838082
now: 1505838083
jail: ./resource: exited on signal 11

Resource Control Rules

To check what resource control rules are applied, simply a “rctl” command will do.

# rctl 

Clearing Resource Rules

Clearing the existing rules can be as simple as using the “-r” command, for example:

# rctl -r jail:myjail:memoryuse
# rctl -r jail:myjail
# rctl -r jail:

Actual Resource Used

Last but not the last, usage of a jail can be listed.

# rctl -u jail:myjail

Other Thoughts

In fact, the resource control can be also applied per process, per user, per group, etc.  The command is mostly similar, just replace “jail” with “process”, “user”, “group”, etc.  Say, a computer is being shared among a few friends.  Even if they do not want to be in jails (who wants?  Pun intended…), their resource usage can be constrained and accounted by the command.

In addition to the processor and memory usage, the resource control can also put limitations on other items like disk read / write (hint: use the word throttle instead of deny), which is useful when the IOPs is counted in a public cloud…

In the next article, we will explore how to put a complicated program, with dynamic linked libraries, into a jail.

Proxy Server with FreeBSD and Squid (Part 2)


Previously, I discussed how to configure a Squid proxy.  The proxy is opaque that the web browsers have to be configured.  I continue to explain how a proxy can be made transparent; when web browsers go to the Internet, the requests gets intercepted and be processed by the proxy.  Like before, I use PF firewall and let it redirect the packets for me.

Step 1: Configure Network Gateway

In order to configure a a network router, it needs to have two network interfaces, virtual or physical.  One of them connects to the external world (through another router, maybe).  Another one connects to the intranet.  In PF, it is recommended to set up macros to determine the external and internal interfaces.  An example rule set will be as follows, where a Realtek was used as external and a Broadcom as internal.

nat pass on $extif from $intif:network to any -> ($extif)
pass in quick from $intif:network to any
pass out quick

In order for a FreeBSD server act as a router, it has to have the gateway variable enabled in /etc/rc.conf:


Once these are configured, reload the firewall rules for a smoke test.  Good luck.

Step 2: Configure Network Clients

Pick a computer and configure its network traffic through the router.  Technically, we change the gateway.

Microsoft Windows: Control Panel > Network and Sharing Centre > Network Interfaces > Properties > TCP/IP Version 4 > Configure > Gateway

Mac OS X: System Preference > Network > Gateway

FreeBSD: Update variable “defaultrouter” in /etc/rc.conf, then reboot

Everything should behave similar, except the network goes through the router.  Hopefully, the network link LEDs could give you some hints.  (Sorry being lazy not telling the proper way…)

Step 3: Packet Redirection and Squid

In PF configuration, add this line right after the NAT rule, and then reload:

rdr pass on $intif proto tcp from any to any port 80 -> ($intif) port 3129

In Squid configuration, add this line right after the original http port statement:

http_port 3129 intercept

I may explain what ‘intercept’ mode means in the next article…

Step 4: Testing

Use the client configured in the step 2 to browse the web.  Like last time, there should be some pages cached.  But make sure you visit pages that are not encrypted (like https); otherwise the proxy will not take effect.

Step 5: To be Continued

In the part 3 of this series, I will explain how to to intercept HTTPS connections as well.

Highly Available Web Pages, Apache and PHP as an Example


Several weeks ago, I discussed how to have a highly available file storage and a highly available relational database.  With a robust supply of file system and database service, we can stack more services on top.  Today, I take Apache and PHP as an example how to have a highly available web server.  As usual, I use FreeBSD for the purpose.

Why Apache and PHP!?

Quite some people would argue Apache and PHP are outdated.  I am not going to make a comment on this.  I hope you will appreciate the shortness of this article because of this choice.  The concepts you acquire from this example can be applied with your favourite platform.

Active / Active Web Servers

The example today is active / active pair.  The two servers can serve webpages together without error.  In fact, one can have unlimited amount of hosts working together.  In contrast, in the previous examples of NFS, only one server is active and another server is passively receiving updates and standby to take over.

To make this happen, one needs to ensure intermediate state of executions of one servers are saved so they can be taken up by the other servers.  In this example, we will move these state to the highly available storage.  With role separation, those intermedia data will not be lost no matter what happens to the web servers.  Another benefit is that web servers can be easily added (handle more load), replaced (for system updates), or reduced (for economy) without affecting any user sessions.

Session Data

When running a cluster of web hosting servers, one needs to take care of the session data.  HTTP servers are stateless by itself.  Cookies (notes to Europeans) are saved on the client side so servers need not to worry about.  Sessions storage, provided by most web programming environments, expect the storage on the server is consistent and persistent for each user session.

Session data is useful storing information that should not be understood or modify the the web users.  This could be some intermediate game states, some login privileges, etc.  It is therefore important to make sure such data is not lost when a user encounters another web server in the cluster.


At the time of writing, Apache 2.4 and PHP 7.1 are the newest.

In order to run the PHP in the most lazy way, install “mod_php” packages.  In addition, install some common PHP modules.  There is a moderate collection called “php71-extensions”.  I would also install “php71-mysqli” and “php71-gd”.  They are for database connections and image processing respectively.

# pkg install mod_php71 php71-mysqli php71-gd

What about Apache?  It is installed automatically as an dependency when you request installing the “mod_php”.

Shared Directories

To build on top of the previous example, I am mounting the NFS prepared previously to /mnt/nfs.  On the shell, overtime you want to mount:

# mkdir /mnt/nfs
# mount /mnt/nfs

Alternatively, in the “/etc/fstab”:

# mkdir /mnt/nfs
# cat >> /etc/fstab << EOF   /mnt/nfs    nfs    rw    0    0
# mount /mnt/nfs

Configuring the Apache

Modify Apache so that the web page and script locations are in the NFS share.  The file to be modified is “/usr/local/etc/apache24/httpd.conf”.  There are originally two directories for normal web pages and CGI pages in “/usr/local/www/apache24”.  We are moving it to “/mnt/nfs”.

# sed -ibak 's|/usr/local/www/apache24|/mnt/nfs|' \
# cp -a /usr/local/www/apache24/cgi-bin /mnt/nfs/
# cp -a /usr/local/www/apache24/data /mnt/nfs/
# cat >> /etc/rc.conf << EOF

Also, update the “/usr/local/etc/apache24/httpd.conf” so that it decodes PHP files.  Modify the following parts manually:

<IfModule dir_module>
    DirectoryIndex index.html index.php

<FilesMatch "\.php$">
    SetHandler application/x-httpd-php
<FilesMatch "\.phps$">
    SetHandler application/x-httpd-php-source

Configuring the PHP

PHP requires only copying the default configuration, and then modifying the session path.  One can also modify the upload path similarly, but I do not think it is necessary.

# cd /usr/local/etc/
# cp php.ini-production php.ini
# vi php.ini

The line to be added is just as follows:

;session.save_path = "/tmp"
session.save_path = "/mnt/nfs/tmp"

Example Code

To make effect, reboot Apache24.  Here is a simple code that prints a counter every time it is loaded.  Write a file “/mnt/nfs/data/counter.php”

if (isset($_SESSION['counter']))
  $_SESSION['counter'] += 1;
  $_SESSION['counter'] = 1;


In order to test the service, we first turn on the web servers.  Then, we alternatively point a domain name to either one of the IP addresses.  The counter code above is repetitively executed as the map changes.  If successful, the counter continues increasing.

The laziest way to change the domain name mapping is of course editing the host table where the web browser is run.  For BSD and BSD-like systems, edit “/etc/hosts”.  For Windows, edit “C:\Windows\System32\Drivers\etc\hosts”.

Upcoming Steps

Once there are multiple web servers ready, one can consider having a load balancer or a content distribution network.  A load balancer service can be found in some value-added cloud service providers.  (Or else, you can set up one yourself.)  A content distribution network can be found anywhere around the globe.  Put aside their fundamental difference, they both accept multiple web server addresses and route network traffic accordingly.


Getting a blank page of PHP:  If you follow my instructions, you have configured PHP in a production mode.  Errors messages are not displayed but saved to the Apache log, which is “/var/log/httpd-error.log” in our context.  Read it and you will get an idea.

Function “session_start” undefined:  Make sure you have installed the package “php71-session” and restarted Apache.  It should be installed as an dependency when you install the package “php71-extensions”.

Appending Distribution Files after Installing FreeBSD


Previously, it was discussed how to install FreeBSD with the installer.  In the Question 4, The installer allows administrators to select what distribution to be installed – 32-bit compatibility libraries, source code, debug symbols, etc.

Sometimes, maybe due to a mistaken omission, or maybe due to a new purpose, more distribution files have to be added.  In the good old days of FreeBSD 4.x, I could easily run the “/stand/install” again and let it be reconfigured.  The new installer since 9.x becomes unknown to me and I get to do it myself.

Thankfully, it is much easier than one could have thought of.

Downloading the Files

Downloading the distribution file is relatively simple with FTP.  There is an FTP client coming with the default minimal FreeBSD installation.  From there, we can download the distributions files.  For simplicity, I have skipped the directory listing messages.  The filenames will be self-explanatory as you encounter them.

# ftp -a
Connected to
(Output truncated)
220 This is - hosted at
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> cd pub/FreeBSD/releases
ftp> ls
150 Here comes the directory listing.
(Output truncated)
226 Directory send OK.
ftp> cd amd64
ftp> ls
150 Here comes the directory listing.
(Output truncated)
226 Directory send OK.
ftp> cd 11.0-RELEASE
ftp> ls
150 Here comes the directory listing.
(Output truncated)
226 Directory send OK.
ftp> mget kernel-dbg.txz base-dbg.txz
mget kernel-dbg.txz [anpqy?]? a
Prompting off for duration of mget
229 Entering Extended Passive Mode
150 Opening BINARY mode data connection for kernel-dbg.txz
226 Transfer complete
229 Entering Extended Passive Mode for base-dbg.txz
226 Transfer complete
ftp> exit
221 Goodbye

Installing the Files

If you want to preview what files are inside, you can use “tar tf” command directly, such as…

# tar tf kernel-dbg.tgz
# tar tf base-dbg.tgz

Installing the files is a simple Bzip2 tarball decompression to the root directory.  For example…

# tar jxf kernel-dbg.txz -C /
# tar jxf base-dbg.txz -C /

Here, the “j” stands for Bzip2, “x” stands for decompress, “f” stands for filename, and “C” stands for changing to a given directory (which is the root in our case).

Updating FreeBSD

It is likely the system has been patched since the “release” installation.  To make sure the files you installed match with your updated system, you can consider running the FreeBSD update once.  Please note the commands have to be run on interactive terminals.  Make backups if the system holds files that you cannot lose.

# freebsd-update fetch
# freebsd-update install

Installing without Installer?

Replying questions of the FreeBSD Installer can be boring.  Technically, installing a minimal FreeBSD can be as simple as:

  1. Boot a temporary operating system environment (like live CD)
  2. Partition the drives and install the boot loader (like Question 8 of here)
  3. Download and decompress the distribution files “kernel.txz” and “base.txz”
  4. Configure the essential config files, “/etc/fstab” and “/etc/rc.conf”
  5. Remove any temporary boot media and reboot

Will it work?  Well…