Getting FreeBSD Jail to Run

Standard

In this short article, I discuss the way to run a full-blown FreeBSD jail, with some common tunings to get it more “normal”.

Assumptions

I assume the host has two network interfaces.  The “vtnet0” is Internet-facing and “vtnet1” is the intranet, where the jail will be running on.  This way, the jail does not have direct access to the Internet.  I also assume we set up the jail at “/root/myjail”.  You can define other locations you want.

Installing a Jail

To run an operating system, one need to install it on some media.  In context of a jail, we install it to a directory.  The FreeBSD handbook tells you to install from source and it is not necessary since BSD system files are mostly tar balls.  Download what you require like this.  For a minimal system, you will need the “base.txz”.  Hmm, no, “kernel.txz” is not required for the jail.

# mkdir /root/myjail
# tar Jxvf base.txz -C /root/myjail

Running a Jail

To run a jail, in particular, with networking, try the following command.

# jail -c path=/root/myjail name=myjail \
  interface=vtnet1 ip4.addr=10.0.250.240 \
  exec.start="/bin/sh /etc/rc"

Where are We?螢幕快照 2017-10-15 下午11.00.02

Here is what it looks like.  No, we are not inside the jail.  The so-called resource configuration script completed execution and we get back to the shell of the host.

Attaching to Jail

Attaching to a jail is simple.  We can simply start one more process in the jail.  (There is no such requirement all processes originate from the same process tree.)

# jexec myjail /bin/tcsh

When finished, you can get out by hanging up the shell (Ctrl-D).  That’s right, you won’t kill the jail by doing this.

Intranet Connection

Inside the jail, try ping other places, like the host IP…

# ping 10.0.250.1
ping: ssend socket: Operation not permitted

To allow ping, you need to change the following, in the host.  If you want to allow this from the beginning, the “allow.raw_sockets=1” option can be passed when the jail is initiated next time.

# jail -m name=myjail allow.raw_sockets=1

Internet Connection

The Intranet now works.  What about going to the Internet?

# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes

Seems there is something wrong with the routing.  Modify the PF rules in the host.  Put this line before the first block / pass statement.  (I assume you have a PF firewall installed.). After you reload the rules (“service pf reload”), it should work.

nat pass vtnet0 from vtnet1:network to any -> (vtnet0)

Domain Name Resolve

Let’s randomly pick a domain name to ping.

# ping yahoo.com
ping: cannot resolve yahoo.com: Host name lookup failure

Ask your network administrators if there are suggested name servers.  If you have nothing in concern, use anything that works.  For my laziness, I use the shortest IP addresses I can recite.  Do this inside the guest:

# cat >> /etc/resolv.conf << EOF
nameserver 8.8.8.8
nameserver 8.8.4.4
EOF

Or, in the host:

# cat >> /root/myjail/etc/resolv.conf << EOF
nameserver 8.8.8.8
nameserver 8.8.4.4
EOF

Listing Processes

System tools like “ps” does not work by default:

# ps
ps: empty file: invalid argument

I think there could be better ways.  But the simplest way is to mount the device file system:

# mount -t devfs devfs /root/myjail/dev

Please note it is not necessary a good idea to people you don’t trust.

Advertisements

Installing FreeBSD without the Installer

Standard

Here are the steps to install FreeBSD without a installer (here is the one using installer, and here is the one appending packages later on).  First of all, one will need a minimum boot media to boot (maybe a USB flash drive) and get into a shell environment.

Creation of the volumes, boot sector and boot code:

Assume you have a fresh SAS drive named /dev/da0 and you want FreeBSD be there.  You first create a GPT partition scheme and then three partitions.  The first is a boot code.  The second is swap.  The third is the root file system.  The latter two file systems are aligned to 1-megabyte boundaries.

Usually, people put the root file system as the second and the swap as the third.  I insist doing the reverse since this allows me to expand the root file system when the disk expands.

gpart create -s gpt /dev/da0
gpart add -t freebsd-boot -s 512K /dev/da0
gpart add -t freebsd-swap -s 2047M -a 1M /dev/da0
gpart add -t freebsd-ufs -a 1M /dev/da0
gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 /dev/da0
newfs -U /dev/da0p3

Network connection:

Making a network connection outside is easy, if your cables are already plugged.  Assume your network card is vtnet0…  (The numbers need to be replaced.)

ifconfig vtnet0 inet 10.0.250.1 net mask 255.255.255.0

Extract the minimum installation files:

Here, I assume you have your “.txz” files ready.  If not, you may want to find it in the same subnet (an existing computer, for example).  Afterwards, we change root into the destination as if we already booted the computer with it.

mount /dev/da0p3 /mnt
cat kernel.txz | tar -Jxvf - -C /mnt
cat base.txz | tar -Jxvf - -C /mnt
chroot /mnt

File system table:

Even if you want to forget all the upcoming steps, this current one is the last one important.  A proper file system table is crucial for FreeBSD to boot.  (This is unlike some other operating system the whereabout of the root file system is hardcoded in some strange place.)

cat > /etc/fstab << EOF
/dev/da0p2 none swap sw 0 0
/dev/da0p3 /    ufs  rw 1 1
EOF

Other configuration files:

Up to your taste, modify as many as you can…

cat > /etc/rc.conf << EOF
hostname=freebsd2
ifconfig_vtnet0=DHCP
ifconfig_vtnet1=inet 10.0.250.1 net mask 255.255.255.0
dumpdev=NO
sendmail_enable=NO
EOF

cat > /boot/loader.conf << EOF
kern.racct.enable=1
EOF

cat > /etc/sysctl.conf << EOF
security.jail.allow_raw_sockets=1
EOF

Boot

Switch off.  Remove the boot media.  Boot again.

Appending Distribution Files after Installing FreeBSD

Standard

Previously, it was discussed how to install FreeBSD with the installer.  In the Question 4, The installer allows administrators to select what distribution to be installed – 32-bit compatibility libraries, source code, debug symbols, etc.

Sometimes, maybe due to a mistaken omission, or maybe due to a new purpose, more distribution files have to be added.  In the good old days of FreeBSD 4.x, I could easily run the “/stand/install” again and let it be reconfigured.  The new installer since 9.x becomes unknown to me and I get to do it myself.

Thankfully, it is much easier than one could have thought of.

Downloading the Files

Downloading the distribution file is relatively simple with FTP.  There is an FTP client coming with the default minimal FreeBSD installation.  From there, we can download the distributions files.  For simplicity, I have skipped the directory listing messages.  The filenames will be self-explanatory as you encounter them.

# ftp -a ftp.freebsd.org
Connected to ftp.geo.freebsd.org.
(Output truncated)
220 This is ftp.geo.freebsd.org - hosted at ISC.org
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> cd pub/FreeBSD/releases
ftp> ls
150 Here comes the directory listing.
(Output truncated)
226 Directory send OK.
ftp> cd amd64
ftp> ls
150 Here comes the directory listing.
(Output truncated)
226 Directory send OK.
ftp> cd 11.0-RELEASE
ftp> ls
150 Here comes the directory listing.
(Output truncated)
226 Directory send OK.
ftp> mget kernel-dbg.txz base-dbg.txz
mget kernel-dbg.txz [anpqy?]? a
Prompting off for duration of mget
229 Entering Extended Passive Mode
150 Opening BINARY mode data connection for kernel-dbg.txz
226 Transfer complete
229 Entering Extended Passive Mode for base-dbg.txz
226 Transfer complete
ftp> exit
221 Goodbye

Installing the Files

If you want to preview what files are inside, you can use “tar tf” command directly, such as…

# tar tf kernel-dbg.tgz
# tar tf base-dbg.tgz

Installing the files is a simple Bzip2 tarball decompression to the root directory.  For example…

# tar jxf kernel-dbg.txz -C /
# tar jxf base-dbg.txz -C /

Here, the “j” stands for Bzip2, “x” stands for decompress, “f” stands for filename, and “C” stands for changing to a given directory (which is the root in our case).

Updating FreeBSD

It is likely the system has been patched since the “release” installation.  To make sure the files you installed match with your updated system, you can consider running the FreeBSD update once.  Please note the commands have to be run on interactive terminals.  Make backups if the system holds files that you cannot lose.

# freebsd-update fetch
# freebsd-update install

Installing without Installer?

Replying questions of the FreeBSD Installer can be boring.  Technically, installing a minimal FreeBSD can be as simple as:

  1. Boot a temporary operating system environment (like live CD)
  2. Partition the drives and install the boot loader (like Question 8 of here)
  3. Download and decompress the distribution files “kernel.txz” and “base.txz”
  4. Configure the essential config files, “/etc/fstab” and “/etc/rc.conf”
  5. Remove any temporary boot media and reboot

Will it work?  Well…

Installing FreeBSD from Scratch and Reinstalling the Boot Loader

Standard

There are cases the default image does not suit for one.  In this exercise, I practice installing FreeBSD version 11 from scratch.  I go beyond the standard procedure by partitioning the drive manually with commands. This is to leave space I can create partitions purely for payload later.   (If you just want to go automatic, you can refer to the FreeBSD handbook.)

Some errors take place so I get to correct the boot loader manually.  If you have tried fixing the boot loader of some other “freedom” operating system, you will appreciate how easy it is!

Inserting the Disc and Boot

Instead of selecting the default boot image, we pick an installation disc.  In Vultr, There are two ways.  The first way is to let the system download the installation disc.  For example, you find a link for the FreeBSD installation disc, copy the URL, and pass it to the interface.  The second way is to reuse the existing library of installation discs.

It takes quite some time for the system to boot.  Depending whether you are lucky or not, you may or may not see the beastie welcome screen.  This is so-called the boot loader, or simply the loader, with just a few tens of kilobytes.

Screen Shot 2017-04-13 at 9.33.34 pm

Inside the Installer

The system boots and the installer (precisely, “bsdinstall”) automatically executes.  From now on, there are a few keystrokes you need to know.  The action buttons, quoted in brackets, can be selected with left and right arrow keys.  To toggle the action button, press enter key.  The items above the action buttons are selected with up and down.  To toggle the item on or off, press spacebar.  At any one time, an action button and a selectable item are highlighted.  When there are multiple fields, press the tab, not enter, to jump between.

Question 1 – mode selection: In the screen below, you can press enter to run the installer.  You can alternatively press right arrow to select the shell, then enter to run the shell.  Here we select “install” directly.

Screen Shot 2017-04-13 at 9.34.07 pm

Question 2 – keymap: If you want to select an alternative keymap, use up and down arrow keys, and press spacebar to select.  Then, press enter to confirm.

Screen Shot 2017-04-13 at 9.34.17 pm

Question 3 – hostname: You are going to enter a hostname.  If you are creating a machine to be cloned, you can pick a generic name.

Question 4 – distributions: You are asked what distribution components to select.  Usually I just pick “lib32” only.  By default, they propose installing “ports”, I deselect it (with spacebar) most of the time.  The updated ports can be downloaded by “postsnap” command later.

Partitioning and Formatting the Drive

Question 5 – partition method: You are given several ways to partition, the “auto” one are the most easy but they may generate something you do not like.  The “manual” shows a dialog where you can create the partitions yourself, but not control the partition alignments.  So let us select “shell”.

Screen Shot 2017-04-13 at 9.35.40 pm.png

Question 6 – partition: You are given a shell and instructed to type in commands, edit a file, and mount the effective file system.  Use the following commands to partition the only virtual hard drive, “vtbd0”, and then install the bootloader.

Screen Shot 2017-04-13 at 9.35.50 pm

# gpart show
# gpart create -s gpt /dev/vtbd0
vtbd0 created
# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40  52428720       - free - (25G)

# gpart add -t freebsd-boot -a 512K -s 512K /dev/vtbd0
vtbd0p1 added
# gpart add -t freebsd-swap -a 1M -s 2047M /dev/vtbd0
vtbd0p2 added
# gpart add -t freebsd-ufs -a 1M -s 5120M /dev/vtbd0
vtbd0p3 added
# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40       984       - free - (492K)
      1024      1024     1 freebsd-boot (512K)
      2048   4192256     2 freebsd-swap (2.0G)
   4194304  10485760     3 freebsd-ufs (5.0G)
  14680064  37748696       - free - (18.0G)
# gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 /dev/vtbd0
bootcode written to /dev/vtbd0

Previous step, we partition the drive into three, a boot partition, a swap partition, and a unix file system partition.  We install the GPT boot loader into the boot partition.  Then, format the last partition, define the file system table as previously instructed, then we are done.  The installer starts installation without a question asked.

# newfs -U /dev/vtbd0p3
(message truncated)

# mount /dev/vtbd0p3 /mnt
# cat >> /tmp/bsdinstall_etc/fstab << EOF
/dev/vtbd0p2 none swap sw 0 0
/dev/vtbd0p3 /    ufs  rw 1 1
EOF

# exit

Screen Shot 2017-04-13 at 9.59.14 pm

Final Touches to the Installation

Question 7 – root password: Pick and enter a password carefully, twice.

Question 8 – network configuration: You are asked what network devices you like to configure.  Select the only virtual network device, “vtnet0”.  Enable IPv4 and DHCP.  Disable IPv6 (unless you know why not).

Question 9 – name resolver configuration: Simply press “ok” for the DNS configuration.  The DNS server setting will be overridden soon.

Question 10 – time zone selection: Select the continent you are in, and then the city.  You are then asked if the abbreviation is appropriate, and confirm the system date and time.

Question 11 – services: I would select “local_unbound”, “sshd”, and “ntpd”.

Screen Shot 2017-04-13 at 10.01.51 pm

Question 12 – security: Since version 11, the FreeBSD installer asks if the user wants any additional security measures.  I think most of them can be enabled, except the debugging.  (This is because I do debug programs.)

Screen Shot 2017-04-13 at 10.03.21 pm

Question 13 – additional users: This is up to you.  I prefer customisation before user creation.

Question 14 – final configuration: Just skip…

Question 15 – final modification: Just skip…

Question 16 – what next: Instead of rebooting, I prefer going to the live CD mode, login and “poweroff”.

Remaining Activities

Take a snapshot before booting the system again.  On the first system boot, the SSH generates its identities.  If you want multiple hosts having their distinct identities, taking the snapshot before the first boot is the laziest and the most correct way.

Last but not least, remove the virtual optical drive image.  Then you are good to boot from the virtual hard drive.

Troubleshooting and Fixing the Boot Loader

Missing boot loader: When generating the screenshots, I forgot to install the boot code.  The boot screen looks like this and is stuck.  This is a sign of missing the boot loader.  I booted with the installation disc again, then choose shell mode, and finally rerun the “gpart bootcode” command.

Screen Shot 2017-04-13 at 10.05.10 pm

# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40       984       - free - (492K)
      1024      1024     1 freebsd-boot (512K)
      2048   4192256     2 freebsd-swap (2.0G)
   4194304  10485760     3 freebsd-ufs (5.0G)
  14680064  37748696       - free - (18G)
# gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 /dev/vtbd0
bootcode written to /dev/vtbd0

Damaged file system table: On the next boot attempt, I drop into single user mode because of bad file system table.  This was because I wrote “rw” instead of “sw” for the swap.  I then corrected the “/etc/fstab” with an editor.  Then I “exit” to continue the boot.

Screen Shot 2017-04-13 at 10.11.27 pm.png

Security Settings

For you reference, the security options I made in installation turns out to be the following.  So they can be incorporated in other installation tools, without actually running the “bsdinstall”.

/etc/rc.conf

clear_tmp_enable="YES"
syslogd_flags="-ss"
local_unbound_enable="YES"

/etc/sysctl.conf

security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.stack_guard_page = 1

/etc/resolv.conf

nameserver 127.0.0.1
options edns0

To be Continued

In the upcoming articles, I will use the snapshots created here to build a highly available block device, and then highly available file systems and database systems.