Highly Available Network File System

Standard

In the previous article, we discussed the way to create a highly available block device by replication.  We continue and attempt making a network file system (NFS) on top of it.  We first discuss the procedures to start and stop the service.  Then we have the script…  Some parts are deliberately missing due to my conflict of interest.

NFS Configuration

Since it is not our goal here, we only do minimal NFS configuration in this example.  In short, the export(5) file “/etc/exports” is being modified like as follows.  This implies the directory “/nfs” is shared with the given two IP subnets.

/nfs -network=10.65.10.0/24
/nfs -network=127.0.0.0/24

Unlike previous setting, we do not use the “/etc/rc.conf” file to start the service.  This is because we like to control when a service is started, instead of blindly just after boot.  In FreeBSD, services can be started with the “onestart” command.

Firewall Configuration

Configuring NFS for a tight firewall is tricky, because it uses random ports.  For convenience, a simple IP address-based whitelist can be implemented.  In this example, we have the server IP 10.65.10.13 (see later), and the client IP 10.65.10.21.  If you simply do not have a firewall, skip this part.  On the server side, the PF can be configured with:

pass in quick on vtnet1 from 10.65.10.21 to 10.65.10.13 keep state

On the client side, the PF can be configured with:

pass in quick on vtnet1 from 10.65.10.13 to 10.65.10.21 keep state

Starting the Service

When we start the service, we want the following to happen:

  1. Acquire the IP address, say 10.65.10.13, regardless which machine it is running.
  2. Activate the HAST subsystem so to become the primary role.
  3. Wait for the HAST device to be available.  If the device is in secondary role, the device file in “/dev/hast” will not appear so we can go to sleep a while.
  4. Run the file system check just in case the file system was corrupted in the last unmount.
  5. Mount the file system for use (in this example, “/nfs”)
  6. Start the NFS-related services in order: the remote procedural call binding daemon, the mount daemon, the network file system daemon, the statistic daemon, and the lock daemon.
  7. Once the step 5 completes, the service is available to the given clients as instructed to the NFS and allowed by the firewall.

For the inpatient, one can jump to the second last section for the actual source code.

Stopping the Service

Stopping the service is the reverse of starting, except some steps can be less serious.

  1. Stop the NFS-related services in order: the lock daemon, the statistic daemon, the network file system daemon, the mount daemon, and finally the remote procedural call binding daemon.
  2. Unmount the file system.
  3. Make the HAST device in secondary role.
  4. Release the iP address so neighbours can reuse.

Also, one can jump to the second last section for the actual source code.

Service Check Script

There are two types of checking.  The first one ensures all the components (like the IP address, mount point service, etc) are present and valid.  The procedure returns success (zero) only when the components are all turned on.  Whenever a component is missing, it will be reported as a failure (non-zero return code).

The second one ensures all the components are simply turned off, so that the service can be started on elsewhere.  The procedure returns success (zero) only when all the components are turned off.  Whenever a component is present, it will be reported as a failure (non-zero return code).

What is Missing

Once we master how to start and stop the service on one node, we need the mechanism to automatically start and stop the service as appropriate.  In particular, it is utmost important not to run the service concurrently on two hosts, as this may damage the file system and confuse the TCP/IP network.  This part should be done out of the routine script.

The Script

Finally, the script is as follows…

#!/bin/sh -x

start() {
  ifconfig vtnet1 add 10.65.10.13/24
  hastctl role primary nfs_block
  while [ ! -e /dev/hast/nfs_block ]
  do
    sleep 1
  done
  fsck -t ufs /dev/hast/nfs_block
  mount /dev/hast/nfs_block /nfs
  service rpcbind onestart
  service mountd onestart
  service nfsd onestart
  service statd onestart
  service lockd onestart
}

stop() {
  service lockd onestop
  service statd onestop
  service nfsd onestop
  service mountd onestop
  service rpcbind onestop
  umount /nfs
  hastctl role secondary nfs_block
  ifconfig vtnet1 delete 10.65.10.13
}

status() {
  ifconfig vtnet1 | grep 10.65.10.13 && \
  service rpcbind onestatus && \
  showmount -e | grep /nfs && \
  mount | grep /nfs && \
  ls /dev/hast/nfs_block
}

residue() {
  ifconfig vtnet1 | grep 10.65.10.13 || \
  (service rpcbind onestatus && showmount -e | grep /nfs) || \
  mount | grep /nfs || \
  ls /dev/hast/nfs_block
}

clean() {
  residue
  if [ $? -ne 0 ]
  then
    exit 0
  fi
  exit 1
}

if [ "$1" == "start" ]
then
  start
elif [ "$1" == "stop" ]
then
  stop
elif [ "$1" == "status" ]
then
  status
elif [ "$1" == "clean" ]
then
  clean
fi

Testing

To test, fine the designated computer and mount the file system.  Assume the file system has been running on the host “store1”, make a manual failover to see…  The file client does not need to explicitly remount the file system; it cab be remounted automatically.

client# mount 10.65.10.13:/nfs /mnt
client# ls /mnt
.snap
client# touch /mnt/helloworld
store1# ./nfs_service.sh stop
store2# ./nfs_service.sh start
client# ls /mnt
.snap helloworld

Highly Available Storage Target on FreeBSD

Standard

To achieve highly available service, it is vital to have the latest data available for restarting the service elsewhere.  In enterprise environments, multipath SAS drives allows each drive to be accessible from multiple hosts.  What about the rest of us, living in the cloud of / or inexpensive SATA drives?  Highly Available Storage Target (HAST) is the answer.  It relatively a new feature in FreeBSD 8.1.  It is useful to keep two copies of drive content on two loosely coupled computers.

In this article, I demonstrate how HAST can be setup, without bothering the actual failover logic.  I assume the two virtual machines are prepared from scratch with some unallocated space.  Alternatively, you can prepare a virtual machine with multiple drives (which is not quite feasible in my setting).

Preparing the Partitions

First, examine the drives.  Here we have 18 (no, indeed 17.9 something) gigabytes of space available.

# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40       984       - free - (492K)
      1024      1024     1 freebsd-boot (512K)
      2048   4192256     2 freebsd-swap (2.0G)
   4194304  10485760     3 freebsd-ufs (5.0G)
  14680064  37748696       - free - (18G)

Then, we can add two more partitions.  Since we have two hosts, we set two partitions so that each host can run the service on one partition.  This is so-called active-active setup.  I dedicate one for NFS and one for database.  Once you finish working on one virtual machine, do not forget performing the same on another machine.

# gpart add -t freebsd-ufs -l nfs_block -a 1M -s 8960M /dev/vtbd0
vtbd0p4 added.
# gpart add -t freebsd-ufs -l db_block -a 1M -s 8960M /dev/vtbd0
vtbd0p5 added.

This is the result.  Two block devices are created and made available inside /dev/gpt with their appropriate labels.

# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40       984       - free - (492K)
      1024      1024     1 freebsd-boot (512K)
      2048   4192256     2 freebsd-swap (2.0G)
   4194304  10485760     3 freebsd-ufs (5.0G)
  14680064  18350080     4 freebsd-ufs (8.8G)
  33030144  18350080     5 freebsd-ufs (8.8G)
  51380224   1048536       - free -  (512M)
# ls /dev/gpt
db_block nfs_block

HAST Daemon Setup

Here is an sample of defining the HAST configuration, hast.conf(5).  In short, there are two hosts and two resource items.  The host “store1” has its remote partner “store2” and vice versa.  Since we use the short host names “store1” and “store2”, do not forget to update the host(5) file to make them resolvable.  Remember to repeat these another machine.  Thankfully, the HAST configuration need not to be customised for each member host.

# sysrc hastd_enable="YES"
# cat > /etc/hast.conf << EOF
resource nfs_block {
  on store1 {
    local /dev/gpt/nfs_block
    remote store2
  }
  on store2 {
    local /dev/gpt/nfs_block
    remote store1
  }
}
resource db_block {
  on store1 {
    local /dev/gpt/db_block
    remote store2
  }
  on store2 {
    local /dev/gpt/db_block
    remote store1
  }
}
EOF
# service hastd start

Firewall Rules

If you are having a firewall, remember to open the port number 8457 opened.  For example, in PF, add these three lines to the two hosts.  Remember the replace the IP addresses as appropriate.

geompeers="{10.65.10.11,10.65.10.12}"
geomports="{8457}"
pass in quick inet proto tcp from $geompeers to any port $geomports keep state

HAST Daemon Status

Once the HAST daemon is started, the status of the blocks can be checked.  Since we have defined two resource items, there are two HAST device status reported.  For example, in the host “store1”, it says there are two components for each resource item: one block device, and one remote host.  At first, the resource items are in “initialisation” state:

store1# hastctl status
Name      Status   Role      Components
nfs_block -        init      /dev/gpt/nfs_block store2
db_block  -        init      /dev/gpt/db_block store2

To turn on the device for operation, use the “role” subcommand on “store1”

store1# hastctl role primary db_block
store1# hastctl status
Name      Status   Role      Components
nfs_block -        init      /dev/gpt/nfs_block store2
db_block  degraded primary   /dev/gpt/db_block store2

Similarly, use the “role” command on “store2”, but this time we set it secondary:

store2# hastctl role secondary db_block
store2# hastctl status
Name      Status   Role      Components
nfs_block -        init      /dev/gpt/nfs_block store1
db_block  -        secondary /dev/gpt/db_block store1

When the synchronisation completes, the status is marked complete:

store2# hastctl status
Name      Status   Role      Components
nfs_block -        init      /dev/gpt/nfs_block store1
db_block  complete secondary /dev/gpt/db_block store1

Formatting the Block Devices

We can format the block device like usual, just that we should format the device under the device directory “/dev/hast” instead of “/dev/gpt”.  Since currently the “db_block” is active on the host “store1”, it has to be executed over there.

store1# newfs -J /dev/hast/db_block
/dev/hast/db_block: 8960.0MB (18350064 sectors) block size 32768, fragment size 4096
using 15 cylinder groups of 626.09MB, 20035 blks, 80256 inodes.
super-block backups (for fsck_ffs -b #) at:
192, 1282432, 2564672, 3846912, 5129152, 6411392, 7693632, 8975872, 10258112, 11540352, 12822592, 14104832, 15387072, 16669312, 17951552

One thing to note is, the raw device was 18350080 sectors.  When HAST takes it for service, there is only 18350064 blocks left for payload.  Next, we can mount the file system.  We do not use the fstab(5) like before, because they do not need to execute every time boot.

store1# mkdir /db
store1# mount /dev/hast/db_block /db

Switching Over

In order to switch over, the procedure is as follows.

store1# umount /db
store1# hastctl role secondary db_block
store2# hastctl role primary db_block
store2# mkdir /db
store2# mount /dev/hast/db_block /db

What if, in an error, the backend device in “/dev/gpt” is being used for mounting?  It will say the following.  The chance to go wrong is really not heavy.

store2# mount /dev/gpt/db_block /db
/dev/gpt/db_block: Operation not permitted

For automatic switching, it will be discussed in a separated article.

Troubleshooting

Nothing can go really wrong without a serious application on top.  Nevertheless, the following message troubled me for a few hours.

We act as primary for the resource and not as secondary as secondary“: check the HAST configuration.  Likely a host is configured to have itself as the remote partner.

To be Continued

In the upcoming articles, I will cover how to make use of this highly-available block storage for shared file system and database.

Installing FreeBSD from Scratch and Reinstalling the Boot Loader

Standard

There are cases the default image does not suit for one.  In this exercise, I practice installing FreeBSD version 11 from scratch.  I go beyond the standard procedure by partitioning the drive manually with commands. This is to leave space I can create partitions purely for payload later.   (If you just want to go automatic, you can refer to the FreeBSD handbook.)

Some errors take place so I get to correct the boot loader manually.  If you have tried fixing the boot loader of some other “freedom” operating system, you will appreciate how easy it is!

Inserting the Disc and Boot

Instead of selecting the default boot image, we pick an installation disc.  In Vultr, There are two ways.  The first way is to let the system download the installation disc.  For example, you find a link for the FreeBSD installation disc, copy the URL, and pass it to the interface.  The second way is to reuse the existing library of installation discs.

It takes quite some time for the system to boot.  Depending whether you are lucky or not, you may or may not see the beastie welcome screen.  This is so-called the boot loader, or simply the loader, with just a few tens of kilobytes.

Screen Shot 2017-04-13 at 9.33.34 pm

Inside the Installer

The system boots and the installer (precisely, “bsdinstall”) automatically executes.  From now on, there are a few keystrokes you need to know.  The action buttons, quoted in brackets, can be selected with left and right arrow keys.  To toggle the action button, press enter key.  The items above the action buttons are selected with up and down.  To toggle the item on or off, press spacebar.  At any one time, an action button and a selectable item are highlighted.  When there are multiple fields, press the tab, not enter, to jump between.

Question 1 – mode selection: In the screen below, you can press enter to run the installer.  You can alternatively press right arrow to select the shell, then enter to run the shell.  Here we select “install” directly.

Screen Shot 2017-04-13 at 9.34.07 pm

Question 2 – keymap: If you want to select an alternative keymap, use up and down arrow keys, and press spacebar to select.  Then, press enter to confirm.

Screen Shot 2017-04-13 at 9.34.17 pm

Question 3 – hostname: You are going to enter a hostname.  If you are creating a machine to be cloned, you can pick a generic name.

Question 4 – distributions: You are asked what distribution components to select.  Usually I just pick “lib32” only.  By default, they propose installing “ports”, I deselect it (with spacebar) most of the time.  The updated ports can be downloaded by “postsnap” command later.

Partitioning and Formatting the Drive

Question 5 – partition method: You are given several ways to partition, the “auto” one are the most easy but they may generate something you do not like.  The “manual” shows a dialog where you can create the partitions yourself, but not control the partition alignments.  So let us select “shell”.

Screen Shot 2017-04-13 at 9.35.40 pm.png

Question 6 – partition: You are given a shell and instructed to type in commands, edit a file, and mount the effective file system.  Use the following commands to partition the only virtual hard drive, “vtbd0”, and then install the bootloader.

Screen Shot 2017-04-13 at 9.35.50 pm

# gpart show
# gpart create -s gpt /dev/vtbd0
vtbd0 created
# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40  52428720       - free - (25G)

# gpart add -t freebsd-boot -a 512K -s 512K /dev/vtbd0
vtbd0p1 added
# gpart add -t freebsd-swap -a 1M -s 2047M /dev/vtbd0
vtbd0p2 added
# gpart add -t freebsd-ufs -a 1M -s 5120M /dev/vtbd0
vtbd0p3 added
# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40       984       - free - (492K)
      1024      1024     1 freebsd-boot (512K)
      2048   4192256     2 freebsd-swap (2.0G)
   4194304  10485760     3 freebsd-ufs (5.0G)
  14680064  37748696       - free - (18.0G)
# gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 /dev/vtbd0
bootcode written to /dev/vtbd0

Previous step, we partition the drive into three, a boot partition, a swap partition, and a unix file system partition.  We install the GPT boot loader into the boot partition.  Then, format the last partition, define the file system table as previously instructed, then we are done.  The installer starts installation without a question asked.

# newfs -U /dev/vtbd0p3
(message truncated)

# mount /dev/vtbd0p3 /mnt
# cat >> /tmp/bsdinstall_etc/fstab << EOF
/dev/vtbd0p2 none swap sw 0 0
/dev/vtbd0p3 /    ufs  rw 1 1
EOF

# exit

Screen Shot 2017-04-13 at 9.59.14 pm

Final Touches to the Installation

Question 7 – root password: Pick and enter a password carefully, twice.

Question 8 – network configuration: You are asked what network devices you like to configure.  Select the only virtual network device, “vtnet0”.  Enable IPv4 and DHCP.  Disable IPv6 (unless you know why not).

Question 9 – name resolver configuration: Simply press “ok” for the DNS configuration.  The DNS server setting will be overridden soon.

Question 10 – time zone selection: Select the continent you are in, and then the city.  You are then asked if the abbreviation is appropriate, and confirm the system date and time.

Question 11 – services: I would select “local_unbound”, “sshd”, and “ntpd”.

Screen Shot 2017-04-13 at 10.01.51 pm

Question 12 – security: Since version 11, the FreeBSD installer asks if the user wants any additional security measures.  I think most of them can be enabled, except the debugging.  (This is because I do debug programs.)

Screen Shot 2017-04-13 at 10.03.21 pm

Question 13 – additional users: This is up to you.  I prefer customisation before user creation.

Question 14 – final configuration: Just skip…

Question 15 – final modification: Just skip…

Question 16 – what next: Instead of rebooting, I prefer going to the live CD mode, login and “poweroff”.

Remaining Activities

Take a snapshot before booting the system again.  On the first system boot, the SSH generates its identities.  If you want multiple hosts having their distinct identities, taking the snapshot before the first boot is the laziest and the most correct way.

Last but not least, remove the virtual optical drive image.  Then you are good to boot from the virtual hard drive.

Troubleshooting and Fixing the Boot Loader

Missing boot loader: When generating the screenshots, I forgot to install the boot code.  The boot screen looks like this and is stuck.  This is a sign of missing the boot loader.  I booted with the installation disc again, then choose shell mode, and finally rerun the “gpart bootcode” command.

Screen Shot 2017-04-13 at 10.05.10 pm

# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40       984       - free - (492K)
      1024      1024     1 freebsd-boot (512K)
      2048   4192256     2 freebsd-swap (2.0G)
   4194304  10485760     3 freebsd-ufs (5.0G)
  14680064  37748696       - free - (18G)
# gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 /dev/vtbd0
bootcode written to /dev/vtbd0

Damaged file system table: On the next boot attempt, I drop into single user mode because of bad file system table.  This was because I wrote “rw” instead of “sw” for the swap.  I then corrected the “/etc/fstab” with an editor.  Then I “exit” to continue the boot.

Screen Shot 2017-04-13 at 10.11.27 pm.png

Security Settings

For you reference, the security options I made in installation turns out to be the following.  So they can be incorporated in other installation tools, without actually running the “bsdinstall”.

/etc/rc.conf

clear_tmp_enable="YES"
syslogd_flags="-ss"
local_unbound_enable="YES"

/etc/sysctl.conf

security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.stack_guard_page = 1

/etc/resolv.conf

nameserver 127.0.0.1
options edns0

To be Continued

In the upcoming articles, I will use the snapshots created here to build a highly available block device, and then highly available file systems and database systems.

OpenStack with FreeBSD

Standard

Recently, I come across NFV Express through the FreeBSDNews article.  The installation code is already openly available on Github.  At the first glance, the code is so amazing to be clean!  (You definitely should look at them if you know programming.)  When there is time, I will try them with my stack of FreeBSD computers…

Openstack is used to be Linux-centric, yet it is mostly written in Python scripts.  Theoretically, it is possible to make it running on any capable platforms… if you do not hate working with Python scripts.

Sorry for this short article.  This is indeed just a reminder for me to follow up.  🙂

Update on 4 Apr 2017: the tool requires Xen to be installed.  I tried installing it on the new computer but UEFI is not supported.  I am going to schedule another older computer to try.

Proxy Server with FreeBSD and Squid (Part 1)

Standard

When one has multiple machines within a cloud network, it is natural to ask for centralised network traffic, data files, credential service, etc.  In this article, I focus on centralising network traffic, in particular, the world wide web.

With the world wide web traffic centralised, the proxy server can accelerate the download processes of some frequently accessed files, such as operating system patches.

In this part, we install Squid as an opaque proxy server, and configure some clients to use it.  We will handle the transparent proxy in the next part.

Step 1: Install Squid

Installing Squid is easy as usual.  But to make it even smoother, install the SSL root certificates.

# pkg install squid ca_root_nss

The squid requires Perl which, since FreeBSD 5.0, no longer installed by default.  The package installer will handle it.

Step 2: Smoke Test

Even though there are quite a page of notice upon installing Squid, the default Squid installation is good enough to fire without configuration:

# sysrc squid_enable=YES
# service squid start

or, for the old style…

# echo squid_enable=YES >> /etc/rc.conf
# /usr/local/etc/rc.d/squid start

Step 3: Configure Squid

The Squid refers to its configuration file in “/usr/local/etc/squid”.

# ee /usr/local/etc/squid/squid.conf

 

What shall we configure?  Here are some suggestions:

  1. Configure the intranet address ranges.  By the configuration file, only networks defined as “localnet” is granted access.  For example, I only use the class A private subnet, so I disable all others.
    acl localnet src 10.0.0.0/8     
    #acl localnet src 172.16.0.0/12 
    #acl localnet src 192.168.0.0/16
    #acl localnet src fc00::/7      
    #acl localnet src fe80::/10
  2. Deny the squid accessing the so-called “localhost”.  Internal network of the squid is not supposed to be accessed by others.
    http_access deny to_localhost
  3. Configure the cache directory.  For example, here I configure a 10000 MB cache space.  There are two levels of directories, with 16 and 256 directories per level respectively.
    cache_dir ufs /var/squid/cache 10000 16 256
  4. Prefer IPv4 whenever possible.  This is great if your network is not IPv6 ready.
    dns_v4_first on

We then stop the Squid, create the cache directory, and restart.

# service squid stop
# squid -z -N
# chown -R squid:squid /var/squid/cache
# service squid start

Step 4: Update Firewall

If you are using PF firewall, you will need to update the rules to allow the port.  For example, I only use the “vtnet1” as intranet, and it is accepting port 3128 (the default squid service port).  You may refer to the configuration of the VPN notes as example.

# cat >> /etc/pf.conf << EOF
intif="vtnet1"
inttcpports="{3128}"
pass in quick on $intif inet proto tcp from any to any port $inttcpports keep state
EOF
# service pf reload

Step 5: Playing with Telnet

The title is not a typo.  We now test the system with the most geeky way—enter the HTTP request manually.  FreeBSD comes with a telnet client and it is handy for various testing.  Suppose the address is 10.65.0.1 and we connect to the port 3128 of it.  After typing the GET statement, remember to hit the enter key twice to complete the request.

# telnet 10.65.0.1 3128
GET http://www.freebsd.org/ HTTP/1.0

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Tue, 07 Mar 2017 16:04:30 GMT
Content-Type: text/html
Content-Length: 178
Location: https://www.freebsd.org/
X-Cache: MISS from singapura-roteador
X-Cache-Lookup: HIT from singapura-roteador:3128
Via: 1.1 singapura-roteador (squid/3.5.23)
Connection: close

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
Connection closed by foreign host.

The HTTP return code 301 indicates another URL for access.  What about following it?

# telnet 10.65.0.1 3128
GET https://www.freebsd.org/ HTTP/1.0

Step 6: Troubleshoot

Here are some errors you may encounter:

Unable to connect to the server: try disabling the “pf” service briefly and retry.  If it works, you will need to update your firewall rules.

Squid is not starting after enabling cache: make sure you create the cache directory and set the proper ownership with the chown command (as above) .

Getting 503 error with ERR_CONNECT_FAIL: review the error message and if the address is an IPv6 address, there could be issues with the IPv6 routing and you can consider using the “dns_v4_first” option (as above).

Getting 503 error with ERR_SECURE_CONNECT_FAIL: review the website with your own browser, ensure you have installed the common root certificates “ca_root_nss” package (as above).

Step 7: Configure Client

Since our proxy is not transparent yet, the clients have to be manually configured in order to use the proxy.  Depending on the hosts you have on the same intranet, here are the configuration methods for your reference:

Mac OS X: System Preferences > Network > Advanced > Proxies > Web Proxy / Secure Web Proxy

Internet Explorer on Windows: Tools > Internet Options > Connections > LAN Setting

Firefox on Windows: Tools > Internet Options > Network > Connections > Settings

FreeBSD shell: export “HTTP_PROXY” variable in “.cshrc”, like “setenv HTTP_PROXY http://10.65.0.1:3128&#8221;

Step 8: Is the Proxy Really Running?

There are some barbaric ways to check if the proxy is running:

Check the number of files the cache: “find /var/squid/cache | wc -l”

Check the Squid access log: “less /var/log/squid/access.log”

Step 9: To be Continued

A few weeks later, we will revisit Squid and configure the network routers to transparently direct world wide web requests to the proxy server, etc.

Later this week, we may come up with an article not related with FreeBSD.  The next FreeBSD-related article, hopefully next week, will be likely related to programming.  Stay tuned.

Details of Configuring FreeBSD

Standard

We have come across some of my preferred customisations.  In this article, I explain how the files are being modified and their relationships with the system.  It is hoped that this will give new comers an idea why and what to configure with a fresh FreeBSD installation.

What to Configure?

To understand how FreeBSD can be configured, we first understand the parts that can be configured.  The startup process is roughly as follows and I will cover configuration of each stage in later sections, arranged according to the likeness one will use.

  1. The computer loads and execute the loader (more on step 6)
  2. The loader mounts according to the file system table (more on step 5)
  3. The loader loads the kernel into memory and execute (more on step 4)
  4. The init program is executed as the first program (not covered)
  5. The device file system is mounted (not covered)
  6. The rc program prepares the user space (more on step 1)
  7. Network neighbours are recognised (more on step 2)
  8. The periodic jobs are run (more on step 3)

Step 0: Modifying Files

Modifying a configuration file could be easy, but there are some geeky ways to do it right.

To read a configuration file, one uses the cat(1) command.  Before some fluffy haters jump out and complain, let me say, cat means concatenate.  That said, the concatenation tool is also useful dumping content of a single file.

Quite some FreeBSD configuration files are in name-value pairs, such as:

# cat /etc/rc.conf
hostname="raquel"

In these cases, the sysrc(8) tool acts as a safe shortcut to modify the files.  For example:

# sysrc -f /etc/rc.conf hostname="lea"
# cat /etc/rc.conf
hostname="lea"

You can use the same cat to pipe in the output.  The >> symbol means appending to a file; contents following the command will be appended.  (If you use > instead, you overwrite it instead.)  The << defines the termination string, the termination string will not be inserted and it will be returned to the shell.

# cat >> /etc/rc.conf << EOF
hostname="raquel"
EOF
# cat /etc/rc.conf
hostname="lea"
hostname="raquel"

Since these are just name-value pairs, it does not harm more than causing your own confusions.  The latest value assignment takes effect.

Last but not the least, for the files you want to manually edit, you can always use the ee(1) easy editor, or the visually (difficult) vi(1) editor.

Step 1: Run Command

To a new user, the most likely item to configure first is the run command.  At later stage of system start, the rc(8) starts the desired processes for the users.  Most of these configuration are done in two files.  The values desired by the user goes to “/etc/rc.conf”.  When a value required is absent, the default value is loaded from “/etc/defaults/rc.conf”.  The former is empty or even absent after a fresh system install.  The latter is installed and updated automatically and is very lengthy.  By the way, the lengthy file can also act as your configuration guide; more on these later.

Step 1.1: Network Configuration

FreeBSD uses ifconfig(8) to configure network interfaces.  For example, to configure network addresses of interface vtnet1 (the second virtual network interface), the default gateway and the hostname, one can use:

# sysrc vtnet1_ifconfig="inet 10.65.0.11 netmask 255.255.255.0"
vtnet1_ifconfig:  -> inet 10.65.0.11 netmask 255.255.255.0
# sysrc defaultrouter="10.65.0.1"
defaultrouter: NO -> 10.65.0.1
# sysrc hostname="myhost"
hostname:  -> myhost

In the example above, vtnet1 device exists because it exists in the hardware layer.  Sometimes, the interfaces are purely software, like the gif(4) interface we mentioned in building virtual network.  They can be generated in boot time with the cloned interface.  Note there is a plus sign before the equal.  This appends new values to the existing values.  Once the interfaces are created, they can be configured with the typical “ifconfig” above.

# sysrc cloned_interfaces+="gif0"
cloned_interfaces:  -> gif0
# sysrc cloned_interfaces+="gif1"
cloned_interfaces: gif0 -> gif0 gif1

Step 1.2: Service Configuration

The run command file allows defining what services to be switched on at boot.  These services will be switched off gracefully at shutdown as well.  For example, to enable the sshd(8) daemon, one will write this in “/etc/rc.conf”.

sshd_enable="YES"

As an unofficial but nevertheless useful way, one can search for “_enabled” in the “/etc/defaults/rc.conf” to see what can be enabled or disabled:

# grep _enabled /etc/defaults/rc.conf
apm_enable="NO" # Set to YES to enable APM BIOS functions (or NO).
apmd_enable="NO" # Run apmd to handle APM event from userland.
ddb_enable="NO" # Set to YES to load ddb scripts at boot.
devd_enable="YES" # Run devd, to trigger programs on device tree changes.
kldxref_enable="NO" # Build linker.hints files with kldxref(8).
powerd_enable="NO" # Run powerd to lower our power usage.
...

In short, some interesting services are:

  • ntpd(8): network time daemon, keep system time synchronised
  • pf(4): packet filter (or, firewall), useful for protecting your system
  • sendmail(1): mail server, in the previous page I recommend disabling it
  • sshd(8): secure shell server, essential for you to login remotely

Step 1.3: Startup Script

One more place to configure run command is to directly type them into “/etc/rc.local”.  Indeed, this was how FreeBSD 4.x or before worked.  Every time a service was installed, the administrator opened an editor and input the required startup scripts there.  It is still the preferred way for some other BSD family operating systems.

Step 2: Host Tables and Resolvers

When you have multiple machines connected in a network, you will want alias to them.  FreeBSD provides a hosts(5) table “/etc/hosts”.  As long as you can handle and synchronise the files on the different hosts, this is the most reliable and straight-forward method.  (If not, you will need a centralised place, like a name server, for this purpose, and it is out of my scope today.)

One more network configuration is the domain name service, transforming domain names to IP addresses which are understandable to the network.  It is configured with the resolv.conf(5) in “/etc/resolv.conf”.  If you are using a cloud, it is usually automatically configured with DHCP.

Step 3: Periodic Jobs

There are quite a few ways to let jobs run periodically.  I cover the maintenance and cron jobs here.  The former is more common to execute some system default cleanup activities.  The latter is more common for administrators’ customisations.  In the previous article, I changed some maintenance job settings and used a cron job to periodically clean the network offenders (in case those are false positives).

Step 3.1: Maintenance Jobs

FreeBSD has periodic(8) for running periodic maintenance jobs.  The settings are in “/etc/periodic.conf” and the default settings are in “/etc/defaults/periodic.conf”.

The file is read whenever the periodic job is run.  After updating the periodic configuration, you can just let the next periodic job to run.  Nothing special is needed to reload the configuration.

By default, the periodic jobs output are sent to the local mail server.  The step 2 of the customisation script disables these mails and direct the output to the file.  This saves you a few daily emails per server and provides one less attack vector.

Step 3.2: Cron Jobs

FreeBSD has cron(8) for running arbitrary commands on given time.  There is a master cron job file available as plain crontab(5) table “/etc/crontab”, and a crontab(1) database for each user.  Readers may feel strange the two items are both called crontab, with different numbers in the suffix.  In short, the (5) is a file, and (1) is a command.

When editing the crontab(5), the format is as follows:

minute hour day month weekday user command args...

For example, the default FreeBSD comes with cron jobs that run the periodic jobs mentioned in the previous subsection.  This means at every 03:01, command “periodic daily” is executed as the root user; at every 4:15 on Saturday, command “periodic weekly” is executed as the root user, etc.

 1 3 * * *  root  periodic daily
15 4 * * 6  root  periodic weekly
30 5 1 * *  root  periodic monthly

When editing with crontab(1), one uses “crontab -e” to edit his own list.  This gives the user advantage of defining his own jobs without modifying the master table.  The format is as follows, which is similar, yet the username is skipped for the obvious reason.

minute hour day month weekday command args...

Please note in the crontabs, it sometimes lacks the conventional environment variables.  In particular, you may need to be more specific in pathnames, such as calling “/sbin/pfctl” instead of plainly “pfctl”.

Step 4: Kernel System Settings

FreeBSD provides sysctl(8) utility for configuring kernel and kernel module settings, and reading some system status as well.

Kernel settings can be configured in “/etc/sysctl.conf”, with a default file in “/etc/defaults/sysctl.conf”.  The files are read on boot time.  After adding new system configurations, you can either reboot to make it in effect, or run this command to reload immediately: “sysctl -f /etc/sysctl.conf”

These variables are useful when a user wants to have more control on the kernel behaviour.  More on these can be found on the manual page tuning(7).  You can list them by calling “sysctl -a”.  The output is long, prepare your terminal to scroll back.

To a cloud server, most of the activities are network transfer; some interesting configurations are:

  • net.inet.tcp.sendspace: bytes of initial space for sending for each TCP session
  • net.inet.tcp.sendbuf_inc: bytes of sending space increment per session
  • net.inet.tcp.sendbuf_max: bytes of maximum space for receiving per session
  • net.inet.tcp.recvspace: bytes of initial space for receiving for each TCP session
  • net.inet.tcp.recvbuf_inc: bytes of receiving space increment for session
  • net.inet.tcp.recvbuf_max: bytes of maximum space for receiving per session
  • kern.ipc.nmbclusters: total amount of space for network memory (in 2 KiB blocks)
  • kern.ipc.somaxconn: maximum pending connections

By default, the network space values make sense.  The sending space and receiving space are 32 KiB and 64 KiB respectively.  This incurs 96 KiB when a connection is made.  These can go up to 2 MiB for sending and another 2 MiB for receiving.  The system reserves around 100 MiB network memory by default, which is enough for even for gigabit network.

If you find a real need, you can consider adjusting these values depending on the nature of the server.  These values can be lowered if the server has a lot of connections serving small files and messages.  They can be raised if the server has only a few connections and each serving large files and messages.  If the server has a lot of connections each serving large files, the total amount of space for network memory has to be increased.  You can check the buffer usage with command “netstat -m”.

The maximum pending connections, is set to be 128 as default value.  That means, if there are 129 incoming connections together and the server software is not yet capable of completing any of the handshakes, the 129th will be lost.  It is suggested to be set to 1024 or even higher.  This will prevent the system being jammed easily by denial-of-service attack.  If you have applied the customisation script, a client will be blocked if it tries to make too many connections in a short time.

Step 5: File System Table

FreeBSD uses file system table fstab(5) to define the file systems to be mounted.  The table is read by the loader (below) when the system boots.  (A file is read from the volume before the volume is officially mounted.  This is quite counter-intuitive if you think again.  But it make things much simpler.)  If there are errors, one may end up locking the system from booting.

The file format is as follows:

block mount fstype options dump pass

For example, one of my virtual machine says:

/dev/ufs/rootfs  /  ufs  rw  1  1

This means the UFS volume labeled “rootfs” to be mounted at the root directory /, as an UFS file system, read and write enabled, dump enabled, and checked for consistency first after rebooting from system crash.

Normally, this file is generated when you have a virtual machine ready.  It is useful to modify it later when you insert disks to a virtual (or physical) machine.

Step 6: Loader Settings

Eventually, a user may want to configure how a kernel is loaded, and what kernel modules to be loaded together.  FreeBSD comes with a simple yet efficient loader(8) for the purpose.  The configuration values are in “/boot/loader.conf” and defaults in “/boot/defaults/loader.conf”.  I recommend against changing these files randomly because they really affect how the kernel is loaded.  Some settings I ever changed are:

  • kern.maxproc: maximum processes in a system
  • kern.maxfiles: maximum open file handles (including sockets) in a system
  • geom_mirror_load: load the software RAID module (for booting the kernel)
  • coretemp_load: load the module to detect Intel Core™ processor temperature
  • amdtemp_load: load the module to detect AMD processor temperature

The maximum processes, and maximum open file handles are set according to the memory available in the machine.  They are suggested to be set to a predefined value if you have a lot of open files or network sockets, or when the machine is to be shared by a lot of users.  The process table and file table are initialised once and only once when the kernel loads, thus they are to be set in here instead of sysctl.

 

Step 7: Skeleton Files

Few accounts use only the default “root” user account.  To have consistent personalisation across various created users, administrators often prepare skeleton files.  The skeleton files are located at “/usr/share/skel”.  For example, “/usr/share/skel/dot.cshrc” will be copied to “$HOME/.cshrc” of the newly created user.

Once the account is created, the file is owned by the created user.  There are not automatic ways to update across multiple users.  It is therefore important to have the customisations right before creating the users.

Conclusion

We have gone through various parts in FreeBSD that can be configured, especially in context to a new comer.  We discussed how to configure the rc.conf, the host table, the resolver configuration, placing periodic jobs, adjusting kernel settings, setting file system mount points, kernel loading options, and finally skeleton files.

Virtual Private Network with FreeBSD (Part 2)

Standard

This is the second and the final part on setting up a private virtual network.  The part 1 has been uploaded about one week ago.

If you are following the guide in FreeBSD handbook, you must be frustrated because of the complicated configurations.  Those configurations are good in the sense it restricts the IPsec to particular IP pairs but they are also too tough for purpose of connecting just two routers.  In this example, I break these assumptions and go as minimal as possible so the readers can have a smoother learning curve.  I protect the private ports from unsolicited connections with the help of PF firewall instead of the IPsec configuration.

Step 7: Install Packages

Install “ipsec-tools” package on the routers.

pkg install ipsec-tools

Step 8: Configure IPsec Rules

Set the rules in IPsec in japao-roteador.  These two lines say:

  1. IP packets, in any protocols, from the Japan network (10.81.0.0) to the Singapore network (10.65.0.0) is sent out encapsulated in tunnel mode, from the japao-roteador (45.76.207.156) to cingapura-roteador (47.76.133.70), with the encryption mandatory.
  2. IP packets, in any protocols, from the Singapore network to the Japan network is accepted in encapsulated in tunnel mode, from cingapura-roteador to japao-roteador, with the encryption mandatory.
    # touch /usr/local/etc/racoon/setkey.conf
    # cat >> /usr/local/etc/racoon/setkey.conf << EOF
    spdadd 10.81.0.0/24 10.65.0.0/24 any -P out ipsec esp/tunnel/45.76.207.156-45.76.153.70/require;
    spdadd 10.65.0.0/24 10.81.0.0/24 any -P in ipsec esp/tunnel/45.76.153.70-45.76.207.156/require;
    EOF

On another router cingapura-roteador, the in and out are reversed for the obvious reason:

  1. Incoming: the Japan network to the Singapore network
  2. Outgoing: the Singapore network to the Japan network
  3. # touch /usr/local/etc/racoon/setkey.conf
    # cat >> /usr/local/etc/racoon/setkey.conf << EOF
    spdadd 10.81.0.0/24 10.65.0.0/24 any -P in ipsec esp/tunnel/45.76.207.156-45.76.153.70/require;
    spdadd 10.65.0.0/24 10.81.0.0/24 any -P out ipsec esp/tunnel/45.76.153.70-45.76.207.156/require;
    EOF

Please note I used the word “require” instead of “use” as written by some other documentations.  It means the message must be in the expected form to be accepted.  If the encryption is turned off accidentally, it should be disregarded rather than passing through.

Step 9: Configure Racoon

Racoon, as installed with the “ipsec-tools” package, is going to help us the IPsec negotiation phase.  Setting up the pre-shared keys is as easy as updating a file, with each line of the IP address, and then a secret string.  The file will be referenced by the Racoon script.  Setting up the Racoon main configuration can be much more straight forward than the handbook.  I copied an example in “/usr/local/share/ipsec-tools” and customised it.  The two routers will have very similar configuration except the listening IP.

  1. On japao-roteador:
    # touch /usr/local/etc/racoon/psk.txt
    # chmod 0600 /usr/local/etc/racoon/psk.txt
    # cat >> /usr/local/etc/racoon/psk.txt << EOF
    45.76.207.156 pleasedonotusethis
    EOF
    
    # cat > /usr/local/etc/racoon/racoon.conf << EOF
    path pre_shared_key "/usr/local/etc/racoon/psk.txt";
    log debug;
    
    listen
    {
      isakmp          45.76.153.70;
    }
    
    remote anonymous
    {
      exchange_mode     main,aggressive;
      lifetime          time 8 hour;
      passive           off;
      proposal_check    obey;
      nat_traversal     off;
      generate_policy   off;
      proposal {
        encryption_algorithm   3des;
        hash_algorithm         sha1;
        authentication_method  pre_shared_key;
        dh_group               1;
      }
    }
    sainfo anonymous
    {
      pfs_group                 1;
      lifetime                  time 12 hour;
      encryption_algorithm      3des;
      authentication_algorithm  hmac_sha1;
      compression_algorithm     deflate;
    }
    EOF
  2. On cingapura-roteador:
    # touch /usr/local/etc/racoon/psk.txt
    # chmod 0600 /usr/local/etc/racoon/psk.txt
    # cat > /usr/local/etc/racoon/psk.txt << EOF
    45.76.153.70 pleasedonotusethis
    EOF
    
    # cat > /usr/local/etc/racoon/racoon.conf << EOF
    path pre_shared_key "/usr/local/etc/racoon/psk.txt";
    log debug;
    
    listen
    {
      isakmp          45.76.207.156;
    }
    
    remote anonymous
    {
      exchange_mode     main,aggressive;
      lifetime          time 8 hour;
      passive           off;
      proposal_check    obey;
      nat_traversal     off;
      generate_policy   off;
      proposal {
        encryption_algorithm   3des;
        hash_algorithm         sha1;
        authentication_method  pre_shared_key;
        dh_group               1;
      }
    }
    sainfo anonymous
    {
      pfs_group                 1;
      lifetime                  time 12 hour;
      encryption_algorithm      3des;
      authentication_algorithm  hmac_sha1;
      compression_algorithm     deflate;
    }
    EOF

Step 10: Enable Service

Almost finish.  We enable the services and start them as soon as possible.

# cat >> /etc/rc.conf << EOF
racoon_enable="YES"
ipsec_enable="YES"
ipsec_program="/usr/local/sbin/setkey"
ipsec_file="/usr/local/etc/racoon/setkey.conf"
EOF
# service racoon start
# service ipsec start

Step 11: Update Firewall Rules

We have been using firewall.  In the step 3 of part 1, we allowed peers to pass through with encapsulated messages.  Now, we let a few more items to go through: the authentication AH packets, the encapsulated ESP messages.

# cat >> /etc/rc.conf << EOF
pass in quick inet proto {ah,esp,ipencap} from $encappeers to any
pass in quick inet proto udp from $encappeers port = 500 to any port = 500
EOF

Step 12: Try to Break Something

Hopefully, things should go well when you ping the two hosts.  If not, you can watch the debug log “/var/log/debug.log” for a bit more insights.  The next step is to break the tunnel.  Here are some examples.

  1. Try stopping Racoon on one of the routers
  2. Try stopping IPsec on one of the routers
  3. Try using different pre-shared keys on the two routers
  4. Repeat step 1, 2, 3 with “use” rather than “require” in the IPsec configuration
  5. Deleting the route on one of the routers