Highly Available Storage Target on FreeBSD

Standard

To achieve highly available service, it is vital to have the latest data available for restarting the service elsewhere.  In enterprise environments, multipath SAS drives allows each drive to be accessible from multiple hosts.  What about the rest of us, living in the cloud of / or inexpensive SATA drives?  Highly Available Storage Target (HAST) is the answer.  It relatively a new feature in FreeBSD 8.1.  It is useful to keep two copies of drive content on two loosely coupled computers.

In this article, I demonstrate how HAST can be setup, without bothering the actual failover logic.  I assume the two virtual machines are prepared from scratch with some unallocated space.  Alternatively, you can prepare a virtual machine with multiple drives (which is not quite feasible in my setting).

Preparing the Partitions

First, examine the drives.  Here we have 18 (no, indeed 17.9 something) gigabytes of space available.

# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40       984       - free - (492K)
      1024      1024     1 freebsd-boot (512K)
      2048   4192256     2 freebsd-swap (2.0G)
   4194304  10485760     3 freebsd-ufs (5.0G)
  14680064  37748696       - free - (18G)

Then, we can add two more partitions.  Since we have two hosts, we set two partitions so that each host can run the service on one partition.  This is so-called active-active setup.  I dedicate one for NFS and one for database.  Once you finish working on one virtual machine, do not forget performing the same on another machine.

# gpart add -t freebsd-ufs -l nfs_block -a 1M -s 8960M /dev/vtbd0
vtbd0p4 added.
# gpart add -t freebsd-ufs -l db_block -a 1M -s 8960M /dev/vtbd0
vtbd0p5 added.

This is the result.  Two block devices are created and made available inside /dev/gpt with their appropriate labels.

# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40       984       - free - (492K)
      1024      1024     1 freebsd-boot (512K)
      2048   4192256     2 freebsd-swap (2.0G)
   4194304  10485760     3 freebsd-ufs (5.0G)
  14680064  18350080     4 freebsd-ufs (8.8G)
  33030144  18350080     5 freebsd-ufs (8.8G)
  51380224   1048536       - free -  (512M)
# ls /dev/gpt
db_block nfs_block

HAST Daemon Setup

Here is an sample of defining the HAST configuration, hast.conf(5).  In short, there are two hosts and two resource items.  The host “store1” has its remote partner “store2” and vice versa.  Since we use the short host names “store1” and “store2”, do not forget to update the host(5) file to make them resolvable.  Remember to repeat these another machine.  Thankfully, the HAST configuration need not to be customised.

# sysrc hastd_enable="YES"
# cat > /etc/hast.conf << EOF
resource nfs_block {
  on store1 {
    local /dev/gpt/nfs_block
    remote store2
  }
  on store2 {
    local /dev/gpt/nfs_block
    remote store1
  }
}
resource db_block {
  on store1 {
    local /dev/gpt/db_block
    remote store2
  }
  on store2 {
    local /dev/gpt/db_block
    remote store1
  }
}
EOF
# service hastd start

Firewall Rules

If you are having a firewall, remember to open the port number 8457 opened.  For example, in PF, add these three lines to the two hosts.  Remember the replace the IP addresses as appropriate.

geompeers="{10.65.10.11,10.65.10.12}"
geomports="{8457}"
pass in quick inet proto tcp from $geompeers to any port $geomports keep state

HAST Daemon Status

Once the HAST daemon is started, the status of the blocks can be checked.  Since we have defined two resource items, there are two HAST device status reported.  For example, in the host “store1”, it says there are two components for each resource item: one block device, and one remote host.  At first, the resource items are in “initialisation” state:

store1# hastctl status
Name      Status   Role      Components
nfs_block -        init      /dev/gpt/nfs_block store2
db_block  -        init      /dev/gpt/db_block store2

To turn on the device for operation, use the “role” subcommand on “store1”

store1# hastctl role primary db_block
store1# hastctl status
Name      Status   Role      Components
nfs_block -        init      /dev/gpt/nfs_block store2
db_block  degraded primary   /dev/gpt/db_block store2

Similarly, use the “role” command on “store2”, but this time we set it secondary:

store2# hastctl role secondary db_block
store2# hastctl status
Name      Status   Role      Components
nfs_block -        init      /dev/gpt/nfs_block store1
db_block  -        secondary /dev/gpt/db_block store1

When the synchronisation completes, the status is marked complete:

store2# hastctl status
Name      Status   Role      Components
nfs_block -        init      /dev/gpt/nfs_block store1
db_block  complete secondary /dev/gpt/db_block store1

Formatting the Block Devices

We can format the block device like usual, just that we should format the device under the device directory “/dev/hast” instead of “/dev/gpt”.  Since currently the “db_block” is active on the host “store1”, it has to be executed over there.

store1# newfs -J /dev/hast/db_block
/dev/hast/db_block: 8960.0MB (18350064 sectors) block size 32768, fragment size 4096
using 15 cylinder groups of 626.09MB, 20035 blks, 80256 inodes.
super-block backups (for fsck_ffs -b #) at:
192, 1282432, 2564672, 3846912, 5129152, 6411392, 7693632, 8975872, 10258112, 11540352, 12822592, 14104832, 15387072, 16669312, 17951552

One thing to note is, the raw device was 18350080 sectors.  When HAST takes it for service, there is only 18350064 blocks left for payload.  Next, we can mount the file system.  We do not use the fstab(5) like before, because they do not need to execute every time boot.

store1# mkdir /db
store1# mount /dev/hast/db_block /db

Switching Over

In order to switch over, the procedure is as follows.

store1# umount /db
store1# hastctl role secondary db_block
store2# hastctl role primary db_block
store2# mkdir /db
store2# mount /dev/hast/db_block /db

What if, in an error, the backend device in “/dev/gpt” is being used for mounting?  It will say the following.  The chance to go wrong is really not heavy.

store2# mount /dev/gpt/db_block /db
/dev/gpt/db_block: Operation not permitted

For automatic switching, it will be discussed in a separated article.

Troubleshooting

Nothing can go really wrong without a serious application on top.  Nevertheless, the following message troubled me for a few hours.

We act as primary for the resource and not as secondary as secondary“: check the HAST configuration.  Likely a host is configured to have itself as the remote partner.

To be Continued

In the upcoming articles, I will cover how to make use of this highly-available block storage for database and shared file system.

Installing FreeBSD from Scratch and Reinstalling the Boot Loader

Standard

There are cases the default image does not suit for one.  In this exercise, I practice installing FreeBSD version 11 from scratch.  I go beyond the standard procedure by partitioning the drive manually with commands. This is to leave space I can create partitions purely for payload later.   (If you just want to go automatic, you can refer to the FreeBSD handbook.)

Some errors take place so I get to correct the boot loader manually.  If you have tried fixing the boot loader of some other “freedom” operating system, you will appreciate how easy it is!

Inserting the Disc and Boot

Instead of selecting the default boot image, we pick an installation disc.  In Vultr, There are two ways.  The first way is to let the system download the installation disc.  For example, you find a link for the FreeBSD installation disc, copy the URL, and pass it to the interface.  The second way is to reuse the existing library of installation discs.

It takes quite some time for the system to boot.  Depending whether you are lucky or not, you may or may not see the beastie welcome screen.  This is so-called the boot loader, or simply the loader, with just a few tens of kilobytes.

Screen Shot 2017-04-13 at 9.33.34 pm

Inside the Installer

The system boots and the installer (precisely, “bsdinstall”) automatically executes.  From now on, there are a few keystrokes you need to know.  The action buttons, quoted in brackets, can be selected with left and right arrow keys.  To toggle the action button, press enter key.  The items above the action buttons are selected with up and down.  To toggle the item on or off, press spacebar.  At any one time, an action button and a selectable item are highlighted.  When there are multiple fields, press the tab, not enter, to jump between.

Question 1 – mode selection: In the screen below, you can press enter to run the installer.  You can alternatively press right arrow to select the shell, then enter to run the shell.  Here we select “install” directly.

Screen Shot 2017-04-13 at 9.34.07 pm

Question 2 – keymap: If you want to select an alternative keymap, use up and down arrow keys, and press spacebar to select.  Then, press enter to confirm.

Screen Shot 2017-04-13 at 9.34.17 pm

Question 3 – hostname: You are going to enter a hostname.  If you are creating a machine to be cloned, you can pick a generic name.

Question 4 – distributions: You are asked what distribution components to select.  Usually I just pick “lib32” only.  By default, they propose installing “ports”, I deselect it (with spacebar) most of the time.  The updated ports can be downloaded by “postsnap” command later.

Partitioning and Formatting the Drive

Question 5 – partition method: You are given several ways to partition, the “auto” one are the most easy but they may generate something you do not like.  The “manual” shows a dialog where you can create the partitions yourself, but not control the partition alignments.  So let us select “shell”.

Screen Shot 2017-04-13 at 9.35.40 pm.png

Question 6 – partition: You are given a shell and instructed to type in commands, edit a file, and mount the effective file system.  Use the following commands to partition the only virtual hard drive, “vtbd0”, and then install the bootloader.

Screen Shot 2017-04-13 at 9.35.50 pm

# gpart show
# gpart create -s gpt /dev/vtbd0
vtbd0 created
# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40  52428720       - free - (25G)

# gpart add -t freebsd-boot -a 512K -s 512K /dev/vtbd0
vtbd0p1 added
# gpart add -t freebsd-swap -a 1M -s 2047M /dev/vtbd0
vtbd0p2 added
# gpart add -t freebsd-ufs -a 1M -s 5120M /dev/vtbd0
vtbd0p3 added
# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40       984       - free - (492K)
      1024      1024     1 freebsd-boot (512K)
      2048   4192256     2 freebsd-swap (2.0G)
   4194304  10485760     3 freebsd-ufs (5.0G)
  14680064  37748696       - free - (18.0G)
# gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 /dev/vtbd0
bootcode written to /dev/vtbd0

Previous step, we partition the drive into three, a boot partition, a swap partition, and a unix file system partition.  We install the GPT boot loader into the boot partition.  Then, format the last partition, define the file system table as previously instructed, then we are done.  The installer starts installation without a question asked.

# newfs -U /dev/vtbd0p3
(message truncated)

# mount /dev/vtbd0p3 /mnt
# cat >> /tmp/bsdinstall_etc/fstab << EOF
/dev/vtbd0p2 none swap sw 0 0
/dev/vtbd0p3 /    ufs  rw 1 1
EOF

# exit

Screen Shot 2017-04-13 at 9.59.14 pm

Final Touches to the Installation

Question 7 – root password: Pick and enter a password carefully, twice.

Question 8 – network configuration: You are asked what network devices you like to configure.  Select the only virtual network device, “vtnet0”.  Enable IPv4 and DHCP.  Disable IPv6 (unless you know why not).

Question 9 – name resolver configuration: Simply press “ok” for the DNS configuration.  The DNS server setting will be overridden soon.

Question 10 – time zone selection: Select the continent you are in, and then the city.  You are then asked if the abbreviation is appropriate, and confirm the system date and time.

Question 11 – services: I would select “local_unbound”, “sshd”, and “ntpd”.

Screen Shot 2017-04-13 at 10.01.51 pm

Question 12 – security: Since version 11, the FreeBSD installer asks if the user wants any additional security measures.  I think most of them can be enabled, except the debugging.  (This is because I do debug programs.)

Screen Shot 2017-04-13 at 10.03.21 pm

Question 13 – additional users: This is up to you.  I prefer customisation before user creation.

Question 14 – final configuration: Just skip…

Question 15 – final modification: Just skip…

Question 16 – what next: Instead of rebooting, I prefer going to the live CD mode, login and “poweroff”.

Remaining Activities

Take a snapshot before booting the system again.  On the first system boot, the SSH generates its identities.  If you want multiple hosts having their distinct identities, taking the snapshot before the first boot is the laziest and the most correct way.

Last but not least, remove the virtual optical drive image.  Then you are good to boot from the virtual hard drive.

Troubleshooting and Fixing the Boot Loader

Missing boot loader: When generating the screenshots, I forgot to install the boot code.  The boot screen looks like this and is stuck.  This is a sign of missing the boot loader.  I booted with the installation disc again, then choose shell mode, and finally rerun the “gpart bootcode” command.

Screen Shot 2017-04-13 at 10.05.10 pm

# gpart show
=>      40  52428720 vtbd0 GPT (25G)
        40       984       - free - (492K)
      1024      1024     1 freebsd-boot (512K)
      2048   4192256     2 freebsd-swap (2.0G)
   4194304  10485760     3 freebsd-ufs (5.0G)
  14680064  37748696       - free - (18G)
# gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 /dev/vtbd0
bootcode written to /dev/vtbd0

Damaged file system table: On the next boot attempt, I drop into single user mode because of bad file system table.  This was because I wrote “rw” instead of “sw” for the swap.  I then corrected the “/etc/fstab” with an editor.  Then I “exit” to continue the boot.

Screen Shot 2017-04-13 at 10.11.27 pm.png

Security Settings

For you reference, the security options I made in installation turns out to be the following.  So they can be incorporated in other installation tools, without actually running the “bsdinstall”.

/etc/rc.conf

clear_tmp_enable="YES"
syslogd_flags="-ss"
local_unbound_enable="YES"

/etc/sysctl.conf

security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.stack_guard_page = 1

/etc/resolv.conf

nameserver 127.0.0.1
options edns0

To be Continued

In the upcoming articles, I will use the snapshots created here to build a highly available block device, and then highly available file systems and database systems.

Basic C TCP/IP Programming

Standard

In this article, I share how to have some TCP/IP programming with the C language.  I am using FreeBSD because it is my most familiar platform but it does not prevent you trying the source code elsewhere.  There are of course thousands of examples online.  To show that I am different, I will present my code in an unorthodox but effective way.  (I am a protestant, if you force me to answer.  🙂

I am using the word TCP/IP in the title, in case I want to have things like libibverbs in another time.  In this article, when I use the word “network”, I refer to TCP/IP.

Header Files

Some header files are required, such as standard system call headers, data type definitions, string operations, etc.

#include <stdio.h>
#include <unistd.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/wait.h>
#include <netdb.h>
#include <string.h>

System Calls and File Descriptors

In Unix system, a lot of things are presented as file descriptors and these include the network sockets we discuss here.  There are various system calls to get file descriptors of different types, such as open(2) for a named file, pipe(2) for inter-process communications, and socket(2) for a network socket.  Even so, once a file descriptor is prepared, the operations are similar: read(2) for withdrawing message, write(2) for depositing message, and close(2) for finishing after use.

Things are easy to be said than to be done.  To make a network connection, there are quite some steps that must go through.  Here I split into two roles, a server that receives connection, and a client that initiates connection.

Server:

  1. socket(2) to create a network socket
  2. bind(2) to attach to a particular defined network port
  3. listen(2) to create a connection queue
  4. accept(2) to accept a connection request
  5. read(2) or write(2) to communicate
  6. close(2) to finish

Client:

  1. socket(2) to create a network socket
  2. connect(2) to connect to a particular address
  3. read(2) or write(2) to communicate
  4. close(2) to finish

Obtaining Socket Address

In order to bind or connect, one needs to provide a socket address.  It is a structure with complex data structure.  Some writers would tell readers to fill in the structures one by one, tell them to be caution about the endiness, etc.  I am lazy and just use the getaddrinfo(3) function.  It is also the most reliable method if you want to handle different network types.  It takes the hostname, port number, and optionally hints as input.  It generates a data structure and return the pointer through a pointer of pointer.  The most difficult part is filling in the hints, but these can be blindly copied.  For example, to get a TCP over IPv4, we request SOCK_STREAM and AF_INET like…

memset(&hint, 0, sizeof(hint));
hint.ai_family = AF_INET;
hint.ai_socktype = SOCK_STREAM;
hint.ai_protocol = 0;
getaddrinfo(host, port, &hint, &info);

Given there are not errors, the info structure now contains the answer.  “info->ai_addr” points to the required sockaddr structure, and “info->ai_addrlen” points to the length of the answer.  Just in case you did not define the numbers, also helps by filling in the “info->ai_family,” “info->ai_socktype”, and “info->ai_protocol” according to the hints.  These are useful in creating the first socket.

Please note the address info can be a linked list when there are multiple options for the connection.  To be robust, one may want to try out all the connections.  For demonstration, trying one is enough.  After use, it can be cleanup with the corresponding freeaddrinfo(3) function.

Error Handling

Most system calls come with exceptional situations, like when a file is not found, or a network socket is not connectable.  It is a good practice to check the return value of each system call.  The code becomes very messy and this is when people yell for the “exception” feature of a language — put the error handling code out of the normal execution!  Hold on, checking return value can be trivial with C macro functions.  I learned this trick from a famous book but I do not recall the name.

#define pt {fprintf(stderr, "%s:%d: ", __FILE__, __LINE__); perror("");}
#define ez(x) {if ((x) != 0) {pt; goto error;}}
#define ep(x) {if ((x) <= 0) {pt; goto error;}}
#define ezp(x) {if ((x) < 0) {pt; goto error;}}

Whenever I expect it to return zero, I use ez (expect zero); also ep for expecting positive, and zp for expecting natural numbers.  With these, the error checking can be much easier, for example,

if (listen(fd, 1) != 0) {
        fprintf(stderr, "%s:%d: ", __FILE__, __LINE__);
        perror("");
        goto error;
}

can be replaced as

ez(listen(fd, 1));

At the end of the function, I just define a label to catch these errors and perform cleanup.  This is now even simpler than handling exceptions.  The actual cleanup code will be shown later.

The Server Code

The server first prepares the hints and obtains the socket address accordingly.  Then it calls the socket, bind, listen, accept, write system call accordingly.  There are two file descriptors, one for binding to the particular port, and one (or more) for communicating with the clients.  After use, it cleans up cautiously.  If there is ever error, the flow jumps to “error” immediately and the variable is updated accordingly.

int server(const char* host, const char* port)
{
        int fd = -1;
        int fd2 = -1;
        int error = 0;
        struct addrinfo* info = 0;
        struct addrinfo  hint;
        char message[16] = "hello world";
        memset(&hint, 0, sizeof(hint));
        hint.ai_family = AF_INET;
        hint.ai_socktype = SOCK_STREAM;
        hint.ai_protocol = 0;
        ez(getaddrinfo(host, port, &hint, &info));

        ezp(fd = socket(info->ai_family, info->ai_socktype, info->ai_protocol));
        ez(bind(fd, info->ai_addr, info->ai_addrlen));
        ez(listen(fd, 1));
        ezp(fd2 = accept(fd, 0, 0));
        ep(write(fd2, message, sizeof(message)));

cleanup:
        if (info != 0) freeaddrinfo(info);
        if (fd2 != -1) close(fd2);
        if (fd != -1) close(fd);
        return error;

error:
        error = 1;
        goto cleanup;
}

The Client Code

The client code is similar to the server, except it only has one file descriptor and it connects rather than bind or connect.  After reading the message from the server, it prints it out and finish.

int server(const char* host, const char* port)
{
        int fd = -1;
        int fd2 = -1;
        int error = 0;
        struct addrinfo* info = 0;
        struct addrinfo  hint;
        char message[16] = "";

        memset(&hint, 0, sizeof(hint));
        hint.ai_family = AF_INET;
        hint.ai_socktype = SOCK_STREAM;
        hint.ai_protocol = 0;
        ez(getaddrinfo(host, port, &hint, &info));
        ezp(fd = socket(info->ai_family, info->ai_socktype, info->ai_protocol));
        ez(connect(fd, info->ai_addr, info->ai_addrlen));
        ep(read(fd, message, sizeof(message)));
        printf("client received: %s\n", message);

cleanup:
        if (info != 0) freeaddrinfo(info);
        if (fd2 != -1) close(fd2);
        if (fd != -1) close(fd);
        return error;

error:
        error = 1;
        goto cleanup;
}

Putting Them Together

Finally, we need a main function to run a server and a client about the same time.  Here we use fork(2) and wait(3) calls.  The client is delayed 1 second to ensure the server has got ready before a connection is established.  The code pasting will be left as an exercise.  In short…

  1. The header files
  2. The error handling macros
  3. The server code
  4. The client code
  5. The main function

The main function is as follows.  After fork, there are two processes for the two roles.  The server is started to accept connections from port 8080 of any given IP addresses.  The client is targeting localhost port 8080.

int main(int argc, char** argv)
{
        int status = 0;
        pid_t pid = fork();

        // Forked as a parent
        if (pid > 0) {
                server(0, "8080");
                waitpid(pid, &status, 0);
        }

        // Forked as a child
        if (pid == 0) {
                sleep(1);
                client("localhost", "8080");
        }

        // Something went wrong with forking
        if (pid < 0) {
                perror("fork");
        }
}

And the program execution is as simple as…

# clang net.c -o net
# ./net
client received: hello world

Basic C Programming on FreeBSD

Standard

In this article, I share some programming techniques with the C programming language on FreeBSD.  As of FreeBSD 11.0, the basic installation comes with clang(1) tools for programming in C, C++, and Obj-C languages.

When I was a teaching assistant, I used to give my students a crash course in C programming.  They already know C++ from their year 1.  Once they understand the differences, they picked up C real quick.

C is one of my favourite programming languages.  It’s features are minimal yet comprehensive.  Because of its simplicity, it is also very popular in system implementation.  Once somebody understands it, he is able to read and understand a lot of system implementation work.  (Warning: he will also start to dislike some particular kernel implementations, because they are really badly written.)

Why is programming related to this blog?  Indeed, one of the long-term goals here is to build a distributed software transactional memory system and deploy it on the cloud.  Time will tell if I am capable of this accomplishment.

As a side note, in Cantonese, “C” has similar pronunciation as “詩” (poem), which is a popular character for feminine names, such as “李慧詩“.  It is quite romantic to say “寫詩” (write poems) in place of “programming in that low-level language”… unless one broke up with one of those girls.  Ah.  The weather today is no good; and life is really harsh!  Let’s carry (not Carrie) on.

The Euclidean Algorithm

Today, we take the Euclidean Algorithm as an example.  It is useful for finding the greatest common divisor (GCD) of two integers.  I am not a mathematician, so please let me jump to my conclusion, as copied from the Wikipedia:

function gcd(a, b)
    while b ≠ 0
       t := b; 
       b := a mod b; 
       a := t; 
    return a;

In short:  There are two numbers.  The larger number is replaced with the modulus between the two number.  The logic repeats until a number turns into nothing.  The remaining number is the answer.

The First Attempt

Open up your favourite text editor and write to a file “gcd.c”.  I use vi(1), but you can always use the easier ee(1).  There are two functions in this program.  The “gcd” function is responsible computing the greatest common divisor.  The “main” function is the entry point.  It uses scanf(3) to input two integers, “a” and “b”, and then uses printf(3) to output the result.  In order to use these two functions, a header file “stdio.h” has to be included, as hinted by the manual pages.

There are “&” signs near the “a” and “b” and they mean pass-by-pointer.  Normally, variables are only pass-by-value.  The pass-by-pointer strategy allows the scanf(3) function to take the pointers and update the variables directly.  The arguments “argc” and “argv” are the argument counts (a number) and argument vectors (array of strings, or array of array of characters).  We do not use the arguments so we can leave them untouched.

#include <stdio.h>

int gcd(int a, int b) {
        int c;
        while (b != 0) {
                c = a % b;
                a = b;
        }
}

int main(int argc, char** argv) {
        int a, int b;
        scanf("%d %d", &a, &b);
        printf("%d\n", gcd(a, b));
        return 0;
}

Compilation Errors

Compile with the cc(1) command.  Like the way I demonstrated to my students, the first attempt is usually a failed one…

# cc gcd.c -o gcd
gcd.c:9:1: warning: control reaches end of non-void function
}
^
gcd.c:12:9: error: expected identifier or '('
        int a, int b;
               ^
gcd.c:12:8: error: expected ';' at end of declaration
        int a, int b;
              ^
              ;
1 warning and 2 errors generated.

The Second Attempt

It seems that semicolon should be used to separate declaration of the two variables.  (I know, comma works another way, but I am not going to tell…)

#include <stdio.h>

int gcd(int a, int b) {
        int c;
        while (b != 0) {
                c = a % b;
                a = b;
        }
}

int main(int argc, char** argv) {
        int a;  // <-- this line
        int b;  // <-- this line
        scanf("%d %d", &a, &b);
        printf("%d\n", gcd(a, b));
        return 0;
}

Runtime Error

Let’s compile and try again.  The program does compile despite the warning.  For dramatic sake, ignore the warning and try.  In the smoke test, we are supposed to enter the two numbers into the standard input.  After inputting the two numbers, press enter a few times and …

# cc gcd.c -o gcd
gcd.c:9:1: warning: control reaches end of non-void function
}
^
1 warning generated.
# ./gcd
4 24

the program loops forever, press Ctrl-C to break out.

First Time Debugging

To debug, let us use the LLVM debugger. (This is a new standard feature in FreeBSD 11.0.  We used to have another debugger in the past.)  Issue the lldb command, run with command “run”, input some text, then break it with Ctrl-C.  Unlike last time we break to the shell, this time we end up seeing some assembly code dump.

# lldb gcd
(lldb) target create "gcd"
Current executable set to 'gcd' (x86_64).
(lldb) run
Process 93041 launching
Process 93041 launched: '/root/gcd' (x86_64)
4 24

^C
Process 93041 stopped
* thread #1: tid = 100067, 0x00000000004007ab gcd`gcd + 27, stop reason = signal SIGSTOP
    frame #0: 0x00000000004007ab gcd`gcd + 27
gcd`gcd:
->  0x4007ab <+27>: movl   %edx, -0x10(%rbp)
    0x4007ae <+30>: movl   -0xc(%rbp), %edx
    0x4007b1 <+33>: movl   %edx, -0x8(%rbp)
    0x4007b4 <+36>: jmp    0x40079a                  ; <+10>
(lldb) quit
Quitting LLDB will kill one or more processes. Do you really want to proceed: [Y/n] y

As part of my plan, you see nothing useful unless you understand assembly code.  Quit and say yes to confirm.  Then, we try compiling again with debug symbols.

# cc gcd.c -o gcd
gcd.c:9:1: warning: control reaches end of non-void function
}
^
1 warning generated.
# lldb gcd
(lldb) target create "gcd"
Current executable set to 'gcd' (x86_64).
(lldb) run
Process 93075 launching
Process 93075 launched: '/root/gcd' (x86_64)
4 24

^C
Process 93075 stopped
* thread #1: tid = 100096, 0x00000000004007ab gcd`gcd(a=24, b=24) + 27 at gcd.c:6, stop reason = signal SIGSTOP
    frame #0: 0x00000000004007ab gcd`gcd(a=24, b=24) + 27 at gcd.c:6
   3   int gcd(int a, int b) {
   4     int c;
   5     while (b != 0) {
-> 6       c = a % b;
   7       a = b;
   8     }
   9   }
(lldb)

To step through the program, use the command “print” to print a variable content, and “next” to step to the next statement.  Repeat this a few times, we see the variable content does not change.  The condition to break out of the loop will not be satisfiable.

(lldb) print b
(int) $0 = 24
(lldb) next
Process 93075 stopped
* thread #1: tid = 100096, 0x00000000004007ae gcd`gcd(a=24, b=24) + 30 at gcd.c:7, stop reason = step over
    frame #0: 0x00000000004007ae gcd`gcd(a=24, b=24) + 30 at gcd.c:7
   4     int c;
   5     while (b != 0) {
   6       c = a % b;
-> 7       a = b;
   8     }
   9   }
   10  
(lldb) print b
(int) $1 = 24
(lldb) next
Process 93075 stopped
* thread #1: tid = 100096, 0x00000000004007b4 gcd`gcd(a=24, b=24) + 36 at gcd.c:5, stop reason = step over
    frame #0: 0x00000000004007b4 gcd`gcd(a=24, b=24) + 36 at gcd.c:5
   2   
   3   int gcd(int a, int b) {
   4     int c;
-> 5     while (b != 0) {
   6       c = a % b;
   7       a = b;
   8     }
(lldb) print b
(int) $2 = 24
(lldb) next
Process 93075 stopped
* thread #1: tid = 100096, 0x00000000004007a4 gcd`gcd(a=24, b=24) + 20 at gcd.c:6, stop reason = step over
    frame #0: 0x00000000004007a4 gcd`gcd(a=24, b=24) + 20 at gcd.c:6
   3   int gcd(int a, int b) {
   4     int c;
   5     while (b != 0) {
-> 6       c = a % b;
   7       a = b;
   8     }
   9   }
(lldb) print b
(int) $3 = 24

The Third Attempt

In order to make the loop terminate, the variable “b” must change.  We take revision on the source code and noticed a line is missing.  Here is another iteration:

#include <stdio.h>

int gcd(int a, int b) {
        int c;
        while (b != 0) {
                c = a % b;
                a = b;
                b = c;  // <-- this line
        }
}

int main(int argc, char** argv) {
        int a;
        int b;
        scanf("%d %d", &a, &b);
        printf("%d\n", gcd(a, b));
        return 0;
}

Second Time Debugging

I am going to save time and tell you the code does not work.  It consistently returns “0” in my case.  Let us jump into the debugger again.  Because the program does not loop indefinitely this time, we need another way to stop the program before it finishes.  In this example, I used a “breakpoint set” command with “–file” and “–line” option.  Then I used “step” command to step inside the “gcd” function call.

# cc gcd.c -o gcd
gcd.c:9:1: warning: control reaches end of non-void function
}
^
1 warning generated.
# lldb gcd
(lldb) target create "gcd"
Current executable set to 'gcd' (x86_64).
(lldb) breakpoint set --file gcd.c --line 15
Breakpoint 1: where = gcd`main + 53 at gcd.c:15, address = 0x0000000000400805
(lldb) run
Process 93155 launching
Process 93155 launched: '/root/gcd' (x86_64)
4 24

^C
Process 93155 stopped
* thread #1: tid = 100085, 0x0000000000400805 gcd`main(argc=1, argv=0x00007fffffffeb60) + 53 at gcd.c:15, stop reason = breakpoint 1.1
    frame #0: 0x0000000000400805 gcd`main(argc=1, argv=0x00007fffffffeb60) + 53 at gcd.c:15
   12  int main(int argc, char** argv) {
   13  int a; int b;
   14  scanf("%d %d", &a, &b);
-> 15  printf("%d\n", gcd(a, b));
   16  return 0;
   17  }
(lldb) print a
(int) $0 = 4
(lldb) print b
(int) $1 = 24
(lldb) step
Process 93155 stopped
* thread #1: tid = 100085, 0x000000000040079a gcd`gcd(a=4, b=24) + 10 at gcd.c:5, stop reason = step in
    frame #0: 0x000000000040079a gcd`gcd(a=4, b=24) + 10 at gcd.c:5
   2   
   3   int gcd(int a, int b) {
   4     int c;
-> 5     while (b != 0) {
   6       c = a % b;
   7       a = b;
   8       b = c;
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007a4 gcd`gcd(a=4, b=24) + 20 at gcd.c:6, stop reason = step in
    frame #0: 0x00000000004007a4 gcd`gcd(a=4, b=24) + 20 at gcd.c:6
   3   int gcd(int a, int b) {
   4     int c;
   5     while (b != 0) {
-> 6       c = a % b;
   7       a = b;
   8       b = c;
   9     }
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007ae gcd`gcd(a=4, b=24) + 30 at gcd.c:7, stop reason = step in
    frame #0: 0x00000000004007ae gcd`gcd(a=4, b=24) + 30 at gcd.c:7
   4     int c;
   5     while (b != 0) {
   6       c = a % b;
-> 7       a = b;
   8       b = c;
   9     }
   10  }
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007b4 gcd`gcd(a=24, b=24) + 36 at gcd.c:8, stop reason = step over
    frame #0: 0x00000000004007b4 gcd`gcd(a=24, b=24) + 36 at gcd.c:8
   5     while (b != 0) {
   6       c = a % b;
   7       a = b;
-> 8       b = c;
   9     }
   10  }
   11  
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007ba gcd`gcd(a=24, b=4) + 42 at gcd.c:5, stop reason = step over
    frame #0: 0x00000000004007ba gcd`gcd(a=24, b=4) + 42 at gcd.c:5
   2   
   3   int gcd(int a, int b) {
   4     int c;
-> 5     while (b != 0) {
   6     c = a % b;
   7     a = b;
   8     b = c;
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007a4 gcd`gcd(a=24, b=4) + 20 at gcd.c:6, stop reason = step over
    frame #0: 0x00000000004007a4 gcd`gcd(a=24, b=4) + 20 at gcd.c:6
   3   int gcd(int a, int b) {
   4     int c;
   5     while (b != 0) {
-> 6       c = a % b;
   7       a = b;
   8       b = c;
   9     }
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007ae gcd`gcd(a=24, b=4) + 30 at gcd.c:7, stop reason = step over
    frame #0: 0x00000000004007ae gcd`gcd(a=24, b=4) + 30 at gcd.c:7
   4     int c;
   5     while (b != 0) {
   6       c = a % b;
-> 7       a = b;
   8       b = c;
   9     }
   10  }
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007b4 gcd`gcd(a=4, b=4) + 36 at gcd.c:8, stop reason = step over
    frame #0: 0x00000000004007b4 gcd`gcd(a=4, b=4) + 36 at gcd.c:8
   5     while (b != 0) {
   6       c = a % b;
   7       a = b;
-> 8       b = c;
   9     }
   10  }
   11  
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007ba gcd`gcd(a=4, b=0) + 42 at gcd.c:5, stop reason = step over
    frame #0: 0x00000000004007ba gcd`gcd(a=4, b=0) + 42 at gcd.c:5
   2   
   3   int gcd(int a, int b) {
   4     int c;
-> 5     while (b != 0) {
   6       c = a % b;
   7       a = b;
   8       b = c;
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007bf gcd`gcd(a=4, b=0) + 47 at gcd.c:10, stop reason = step over
    frame #0: 0x00000000004007bf gcd`gcd(a=4, b=0) + 47 at gcd.c:10
   7       a = b;
   8       b = c;
   9     }
-> 10  }
   11  
   12  int main(int argc, char** argv) {
   13  int a; int b;
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x0000000000400813 gcd`main(argc=1, argv=0x00007fffffffeb60) + 67 at gcd.c:15, stop reason = step over
    frame #0: 0x0000000000400813 gcd`main(argc=1, argv=0x00007fffffffeb60) + 67 at gcd.c:15
   12  int main(int argc, char** argv) {
   13    int a; int b;
   14    scanf("%d %d", &a, &b);
-> 15    printf("%d\n", gcd(a, b));
   16    return 0;
   17  }

We see the values of variables “a” and “b” decreases alone the time line.  Pay attention to the transition between the “gcd” function to the “main” function.  There is not a return value.  This is why we kept having the warning.  Compiler warnings are often more useful than what programmers think.  Sometimes, I am amazed why the open source software packages contain some many warnings in compilations and they still work.

The Final Version

We get to return the answer in the “gcd” function.  Here is how it the code being corrected.

#include <stdio.h>

int gcd(int a, int b) {
        int c;
        while (b != 0) {
                c = a % b;
                a = b;
                b = c;
        }
        return a;  // <-- this line
}

int main(int argc, char** argv) {
        int a;
        int b;
        scanf("%d %d", &a, &b);
        printf("%d\n", gcd(a, b));
        return 0;
}

The Final Testing

# cc gcd.c -o gcd
# ./gcd
0 5
5
# ./gcd
5 0
5
# ./gcd
1 9
1
# ./gcd
9 1
1
# ./gcd
4 36
4
# ./gcd
36 4
4
# ./gcd
24 36
12
# ./gcd
36 24
12

 

What can Go Wrong Building a Computer from Parts

Standard

Recently, I get itchy and wanted to build a gaming machine.  (Not running BSD, sorry)  Originally, I was thinking buying a Kaby Lake-based Intel Pentium G4560.  I was greedy and wanted a ECC capable motherboard.  The cost spiralled quickly close to a secondhand CPU-motherboard set with Intel Xeon E5-2670.  This particular Xeon processor model has a meaning to me—one of my important customers uses hundreds of these for high performance numerical simulations.  I just think, it is nice to own one of these as my souvenir, right?

I ordered stuffs on an online shopping mall somewhere.  (In order not to hurt their feeling, I decide not to name which country it is.)  The nightmare began.

Ordering

What can go wrong with ordering?  In that online mall, there are numerous shops run by different people of different service quality.  You are gambling with your luck if you order things from a new vendor.

Among the computer parts, I get rejected after payment for the memory modules.  I selected DDR3 ECC UDIMM for the first order, because it works with another of my computer with AMD 1055T processor as well.  Upon the payment, the shopkeeper was friendly and asked me to double check if the memory is compatible.  Once I verified it works, the shopkeeper told me to refund because it is out of stock.  I switched to buy DDR3 ECC RDIMM, which is less compatible to desktop computers, but much more readily available.

I also got rejected payment, indeed twice, for the keyboards.  I was greedy and ordered a niche second-hand UNICOMP keyboard with a red stick (ultranav?) with PS/2 plugs.  At last I gave up and just left it open, maybe sooner with a second-hand IBM Spacesaver Model M4-1… if I dare to try again.

Eventually, the products that could arrive, arrived in good shape, especially I ordered them through a Hong Kong-based consolidation shipping company.

Accuracy and Quality

The quality of the products are not satisfactory.  The only thing that I have no complaint is the memory modules, because they just sit in and work.

The Motherboard

Motherboard plays an important role in the computer.  While I do not expect top-notch quality, I do have some reasonable expectation to it.

Some people like smelling motherboards.  I was curious and gave my board a shot.  It was definitely different, awfully different, but this is just the beginning.  While the board is claimed to be new, the south-bridge heatsinks are obviously dented.

相片 2-3-2017 下午12 57 13

The original product picture found on the online store is as follows.  Can you spot two more problems?  I am really disappointed that they replaced a PCIe for PCI, and also left my flash drive port empty.

Screen Shot 2017-03-10 at 11.36.06 pm.png

At later stage of the build, I wanted to unplug the USB3 header cable temporary for cable management.  The plastic part of the header came out with the cable, just in one unplug!  I resorted to plug the USB3 cable onto the bare pins, without the plastic piece.

相片 3-3-2017 下午7 40 44

This motherboard make is now in my black list.

The Computer Case

I ordered the computer case with the fans and LED strips.  Frankly speaking, the computer case exceeds my expectation.  What disappoints me are the complimentary fans and the LED strips.  Instead of 3 or 4-pin fan headers, they all use MOLEX power plugs.  I got to have a dedicated MOLEX cable going out from the cable-management power supply, which is quite ugly if you ask me.

The Heatsink

Because the Xeon processor can go as much as 120W in peak load, I went to buy a very big heatsink with push-pull technology.  But it ends up being too tall with the computer case.  I was so amazed I made such a big mistake.  I should have checked the height clearance of the computer case and the heatsink dimensions.

Compatibility

After such a painful experience of building, I was actually quite surprise the setup boots up immediately without complaints.

The only issue happened when I try to install the NVidia graphics driver.  In order to get it installed, the computer has to boot into the text mode.  Yet, the text mode of that operating system is broken.  (No, it is not FreeBSD.)  I thought the computer did not show up the login screen because the systemd acted wrongly.  I tried reinstalling and also booting from the installation flash disk many times.  It eventually took me half a day to realise it was the graphics mode.

I am not quite sure what I can do if the computer fails to boot into graphical mode after an update.  Anyway, treat it as a game machine and I will be alright.

Bill of Materials

This is the bill of materials I used, including the ones I purchased before this.

  • Intel Xeon E5-2670 Processor, Qty: 1
  • X79 Motherboard, Qty: 1
  • DDR3-1600 ECC UDIMM 8 GB, Qty: 4
  • DDR3-1333 ECC RDIMM 8 GB, Qty: 4
  • Gigabyte GeForce GTX1050Ti Windforce OC 4G, Qty: 1
  • Antec High Gamer Power 520W, Qty: 1
  • Cooler Master T400 Pro, Qty: 1
  • Aigo ATX Computer Case, Qty: 1
  • UNICOMP Model M keyboard, Qty: 1
  • Cooler Master Rapid Fire 87-Key Keyboard, Qty: 1
  • Logitech Wireless Mouse M280, Qty: 1
  • Dell Ultrasharp U2412M Display, Qty: 1

If I only build from only new parts, the first three items becomes as follows.  The core count and memory channels are different.  Hopefully, the AVX2 instruction sets and the DDR4 memory would cover the difference.

  • Intel Xeon E3-1230 v5 Processor, Qty: 1
  • Gigabyte GA-X150-Pro ECC Motherboard, Qty: 1
  • DDR4-2133 ECC UDIMM 8 GB: Qty: 4

At the end, I saved around 300 USD, at the cost not having warranty on the parts.  Well, indeed, owning a computer with Xeon E5-2670 is priceless for me…

Lessons Learned

  • Place no expectation on the fans coming with a computer case.
  • Whenever possible, buy from a local brick-and-mortar store, even with a premium.
  • When buying niche stuff, be more verbose and ask for availability before payment.

OpenStack with FreeBSD

Standard

Recently, I come across NFV Express through the FreeBSDNews article.  The installation code is already openly available on Github.  At the first glance, the code is so amazing to be clean!  (You definitely should look at them if you know programming.)  When there is time, I will try them with my stack of FreeBSD computers…

Openstack is used to be Linux-centric, yet it is mostly written in Python scripts.  Theoretically, it is possible to make it running on any capable platforms… if you do not hate working with Python scripts.

Sorry for this short article.  This is indeed just a reminder for me to follow up.  🙂

Update on 4 Apr 2017: the tool requires Xen to be installed.  I tried installing it on the new computer but UEFI is not supported.  I am going to schedule another older computer to try.

Proxy Server with FreeBSD and Squid (Part 1)

Standard

When one has multiple machines within a cloud network, it is natural to ask for centralised network traffic, data files, credential service, etc.  In this article, I focus on centralising network traffic, in particular, the world wide web.

With the world wide web traffic centralised, the proxy server can accelerate the download processes of some frequently accessed files, such as operating system patches.

In this part, we install Squid as an opaque proxy server, and configure some clients to use it.  We will handle the transparent proxy in the next part.

Step 1: Install Squid

Installing Squid is easy as usual.  But to make it even smoother, install the SSL root certificates.

# pkg install squid ca_root_nss

The squid requires Perl which, since FreeBSD 5.0, no longer installed by default.  The package installer will handle it.

Step 2: Smoke Test

Even though there are quite a page of notice upon installing Squid, the default Squid installation is good enough to fire without configuration:

# sysrc squid_enable=YES
# service squid start

or, for the old style…

# echo squid_enable=YES >> /etc/rc.conf
# /usr/local/etc/rc.d/squid start

Step 3: Configure Squid

The Squid refers to its configuration file in “/usr/local/etc/squid”.

# ee /usr/local/etc/squid/squid.conf

 

What shall we configure?  Here are some suggestions:

  1. Configure the intranet address ranges.  By the configuration file, only networks defined as “localnet” is granted access.  For example, I only use the class A private subnet, so I disable all others.
    acl localnet src 10.0.0.0/8     
    #acl localnet src 172.16.0.0/12 
    #acl localnet src 192.168.0.0/16
    #acl localnet src fc00::/7      
    #acl localnet src fe80::/10
  2. Deny the squid accessing the so-called “localhost”.  Internal network of the squid is not supposed to be accessed by others.
    http_access deny to_localhost
  3. Configure the cache directory.  For example, here I configure a 10000 MB cache space.  There are two levels of directories, with 16 and 256 directories per level respectively.
    cache_dir ufs /var/squid/cache 10000 16 256
  4. Prefer IPv4 whenever possible.  This is great if your network is not IPv6 ready.
    dns_v4_first on

We then stop the Squid, create the cache directory, and restart.

# service squid stop
# squid -z -N
# chown -R squid:squid /var/squid/cache
# service squid start

Step 4: Update Firewall

If you are using PF firewall, you will need to update the rules to allow the port.  For example, I only use the “vtnet1” as intranet, and it is accepting port 3128 (the default squid service port).  You may refer to the configuration of the VPN notes as example.

# cat >> /etc/pf.conf << EOF
intif="vtnet1"
inttcpports="{3128}"
pass in quick on $intif inet proto tcp from any to any port $inttcpports keep state
EOF
# service pf reload

Step 5: Playing with Telnet

The title is not a typo.  We now test the system with the most geeky way—enter the HTTP request manually.  FreeBSD comes with a telnet client and it is handy for various testing.  Suppose the address is 10.65.0.1 and we connect to the port 3128 of it.  After typing the GET statement, remember to hit the enter key twice to complete the request.

# telnet 10.65.0.1 3128
GET http://www.freebsd.org/ HTTP/1.0

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Tue, 07 Mar 2017 16:04:30 GMT
Content-Type: text/html
Content-Length: 178
Location: https://www.freebsd.org/
X-Cache: MISS from singapura-roteador
X-Cache-Lookup: HIT from singapura-roteador:3128
Via: 1.1 singapura-roteador (squid/3.5.23)
Connection: close

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
Connection closed by foreign host.

The HTTP return code 301 indicates another URL for access.  What about following it?

# telnet 10.65.0.1 3128
GET https://www.freebsd.org/ HTTP/1.0

Step 6: Troubleshoot

Here are some errors you may encounter:

Unable to connect to the server: try disabling the “pf” service briefly and retry.  If it works, you will need to update your firewall rules.

Squid is not starting after enabling cache: make sure you create the cache directory and set the proper ownership with the chown command (as above) .

Getting 503 error with ERR_CONNECT_FAIL: review the error message and if the address is an IPv6 address, there could be issues with the IPv6 routing and you can consider using the “dns_v4_first” option (as above).

Getting 503 error with ERR_SECURE_CONNECT_FAIL: review the website with your own browser, ensure you have installed the common root certificates “ca_root_nss” package (as above).

Step 7: Configure Client

Since our proxy is not transparent yet, the clients have to be manually configured in order to use the proxy.  Depending on the hosts you have on the same intranet, here are the configuration methods for your reference:

Mac OS X: System Preferences > Network > Advanced > Proxies > Web Proxy / Secure Web Proxy

Internet Explorer on Windows: Tools > Internet Options > Connections > LAN Setting

Firefox on Windows: Tools > Internet Options > Network > Connections > Settings

FreeBSD shell: export “HTTP_PROXY” variable in “.cshrc”, like “setenv HTTP_PROXY http://10.65.0.1:3128&#8221;

Step 8: Is the Proxy Really Running?

There are some barbaric ways to check if the proxy is running:

Check the number of files the cache: “find /var/squid/cache | wc -l”

Check the Squid access log: “less /var/log/squid/access.log”

Step 9: To be Continued

A few weeks later, we will revisit Squid and configure the network routers to transparently direct world wide web requests to the proxy server, etc.

Later this week, we may come up with an article not related with FreeBSD.  The next FreeBSD-related article, hopefully next week, will be likely related to programming.  Stay tuned.