Basic C Programming on FreeBSD

Standard

In this article, I share some programming techniques with the C programming language on FreeBSD.  As of FreeBSD 11.0, the basic installation comes with clang(1) tools for programming in C, C++, and Obj-C languages.

When I was a teaching assistant, I used to give my students a crash course in C programming.  They already know C++ from their year 1.  Once they understand the differences, they picked up C real quick.

C is one of my favourite programming languages.  It’s features are minimal yet comprehensive.  Because of its simplicity, it is also very popular in system implementation.  Once somebody understands it, he is able to read and understand a lot of system implementation work.  (Warning: he will also start to dislike some particular kernel implementations, because they are really badly written.)

Why is programming related to this blog?  Indeed, one of the long-term goals here is to build a distributed software transactional memory system and deploy it on the cloud.  Time will tell if I am capable of this accomplishment.

As a side note, in Cantonese, “C” has similar pronunciation as “詩” (poem), which is a popular character for feminine names, such as “李慧詩“.  It is quite romantic to say “寫詩” (write poems) in place of “programming in that low-level language”… unless one broke up with one of those girls.  Ah.  The weather today is no good; and life is really harsh!  Let’s carry (not Carrie) on.

The Euclidean Algorithm

Today, we take the Euclidean Algorithm as an example.  It is useful for finding the greatest common divisor (GCD) of two integers.  I am not a mathematician, so please let me jump to my conclusion, as copied from the Wikipedia:

function gcd(a, b)
    while b ≠ 0
       t := b; 
       b := a mod b; 
       a := t; 
    return a;

In short:  There are two numbers.  The larger number is replaced with the modulus between the two number.  The logic repeats until a number turns into nothing.  The remaining number is the answer.

The First Attempt

Open up your favourite text editor and write to a file “gcd.c”.  I use vi(1), but you can always use the easier ee(1).  There are two functions in this program.  The “gcd” function is responsible computing the greatest common divisor.  The “main” function is the entry point.  It uses scanf(3) to input two integers, “a” and “b”, and then uses printf(3) to output the result.  In order to use these two functions, a header file “stdio.h” has to be included, as hinted by the manual pages.

There are “&” signs near the “a” and “b” and they mean pass-by-pointer.  Normally, variables are only pass-by-value.  The pass-by-pointer strategy allows the scanf(3) function to take the pointers and update the variables directly.  The arguments “argc” and “argv” are the argument counts (a number) and argument vectors (array of strings, or array of array of characters).  We do not use the arguments so we can leave them untouched.

#include <stdio.h>

int gcd(int a, int b) {
        int c;
        while (b != 0) {
                c = a % b;
                a = b;
        }
}

int main(int argc, char** argv) {
        int a, int b;
        scanf("%d %d", &a, &b);
        printf("%d\n", gcd(a, b));
        return 0;
}

Compilation Errors

Compile with the cc(1) command.  Like the way I demonstrated to my students, the first attempt is usually a failed one…

# cc gcd.c -o gcd
gcd.c:9:1: warning: control reaches end of non-void function
}
^
gcd.c:12:9: error: expected identifier or '('
        int a, int b;
               ^
gcd.c:12:8: error: expected ';' at end of declaration
        int a, int b;
              ^
              ;
1 warning and 2 errors generated.

The Second Attempt

It seems that semicolon should be used to separate declaration of the two variables.  (I know, comma works another way, but I am not going to tell…)

#include <stdio.h>

int gcd(int a, int b) {
        int c;
        while (b != 0) {
                c = a % b;
                a = b;
        }
}

int main(int argc, char** argv) {
        int a;  // <-- this line
        int b;  // <-- this line
        scanf("%d %d", &a, &b);
        printf("%d\n", gcd(a, b));
        return 0;
}

Runtime Error

Let’s compile and try again.  The program does compile despite the warning.  For dramatic sake, ignore the warning and try.  In the smoke test, we are supposed to enter the two numbers into the standard input.  After inputting the two numbers, press enter a few times and …

# cc gcd.c -o gcd
gcd.c:9:1: warning: control reaches end of non-void function
}
^
1 warning generated.
# ./gcd
4 24

the program loops forever, press Ctrl-C to break out.

First Time Debugging

To debug, let us use the LLVM debugger. (This is a new standard feature in FreeBSD 11.0.  We used to have another debugger in the past.)  Issue the lldb command, run with command “run”, input some text, then break it with Ctrl-C.  Unlike last time we break to the shell, this time we end up seeing some assembly code dump.

# lldb gcd
(lldb) target create "gcd"
Current executable set to 'gcd' (x86_64).
(lldb) run
Process 93041 launching
Process 93041 launched: '/root/gcd' (x86_64)
4 24

^C
Process 93041 stopped
* thread #1: tid = 100067, 0x00000000004007ab gcd`gcd + 27, stop reason = signal SIGSTOP
    frame #0: 0x00000000004007ab gcd`gcd + 27
gcd`gcd:
->  0x4007ab <+27>: movl   %edx, -0x10(%rbp)
    0x4007ae <+30>: movl   -0xc(%rbp), %edx
    0x4007b1 <+33>: movl   %edx, -0x8(%rbp)
    0x4007b4 <+36>: jmp    0x40079a                  ; <+10>
(lldb) quit
Quitting LLDB will kill one or more processes. Do you really want to proceed: [Y/n] y

As part of my plan, you see nothing useful unless you understand assembly code.  Quit and say yes to confirm.  Then, we try compiling again with debug symbols.

# cc gcd.c -o gcd
gcd.c:9:1: warning: control reaches end of non-void function
}
^
1 warning generated.
# lldb gcd
(lldb) target create "gcd"
Current executable set to 'gcd' (x86_64).
(lldb) run
Process 93075 launching
Process 93075 launched: '/root/gcd' (x86_64)
4 24

^C
Process 93075 stopped
* thread #1: tid = 100096, 0x00000000004007ab gcd`gcd(a=24, b=24) + 27 at gcd.c:6, stop reason = signal SIGSTOP
    frame #0: 0x00000000004007ab gcd`gcd(a=24, b=24) + 27 at gcd.c:6
   3   int gcd(int a, int b) {
   4     int c;
   5     while (b != 0) {
-> 6       c = a % b;
   7       a = b;
   8     }
   9   }
(lldb)

To step through the program, use the command “print” to print a variable content, and “next” to step to the next statement.  Repeat this a few times, we see the variable content does not change.  The condition to break out of the loop will not be satisfiable.

(lldb) print b
(int) $0 = 24
(lldb) next
Process 93075 stopped
* thread #1: tid = 100096, 0x00000000004007ae gcd`gcd(a=24, b=24) + 30 at gcd.c:7, stop reason = step over
    frame #0: 0x00000000004007ae gcd`gcd(a=24, b=24) + 30 at gcd.c:7
   4     int c;
   5     while (b != 0) {
   6       c = a % b;
-> 7       a = b;
   8     }
   9   }
   10  
(lldb) print b
(int) $1 = 24
(lldb) next
Process 93075 stopped
* thread #1: tid = 100096, 0x00000000004007b4 gcd`gcd(a=24, b=24) + 36 at gcd.c:5, stop reason = step over
    frame #0: 0x00000000004007b4 gcd`gcd(a=24, b=24) + 36 at gcd.c:5
   2   
   3   int gcd(int a, int b) {
   4     int c;
-> 5     while (b != 0) {
   6       c = a % b;
   7       a = b;
   8     }
(lldb) print b
(int) $2 = 24
(lldb) next
Process 93075 stopped
* thread #1: tid = 100096, 0x00000000004007a4 gcd`gcd(a=24, b=24) + 20 at gcd.c:6, stop reason = step over
    frame #0: 0x00000000004007a4 gcd`gcd(a=24, b=24) + 20 at gcd.c:6
   3   int gcd(int a, int b) {
   4     int c;
   5     while (b != 0) {
-> 6       c = a % b;
   7       a = b;
   8     }
   9   }
(lldb) print b
(int) $3 = 24

The Third Attempt

In order to make the loop terminate, the variable “b” must change.  We take revision on the source code and noticed a line is missing.  Here is another iteration:

#include <stdio.h>

int gcd(int a, int b) {
        int c;
        while (b != 0) {
                c = a % b;
                a = b;
                b = c;  // <-- this line
        }
}

int main(int argc, char** argv) {
        int a;
        int b;
        scanf("%d %d", &a, &b);
        printf("%d\n", gcd(a, b));
        return 0;
}

Second Time Debugging

I am going to save time and tell you the code does not work.  It consistently returns “0” in my case.  Let us jump into the debugger again.  Because the program does not loop indefinitely this time, we need another way to stop the program before it finishes.  In this example, I used a “breakpoint set” command with “–file” and “–line” option.  Then I used “step” command to step inside the “gcd” function call.

# cc gcd.c -o gcd
gcd.c:9:1: warning: control reaches end of non-void function
}
^
1 warning generated.
# lldb gcd
(lldb) target create "gcd"
Current executable set to 'gcd' (x86_64).
(lldb) breakpoint set --file gcd.c --line 15
Breakpoint 1: where = gcd`main + 53 at gcd.c:15, address = 0x0000000000400805
(lldb) run
Process 93155 launching
Process 93155 launched: '/root/gcd' (x86_64)
4 24

^C
Process 93155 stopped
* thread #1: tid = 100085, 0x0000000000400805 gcd`main(argc=1, argv=0x00007fffffffeb60) + 53 at gcd.c:15, stop reason = breakpoint 1.1
    frame #0: 0x0000000000400805 gcd`main(argc=1, argv=0x00007fffffffeb60) + 53 at gcd.c:15
   12  int main(int argc, char** argv) {
   13  int a; int b;
   14  scanf("%d %d", &a, &b);
-> 15  printf("%d\n", gcd(a, b));
   16  return 0;
   17  }
(lldb) print a
(int) $0 = 4
(lldb) print b
(int) $1 = 24
(lldb) step
Process 93155 stopped
* thread #1: tid = 100085, 0x000000000040079a gcd`gcd(a=4, b=24) + 10 at gcd.c:5, stop reason = step in
    frame #0: 0x000000000040079a gcd`gcd(a=4, b=24) + 10 at gcd.c:5
   2   
   3   int gcd(int a, int b) {
   4     int c;
-> 5     while (b != 0) {
   6       c = a % b;
   7       a = b;
   8       b = c;
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007a4 gcd`gcd(a=4, b=24) + 20 at gcd.c:6, stop reason = step in
    frame #0: 0x00000000004007a4 gcd`gcd(a=4, b=24) + 20 at gcd.c:6
   3   int gcd(int a, int b) {
   4     int c;
   5     while (b != 0) {
-> 6       c = a % b;
   7       a = b;
   8       b = c;
   9     }
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007ae gcd`gcd(a=4, b=24) + 30 at gcd.c:7, stop reason = step in
    frame #0: 0x00000000004007ae gcd`gcd(a=4, b=24) + 30 at gcd.c:7
   4     int c;
   5     while (b != 0) {
   6       c = a % b;
-> 7       a = b;
   8       b = c;
   9     }
   10  }
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007b4 gcd`gcd(a=24, b=24) + 36 at gcd.c:8, stop reason = step over
    frame #0: 0x00000000004007b4 gcd`gcd(a=24, b=24) + 36 at gcd.c:8
   5     while (b != 0) {
   6       c = a % b;
   7       a = b;
-> 8       b = c;
   9     }
   10  }
   11  
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007ba gcd`gcd(a=24, b=4) + 42 at gcd.c:5, stop reason = step over
    frame #0: 0x00000000004007ba gcd`gcd(a=24, b=4) + 42 at gcd.c:5
   2   
   3   int gcd(int a, int b) {
   4     int c;
-> 5     while (b != 0) {
   6     c = a % b;
   7     a = b;
   8     b = c;
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007a4 gcd`gcd(a=24, b=4) + 20 at gcd.c:6, stop reason = step over
    frame #0: 0x00000000004007a4 gcd`gcd(a=24, b=4) + 20 at gcd.c:6
   3   int gcd(int a, int b) {
   4     int c;
   5     while (b != 0) {
-> 6       c = a % b;
   7       a = b;
   8       b = c;
   9     }
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007ae gcd`gcd(a=24, b=4) + 30 at gcd.c:7, stop reason = step over
    frame #0: 0x00000000004007ae gcd`gcd(a=24, b=4) + 30 at gcd.c:7
   4     int c;
   5     while (b != 0) {
   6       c = a % b;
-> 7       a = b;
   8       b = c;
   9     }
   10  }
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007b4 gcd`gcd(a=4, b=4) + 36 at gcd.c:8, stop reason = step over
    frame #0: 0x00000000004007b4 gcd`gcd(a=4, b=4) + 36 at gcd.c:8
   5     while (b != 0) {
   6       c = a % b;
   7       a = b;
-> 8       b = c;
   9     }
   10  }
   11  
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007ba gcd`gcd(a=4, b=0) + 42 at gcd.c:5, stop reason = step over
    frame #0: 0x00000000004007ba gcd`gcd(a=4, b=0) + 42 at gcd.c:5
   2   
   3   int gcd(int a, int b) {
   4     int c;
-> 5     while (b != 0) {
   6       c = a % b;
   7       a = b;
   8       b = c;
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x00000000004007bf gcd`gcd(a=4, b=0) + 47 at gcd.c:10, stop reason = step over
    frame #0: 0x00000000004007bf gcd`gcd(a=4, b=0) + 47 at gcd.c:10
   7       a = b;
   8       b = c;
   9     }
-> 10  }
   11  
   12  int main(int argc, char** argv) {
   13  int a; int b;
(lldb) next
Process 93155 stopped
* thread #1: tid = 100085, 0x0000000000400813 gcd`main(argc=1, argv=0x00007fffffffeb60) + 67 at gcd.c:15, stop reason = step over
    frame #0: 0x0000000000400813 gcd`main(argc=1, argv=0x00007fffffffeb60) + 67 at gcd.c:15
   12  int main(int argc, char** argv) {
   13    int a; int b;
   14    scanf("%d %d", &a, &b);
-> 15    printf("%d\n", gcd(a, b));
   16    return 0;
   17  }

We see the values of variables “a” and “b” decreases alone the time line.  Pay attention to the transition between the “gcd” function to the “main” function.  There is not a return value.  This is why we kept having the warning.  Compiler warnings are often more useful than what programmers think.  Sometimes, I am amazed why the open source software packages contain some many warnings in compilations and they still work.

The Final Version

We get to return the answer in the “gcd” function.  Here is how it the code being corrected.

#include <stdio.h>

int gcd(int a, int b) {
        int c;
        while (b != 0) {
                c = a % b;
                a = b;
                b = c;
        }
        return a;  // <-- this line
}

int main(int argc, char** argv) {
        int a;
        int b;
        scanf("%d %d", &a, &b);
        printf("%d\n", gcd(a, b));
        return 0;
}

The Final Testing

# cc gcd.c -o gcd
# ./gcd
0 5
5
# ./gcd
5 0
5
# ./gcd
1 9
1
# ./gcd
9 1
1
# ./gcd
4 36
4
# ./gcd
36 4
4
# ./gcd
24 36
12
# ./gcd
36 24
12

 

What can Go Wrong Building a Computer from Parts

Standard

Recently, I get itchy and wanted to build a gaming machine.  (Not running BSD, sorry)  Originally, I was thinking buying a Kaby Lake-based Intel Pentium G4560.  I was greedy and wanted a ECC capable motherboard.  The cost spiralled quickly close to a secondhand CPU-motherboard set with Intel Xeon E5-2670.  This particular Xeon processor model has a meaning to me—one of my important customers uses hundreds of these for high performance numerical simulations.  I just think, it is nice to own one of these as my souvenir, right?

I ordered stuffs on an online shopping mall somewhere.  (In order not to hurt their feeling, I decide not to name which country it is.)  The nightmare began.

Ordering

What can go wrong with ordering?  In that online mall, there are numerous shops run by different people of different service quality.  You are gambling with your luck if you order things from a new vendor.

Among the computer parts, I get rejected after payment for the memory modules.  I selected DDR3 ECC UDIMM for the first order, because it works with another of my computer with AMD 1055T processor as well.  Upon the payment, the shopkeeper was friendly and asked me to double check if the memory is compatible.  Once I verified it works, the shopkeeper told me to refund because it is out of stock.  I switched to buy DDR3 ECC RDIMM, which is less compatible to desktop computers, but much more readily available.

I also got rejected payment, indeed twice, for the keyboards.  I was greedy and ordered a niche second-hand UNICOMP keyboard with a red stick (ultranav?) with PS/2 plugs.  At last I gave up and just left it open, maybe sooner with a second-hand IBM Spacesaver Model M4-1… if I dare to try again.

Eventually, the products that could arrive, arrived in good shape, especially I ordered them through a Hong Kong-based consolidation shipping company.

Accuracy and Quality

The quality of the products are not satisfactory.  The only thing that I have no complaint is the memory modules, because they just sit in and work.

The Motherboard

Motherboard plays an important role in the computer.  While I do not expect top-notch quality, I do have some reasonable expectation to it.

Some people like smelling motherboards.  I was curious and gave my board a shot.  It was definitely different, awfully different, but this is just the beginning.  While the board is claimed to be new, the south-bridge heatsinks are obviously dented.

相片 2-3-2017 下午12 57 13

The original product picture found on the online store is as follows.  Can you spot two more problems?  I am really disappointed that they replaced a PCIe for PCI, and also left my flash drive port empty.

Screen Shot 2017-03-10 at 11.36.06 pm.png

At later stage of the build, I wanted to unplug the USB3 header cable temporary for cable management.  The plastic part of the header came out with the cable, just in one unplug!  I resorted to plug the USB3 cable onto the bare pins, without the plastic piece.

相片 3-3-2017 下午7 40 44

This motherboard make is now in my black list.

The Computer Case

I ordered the computer case with the fans and LED strips.  Frankly speaking, the computer case exceeds my expectation.  What disappoints me are the complimentary fans and the LED strips.  Instead of 3 or 4-pin fan headers, they all use MOLEX power plugs.  I got to have a dedicated MOLEX cable going out from the cable-management power supply, which is quite ugly if you ask me.

The Heatsink

Because the Xeon processor can go as much as 120W in peak load, I went to buy a very big heatsink with push-pull technology.  But it ends up being too tall with the computer case.  I was so amazed I made such a big mistake.  I should have checked the height clearance of the computer case and the heatsink dimensions.

Compatibility

After such a painful experience of building, I was actually quite surprise the setup boots up immediately without complaints.

The only issue happened when I try to install the NVidia graphics driver.  In order to get it installed, the computer has to boot into the text mode.  Yet, the text mode of that operating system is broken.  (No, it is not FreeBSD.)  I thought the computer did not show up the login screen because the systemd acted wrongly.  I tried reinstalling and also booting from the installation flash disk many times.  It eventually took me half a day to realise it was the graphics mode.

I am not quite sure what I can do if the computer fails to boot into graphical mode after an update.  Anyway, treat it as a game machine and I will be alright.

Bill of Materials

This is the bill of materials I used, including the ones I purchased before this.

  • Intel Xeon E5-2670 Processor, Qty: 1
  • X79 Motherboard, Qty: 1
  • DDR3-1600 ECC UDIMM 8 GB, Qty: 4
  • DDR3-1333 ECC RDIMM 8 GB, Qty: 4
  • Gigabyte GeForce GTX1050Ti Windforce OC 4G, Qty: 1
  • Antec High Gamer Power 520W, Qty: 1
  • Cooler Master T400 Pro, Qty: 1
  • Aigo ATX Computer Case, Qty: 1
  • UNICOMP Model M keyboard, Qty: 1
  • Cooler Master Rapid Fire 87-Key Keyboard, Qty: 1
  • Logitech Wireless Mouse M280, Qty: 1
  • Dell Ultrasharp U2412M Display, Qty: 1

If I only build from only new parts, the first three items becomes as follows.  The core count and memory channels are different.  Hopefully, the AVX2 instruction sets and the DDR4 memory would cover the difference.

  • Intel Xeon E3-1230 v5 Processor, Qty: 1
  • Gigabyte GA-X150-Pro ECC Motherboard, Qty: 1
  • DDR4-2133 ECC UDIMM 8 GB: Qty: 4

At the end, I saved around 300 USD, at the cost not having warranty on the parts.  Well, indeed, owning a computer with Xeon E5-2670 is priceless for me…

Lessons Learned

  • Place no expectation on the fans coming with a computer case.
  • Whenever possible, buy from a local brick-and-mortar store, even with a premium.
  • When buying niche stuff, be more verbose and ask for availability before payment.

OpenStack with FreeBSD

Standard

Recently, I come across NFV Express through the FreeBSDNews article.  The installation code is already openly available on Github.  At the first glance, the code is so amazing to be clean!  (You definitely should look at them if you know programming.)  When there is time, I will try them with my stack of FreeBSD computers…

Openstack is used to be Linux-centric, yet it is mostly written in Python scripts.  Theoretically, it is possible to make it running on any capable platforms… if you do not hate working with Python scripts.

Sorry for this short article.  This is indeed just a reminder for me to follow up.  🙂

Proxy Server with FreeBSD and Squid (Part 1)

Standard

When one has multiple machines within a cloud network, it is natural to ask for centralised network traffic, data files, credential service, etc.  In this article, I focus on centralising network traffic, in particular, the world wide web.

With the world wide web traffic centralised, the proxy server can accelerate the download processes of some frequently accessed files, such as operating system patches.

In this part, we install Squid as an opaque proxy server, and configure some clients to use it.  We will handle the transparent proxy in the next part.

Step 1: Install Squid

Installing Squid is easy as usual.  But to make it even smoother, install the SSL root certificates.

# pkg install squid ca_root_nss

The squid requires Perl which, since FreeBSD 5.0, no longer installed by default.  The package installer will handle it.

Step 2: Smoke Test

Even though there are quite a page of notice upon installing Squid, the default Squid installation is good enough to fire without configuration:

# sysrc squid_enable=YES
# service squid start

or, for the old style…

# echo squid_enable=YES >> /etc/rc.conf
# /usr/local/etc/rc.d/squid start

Step 3: Configure Squid

The Squid refers to its configuration file in “/usr/local/etc/squid”.

# ee /usr/local/etc/squid/squid.conf

 

What shall we configure?  Here are some suggestions:

  1. Configure the intranet address ranges.  By the configuration file, only networks defined as “localnet” is granted access.  For example, I only use the class A private subnet, so I disable all others.
    acl localnet src 10.0.0.0/8     
    #acl localnet src 172.16.0.0/12 
    #acl localnet src 192.168.0.0/16
    #acl localnet src fc00::/7      
    #acl localnet src fe80::/10
  2. Deny the squid accessing the so-called “localhost”.  Internal network of the squid is not supposed to be accessed by others.
    http_access deny to_localhost
  3. Configure the cache directory.  For example, here I configure a 10000 MB cache space.  There are two levels of directories, with 16 and 256 directories per level respectively.
    cache_dir ufs /var/squid/cache 10000 16 256
  4. Prefer IPv4 whenever possible.  This is great if your network is not IPv6 ready.
    dns_v4_first on

We then stop the Squid, create the cache directory, and restart.

# service squid stop
# squid -z -N
# chown -R squid:squid /var/squid/cache
# service squid start

Step 4: Update Firewall

If you are using PF firewall, you will need to update the rules to allow the port.  For example, I only use the “vtnet1” as intranet, and it is accepting port 3128 (the default squid service port).  You may refer to the configuration of the VPN notes as example.

# cat >> /etc/pf.conf << EOF
intif="vtnet1"
inttcpports="{3128}"
pass in quick on $intif inet proto tcp from any to any port $inttcpports keep state
EOF
# service pf reload

Step 5: Playing with Telnet

The title is not a typo.  We now test the system with the most geeky way—enter the HTTP request manually.  FreeBSD comes with a telnet client and it is handy for various testing.  Suppose the address is 10.65.0.1 and we connect to the port 3128 of it.  After typing the GET statement, remember to hit the enter key twice to complete the request.

# telnet 10.65.0.1 3128
GET http://www.freebsd.org/ HTTP/1.0

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Tue, 07 Mar 2017 16:04:30 GMT
Content-Type: text/html
Content-Length: 178
Location: https://www.freebsd.org/
X-Cache: MISS from singapura-roteador
X-Cache-Lookup: HIT from singapura-roteador:3128
Via: 1.1 singapura-roteador (squid/3.5.23)
Connection: close

<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
Connection closed by foreign host.

The HTTP return code 301 indicates another URL for access.  What about following it?

# telnet 10.65.0.1 3128
GET https://www.freebsd.org/ HTTP/1.0

Step 6: Troubleshoot

Here are some errors you may encounter:

Unable to connect to the server: try disabling the “pf” service briefly and retry.  If it works, you will need to update your firewall rules.

Squid is not starting after enabling cache: make sure you create the cache directory and set the proper ownership with the chown command (as above) .

Getting 503 error with ERR_CONNECT_FAIL: review the error message and if the address is an IPv6 address, there could be issues with the IPv6 routing and you can consider using the “dns_v4_first” option (as above).

Getting 503 error with ERR_SECURE_CONNECT_FAIL: review the website with your own browser, ensure you have installed the common root certificates “ca_root_nss” package (as above).

Step 7: Configure Client

Since our proxy is not transparent yet, the clients have to be manually configured in order to use the proxy.  Depending on the hosts you have on the same intranet, here are the configuration methods for your reference:

Mac OS X: System Preferences > Network > Advanced > Proxies > Web Proxy / Secure Web Proxy

Internet Explorer on Windows: Tools > Internet Options > Connections > LAN Setting

Firefox on Windows: Tools > Internet Options > Network > Connections > Settings

FreeBSD shell: export “HTTP_PROXY” variable in “.cshrc”, like “setenv HTTP_PROXY http://10.65.0.1:3128&#8221;

Step 8: Is the Proxy Really Running?

There are some barbaric ways to check if the proxy is running:

Check the number of files the cache: “find /var/squid/cache | wc -l”

Check the Squid access log: “less /var/log/squid/access.log”

Step 9: To be Continued

A few weeks later, we will revisit Squid and configure the network routers to transparently direct world wide web requests to the proxy server, etc.

Later this week, we may come up with an article not related with FreeBSD.  The next FreeBSD-related article, hopefully next week, will be likely related to programming.  Stay tuned.

Configuring FreeBSD

Standard

We have come across some of my preferred customisations.  In this article, I explain how the files are being modified and their relationships with the system.  It is hoped that this will give new comers an idea why and what to configure with a fresh FreeBSD installation.

What to Configure?

To understand how FreeBSD can be configured, we first understand the parts that can be configured.  The startup process is roughly as follows and I will cover configuration of each stage in later sections, arranged according to the likeness one will use.

  1. The computer loads and execute the loader (more on step 6)
  2. The loader mounts according to the file system table (more on step 5)
  3. The loader loads the kernel into memory and execute (more on step 4)
  4. The init program is executed as the first program (not covered)
  5. The device file system is mounted (not covered)
  6. The rc program prepares the user space (more on step 1)
  7. Network neighbours are recognised (more on step 2)
  8. The periodic jobs are run (more on step 3)

Step 0: Modifying Files

Modifying a configuration file could be easy, but there are some geeky ways to do it right.

To read a configuration file, one uses the cat(1) command.  Before some fluffy haters jump out and complain, let me say, cat means concatenate.  That said, the concatenation tool is also useful dumping content of a single file.

Quite some FreeBSD configuration files are in name-value pairs, such as:

# cat /etc/rc.conf
hostname="raquel"

In these cases, the sysrc(8) tool acts as a safe shortcut to modify the files.  For example:

# sysrc -f /etc/rc.conf hostname="lea"
# cat /etc/rc.conf
hostname="lea"

You can use the same cat to pipe in the output.  The >> symbol means appending to a file; contents following the command will be appended.  (If you use > instead, you overwrite it instead.)  The << defines the termination string, the termination string will not be inserted and it will be returned to the shell.

# cat >> /etc/rc.conf << EOF
hostname="raquel"
EOF
# cat /etc/rc.conf
hostname="lea"
hostname="raquel"

Since these are just name-value pairs, it does not harm more than causing your own confusions.  The latest value assignment takes effect.

Last but not the least, for the files you want to manually edit, you can always use the ee(1) easy editor, or the visually (difficult) vi(1) editor.

Step 1: Run Command

To a new user, the most likely item to configure first is the run command.  At later stage of system start, the rc(8) starts the desired processes for the users.  Most of these configuration are done in two files.  The values desired by the user goes to “/etc/rc.conf”.  When a value required is absent, the default value is loaded from “/etc/defaults/rc.conf”.  The former is empty or even absent after a fresh system install.  The latter is installed and updated automatically and is very lengthy.  By the way, the lengthy file can also act as your configuration guide; more on these later.

Step 1.1: Network Configuration

FreeBSD uses ifconfig(8) to configure network interfaces.  For example, to configure network addresses of interface vtnet1 (the second virtual network interface), the default gateway and the hostname, one can use:

# sysrc vtnet1_ifconfig="inet 10.65.0.11 netmask 255.255.255.0"
vtnet1_ifconfig:  -> inet 10.65.0.11 netmask 255.255.255.0
# sysrc defaultrouter="10.65.0.1"
defaultrouter: NO -> 10.65.0.1
# sysrc hostname="myhost"
hostname:  -> myhost

In the example above, vtnet1 device exists because it exists in the hardware layer.  Sometimes, the interfaces are purely software, like the gif(4) interface we mentioned in building virtual network.  They can be generated in boot time with the cloned interface.  Note there is a plus sign before the equal.  This appends new values to the existing values.  Once the interfaces are created, they can be configured with the typical “ifconfig” above.

# sysrc cloned_interfaces+="gif0"
cloned_interfaces:  -> gif0
# sysrc cloned_interfaces+="gif1"
cloned_interfaces: gif0 -> gif0 gif1

Step 1.2: Service Configuration

The run command file allows defining what services to be switched on at boot.  These services will be switched off gracefully at shutdown as well.  For example, to enable the sshd(8) daemon, one will write this in “/etc/rc.conf”.

sshd_enable="YES"

As an unofficial but nevertheless useful way, one can search for “_enabled” in the “/etc/defaults/rc.conf” to see what can be enabled or disabled:

# grep _enabled /etc/defaults/rc.conf
apm_enable="NO" # Set to YES to enable APM BIOS functions (or NO).
apmd_enable="NO" # Run apmd to handle APM event from userland.
ddb_enable="NO" # Set to YES to load ddb scripts at boot.
devd_enable="YES" # Run devd, to trigger programs on device tree changes.
kldxref_enable="NO" # Build linker.hints files with kldxref(8).
powerd_enable="NO" # Run powerd to lower our power usage.
...

In short, some interesting services are:

  • ntpd(8): network time daemon, keep system time synchronised
  • pf(4): packet filter (or, firewall), useful for protecting your system
  • sendmail(1): mail server, in the previous page I recommend disabling it
  • sshd(8): secure shell server, essential for you to login remotely

Step 1.3: Startup Script

One more place to configure run command is to directly type them into “/etc/rc.local”.  Indeed, this was how FreeBSD 4.x or before worked.  Every time a service was installed, the administrator opened an editor and input the required startup scripts there.  It is still the preferred way for some other BSD family operating systems.

Step 2: Host Tables and Resolvers

When you have multiple machines connected in a network, you will want alias to them.  FreeBSD provides a hosts(5) table “/etc/hosts”.  As long as you can handle and synchronise the files on the different hosts, this is the most reliable and straight-forward method.  (If not, you will need a centralised place, like a name server, for this purpose, and it is out of my scope today.)

One more network configuration is the domain name service, transforming domain names to IP addresses which are understandable to the network.  It is configured with the resolv.conf(5) in “/etc/resolv.conf”.  If you are using a cloud, it is usually automatically configured with DHCP.

Step 3: Periodic Jobs

There are quite a few ways to let jobs run periodically.  I cover the maintenance and cron jobs here.  The former is more common to execute some system default cleanup activities.  The latter is more common for administrators’ customisations.  In the previous article, I changed some maintenance job settings and used a cron job to periodically clean the network offenders (in case those are false positives).

Step 3.1: Maintenance Jobs

FreeBSD has periodic(8) for running periodic maintenance jobs.  The settings are in “/etc/periodic.conf” and the default settings are in “/etc/defaults/periodic.conf”.

The file is read whenever the periodic job is run.  After updating the periodic configuration, you can just let the next periodic job to run.  Nothing special is needed to reload the configuration.

By default, the periodic jobs output are sent to the local mail server.  The step 2 of the customisation script disables these mails and direct the output to the file.  This saves you a few daily emails per server and provides one less attack vector.

Step 3.2: Cron Jobs

FreeBSD has cron(8) for running arbitrary commands on given time.  There is a master cron job file available as plain crontab(5) table “/etc/crontab”, and a crontab(1) database for each user.  Readers may feel strange the two items are both called crontab, with different numbers in the suffix.  In short, the (5) is a file, and (1) is a command.

When editing the crontab(5), the format is as follows:

minute hour day month weekday user command args...

For example, the default FreeBSD comes with cron jobs that run the periodic jobs mentioned in the previous subsection.  This means at every 03:01, command “periodic daily” is executed as the root user; at every 4:15 on Saturday, command “periodic weekly” is executed as the root user, etc.

 1 3 * * *  root  periodic daily
15 4 * * 6  root  periodic weekly
30 5 1 * *  root  periodic monthly

When editing with crontab(1), one uses “crontab -e” to edit his own list.  This gives the user advantage of defining his own jobs without modifying the master table.  The format is as follows, which is similar, yet the username is skipped for the obvious reason.

minute hour day month weekday command args...

Please note in the crontabs, it sometimes lacks the conventional environment variables.  In particular, you may need to be more specific in pathnames, such as calling “/sbin/pfctl” instead of plainly “pfctl”.

Step 4: Kernel System Settings

FreeBSD provides sysctl(8) utility for configuring kernel and kernel module settings, and reading some system status as well.

Kernel settings can be configured in “/etc/sysctl.conf”, with a default file in “/etc/defaults/sysctl.conf”.  The files are read on boot time.  After adding new system configurations, you can either reboot to make it in effect, or run this command to reload immediately: “sysctl -f /etc/sysctl.conf”

These variables are useful when a user wants to have more control on the kernel behaviour.  More on these can be found on the manual page tuning(7).  You can list them by calling “sysctl -a”.  The output is long, prepare your terminal to scroll back.

To a cloud server, most of the activities are network transfer; some interesting configurations are:

  • net.inet.tcp.sendspace: bytes of initial space for sending for each TCP session
  • net.inet.tcp.sendbuf_inc: bytes of sending space increment per session
  • net.inet.tcp.sendbuf_max: bytes of maximum space for receiving per session
  • net.inet.tcp.recvspace: bytes of initial space for receiving for each TCP session
  • net.inet.tcp.recvbuf_inc: bytes of receiving space increment for session
  • net.inet.tcp.recvbuf_max: bytes of maximum space for receiving per session
  • kern.ipc.nmbclusters: total amount of space for network memory (in 2 KiB blocks)
  • kern.ipc.somaxconn: maximum pending connections

By default, the network space values make sense.  The sending space and receiving space are 32 KiB and 64 KiB respectively.  This incurs 96 KiB when a connection is made.  These can go up to 2 MiB for sending and another 2 MiB for receiving.  The system reserves around 100 MiB network memory by default, which is enough for even for gigabit network.

If you find a real need, you can consider adjusting these values depending on the nature of the server.  These values can be lowered if the server has a lot of connections serving small files and messages.  They can be raised if the server has only a few connections and each serving large files and messages.  If the server has a lot of connections each serving large files, the total amount of space for network memory has to be increased.  You can check the buffer usage with command “netstat -m”.

The maximum pending connections, is set to be 128 as default value.  That means, if there are 129 incoming connections together and the server software is not yet capable of completing any of the handshakes, the 129th will be lost.  It is suggested to be set to 1024 or even higher.  This will prevent the system being jammed easily by denial-of-service attack.  If you have applied the customisation script, a client will be blocked if it tries to make too many connections in a short time.

Step 5: File System Table

FreeBSD uses file system table fstab(5) to define the file systems to be mounted.  The table is read by the loader (below) when the system boots.  (A file is read from the volume before the volume is officially mounted.  This is quite counter-intuitive if you think again.  But it make things much simpler.)  If there are errors, one may end up locking the system from booting.

The file format is as follows:

block mount fstype options dump pass

For example, one of my virtual machine says:

/dev/ufs/rootfs  /  ufs  rw  1  1

This means the UFS volume labeled “rootfs” to be mounted at the root directory /, as an UFS file system, read and write enabled, dump enabled, and checked for consistency first after rebooting from system crash.

Normally, this file is generated when you have a virtual machine ready.  It is useful to modify it later when you insert disks to a virtual (or physical) machine.

Step 6: Loader Settings

Eventually, a user may want to configure how a kernel is loaded, and what kernel modules to be loaded together.  FreeBSD comes with a simple yet efficient loader(8) for the purpose.  The configuration values are in “/boot/loader.conf” and defaults in “/boot/defaults/loader.conf”.  I recommend against changing these files randomly because they really affect how the kernel is loaded.  Some settings I ever changed are:

  • kern.maxproc: maximum processes in a system
  • kern.maxfiles: maximum open file handles (including sockets) in a system
  • geom_mirror_load: load the software RAID module (for booting the kernel)
  • coretemp_load: load the module to detect Intel Core™ processor temperature
  • amdtemp_load: load the module to detect AMD processor temperature

The maximum processes, and maximum open file handles are set according to the memory available in the machine.  They are suggested to be set to a predefined value if you have a lot of open files or network sockets, or when the machine is to be shared by a lot of users.  The process table and file table are initialised once and only once when the kernel loads, thus they are to be set in here instead of sysctl.

 

Step 7: Skeleton Files

Few accounts use only the default “root” user account.  To have consistent personalisation across various created users, administrators often prepare skeleton files.  The skeleton files are located at “/usr/share/skel”.  For example, “/usr/share/skel/dot.cshrc” will be copied to “$HOME/.cshrc” of the newly created user.

Once the account is created, the file is owned by the created user.  There are not automatic ways to update across multiple users.  It is therefore important to have the customisations right before creating the users.

Conclusion

We have gone through various parts in FreeBSD that can be configured, especially in context to a new comer.  We discussed how to configure the rc.conf, the host table, the resolver configuration, placing periodic jobs, adjusting kernel settings, setting file system mount points, kernel loading options, and finally skeleton files.

Virtual Private Network with FreeBSD (Part 2)

Standard

This is the second and the final part on setting up a private virtual network.  The part 1 has been uploaded about one week ago.

If you are following the guide in FreeBSD handbook, you must be frustrated because of the complicated configurations.  Those configurations are good in the sense it restricts the IPsec to particular IP pairs but they are also too tough for purpose of connecting just two routers.  In this example, I break these assumptions and go as minimal as possible so the readers can have a smoother learning curve.  I protect the private ports from unsolicited connections with the help of PF firewall instead of the IPsec configuration.

Step 7: Install Packages

Install “ipsec-tools” package on the routers.

pkg install ipsec-tools

Step 8: Configure IPsec Rules

Set the rules in IPsec in japao-roteador.  These two lines say:

  1. IP packets, in any protocols, from the Japan network (10.81.0.0) to the Singapore network (10.65.0.0) is sent out encapsulated in tunnel mode, from the japao-roteador (45.76.207.156) to cingapura-roteador (47.76.133.70), with the encryption mandatory.
  2. IP packets, in any protocols, from the Singapore network to the Japan network is accepted in encapsulated in tunnel mode, from cingapura-roteador to japao-roteador, with the encryption mandatory.
    # touch /usr/local/etc/racoon/setkey.conf
    # cat >> /usr/local/etc/racoon/setkey.conf << EOF
    spdadd 10.81.0.0/24 10.65.0.0/24 any -P out ipsec esp/tunnel/45.76.207.156-45.76.153.70/require;
    spdadd 10.65.0.0/24 10.81.0.0/24 any -P in ipsec esp/tunnel/45.76.153.70-45.76.207.156/require;
    EOF

On another router cingapura-roteador, the in and out are reversed for the obvious reason:

  1. Incoming: the Japan network to the Singapore network
  2. Outgoing: the Singapore network to the Japan network
  3. # touch /usr/local/etc/racoon/setkey.conf
    # cat >> /usr/local/etc/racoon/setkey.conf << EOF
    spdadd 10.81.0.0/24 10.65.0.0/24 any -P in ipsec esp/tunnel/45.76.207.156-45.76.153.70/require;
    spdadd 10.65.0.0/24 10.81.0.0/24 any -P out ipsec esp/tunnel/45.76.153.70-45.76.207.156/require;
    EOF

Please note I used the word “require” instead of “use” as written by some other documentations.  It means the message must be in the expected form to be accepted.  If the encryption is turned off accidentally, it should be disregarded rather than passing through.

Step 9: Configure Racoon

Racoon, as installed with the “ipsec-tools” package, is going to help us the IPsec negotiation phase.  Setting up the pre-shared keys is as easy as updating a file, with each line of the IP address, and then a secret string.  The file will be referenced by the Racoon script.  Setting up the Racoon main configuration can be much more straight forward than the handbook.  I copied an example in “/usr/local/share/ipsec-tools” and customised it.  The two routers will have very similar configuration except the listening IP.

  1. On japao-roteador:
    # touch /usr/local/etc/racoon/psk.txt
    # chmod 0600 /usr/local/etc/racoon/psk.txt
    # cat >> /usr/local/etc/racoon/psk.txt << EOF
    45.76.207.156 pleasedonotusethis
    EOF
    
    # cat > /usr/local/etc/racoon/racoon.conf << EOF
    path pre_shared_key "/usr/local/etc/racoon/psk.txt";
    log debug;
    
    listen
    {
      isakmp          45.76.153.70;
    }
    
    remote anonymous
    {
      exchange_mode     main,aggressive;
      lifetime          time 8 hour;
      passive           off;
      proposal_check    obey;
      nat_traversal     off;
      generate_policy   off;
      proposal {
        encryption_algorithm   3des;
        hash_algorithm         sha1;
        authentication_method  pre_shared_key;
        dh_group               1;
      }
    }
    sainfo anonymous
    {
      pfs_group                 1;
      lifetime                  time 12 hour;
      encryption_algorithm      3des;
      authentication_algorithm  hmac_sha1;
      compression_algorithm     deflate;
    }
    EOF
  2. On cingapura-roteador:
    # touch /usr/local/etc/racoon/psk.txt
    # chmod 0600 /usr/local/etc/racoon/psk.txt
    # cat > /usr/local/etc/racoon/psk.txt << EOF
    45.76.153.70 pleasedonotusethis
    EOF
    
    # cat > /usr/local/etc/racoon/racoon.conf << EOF
    path pre_shared_key "/usr/local/etc/racoon/psk.txt";
    log debug;
    
    listen
    {
      isakmp          45.76.207.156;
    }
    
    remote anonymous
    {
      exchange_mode     main,aggressive;
      lifetime          time 8 hour;
      passive           off;
      proposal_check    obey;
      nat_traversal     off;
      generate_policy   off;
      proposal {
        encryption_algorithm   3des;
        hash_algorithm         sha1;
        authentication_method  pre_shared_key;
        dh_group               1;
      }
    }
    sainfo anonymous
    {
      pfs_group                 1;
      lifetime                  time 12 hour;
      encryption_algorithm      3des;
      authentication_algorithm  hmac_sha1;
      compression_algorithm     deflate;
    }
    EOF

Step 10: Enable Service

Almost finish.  We enable the services and start them as soon as possible.

# cat >> /etc/rc.conf << EOF
racoon_enable="YES"
ipsec_enable="YES"
ipsec_program="/usr/local/sbin/setkey"
ipsec_file="/usr/local/etc/racoon/setkey.conf"
EOF
# service racoon start
# service ipsec start

Step 11: Update Firewall Rules

We have been using firewall.  In the step 3 of part 1, we allowed peers to pass through with encapsulated messages.  Now, we let a few more items to go through: the authentication AH packets, the encapsulated ESP messages.

# cat >> /etc/rc.conf << EOF
pass in quick inet proto {ah,esp,ipencap} from $encappeers to any
pass in quick inet proto udp from $encappeers port = 500 to any port = 500
EOF

Step 12: Try to Break Something

Hopefully, things should go well when you ping the two hosts.  If not, you can watch the debug log “/var/log/debug.log” for a bit more insights.  The next step is to break the tunnel.  Here are some examples.

  1. Try stopping Racoon on one of the routers
  2. Try stopping IPsec on one of the routers
  3. Try using different pre-shared keys on the two routers
  4. Repeat step 1, 2, 3 with “use” rather than “require” in the IPsec configuration
  5. Deleting the route on one of the routers

Virtual Private Network with FreeBSD (Part 1)

Standard

With inexpensive and agile cloud services, it is not unusual for a service to have multiple points of presence around the globe.  In each point of presence, there is usually a per-tenant private network for internal (and confidential) system communications.  One can go further and combine these internal networks together with virtual links.   Here I explain the steps how this can be achieved with FreeBSD without relying extra networking services from the cloud vendors.

The article is split into two parts due to the length.  The first part, this, is about creating a tunnel with ip-in-ip encapsulation.  The second part is about encrypting the communications between the two hosts.

Concepts

Let’s say we have two private networks, network 1 (10.81.x.x) in Japan, and network 2 (10.65.x.x) in Singapore.  The two networks are connected through routers in the public internet.  However, these routers are not acknowledged about our own private network addresses, so we cannot send our messages directly between the two sites.  To enable private communication between the two sites, we will need network tunnel technique and network encryption.

A tunnel can be created by establishing one host on each private network as a router.  Each of these routers have a public routable network address.  Whenever communications between the two private networks have to be done, the network packets are sent through their respective routers.  The routers encapsulate the packets in new packets with “proper” addresses, so that the public routers understand how to route.  Once the encapsulated packets arrive the destination router, it is decapsulated for internal forwarding.  We deploy the FreeBSD generic tunnel interfaces for this part.

The messages between the two private networks are usually secret and thus need to be encrypted.  We can configure the routers so that whenever they communicate each other, the network packets have to be encrypted.  In the second half of this topic, we will employ IPSec (alternative link) for this purpose.

Notations

In this article, the commands are prepared in preformatted text, with pound (#) sign in front of each.  There are two types of commands, one that configures the system settings permanently and one that configures the current prevalent configurations.  The former are usually in sysrc(8) and cat(1) commands.  The latter are usually ifconfig(8) and sysctl(8).

Step 0: Environment

In this setup, we will prepare two private networks, each with a end client and a router.  I use the most inexpensive plans as possible.  As of today (Feb 2017), it is common to find virtual machines around 5 US dollars per month.  I used Vultr for the demonstration and they provide virtual machines at 768 MB for 5 dollars a month.  I assume you apply my customisation script for a minimal system and firewall configuration.

  1. Host 1: japao-roteador
    • Site: Tokyo
    • Plan: anything with 512 MB memory
    • Options: enable external and internal network
    • External interface: vtnet0 (DHCP assigned: 45.76.207.156)
    • Internal interface: vtnet1 (inet 10.81.0.1  netmask 255.255.255.0)
      # sysrc ifconfig_vtnet1="inet 10.81.0.1 netmask 255.255.255.0"
      # ifconfig vtnet1 inet 10.81.0.1 netmask 255.255.255.0
  2. Host 2: japao-host
    • Site: Tokyo
    • Plan: anything with 512 MB memory
    • Options: enable internal network
    • External interface: vtnet0 (DHCP assigned: 45.32.11.250)
    • Internal interface: vtnet1 (inet 10.81.0.11  netmask 255.255.255.0)
      # sysrc ifconfig_vtnet1="inet 10.81.0.11 netmask 255.255.255.0"
      # ifconfig vtnet1 inet 10.81.0.11 netmask 255.255.255.0
  3. Host 3: cingapura-roteador
    • Site: Singapore
    • Plan: anything with 512 MB memory
    • Options: enable external and internal network
    • External interface: vtnet0 (DHCP assigned: 45.76.153.70)
    • Internal interface: vtnet1 (inet 10.65.0.1  netmask 255.255.255.0)
      # sysrc ifconfig_vtnet1="inet 10.65.0.1 netmask 255.255.255.0"
      # ifconfig vtnet1 inet 10.65.0.1 netmask 255.255.255.0
  4. Host 4: cingapura-host
    • Site: Singapore
    • Plan: anything with 512 MB memory
    • Options: enable internal network
    • External interface: vtnet0 (DHCP assigned: 45.76.145.49)
    • Internal interface vtnet1 (inet 10.65.0.11  netmask 255.255.255.0)
      # sysrc ifconfig_vtnet1="inet 10.65.0.11 netmask 255.255.255.0"
      # ifconfig vtnet1 inet 10.65.0.11 netmask 255.255.255.0

Step 1: Tunnel and Route between the Routers

Next step, we will need to turn the designated hosts as routers, create generic tunnel network interface (gif) each of the routers, and set the route accordingly.

  1. Host 1: japao-roteador
    1. Internal: 10.81.0.1 (local) to 10.65.0.1 (counterpart)
    2. External: 45.76.207.156 (local) to 45.76.153.70 (counterpart)
    3. Route: 10.65.0.0/24 (network in Singapore) to 10.65.0.1 (router in Singapore)
      # sysrc gateway_enable="YES"
      # sysrc cloned_interfaces+="gif0"
      # sysrc ifconfig_gif0="10.81.0.1 10.65.0.1 netmask 255.255.255.0"
      # sysrc ifconfig_gif0+="tunnel 45.76.207.156 45.76.153.70"
      # sysrc static_routes+="cingapura"
      # sysrc route_cingapura="10.65.0.0/24 10.65.0.1"
      
      
      # sysctl net.inet.ip.forwarding=1
      # ifconfig gif0 create
      # ifconfig gif0 10.81.0.1 10.65.0.1 netmask 255.255.255.0
      # ifconfig gif0 tunnel 45.76.207.156 45.76.153.70
      # route add 10.65.0.1/24 10.65.0.1
  2. Host 3: cingapura-roteador
    1. Internal: 10.65.0.1 (local) to 10.81.0.1 (counterpart)
    2. External: 45.76.153.70 (local) to 45.76.207.156 (counterpart)
    3. Route: 10.81.0.0/24 (network in Japan) to 10.81.0.1 (router in Japan)
      # sysrc gateway_enable="YES"
      # sysrc cloned_interfaces+="gif0"
      # sysrc ifconfig_gif0="10.65.0.1 10.81.0.1 netmask 255.255.255.0"
      # sysrc ifconfig_gif0+="tunnel 45.76.153.70 45.76.207.156"
      # sysrc static_routes+="japao"
      # sysrc route_japao="10.81.0.0/24 10.81.0.1"
      
      # sysctl net.inet.ip.forwarding=1
      # ifconfig gif0 create
      # ifconfig gif0 10.65.0.1 10.81.0.1 netmask 255.255.255.0
      # ifconfig gif0 tunnel 45.76.153.70 45.76.207.156
      # route add 10.81.0.1/24 10.81.0.1

Step 2: Update Firewall Rules of the Routers

You will notice that after setting the items above, the router in Singapore still cannot obtain ping reply from the internal network address of the router in Japan (10.81.0.1), and vice versa.  If the pf firewall on one side is disabled, it works.  This is because the computers are having a firewall enabled (per our standard customisation script).  Let us relax the rules to let these through.

  1. Host 1: japao-roteador
    1. Counterpart peer: cingapura-roteador (45.76.153.70)
    2. Protocol to allow: ipencap
      # cat >> /etc/pf.conf << EOF
      encappeers="{45.76.153.70}"
      pass in quick inet proto ipencap from $encappeers to any
      EOF
      # service pf reload
  2. Host 2: cingapura-roteador
    1. Counterpart peer: japao-roteador (45.76.207.156)
    2. Protocol to allow: ipencap
      # cat >> /etc/pf.conf << EOF
      encappeers="{45.76.207.156}"
      pass in quick inet proto ipencap from $encappeers to any
      EOF
      # service pf reload

 

Step 3: Configure Routing on the Clients

Similarly, we add static routes to the client hosts.  Please note they are subtlety different.  For example:  The router in Japan is directing Singapore traffic to the router in Singapore.  The host in Japan is directing Singapore traffic to the router in Japan instead.

  1. Host 2: japao-host
    1. Route: 10.65.0.0/24 (network in Singapore) to 10.81.0.1 (router in Japan)
      # sysrc static_routes+="cingapura"
      # sysrc route_cingapura="10.65.0.0/24 10.81.0.1"
      # route add 10.65.0.0/24 10.81.0.1
  2. Host 4: cingapura-host
    1. Route: 10.81.0.0/24 (network in Japan) to 10.81.0.1 (router in Singapore)
      # sysrc static_routes+="japao"
      # sysrc route_cingapura="10.81.0.0/24 10.65.0.1"
      # route add 10.81.0.0/24 10.65.0.1

Step 4: Test

  1. To test the routers, use command ping from japao-reteador to 10.65.0.1 (Singapore), and from cingapura-roteador to 10.81.0.1 (Japan).  Nice.  The two sites are 76 milliseconds apart.
    root@japao-roteador ~# ping 10.65.0.1
    PING 10.65.0.1 (10.65.0.1): 56 data bytes
    64 bytes from 10.65.0.1: icmp_seq=0 ttl=64 time=76.978 ms
    64 bytes from 10.65.0.1: icmp_seq=1 ttl=64 time=76.898 ms
    64 bytes from 10.65.0.1: icmp_seq=2 ttl=64 time=76.859 ms
    64 bytes from 10.65.0.1: icmp_seq=3 ttl=64 time=76.903 ms
    64 bytes from 10.65.0.1: icmp_seq=4 ttl=64 time=76.758 ms
    64 bytes from 10.65.0.1: icmp_seq=5 ttl=64 time=77.120 ms
    ^C
    --- 10.65.0.1 ping statistics ---
    6 packets transmitted, 6 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 76.758/76.919/77.120/0.111 ms
  2. To test the hosts, use command ping from japao-host to 10.65.0.11 (Singapore host), and from cingapura-host to 10.81.0.11 (Japan host).  There is slightly one more millisecond in the round trip time.
    root@japao-host ~# ping 10.65.0.11
    PING 10.65.0.11 (10.65.0.11): 56 data bytes
    64 bytes from 10.65.0.11: icmp_seq=0 ttl=62 time=77.892 ms
    64 bytes from 10.65.0.11: icmp_seq=1 ttl=62 time=77.817 ms
    64 bytes from 10.65.0.11: icmp_seq=2 ttl=62 time=78.710 ms
    64 bytes from 10.65.0.11: icmp_seq=3 ttl=62 time=78.227 ms
    64 bytes from 10.65.0.11: icmp_seq=4 ttl=62 time=78.444 ms
    ^C
    --- 10.65.0.11 ping statistics ---
    5 packets transmitted, 5 packets received, 0.0% packet loss
    round-trip min/avg/max/stddev = 77.817/78.218/78.710/0.335 ms

Step 5: Troubleshoot

I hope you do not need this step.  Anyway…

When you are unable to receive a ping reply, there are mostly two reasons.  Either the ping request is unable to reach the destination, or the ping reply could not transverse back to the requestor.  Seeing the packets on both the hosts simultaneously gives you insight which case it is.  To see if there are any traffic in the tunnel, issue on the routers:

# tcpdump -i gif0

Why is a packet unable to appear in the network?  There are mostly two reasons.  Either the network firewall blocks it (unlikely if you follow my instructions) or the route is set incorrectly.  To see if the firewall state table and the route table:

# pfctl -s states
# netstat -rn

Step 6: To be Continued

At this stage, when you use “tcpdump” on the interface, you get messages like:

# tcpdump host 45.76.153.70
16:09:26.759030 IP 45.76.207.156.vultr.com > 45.76.153.70.vultr.com: IP 10.81.0.1 > 10.65.0.1: ICMP echo reply, id 53509, seq 9, length 64 (ipip-proto-4)

And we want it to be encrypted, like:

# tcpdump host 45.76.153.70
16:09:32.314383 IP 45.76.207.156.vultr.com > 45.76.153.70.vultr.com: ESP(spi=0x004a1fe7,seq=0x1c), length 116

In the next part, I will go through the steps to encrypt the tunnel.