Simple Experiment with Jails and Resource Control

Standard

While it has been a good practise to have one computer doing one role, it has also been  asked if a computer can be split for multiple functions as if it is a group of computers.  One would immediately answer virtual machines, but it can be too heavy and complicated, say like having virtual machines inside the virtual machines you ordered in a public cloud.  Another would say container and cgroup in L…, em, what?  In other operating systems, we have other more battle-proven stuffs like zones in Solaris and Jails in FreeBSD.  Jails in FreeBSD had not been too attractive to me because it obviously lacked the resource control so that out-of-control processes can make resource starvation for others.  Thankfully, since FreeBSD 9.0, the rctl(8) has made the world a fairer place.

How simple is it to make a jail and try resource control, and gain confidence it works?  The suggested method in the FreeBSD handbook is to make a directory and reinstall FreeBSD there with the appropriate distribution files.  It is a generic method but it can be more inspiring.  Let’s begin.  In this article, we will demonstrate writing a dummy program, instantiate it in a jail, and fiddle the resource control.

System Setup

First of all, in order to use the resource control, add the following line to the loader.conf(5) /boot/loader.conf.  The file could be missing on a fresh installation, you can create a new one if it is the case.  After this, reboot to make it effective.

kern.racct.enable="1"

The Program and Compilation

Let us go to do some programming.  We want a dummy program that consumes some memory and processing power.  Here is my randomly crafted C program.  It is deliberately running some nonsense loops and leaking some memory.  Once every 10 seconds, it allocates 16 MB (malloc) of memory and uses 1 MB (memset) of it.  (The remaining 15 MB is lazily allocated and forgotten.)  By watching the lines printed, you get a sense how fast the process runs.  The longer it runs, the more memory it consumes.

#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <string.h>
#include <time.h>

int main(int argc, char** argv) {
  time_t then, now;
  int i, j, allocated;
  void* region;

  time(&then);
  time(&now);
  i = j = allocated = 0;
  while (1) {
    while(now - then < 10) {
      for (i = 0; i < (1 << 25); ++i)
        j *= i;
      time(&now);
      printf("now: %ld\n", now);
    }
    then = now;
    region = malloc(1 << 24);
    memset(region, j % 23, (1 << 20));
    allocated += 16;
    printf("allocated: %d\n", allocated);
  }
  return 0;
}

Usually, the program can be compiled simply with clang(1) command.  In other to compile for the jail in the most lazy method, call clang with the “-static” option.  The executable is significantly larger.  Mark down the absolute path of the directory so you can do the next step.

# ls
resource.c
# clang resource.c -o resource -static -Wall
# ls
resource resource.c
# ldd resource
ldd: resource: not a dynamic ELF executable
# pwd
/home/ssching

Running inside a Jail

Then, it’s time to start the jail.  The simplest way is to create a whatever jail(1), encapsulate at the path at the ssching’s home, and execute the executable we compiled.

# jail -c -u ssching path=/home/ssching \
  exec.start='./resource' name=myjail

Open another terminal and issue the top(1) command.  Press key “j” once and you see a process with non-zero jail number.  We see the process is consuming 100% CPU power and the memory usage is increasing.

 PID   JID  ...  SIZE  RES STATE C TIME    WCPU COMMAND
2648    14  ... 8288K 872K CPU1  1 0:09 100.67% resource

Stopping a Jail

Stopping a jail, attached or not, can be as simple as “-r” command.  Upon execution, all the processes in the jail will be killed.

# jail -r myjail

Processor Resource Control

Open yet another terminal and issue the following commands.

# rctl -a jail:myjail:pcpu:deny=50

The usage becomes a bit tamed but it fluctuates a lot from time to time.

 PID   JID ...  SIZE    RES STATE C TIME    WCPU COMMAND     
2648    14 ...  392M 25544K CPU3  3 3:52  50.83% resource

Memory Resource Control

Similarly, the virtual memory usage, or physical memory usage can be constrained as follows.  The virtual memory counts the total memory (SIZE in the top display) allocated, the physical memory counts the actual memory used (RES in the top display).

# rctl -a jail:myjail:vmemoryuse=240M
# rctl -a jail:myjail:memoryuse=120M

If memory usage is beyond the allowed, the system refuses to allocate more memory.  Given the simple design of the program code above, it breaks with signal 11.

now: 1505838082
now: 1505838083
jail: ./resource: exited on signal 11

Resource Control Rules

To check what resource control rules are applied, simply a “rctl” command will do.

# rctl 
jail:myjail:pcpu:deny=50
jail:myjail:vmemoryuse:deny=251658240
jail:myjail:memoryuse:deny=125829120

Clearing Resource Rules

Clearing the existing rules can be as simple as using the “-r” command, for example:

# rctl -r jail:myjail:memoryuse
# rctl -r jail:myjail
# rctl -r jail:

Actual Resource Used

Last but not the last, usage of a jail can be listed.

# rctl -u jail:myjail
cputime=284
datasize=4096
stacksize=0
coredumpsize=0
memoryuse=37724160
memorylocked=0
maxproc=1
openfiles=0
vmemoryuse=595689472
pseudoterminals=0
swapuse=592003072
nthr=1
msgqqueued=0
msgqsize=0
nmsgq=0
nsem=0
nsemop=0
nshm=0
shmsize=0
wallclock=358
pcpu=38
readbps=0
writebps=0
readiops=0
writeiops=0

Other Thoughts

In fact, the resource control can be also applied per process, per user, per group, etc.  The command is mostly similar, just replace “jail” with “process”, “user”, “group”, etc.  Say, a computer is being shared among a few friends.  Even if they do not want to be in jails (who wants?  Pun intended…), their resource usage can be constrained and accounted by the command.

In addition to the processor and memory usage, the resource control can also put limitations on other items like disk read / write (hint: use the word throttle instead of deny), which is useful when the IOPs is counted in a public cloud…

In the next article, we will explore how to put a complicated program, with dynamic linked libraries, into a jail.

Advertisements

OpenStack with FreeBSD

Standard

Recently, I come across NFV Express through the FreeBSDNews article.  The installation code is already openly available on Github.  At the first glance, the code is so amazing to be clean!  (You definitely should look at them if you know programming.)  When there is time, I will try them with my stack of FreeBSD computers…

Openstack is used to be Linux-centric, yet it is mostly written in Python scripts.  Theoretically, it is possible to make it running on any capable platforms… if you do not hate working with Python scripts.

Sorry for this short article.  This is indeed just a reminder for me to follow up.  🙂

Update on 4 Apr 2017: the tool requires Xen to be installed.  I tried installing it on the new computer but UEFI is not supported.  I am going to schedule another older computer to try.