System Performance with FreeBSD (Minecraft Server as Example)

Standard

Quite some time ago, we discussed how to get compile Minecraft and get it running on FreeBSD.  In this article, we take the server as an example how we can monitor system performance.

Minecraft Memory Usage

Minecraft is a Java program.  Java programs generally consume more memory than the counterparts made of unmanaged languages.  Thankfully, Java programs run inside their own sandboxes and have memory usage allocated and constrained.  In the previous article, we defined memory to be 1024 megabytes and expands up to 1024 megabytes only:

java -Xmx1024M -Xms1024M -jar spigot*.jar

It is important not to have Minecraft overrun the system memory.  As folklore, a Java program running on a dedicated computer should not be higher than 60% of total memory.  For example, my virtual machine has 2048 megabytes of memory and 60% of it is about 1200 megabytes.  I deducted myself further for 200 megabytes as safety margin.

General Process Monitoring

FreeBSD provides top(1) utility to check for system saturation and utilisation.  Generally speaking, a system is considered saturated if the number of threads ready to run is higher than the number of processor cores, and considered fully utilised if the utilisation numbers are near 100%.

螢幕快照 2017-06-06 下午10.57.56.png

In the top part, there are three numbers in decimal point, representing the load averages in 1, 5, and 15 minutes.  This is calculated by average number of threads ready to run and is regarded as the saturation of the processors.  In the third and forth row, the overall processor and memory utilisation ratios / rates are shown.

In the picture, the system not yet saturated as there is about 0.6 threads to run per second.  It is under some computation load that around 20% processing power and 80% memory are utilised.  This is a healthy situation that the system is being used, with some slacks to handle possible usage spikes.

In the bottom part, we have detailed breakdown per process.  The fields “TIME” and “WCPU” represents the total time and current portion of processor each process is using. Actual amount of memory usage are represented in the “RES” column.

In the picture, we see the Java process using 44% of current processing power and it has accumulated 295 hours.  Although the Java process is instructed to use only 1024 megabytes of memory, eventually it consumes 1368 megabytes.  This echoes why the folklores recommend only specify 60% of system memory to a Java virtual machine.

System Input / Output Monitoring

FreeBSD provides systat(1) utility for system usage statistics.  Since the general process monitoring can be done by the top(1) command above, it is mostly useful for detailed input and output monitoring.  By default, it shows a “pig” screen that refreshes every 10 seconds.  You can specify what you want to see by specifying in commands, for example, to see network interface usage every second:

systat -ifstat 1

螢幕快照 2017-06-06 下午11.02.03

The screen works like vi(1).  You can press the colon sign (:) and to enter a command.  The first command you want to try is “help”.  It immediate tells you the list of available pages you can switch to.  You can then use the colon sign again and enter the page you want.

I would tell you vmstat is my favourite page.  I learned it on the very first days I was taught FreeBSD.  It shows a lot of comprehensive information from system utilisation to interrupts and disk accesses.

systat -vmstat 1

螢幕快照 2017-06-06 下午10.59.11On the top, we see the system saturation and utilisation as usual.  Immediately under it, we see the detailed breakdown system events like context switches (Csw), traps (Trp), system calls (Sys), etc.  The system is now having thousands of context switch per second.  It would be no good if it were a system for high performance computing, but for our context of gaming with network messages, it is absolutely normal.

Further below, we see an ASCII art of system utilisation break down into system (=), interrupts (+), user space applications (>) and niced user applications (-).  The system is using mild amount of processing power and most of it are for the user space, which is a good thing.

The second bottom section shows the name caches of the virtual file system.  For a gaming system that uses few files, you can expect the hit rate be near 100%.  Otherwise you may want to pay more attention to the name caches.  In the picture, we see the system rarely searches for a file and those requests are handled by the cache perfectly.

The bottom section shows the disk utilisation.  Utilisation and bandwidth of each of the virtual disks are shown.  In the picture, we see the system rarely access the files.

The right most column shows the interrupt statistics.  The “timer” interrupts happens almost all the time in order to hint the operating system to context switch and update system clocks.  It used to be 100 or 1000 per processor core; thankfully, with the advent of more advanced system clocks, systems no longer need to tick as frequent as before.  In the picture, the network card (virtio-network) requires quite some interrupts handling.  As long as the network card interrupts does not go to numbers like 50000 or 100000, they are most likely normal.

The list of pages available in the tool, as of today, are:

  • pigs: shows the processes which consume the most processing power
  • vmstat: (as discussed above)
  • swap: shows the system swap situation
  • zarc: shows the ZFS adaptive read cache situation
  • iostat: raw disk input and output statistics
  • netstat: network socket statistics, such as buffered bytes for each of the connections
  • sctp: stream control transport protocol statistics
  • tcp: transport layer protocol statistics
  • ip: internet protocol statistics
  • ip6: internet protocol statistics for IPv6
  • icmp: internet control message statistics such as ping, etc
  • icmp6: internet control message statistics for IPv6
  • ifstat: raw network interface utilisation statistics

Conclusion

In this article, we go through some performance monitoring tools that come with FreeBSD.  The general process information can be listed by the top(1) command, where you can understand the system saturation and utilisation, and also list the resource consuming processes.  More detailed resource utilisations like network, disks, hardware interrupts can be found in the systat(1) command.  If in doubt, the “vmstat” page can be a great starting point to look for congested system resources.

Minecraft with Java and Tmux on FreeBSD

Standard

A virtual world cannot be complete without a virtual reality environment like Minecraft.  In this article, I quickly go through the steps setting a Minecraft server with FreeBSD.   Specifically, I will be installing a Spigot Minecraft Server.  I like it very much because it comes with lots of performance optimisations.  I assume you prepared your environment with my customisation script or something similar.  The whole process will take less than one hour if you have a good internet connection.

Step 1: Installing Java, Tmux, Bash, and Friends

As usual, installing packages in FreeBSD is easy.  It is simply “pkg install”.  Yet, Java and Bash requires special file systems at “/dev/fd” and “/proc”.  The file “/etc/fstab” is therefore appended and two more file systems are defined.  The file systems are mounted before we proceed on the next step.

pkg install tmux bash git openjdk8
cat >> /etc/fstab << EOF
fdesc /dev/fd fdescfs rw 0 0
proc /proc procfs rw 0 0
EOF
mount /dev/fd
mount /proc

Step 2: Trying Tmux

To try tmux, one can simply issue the command

tmux

Inside the tmux session, one can:

  • Start a new window with keystrokes Ctrl-B, C
  • Switch to the previous window with keystrokes Ctrl-B, P
  • Switch to the next window with keystrokes Ctrl-B, N
  • Switch to a specific window with keystroke Ctrl-B, then the window number
  • Detach from the session with keystrokes Ctrl-B, D

For more, you can easily find information with a search engine.  The last feature, detachment, is very useful.  One can leave a program running inside a tmux session, logout, login, and reattach to the same session.  This feature is particularly useful since a minecraft server is nothing more than a long-running Java application.

Step 3: Execute the Automated Build

In your favourite working directory, download the builder Java archive (JAR) file:

curl "https://hub.spigotmc.org/jenkins/job/BuildTools/lastSuccessfulBuild/artifact/target/BuildTools.jar" -o BuildTools.jar

Then you can execute the JAR file:

java -jar BuildTools.jar

Step 4: Accept the EULA

After about half an hour of compilation, read the “eula.txt” and execute the following if you agree.

sed -ibak s/false/true/ eula.txt

Step 5: Tmux on Boot

Like last time with the firewall table flush, we can append a line to the “/etc/crontab” for a command to automatically start.  Yet, this time we are executing the command once only, we use the “@reboot” magic word.  In the second field, the “root” account is taken for running the program.  You have to stick with the user account you have chosen.  From the third field onward, with the “tmux new-session” command, we generate a new session.

The Java was executed with initial heap size to be 1024 MB, and maximum heap size to be 1024 MB as well.  This should be good enough unless you have a lot of friends.  In that case, you will experience some “OutOfMemoryError” and have to expand the memory accordingly.  Make sure your virtual machine is large enough to handle it.  In the folklores, it is said that one can only have heap size 60% of the total system memory.  If one has 8 GB main memory, at most the Java heap size is 4096 MB or 5120 MB.

cat >> /etc/crontab << EOF
@reboot root /usr/local/bin/tmux new-session -s spigot -d /usr/local/bin/java -Xmx1024M -Xms1024M -jar /root/spigot*.jar
EOF

Step 6: Start for the First Time

Starting it the first time is nothing much different from the command in the crontab.

tmux new-session -s spigot -d java -Xmx1024M -Xms1024M -jar spigot*.jar

Step 7: Firewall Configuration

Add port 25565 to the list of allowed ports.  In our context, we will update the line with “tcpports” in “/etc/pf.conf”:

tcpports="{22,80,443,25565}"

After that, restart the firewall with command:

service pf reload

Step 8: Connect to the Server

Open your Minecraft client, select “Multiplayer”, then “Direct Connect”.  You can then put in your server address.  If you just realise it is a commercial software, sorry, you got to pay.  Some say, one can use unauthorised Minecraft client for playing in his own server with the offline mode.  But the procedures above already assumes you run the server in online mode.

Step 9: Finding Plugins and Writing Startup Scripts

Once you get comfortable starting and stopping the servers, you can find plugins and put them to “plugins” directory.

Also, instead of calling the java command directly, you can write a shell script to encapsulate these complexities.  Sometimes server crash and you prefer it to automatically restart.  You can then add a automatic restart procedure inside the script.

#!/bin/sh
while true
do
  java -Xmx1024M -Xms1024M -jar spigot*.jar
  echo Press Ctrl-C now if you mean to stop instead of restart
  sleep 5
done

The actual procedures are left for you due to my laziness.