FreeS/WAN "make check" guide
Keywords: Linux, IPsec, VPN, security, FreeSWAN, testing, User-Mode-Linux, UML

# How to configure to use "make check"

## What is "make check"

"make check" is a target in the top level makefile. It takes care of running a number of unit and system tests to confirm that FreeSWAN has been compiled correctly, and that no new bugs have been introduced.

As FreeSWAN contains both kernel and userspace components, doing testing of FreeSWAN requires that the kernel be simulated. This is typically difficult to do as a kernel requires that it be run on bare hardware. A technology has emerged that makes this simpler. This is User Mode Linux.

User-Mode Linux is a way to build a Linux kernel such that it can run as a process under another Linux (or in the future other) kernel. Presently, this can only be done for 2.4 guest kernels. The host kernel can be 2.2 or 2.4.

"make check" expects to be able to build User-Mode Linux kernels with FreeSWAN included. To do this it needs to have some files downloaded and extracted prior to running "make check". This is described in the UML testing document.

After having run the example in the UML testing document and successfully brought up the four machine combination, you are ready to use "make check"

## Running "make check"

"make check" works by walking the FreeSWAN source tree invoking the "check" target at each node. At present there are tests defined only for the `klips` directory. These tests will use the UML infrastructure to test out pieces of the `klips` code.

The results of the tests can be recorded. If the environment variable `$REGRESSRESULTS` is non-null, then the results of each test will be recorded. This can be used as part of a nightly regression testing system, see Nightly testing for more details.

"make check" otherwise prints a minimal amount of output for each test, and indicates pass/fail status of each test as they are run. Failed tests do not cause failure of the target in the form of exit codes.

# How to write a "make check" test

## Structure of a test

Each test consists of a set of directories under `testing/`. There are directories for `klips`, `pluto`, `packaging` and `libraries`. Each directory has a list of tests to run is stored in a file called `TESTLIST` in that directory. e.g. `testing/klips/TESTLIST`.

## The TESTLIST

This isn't actually a shell script. It just looks like one. Some tools other than /bin/sh process it. Lines that start with # are comments.

    # test-kind     directory-containing-test       expectation     [PR#]

The first word provides the test type, detailed below.

The second word is the name of the test to run. This the directory in which the test case is to be found..

The third word may be one of:

blank/good
  the test is believed to function, report failure
bad
  the test is known to fail, report unexpected success
suspended
  the test should not be run

The fourth word may be a number, which is a PR# if the test is failing.

## Test kinds

The test types are:

skiptest
  means run no test.
ctltest (TBD)
  means run a single system without input/output.
klipstest
  means run a single system with input/output networks
umlplutotest
  means run a pair of systems
suntest (TBD)
  means run a quad of east/west/sunrise/sunset
roadtest (TBD)
  means run a trio of east-sunrise + warrior
extrudedtest (TBD)
  means run a quad of east-sunrise + warriorsouth + park
mkinsttest
  a test of the "make install" machinery.
kernel_test_patch
  a test of the "make kernelpatch" machinery.

Tests marked (TBD) have yet to be fully defined.

Each test directory has a file in it called `testparams.sh`. This file sets a number of environment variables to define the parameters of the test.

## Common parameters

TESTNAME
  the name of the test (repeated for checking purposes)
TEST_TYPE
  the type of the test (repeat of type type above)
TESTHOST
  the name of the UML machine to run for the test, typically "east" or "west"
TEST_PURPOSE
  The purpose of the test is one of:
  goal
    The goal purpose is where a test is defined for code that is not yet finished. The test indicates when the goals have in fact been reached.
  regress
    This is a test to determine that a previously existing bug has been repaired. This test will initially be created to reproduce the bug in isolation, and then the bug will be fixed.
  exploit
    This is a set of packets/programs that causes a vulnerability to be exposed. It is a specific variation of the regress option.
TEST_GOAL_ITEM
  in the case of a goal test, this is a reference to the requirements document
TEST_PROB_REPORT
  in the case of regression test, this the problem report number from GNATS
TEST_EXPLOIT_URL
  in the case of an exploit, this is a URL referencing the paper explaining the origin of the test and the origin of exploit software
REF_CONSOLE_OUTPUT
  a file in the test directory that contains the sanitized console output against which to compare the output of the actual test.
REF_CONSOLE_FIXUPS
  a list of scripts (found in `klips/test/fixups`) to apply to sanitize the console output of the machine under test. These are typically perl, awk or sed scripts that remove things in the kernel output that change each time the test is run and/or compiled.
INIT_SCRIPT

  a file of commands that is fed into the virtual machine's console in single user mode prior to starting the tests. This file will usually set up any eroute's and SADB entries that are required for the test.

  Lines beginning with # are skipped. Blank lines are skipped. Otherwise, a shell prompted is waited for each time (consisting of `\n#`) and then the command is sent. Note that the prompt is waited for before the command and not after, so completion of the last command in the script is not required. This is often used to invoke a program to monitor the system, e.g. `ipsec pf_key`.

FINAL_SCRIPT

  a file of commands that is fed into the virtual machine's console in single user mode after the final packet is sent. Similar to INIT_SCRIPT, above. If not specified, then the single command "halt" is sent. If specified, then the script should end with a halt command to nicely shutdown the UML.

CONSOLEDIFFDEBUG
  If set to "true" then the series of console fixups (see REF_CONSOLE_FIXUPS) will be output after it is constructed. (It should be set to "false", or unset otherwise)
NETJIGDEBUG
  If set to "true" then the series of console fixups (see REF_CONSOLE_FIXUPS) will be output after it is constructed. (It should be set to "false", or unset otherwise)

## KLIPStest paramaters

The klipstest function starts a program (`testing/utils/uml_netjig/uml_netjig`) to setup a bunch of I/O sockets (that simulate network interfaces). It then exports the references to these sockets to the environment and invokes (using system()) a given script. It waits for the script to finish.

The script invoked (`testing/utils/host-test.tcl`) is a TCL expect script that arranges to start the UML and configure it appropriately for the test. The configuration is done with the script given above for INIT_SCRIPT. The TCL script then forks, leaves the UML in the background and exits. uml_netjig continues. It then starts listening to the simulated network answering ARPs and inserting packets as appropriate.

The klipstest function invokes `uml_netjig` with arguments to capture output from network interface(s) and insert packets as appropriate:

PUBINPUT
  a pcap file to feed in on the public (encrypted) interface. (typically, eth1)
PRIVINPUT
  a pcap file to feed in on the private (plain-text) interface (typically, eth0).
REF_PUB_OUTPUT
  a text file containing tcpdump output. Packets on the public (eth1) interface are captured to a pcap file by `uml_netjig`. The klipstest function then uses tcpdump on the file to produce text output, which is compared to the file given.
REF_PUB_FILTER
  a program that will filter the TCPDUMP output to do further processing. Defaults to "cat".
REF_PRIV_OUTPUT
  a text file containing tcpdump output. Packets on the private (eth0) interface are captured and compared after conversion by tcpdump, as with REFPUBOUTPUT.
REF_PRIV_FILTER
  a program that will filter the TCPDUMP output to do further processing. Defaults to "cat".
< DT>EXITONEMPTY
  a flag for `uml_netjig`. It should contain "--exitonempty" of uml_netjig should exit when all of the input (PUBINPUT,PRIVINPUT) packets have been injected.
ARPREPLY
  a flag for `uml_netjig`. It should contain "--arpreply" if `uml_netjig` should reply to ARP requests. One will typically set this to avoid having to fudge the ARP cache manually.
TCPDUMPFLAGS
  a set of flags for the tcpdump used when converting captured output. Typical values will include "-n" to turn off DNS, and often "-E" to set the decryption key (tcpdump 3.7.1 and higher only) for ESP packets. The "-t" flag (turn off timestamps) is provided automatically

## mkinsttest paramaters

The basic concept of the `mkinsttest` test type is that it performs a "make install" to a temporary $DESTDIR. The resulting tree can then be examined to determine if it was done properly. The files can be uninstalled to determine if the file list was correct, or the contents of files can be examined more precisely.

INSTALL_FLAGS
  If set, then an install will be done. This provides the set of flags to provide for the install. The target to be used (usually "install") must be among the flags.
POSTINSTALL_SCRIPT
  If set, a script to run after initial "make install". Two arguments are provided: an absolute path to the root of the FreeSWAN src tree, and an absolute path to the temporary installation area.
INSTALL2_FLAGS
  If set, a second install will be done using these flags. Similarly to INSTALL_FLAGS, the target must be among the flags.
UNINSTALL_FLAGS
  If set, an uninstall will be done using these flags. Similarly to INSTALL_FLAGS, the target (usually "uninstall") must be among the flags.
REF_FIND_f_l_OUTPUT
  If set, a `find $ROOT ( -type f -or -type -l )` will be done to get a list of a real files and symlinks. The resulting file will be compared to the file listed by this option.
REF_FILE_CONTENTS
  If set, it should point to a file containing records for the form:

        reffile   samplefile

  one record per line. A diff between the provided reference file, and the sample file (located in the temporary installation root) will be done for each record.

## rpm_build_install_test paramaters

The `rpm_build_install_test` type is to verify that the proper packing list is produced by "make rpm", and that the mechanisms for building the kernel modules produce consistent results.

RPM_KERNEL_SOURCE
  Point to an extracted copy of the RedHat kernel source code. Variables from the environment may be used.
REF_RPM_CONTENTS
  This is a file containing one record per line. Each record consists of a RPM name (may contain wildcards) and a filename to compare the contents to. The RPM will be located and a file list will be produced with rpm2cpio.

## libtest paramaters

The libtest test is for testing library routines. The library file is expected to provided an `#ifdef` by the name of library. The libtest type invokes the C compiler to compile this file, links it against `libfreeswan.a` (to resolve any other dependancies) and runs the test with the `-r` argument to invoke a regression test.

The library test case is expected to do a self-test, exiting with status code 0 if everything is okay, and with non-zero otherwise. A core dump (exit code greater than 128) is noted specifically.

Unlike other tests, there are no subdirectories required, or other parameters to set.

## umlplutotest paramaters

The umlplutotest function starts a pair of user mode line processes, connecting them as shown in the multi_netjig diagram (not linked here).

The script invoked (`testing/utils/2host-test.tcl`) is a TCL expect script that arranges to start the UML and configure it appropriately for the test. It then starts listening (using uml_netjig) to the simulated network answering ARPs and inserting packets as appropriate.

The following are used to control the input and output to the system:

EAST_INPUT/WEST_INPUT
  a pcap file to feed in on the private network side of each network
EAST_INIT_SCRIPT/WEST_INIT_SCRIPT

  a file of commands that is fed into the virtual machine's console in single user mode prior to starting the tests. This file will usually set up any eroute's and SADB entries that are required for the test. Similar to INIT_SCRIPT, above.

EAST_FINAL_SCRIPT/WEST_FINAL_SCRIPT

  a file of commands that is fed into the virtual machine's console in single user mode after the final packet is sent. Similar to INIT_SCRIPT, above. If not specified, then the single command "halt" is sent. If specified, then the script should end with a halt command to nicely shutdown the UML.

REF_EAST_CONSOLE_OUTPUT/REF_WEST_CONSOLE_OUTPUT
  Similar to REF_CONSOLE_OUTPUT, above.
REF_PUB_FILTER
  a program that will filter the TCPDUMP output to do further processing. Defaults to "cat".
REF_EAST_OUTPUT/REF_WEST_OUTPUT
  a text file containing tcpdump output. Packets on the private (eth0) interface are captured and compared after conversion by tcpdump, as with REF_PUB_OUTPUT.
REF_EAST_FILTER/REF_WEST_FILTER
  a program that will filter the TCPDUMP output to do further processing. Defaults to "cat".
<
TCPDUMPFLAGS
  a set of flags for the tcpdump used when converting captured output. Typical values will include "-n" to turn off DNS, and often "-E" to set the decryption key (tcpdump 3.7.1 and higher only) for ESP packets. The "-t" flag (turn off timestamps) is provided automatically

There are two additional environment variables that may be set on the command line:

NETJIGTESTDEBUG=netjig export NETJIGTESTDEBUG
  This will enable debugging of the communication with uml_netjig, and turn on debugging in this utility.

HOSTTESTDEBUG=hosttest export HOSTTESTDEBUG
  This will show all interactions with the user-mode-linux consoles

## kernel_patch_test paramaters

The kernel_patch_test function takes some kernel source, copies it with lndir, and then applies the patch as produced by "make kernelpatch".

The following are used to control the input and output to the system:

KERNEL_NAME
  the kernel name, typically something like "linus" or "rh"
KERNEL_VERSION
  the kernel version number, as in "2.2" or "2.4".
KERNEL_${KERNEL_NAME}${KERNEL_VERSION}_SRC
  This variable should set in the environment, probably in ~/freeswan-regress-env.sh. Examples of this variables would be KERNEL_LINUS2_0_SRC or KERNEL_RH7_3_SRC. This variable should point to an extracted copy of the kernel source in question.
REF_PATCH_OUTPUT
  a copy of the patch output to compare against

# Invoking tests manually

If you need to interactively control the UML under test, then you can invoke netjig directly with the right arguments, but give it "sh" as the startup script. An easy way is to do this is to edit "NETJIGDEBUG" in setup.sh to "true" and run the test:

e.g:

    cassidy-[nightly/klips/test/east-pass-01] mcr 1009 %NETJIGDEBUG=true ../../utils/runme.sh
    /c2/freeswan/sandboxes/nightly/testing/utils/uml_netjig/uml_netjig --tcpdump --exitonempty --playprivate ../inputs/01-sunrise-sunset-ping.pcap --recordpublic OUTPUT/spi1-output.pcap --startup expect -f /c2/freeswan/sandboxes/nightly/testing/utils/host-test.tcl /c2/freeswan/sandboxes/nightly/UMLPOOL/east/start.sh spi1.sh

so, substitute "sh" for the startup script.

    /c2/freeswan/sandboxes/nightly/testing/utils/uml_netjig/uml_netjig --tcpdump --exitonempty --playprivate ../inputs/01-sunrise-sunset-ping.pcap --recordpublic OUTPUT/spi1-output.pcap --startup sh

    sh-2.05a$ env | grep UML_
    UML_private_CTL=/tmp/umlz8Xrpn/private/ctl
    UML_public_DATA=/tmp/umlz8Xrpn/public/data
    UML_public_CTL=/tmp/umlz8Xrpn/public/ctl
    UML_private_DATA=/tmp/umlz8Xrpn/private/data

then, in a *new window*, paste those variables in place, and start the UML you need. The start.sh scripts looks for UML_{private,public}_{CTL,DATA} and will connect them to eth0/eth1. Setup the UML as appropriate, etc. When you are ready, exit the above shell, and the uml_netjig will start to inject packets and record them.

# Current pitfalls

"tcpdump dissector" not available.
  This is a non-fatal warning. If uml_netjig is invoked with the -t option, then it will attempt to use tcpdump's dissector to decode each packet that it processes. The dissector is presently not available, so this option it normally turned off at compile time. The dissector library will be released with tcpdump version 4.0.