\section{How to test with physical hardware}

\begin{figure}[ht]
\includegraphics[height=4in,width=5in]{testsetup1.eps} 
\caption{Basic Physical Network configuration}
\label{basicnet}
\end{figure}

The basic network is shown in Figure \ref{basicnet}. The taxonomy for our test setup
is that the Sun rises in the east and the sun sets in the west. Thus one can
easily remember where each host is. 

{\scshape east} and {\scshape west}, shown with firewall icons, are FreeS/WAN IPsec gateway boxes.

{\scshape sunrise} and {\scshape sunset} are just ordinary hosts whose traffic
will be protected by their respective gateways.

The machine {\scshape sky} is used to do network analysis (``sniff''). 
There are frequently problems that occur when trying examine the traffic
produced by a machine itself, so a seperate machine to make unbiased
observations if necessary.\footnote{In particular, on Linux 2.2 or lower,
turning on the packet capture mechanism changes the control structures
attached to the traffic and causes faults relating to policy for the keying
channels' control packets. PR\#48 at 
\htmladdnormallink{http://bugs.freeswan.org:81/bugs/gnatsweb.pl}
{http://bugs.freeswan.org:81/bugs/gnatsweb.pl?&database=freeswan&cmd=view&pr=48}
2.4 has solved this problem}

The two gateway boxes are not directly attached, but rather are connected via 
a router. There are two reasons for this:
\begin{itemize}
\item the current implementation of FreeS/WAN requires a default route to
	operate correctly.
\item a common operational issue is with links where the Maximum Transmission 
	Unit (MTU) is restricted, and this router provides a place to cause
	such an impairment\footnote{FreeS/WAN has adopted the term
	``impairment'' to denote any challenges which are introduced to a
	system or network to permit another part of the system to be tested}
\end{itemize}

This setup is very representative of the typically deployed scenario for
FreeS/WAN systems in a VPN. It does not cover every single situation - most
of the most difficult to reproduce bugs have occured in other setups. More
machines are needed to create such setups.

Aside from the space and cost involved in providing each developer with six
machines (it is often the case that {\tt sky} is the developers desktop),
there are a number of other factors that make this difficult.  

The major problem is maintaining this setup. There are many machines with
many files that must be maintained. The systems must be kept up-to-date so
that the latest kernels can be tested, yet at the same time, testing against
older kernels is necessary. Different distributions need to be tested. The
combinatorics are quite high.

The other major problem is work environment. Sitting in a room with six
computers is a lot of noise. Getting access to each systems' console is
difficult (one can not rely upon network logins!). If a monitor is attached
to each system (vs a monitor switch), then the developer probably gets too
much exersize.

One answer to this is serial consoles. See figure
\ref{basicconsole}. Terminals attached to serial ports was the primary 
way that people used Unix until the advent of the X-terminal, and Linux 
continues this grand tradition.

\begin{figure}[hf]
\includegraphics[height=4in,width=5in]{testsetup2.eps} 
\caption{Basic Network with console access}
\label{basicconsole}
\end{figure}

One simply puts the following in {\tt /etc/lilo.conf}:
\begin{verbatim}
serial=0,38400n

...
image=/boot/vmlinuz-2.4.18-6mdk
        label=linux2418
        root=/dev/hda1
        append=" devfs=mount console=ttyS0,38400 console=tty0"
        read-only

\end{verbatim}

The console then appears on both ``COM1'' and on the VGA screen. In this
situation, the machines may be located in another room, connected to a
console server. One logins from one's (quiet) desktop to the console server,
accessing each machine via a serial port. Serial interfacesd are readily
available with either PCI or USB interfaces. This makes building a 6 port
console server rather easy.

The developer now has ready access to each machine, can reboot each machine,
select different kernels and can configure it without even having networking on.
In addition, kernel panics (``kernel oops'') or other strange output on the
console can be cut and pasted into emails, etc..

\subsection{Still challenging to test}

The serial consoles do not solve the other problems - managing the very many 
different configurations, or coordinating the systems to perform a test case.

The author has used such a setup for many years with many Unix operating
systems. Using the ``expect'' program and the serial consoles one can
automate some of the tests. Some of tests are harder to deal ones - ones that 
fail can cause the system to hang - this will require operator
intervention. Further use of more hardware can solve this problem as well -
relays can toggle reset switches or even power cycles. 

The result, however, is a very complicated testing environment - it can take
weeks to configure it, and mere hours to break. There is far too much
specialized hardware involved, not to mention the software.

There is a better way which will be described, but first, the requirements
for the testing environemnt will be examined in a bit more detail.






	
