Contents Previous Next

Linux FreeS/WAN Troubleshooting Guide

Overview

Before we launch into how to establish, test and troubleshoot a connection step by step, here's a brief overview that may allow you to skim ahead to the most useful part of this document for you.

There are several general places where you might have a problem:

  1. During install.

  2. Before connection negotiation.

  3. During the negotiation process.

  4. Using an established connection.

This document also contains notes which expand on points made in these sections, and tips for problem reporting. If the other end of your connection is not FreeS/WAN, see also our interoperation document.

1. During Install

Instructions and tips are in our install document. If you encounter a problem, it may be:

2. Before Connection Negotiation

2.1 Plan your tunnels

Know your IPSec needs. For most people who begin experimenting with IPSec in the field, your configuration will more or less resemble one of the samples in our configuration document . Other folks may wish to create a testbed network in a lab environment for intensive IPSec testing or proof-of-concept. They may be interested in this description of a sample testbed network.

2.2 Configure Step by Step

Draw a network schematic. This will assist others in helping you should you mail the list for help.

For example:

Sunset==========West------------------East
         corporate LAN     untrusted net

Add IPs to the diagram.

Sunset==========West------------------East
         corporate LAN     untrusted net

192.168.1.5                           N.N.N.235

         eth0(external)=N.N.N.222

         eth1(internal)=192.168.1.1;
         gateway for 192.168.0.0/24

How many tunnels do you need to connect your sites sufficiently for your purposes? Check our configuration document for why you might need multiple tunnels. How would you name each tunnel?

Above, it is possible to make two tunnels: sunset-east and west-east. If East does not need to access any resources on West, and West is not masquerading (see below) we may only want to create sunset-east. Sunset-east is a tunnel between IPSec-enabled machines West and East, created to protect traffic between the net on which Sunset resides, and East. However, it is safest to create all potential tunnels in your configuration. This lowers the risk that you will forget to configure a needed tunnel, and send important data cleartext. As an added safeguard, by default, (at the time of writing) Linux FreeS/WAN prevents you from routing cleartext packets between IPSec gateways which are also linked by a tunnel. Reliance on this behaviour is not a substitute for secure network design.

Take into account any masquerading and Network Address Translation rules on your gateway. If you are masquerading packets from Sunset as they leave West, Linux FreeS/WAN will treat these as though they originated at West, not Sunset. For more detail, see this packet flow diagram.

For each tunnel, think of a packet path that will allow you to test that tunnel. Refer to this discussion.

2.3 Ensure a Clear Path for IKE

Ensure that IKE packets can travel freely between your IPSec gateways, as described here. If they cannot, the connection negotiation, described below, will be stuck in its first state.

3. During Negotiation

3.1 Create a connection

Bring up one of your connections, using the ipsec auto commands at the command line:

    ipsec auto --add west-east

then

    ipsec auto --up west-east

At this stage of testing, do not bring up the connections by the alternate method, using ipsec.conf's auto= directive. You want to view the output as it happens.

If the resulting status report shows that you have established an ISAKMP and an IPSec Security Association (aka "SA", loosely translated as "tunnel" or "connection"), your tunnel is up. Repeat for each tunnel you are testing.

If negotiations for any one tunnel fail, troubleshoot as indicated in the next section. If you have successfully established all desired tunnels, proceed to test your connection(s) below. If you find that each tunnel in a multitunnel config may be created individually, but all may not be created at once, you may have encountered an old (1.6) bug. Update your Linux FreeS/WAN.

3.1.1 Determine Connection State

When you bring a tunnel up from the command line, you see a report on the negotiations involved in creating the connection, as these happen. There are also:

Often, the most relevant state information appears last in a log or status report.

Negotiations will proceed through various states. You will know these are done and a connection is established when you see both messages:

    000 #21: "myconn" STATE_MAIN_I4 (ISAKMP SA established)...
    000 #2: "myconn" STATE_QUICK_I2 (sent QI2, IPsec SA established)...

The key phrases are "ISAKMP SA established" and "IPSec SA established", which should appear with the relevant connection name. Often, this happens at STATE_MAIN_I4 and STATE_QUICK_I2, respectively.

A note on ipsec auto --status: this will tell you what states have been achieved, rather than the current state. Since determining the current state is rather more difficult to do, current state information is not available from Linux FreeS/WAN. If you are actively bringing a connection up, the status report's last states for that connection likely reflect its current state. Beware, though, of the case where a connection was correctly brought up but is now downed: Linux FreeS/WAN will not notice this until it attempts to rekey. Meanwhile, the last known state indicates that the connection has been established.

Linux FreeS/WAN proceeds though IKE (Phase 1, Main Mode, STATE_MAIN_*) negotiations first, then begins IPSec (Phase 2, Quick Mode, STATE_QUICK_*) negotiations. If you do not see success, note the place where negotiations stopped. This information is useful, since there are common errors specific to certain points in the process.

3.2 Find a Negotiation Error

Look for verbose error text in the logs. While ipsec --auto dialog will tell you at which state Linux FreeS/WAN failed, it lacks detail. You can get more detail by modifying it with the --verbose flag on each invocation. For example:

    ipsec auto --verbose --up west-east

Complete information can be gleaned from the log files.

The amount of description in the logs depends on ipsec.conf debug settings, klipsdebug= and plutodebug=. See the ipsec.conf(5) man page for details. You will normally want to set these to either "none" or "all". Note that you must have enabled the klipsdebug compile-time option for the klipsdebug configuration switch to work.

For negotiation problems, plutodebug is most relevant. klipsdebug applies mainly to attempts to use an already-established connection. See also this description of the division of duties within Linux FreeS/WAN.

After raising your debug levels, restart Linux FreeS/WAN to ensure that ipsec.conf is reread, then recreate the error to generate verbose logs.

This is a good time to produce a barf file, a collection of information useful for debugging Linux FreeS/WAN on your system. Use the command

    ipsec barf > barf.west

See also the ipsec_barf(8) man page.

Look at the logs within the resulting file, and find the failure point. Are there a handful of lines which succinctly describe how things are going wrong or contrary to your expectation? Sometimes the failure point is not immediately obvious: Linux FreeS/WAN's errors are usually not marked "Error". Have a look in the FAQ for what some common failures look like. Tip: problems snowball. Focus your efforts on the first problem, which is likely to be the cause of later errors.

Repeat the process to find meaningful error text on the peer IPSec box. If the other end is not Linux FreeS/WAN, get it to produce detailed log output while you replicate the error, then capture that output to a file.

It is useful if both ends store information about the same event from two perspectives. Sometimes you will require information which only one side has. In this case, the peer can merely indicate the presence of an error, and its approximate point in the negotiations. If one side keeps retrying, it may be because there is a show stopper on the other side. Have a look at the other side and figure out what it doesn't like.

3.2.1 Interpret a Negotiation Error

To interpret Linux FreeS/WAN log text, use the following resources:

If you have failed to solve your problem with the help of these resources, send a detailed problem report to the users list, following these guidelines.

4. Using a Connection

4.1 Ping test

Test a connection by sending packets through it. The simplest way to do this is with ping. Remember, in the planning stage , choosing a path which would test the tunnel? Now, ping along that path.

If your ping returns, test any other connections you've brought up. If they all check out, great. You may wish to test with large packets for MTU problems.

If your ping fails to return, generate an ipsec barf debugging report on each IPSec gateway. On a non-Linux FreeS/WAN implementation, gather equivalent information. Use this, and the tips in the next sections, to troubleshoot. Are you sure that both endpoints are capable of hearing and responding to ping?

4.1.1 Check your ping path

IPSec may be dropping your ping packets since it does not "think" they belong in the tunnels you have constructed. This is an error about assumptions, and it takes two forms.

In the first, your ping is not returning because its path does not fall within your tunnel. This ping does not test the tunnel you intend to test. Referring to this discussion about appropriate tests, determine an alternate ping path which would test the tunnel.

In the second form of this error, you have not correctly configured the functionality you want. In the example above, you may have configured one of the possible tunnels between West and East (say west-east) but not the tunnel required to secure the important traffic you're now testing (say a sunset-east tunnel to secure traffic between office net Sunset and laptop East). See also this FAQ titled "I can't ping". NAT and masquerading may have an effect on which tunnels you need to configure; that's discussed in " Before there's trouble".

Both forms show identical symptoms. After all, the difference is one of intent. In both forms, Linux FreeS/WAN receives a packet destined for a peer IPSec gateway. Finding no active tunnel in which this packet belongs, it drops the packet on the floor. If your debug levels are appropriate, it logs this with a "klipsdebug... no eroute" message, which is discussed in this FAQ.

Note: When testing a tunnel that protects a multi-node subnet, you may wish to try several subnet nodes as ping targets, in case one node is routing incorrectly.

4.2 Check Routing and Firewalling

If you've confirmed your configuration assumptions, the problem is almost certainly with routing or firewalling. Isolate the problem using interface statistics, firewall statistics, or a packet sniffer.

Background:

4.2.1 View Interface and Firewall Statistics

Interface reports and firewall statistics can help you track down lost packets at a glance.

Check any firewall statistics you may be keeping on your IPSec gateways, for dropped packets.

Both cat /proc/net/dev and ifconfig display interface statistics, and both are included in an ipsec barf. Use either to check if any interface has dropped packets. If you find that one has, test whether this is related to your ping. While you ping continuously, print that interface's statistics several times. Does its drop count increase in proportion to the ping? If so, check why the packets are dropped there.

Check the firewall rules that apply to that interface. If the interface is an IPSec interface, more information may be available in the log. Grep for the word "drop" in a log which was created with klipsdebug=all as the error happened.

See also this detailed discussion by KLIPS programmer Richard Guy Briggs on interpreting ifconfig.

4.3 Sniff packets

If you have checked configuration assumptions, routing, and firewall rules, and your interface statistics yield no clue, it remains for you to investigate the mystery of the lost packet by the most thorough method: with a packet sniffer. Sniff packets at each interface along the projected ping path until you find where your packets disappear. In this way, you can isolate the problem area, and narrow your troubleshooting focus.

Install an up-to-date sniffer (tcpdump, ethereal, ksnuffle) on your IPSec gateway machines. A sniffer on the ping endpoints is also useful. The sniffer should be somewhat modern (tcpdump 3.3+, ethereal-0.8.18) so that you may view packets on the ipsec virtual interface as well as the underlying physical one.

Working from your schematic, anticipate your ping's path. Which machines will your ping be visible on, in which order? Now, which interfaces will it be visible on, in which order? Within a machine running Linux FreeS/WAN, this packet flow diagram will help you anticipate the packet's path. Note that from the perspective of the tunneled packet, the entire tunnel is one hop. That's explained in this FAQ.

Ping, and as you do, sniff the packets. Examine each interface along the projected path, checking for your ping's arrival. If it doesn't arrive at the next stop, you have narrowed down where to look for it.

Note that an encapsulated IPSec packet will look different, when sniffed, from the plaintext packet which generated it. However, you can observe plaintext packets entering an IPSec interface and the resulting cyphertext packets as they emerge from the corresponding physical interface.

Once you isolate where the packet is lost, take a closer look at firewall rules, routing and configuration assumptions as they affect that specific area. If the packet is lost on an IPSec gateway, comb through klipsdebug output for anomalies.

If the packet goes through both gateways successfully and reaches the ping target, but does not return, suspect routing. Check that the ping target routes packets back to the IPSec gateway.

4.4 Find a Connection Use Error

The guidelines are the same as for the Pluto logs, above.

For connection use problems, set klipsdebug=all. Note that you must have enabled the klipsdebug compile-time option to do this. Restart Linux FreeS/WAN so that it rereads the configuration file, then recreate the error condition. When searching through klipsdebug data, look especially for the keywords "drop" (as in dropped packets) and "error".

Often the problem with connection use is not software error, but rather that the software is behaving contrary to expectation.

4.4.1 Interpret a Connection Use Error

To interpret the Linux FreeS/WAN log text you've found, use the same resources as indicated for troubleshooting connection negotiation: the FAQ , our background document , and the list archives. Looking in the KLIPS code is recommended only for the brave.

If you are still stuck, send a detailed problem report to the users' list.

4.4 Test with Large Packets

If each of your connections passed the ping test, you may wish to test by pinging with large packets (2000 bytes or larger). If it does not return, suspect MTU issues, and see this discussion.

4.5 Stress Tests

In most users' view, a simple ping test, and perhaps a large-packet ping test suffice to indicate a working IPSec connection.

Some people might like to do additional stress tests prior to production use. They may be interested in this testing protocol we use at interoperation conferences, aka "bakeoffs". We also have a testing directory that ships with the release.

5. Problem Reporting

5.1 How to ask for help

Ask for troubleshooting help on the users' mailing list, users@lists.freeswan.org. While sometimes an initial query with a quick description of your intent and error will twig someone's memory of a similar problem, it's often necessary to send a second mail with a complete problem report.

The essay How to Report Bugs Effectively contains good guidelines.

When reporting problems to the mailing list(s), please include:

5.2 Where to ask

To report a problem, send mail about it to the users' list. If you are certain that you have found a bug, report it to the bugs list. If you encounter a problem while doing your own coding on the Linux FreeS/WAN codebase and think it is of interest to the design team, notify the design list. When in doubt, default to the users' list. More information about the mailing lists is found here.

For a number of reasons -- including export-control regulations affecting almost any private discussion of encryption software -- we prefer that problem reports and discussions go to the lists, not directly to the team. Beware that the list goes worldwide; US citizens, read this important information about your export laws. If you're using this software, you really should be on the lists. To get onto them, visit lists.freeswan.org.

If you do send private mail to our coders or want a private reply from them, please make sure that the return address on your mail (From or Reply-To header) is a valid one. They have more important things to do than to unravel addresses that have been mangled in an attempt to confuse spammers.

6. Additional Notes on Troubleshooting

The following sections supplement the Guide: information available on your system; testing between security gateways; ifconfig reports for KLIPS debugging; using GDB on Pluto.

6.1 Information available on your system

6.1.1 Logs used

Linux FreeS/WAN logs to:

Check both places to get full information. If you find nothing, check your syslogd.conf(5) to see where your /etc/syslog.conf or equivalent is directing authpriv messages.

6.1.2 man pages provided

ipsec.conf(5)
Manual page for IPSEC configuration file.
ipsec(8)
Primary man page for ipsec utilities.

Other man pages are on this list and in

6.1.3 Status information

ipsec auto --status
Command to get status report from running system. Displays Pluto's state. Includes the list of connections which are currently "added" to Pluto's internal database; lists state objects reflecting ISAKMP and IPsec SAs being negotiated or installed.
ipsec look
Brief status info.
ipsec barf
Copious debugging info.

6.2 Testing between security gateways

Sometimes you need to test a subnet-subnet tunnel. This is a tunnel between two security gateways, which protects traffic on behalf of the subnets behind these gateways. On this network:

     Sunset==========West------------------East=========Sunrise
                     IPSec gateway         IPSec gateway
           local net       untrusted net       local net

you might name this tunnel sunset-sunrise. You can test this tunnel by having a machine behind one gateway ping a machine behind the other gateway, but this is not always convenient or even possible.

Simply pinging one gateway from the other is not useful. Such a ping does not normally go through the tunnel. The tunnel handles traffic between the two protected subnets, not between the gateways . Depending on the routing in place, a ping might

Neither event tells you anything about the tunnel. You can explicitly create an eroute to force such packets through the tunnel, or you can create additional tunnels as described in our configuration document, but those may be unnecessary complications in your situation.

The trick is to explicitly test between both gateways' private-side IP addresses. Since the private-side interfaces are on the protected subnets, the resulting packets do go via the tunnel. Use either ping -I or traceroute -i, both of which allow you to specify a source interface. (Note: unsupported on older Linuxes). The same principles apply for a road warrior (or other) case where only one end of your tunnel is a subnet.

6.3 ifconfig reports for KLIPS debugging

When diagnosing problems using ifconfig statistics, you may wonder what type of activity increments a particular counter for an ipsecN device. Here's an index, posted by KLIPS developer Richard Guy Briggs:

Here is a catalogue of the types of errors that can occur for which
statistics are kept when transmitting and receiving packets via klips.
I notice that they are not necessarily logged in the right counter.
. . .

Sources of ifconfig statistics for ipsec devices

rx-errors:
- packet handed to ipsec_rcv that is not an ipsec packet.
- ipsec packet with payload length not modulo 4.
- ipsec packet with bad authenticator length.
- incoming packet with no SA.
- replayed packet.
- incoming authentication failed.
- got esp packet with length not modulo 8.

tx_dropped:
- cannot process ip_options.
- packet ttl expired.
- packet with no eroute.
- eroute with no SA.
- cannot allocate sk_buff.
- cannot allocate kernel memory.
- sk_buff internal error.


The standard counters are:

struct enet_statistics
{
        int        rx_packets;                /* total packets received */
        int        tx_packets;                /* total packets transmitted */
        int        rx_errors;                /* bad packets received */
        int        tx_errors;                /* packet transmit problems */
        int        rx_dropped;                /* no space in linux buffers */
        int        tx_dropped;                /* no space available in linux */
        int        multicast;                /* multicast packets received */
        int        collisions;

        /* detailed rx_errors: */
        int        rx_length_errors;
        int        rx_over_errors;                /* receiver ring buff overflow */
        int        rx_crc_errors;                /* recved pkt with crc error */
        int        rx_frame_errors;        /* recv'd frame alignment error */
        int        rx_fifo_errors;                /* recv'r fifo overrun */
        int        rx_missed_errors;        /* receiver missed packet */

        /* detailed tx_errors */
        int        tx_aborted_errors;
        int        tx_carrier_errors;
        int        tx_fifo_errors;
        int        tx_heartbeat_errors;
        int        tx_window_errors;
};

of which I think only the first 6 are useful.

6.4 Using GDB on Pluto

You may need to use the GNU debugger, gdb(1), on Pluto. This should be necessary only in unusual cases, for example if you encounter a problem which the Pluto developer cannot readily reproduce or if you are modifying Pluto.

Here are the Pluto developer's suggestions for doing this:

Can you get a core dump and use gdb to find out what Pluto was doing
when it died?

To get a core dump, you will have to set dumpdir to point to a
suitable directory (see ipsec.conf(5)).

To get gdb to tell you interesting stuff:
        $ script
        $ cd dump-directory-you-chose
        $ gdb /usr/local/lib/ipsec/pluto core
        (gdb) where
        (gdb) quit
        $ exit

The resulting output will have been captured by the script command in
a file called "typescript".  Send it to the list.

Do not delete the core file.  I may need to ask you to print out some
more relevant stuff.

Note that the dumpdir parameter takes effect only when the IPsec subsystem is restarted -- reboot or ipsec setup restart.




Contents Previous Next