IPSEC: Tunnel vs Transport Mode

If you go looking for it, there is whole stack of IPSEC documentation out there. It's mostly fairly dense, and tends to concentrate on explaining the somewhat complex operation and configuration details rather than exploring design choices.

One typical scenario is that you find yourself tasked with managing a multisite topology with redundant paths, and over third-party provider networks (private and public). The result is a requirement to implement encryption for all intersite traffic, for which the usual, and often only contender is IPSEC.

Those implementing IPSEC for the first time find that there are a large number of choices to be made, and all of them may seem to be equally important. As a result, the final implementation often bears a strong resemblance to one of the examples which can be found on the Cisco site (with all the subtleties hidden and important decisions pre-made).

One key decision involves the choice of operating mode: Tunnel or Transport.

Typically you find the differences between the two described in a number of ways such as:

• Tunnel mode is used between gateways while Transport mode is used between endstations.
• Tunnel mode is used for pass-through traffic, while Transport mode is used for end-to-end traffic
• Tunnel mode encrypts the whole packet and provides a new header, while Transport mode only encrypts the data (payload).

These descriptions have hints and clues inside them but they don't really tell you why and when you should use them. But once you understand what the basic choice means, IPSEC suddenly becomes a lot more friendly.

Here's the question I think you should be asking:
"Which mode will best support my routing model?"

So, do you use dynamic routing or static routing? This is important because some of the same reasoning you use to justify your routing choice will be the same reasoning you use in making the Tunnel vs Transport choice.

Let's look at the static routing approach:
"I have a simple network and by using static routes I have complete control over what traffic is sent across my links."

So each static route is created at the 'source' of a link, directing traffic to the other end. This is much the same as a typical IPSEC Tunnel-mode link which uses ACLs to define "interesting traffic" at the 'source', to be sent to the other end.

But in order to get your traffic to traverse that IPSEC link, you must have a static route, and you must have a corresponding (crypto) ACL present. Without both in place, the forwarding won't happen. They must be matched by a mirror route/ACL at the other end. So that's four manual entries for each definable flow which must be updated if subnets or paths change.

What about the dynamic routing approach?:
"I have a complex network with multiple paths which I want to be discovered and utilised as needed by the network"

So someone taking this approach doesn't really want to be clumsily routing by ACL, but dynamically with a routing protocol. Trouble is, IPSEC Tunnel mode only handles unicast traffic, which would leave you with BGP as the only usable routing protocol.

You don't really want your routing configuration to have any dependencies on your IPSEC configuration at all. This is where Transport mode comes in:

Create an IPSEC Transport mode link between your pair of site routers and use it to carry only GRE traffic to create a GRE tunnel with its own /30 subnet and addressing, independent of the IPSEC link addressing.

Now, multicast routing protocols such as OSPF and EIGRP will run over the link and take care of all the other traffic.

The only ACL you need to define for IPSEC is one that identifies the peer router for GRE traffic, which won't change even if routing paths and subnet locations do. So that's four manual entries to take care of any number of definable flows, and which don't need to change unless one of the two site routers actually changes its address.

And that's all there is to it. (at a high level).

So in summary:

• Tunnel mode IPSEC forces you to implement "Routing by Crypto-Map", which is ugly and unscalable, but appropriate for links between your external firewall and some other organisation, for instance.
• Transport mode IPSEC (+GRE) frees up the routing design and makes it independent of encryption implementation; it is therefore ideal for any internal links, WAN or LAN.

This is in some ways, counter intuitive: Use Transport mode to carry tunnels and use Tunnel mode to transport raw packets.

PS: Don't forget if you use Transport mode IPSEC with GRE, that there are now two layers of encapsulation and you will need to take extra care with fragmentation and MTU issues. At a minimum you should have path MTU discovery enabled, ICMP unreachables NOT blocked and DF bits copied from the original IP header to the GRE header.