Two-way NAT / PAT on a VPN (Cisco) Stick

Some time ago I was tasked with interfacing to a couple of other multi-site organisations across a large governmental network similar in operation to the Internet. This was an interim measure prior to integrating aspects of the three networks into a single entity, and prior to having any dedicated WAN links in place.

I had to provide connectivity between a variable number of users and servers across all three networks, and with many overlapping IP ranges in place. The idea was to have a flexible enough configuration that I could easily add and change routes at the far end to keep pace with any integration work.

The last requirement was the support of one or more AD trusts between the organisations, with DNS forwarding.

To make it as light a touch a possible for the far-end IT departments, I went for a single interface Cisco router that could be connected directly to a Firewall DMZ on the far end firewall.

The topology essentially looked like this (with extraneous devices stripped away) and Internet substituted for the (private) government network:

At a more useful level, showing physical interfaces and IP addresses:

NAT

Ideally, in order to allow variable numbers of users to cross the NAT boundary in either direction, one would be able to use PAT in both directions. However, this is only available to the “inside” interface.
As a large number of users were likely to be crossing from the remote site, and only a few from the hub site, I had to make the remote physical interface act as “inside” and the tunnel act as “outside”.
This allowed me to use PAT for remote users and dynamic NAT for hub users.
Servers were easily handled by static NAT in both directions

Non-NAT-Compliant Applications

Nowadays, most applications, including to my surprise, Microsoft domain trusts, work quite well across (Cisco IOS) NAT boundaries. I found only one application which didn’t: an old version of HP Openview ServiceDesk, which embeds the source IP of the HPOV server inside the java client for use in a subsequent return connection.
In this particular instance, the server was based at the hub site, and no IP conflict existed at the remote site. I was able to create an identity NAT for the server in the direction of the Hub which worked fine once supporting routes were in place.

MTU issues

Because the remote firewall has not participated in creating the tunnel endpoint, it can’t respond correctly to hub-destined traffic with DF flag set, so we have to ensure that the remote firewall allows ICMP unreachables to be sent from our router to devices on its internal network.

Design Notes

Some basic notes might be required to clarify where all the addresses are coming from:

IPSEC and GRE tunnel end-points

The physical interfaces representing the Hub and Remote endpoints have internal 192.168 addresses and are mapped to NAT addresses on their upstream firewalls. I have used 10.1.1.10 and 10.2.2.10 respectively.

Inter-Org NAT Allocation

Subnet 192.168.198.0/24 is used to Dynamically NAT all Hub users accessing Remote servers
Subnet 192.168.199.0/24 is used to present Hub servers to Remote users.
IP Address 192.168.200.1 is used to PAT all Remote users accessing Hub servers
Subnet 192.168.200.0/24 is used to present Remote servers to Hub users.

Configuration Fragments

The configurations below have been taken from working devices, with some minimal IP address obfuscation:

Hub Distribution Router:

! IKE Phase 1
crypto isakmp policy 1
 encr aes
 authentication pre-share
 group 5

! Pre-shared key for remote site
crypto isakmp key RemoteSiteKey address 10.2.2.10

! IKE Phase 2
crypto ipsec transform-set AES256_SHA_tra esp-aes 256 esp-sha-hmac 
 mode transport

! Crypto ACL for GRE to remote site
ip access-list extended HUB-INTERNET-REMOTE-CRYACL
 remark Tunneled traffic over the Internet to Remote site
 permit gre host 192.168.12.9 host 10.2.2.10

! Crypto MAP entry for remote site
crypto map INTERNET-CM 15 ipsec-isakmp 
 set peer 10.2.2.10
 set transform-set AES256_SHA_tra 
 match address HUB-INTERNET-REMOTE-CRYACL

! Physical interface for termination of all WAN and Internet tunnels
interface GigabitEthernet0/1
 description Connects to Local Firewall inside
 ip address 192.168.12.9 255.255.255.248
 crypto map INTERNET-CM

! Tunnel to remote site
interface Tunnel14200
 description Tunnel over Internet to Remote Site
 ! low bandwidth used (EIGRP) for backup tunnels over Internet
 bandwidth 1000
 ip address 192.168.14.201 255.255.255.252
 ! Maximum starting MTU (1500 - 8(NAT-T) - 53(AES256) - 24(GRE))
 ip mtu 1415
 ! high delay used (EIGRP) for backup tunnels over Internet
 delay 2000
 tunnel source GigabitEthernet0/1
 tunnel destination 10.2.2.10
 ! Tell GRE to copy DF from inner to outer IP header
 tunnel path-mtu-discovery

ip prefix-list EIGRP-SITETUNNELS-OUT-PL description Route adverts to remote sites
ip prefix-list EIGRP-SITETUNNELS-OUT-PL seq 5 permit 0.0.0.0/0
ip prefix-list EIGRP-SITETUNNELS-OUT-PL seq 10 permit 192.168.0.0/16 le 32

router eigrp 192
 passive-interface GigabitEthernet0/1
 network 192.168.12.0
 network 192.168.14.0
 distribute-list prefix EIGRP-SITETUNNELS-OUT-PL out Tunnel14200
 no auto-summary
 no eigrp log-neighbor-changes

Hub Firewall:

PIX Version 7.2(2)

name 10.1.1.10 dist-rt02_INTERNET
name 192.168.12.9 dist-rt02_G01

object-group service NAT-T udp
 description NAT Traversal
 port-object eq 4500

object-group service IPsec_udp udp
 description UDP protocols used by IPsec
 group-object NAT-T
 port-object eq isakmp

object-group network Cisco_Devices
 description Cisco devices' Internet interfaces
 network-object host remote-rt01_INTERNET

interface Ethernet0
 speed 100
 duplex full
 nameif outside
 security-level 0
 ip address 10.1.1.4 255.255.255.0 standby 10.1.1.5 

interface Ethernet1
 speed 100
 duplex full
 nameif inside
 security-level 100
 ip address 192.168.12.12 255.255.255.248 standby 192.168.12.13

route outside 0.0.0.0 0.0.0.0 10.1.1.1 1
route inside 192.168.0.0 255.255.0.0 192.168.12.9 1

! Mapping the routable Tunnel endpoint
static (inside,outside) dist-rt02_INTERNET dist-rt02_G01 netmask 255.255.255.255 

access-list inside-access-in remark Allow ISAKMP & NAT-T to sites using VPN-over-Internet
access-list inside-access-in extended permit udp host dist-rt02_G01 object-group Cisco_Devices object-group IPsec_udp log disable 

access-group inside-access-in in interface inside

Remote Firewall:

PIX Version 6.3(4)

interface ethernet0 100full
interface ethernet1 100full
interface ethernet4 100full

nameif ethernet0 outside security0
nameif ethernet1 inside security100
nameif ethernet4 HUBDMZ security49

ip address outside 10.2.2.4 255.255.255.0
ip address inside 192.168.0.5 255.255.254.0
ip address HUBDMZ 192.168.22.9 255.255.255.248

failover ip address outside 10.2.2.5
failover ip address inside 192.168.0.6
failover ip address HUBDMZ 192.168.22.10

object-group network HUB
  description HUBDMZ network
  network-object 192.168.22.8 255.255.255.248 
  description HUB users on this subnet
  network-object 192.168.198.0 255.255.255.0 
  description HUB servers on this subnet
  network-object 192.168.199.0 255.255.255.0 

! Minimal ACLs to permit traffic flow – not representative!
access-list inside_access_in permit ip any object-group HUB 
access-group inside_access_in in interface inside

access-list hubdmz_access_in permit icmp any any
access-list hubdmz_access_in permit ip host 192.168.22.13 host 10.1.1.10 
access-list hubdmz_access_in permit ip object-group HUB 192.168.0.0 255.255.0.0 
access-group hubdmz_access_in in interface HUBDMZ

access-list outside_access_in permit udp host 10.1.1.10 host 10.2.2.13 eq 4500 
access-list outside_access_in permit udp host 10.1.1.10 host 10.2.2.13 eq isakmp 
access-group outside_access_in in interface outside

! Bypass NAT for incoming HUB traffic (low security to high security)
access-list NO_NAT_HUBDMZ permit ip object-group HUB 192.168.0.0 255.255.0.0 
nat (HUBDMZ) 0 access-list NO_NAT_HUBDMZ

! Mapping the routable Tunnel endpoint
static (HUBDMZ,outside) 10.2.2.10 192.168.22.13 netmask 255.255.255.255 0 0 

! Hub users and servers respectively
route HUBDMZ 192.168.198.0 255.255.255.0 192.168.22.13 1
route HUBDMZ 192.168.199.0 255.255.255.0 192.168.22.13 1

Remote VPN Router:

! example hub hosts with pre(real) and post nat addresses (hub perspective)
ip host hubhost01 192.168.4.10 192.168.199.5
ip host hubhost02 192.168.4.20 192.168.199.6
! example remote hosts with "pre" and "post"(real) nat (hub perspective)
ip host remotehost01 192.168.200.7 192.168.4.50
ip host remotehost02 192.168.200.8 192.168.4.51

! need inspection to activate ALGs
ip inspect name INSPECT_LIST dns
ip inspect name INSPECT_LIST ftp
ip inspect name INSPECT_LIST https
ip inspect name INSPECT_LIST icmp
ip inspect name INSPECT_LIST imap
ip inspect name INSPECT_LIST pop3
ip inspect name INSPECT_LIST esmtp
ip inspect name INSPECT_LIST sqlnet
ip inspect name INSPECT_LIST streamworks
ip inspect name INSPECT_LIST tftp
ip inspect name INSPECT_LIST tcp
ip inspect name INSPECT_LIST udp
ip inspect name INSPECT_LIST vdolive
ip inspect name INSPECT_LIST kerberos
ip inspect name INSPECT_LIST ldap
ip inspect name INSPECT_LIST microsoft-ds

! IKE Phase 1
crypto isakmp policy 1
 encr aes
 authentication pre-share
 group 5

! Pre-shared key for this site
crypto isakmp key RemoteSiteKey address 10.1.1.10

! IKE Phase 2
crypto ipsec transform-set AES256_SHA_tra esp-aes 256 esp-sha-hmac 
 mode transport

! Crypto ACL for GRE to hub site
ip access-list extended REMOTE-INTERNET-HUB-CRYACL
 remark Traffic tunnelled over Internet to HUB
 permit gre host 192.168.22.13 host 10.1.1.10

! Crypto MAP entry for hub site
crypto map INTERNET-CM 2 ipsec-isakmp 
 set peer 10.1.1.10
 set transform-set AES256_SHA_tra 
 match address REMOTE-INTERNET-HUB-CRYACL

! Single physical interface for LAN and VPN traffic
! in/out ACL not included in config
interface FastEthernet0/0
 description Exit to Internet and Remote LAN via Remote DMZ
 ip address 192.168.22.13 255.255.255.248
 no ip redirects
 ip inspect INSPECT_LIST in
 ! Treat the remote network as inside so we can use PAT
 ip nat inside
 ! enabled automatically with NAT config
 ip virtual-reassembly
 duplex full
 speed 100
 no cdp enable
 crypto map INTERNET-CM

interface Loopback0
 description Remote PAT address for overlapping client subnets
 ip address 192.168.200.1 255.255.255.0

interface Tunnel14200
 description Tunnel over Internet to Hub network
 ! low bandwidth used (EIGRP) for backup tunnels over Internet
 bandwidth 1000
 ip address 192.168.14.202 255.255.255.252
 ! Maximum starting MTU (1500-8(NAT-T)-53(AES256)-24(GRE))
 ip mtu 1415
 ! Required to allow PAT in the opposite direction
 ip nat outside
 ! enabled automatically with NAT config
 ip virtual-reassembly
 ! high delay used (EIGRP) for backup tunnels over Internet
 delay 2000
 tunnel source FastEthernet0/0
 tunnel destination 10.1.1.10
 tunnel path-mtu-discovery

router eigrp 192
 passive-interface Loopback0
 network 192.168.14.0
 network 192.168.200.0
 distribute-list prefix EIGRP-TUNNEL-OUT-PL out Tunnel14200
 no auto-summary
 eigrp stub connected

! Floating default route back to the hub over the tunnel
ip route 0.0.0.0 0.0.0.0 192.168.14.201 200

! Example remote site networks - 192.168.4.0 chosen to demonstrate overlaps
ip route 192.168.0.0 255.255.254.0 192.168.22.9
ip route 192.168.4.0 255.255.254.0 192.168.22.9
ip route 192.168.35.0 255.255.254.0 192.168.22.9

! Explicit route for our tunnel destination to avoid recursion
ip route 10.1.1.0 255.255.255.0 192.168.22.9

! We need the flexibility of PAT to be applied to the remote network
ip nat inside source list REMOTE-USERS interface Loopback0 overload

! Which leaves us on the "outside" using dynamic NAT
ip nat pool HUB-POOL 192.168.198.1 192.168.198.254 prefix-length 24
ip nat outside source list HUB-USERS pool HUB-POOL

! Example remote servers - DNS ALG will use these to translate our queries
ip nat inside source static 192.168.4.50 192.168.200.7
ip nat inside source static 192.168.4.51 192.168.200.8

! Example hub servers - DNS ALG will use these to translate their queries
ip nat outside source static 192.168.4.10 192.168.199.5
ip nat outside source static 192.168.4.20 192.168.199.6

! Define which remote subnets hide behind PAT
ip access-list standard REMOTE-USERS
 remark Remote main site
 permit 192.168.0.0 0.0.1.255
 remark Remote secondary site example
 permit 192.168.35.0 0.0.1.255

! Define which hub subnets hide behind Dynamic NAT
ip access-list standard HUB-USERS
 remark Hub IT department
 permit 192.168.32.0 0.0.0.255
 remark Hub main site
 permit 192.168.125.0 0.0.0.255
 remark Hub secondary site example
 permit 192.168.35.0 0.0.0.255

ip prefix-list EIGRP-TUNNEL-OUT-PL description Routes to be advertised from site
ip prefix-list EIGRP-TUNNEL-OUT-PL seq 5 permit 192.168.200.0/24

Postscript

The creation and ongoing support of Microsoft domain trusts across this two-way NAT boundary was reasonably straight forward. There were a couple of issues, neither of which were caused by or really impinged upon the configuration itself, but might be worth mentioning:

1. Problems creating a domain Trust across two-way NAT
I found it useful to ensure that all DNS servers in both domains could see and forward to each other. In one of the organisations I needed to connect to, this was tiresome because they had at least 6 DCs of which 4 were DNS servers. This requires static NAT entries to be configured for each server.
I also found that physical DCs were more reliable  than VMs, in part due to VMware tools not being installed thoughtfully - the Shared Folders option should not be installed as it causes network (RPC) problems. However, you can't chose in advance which DCs will participate on each side, so it becomes useful to be able to mask off the suspect ones by removing their NAT entries.

2. Kerberos-related fragmentation
Depending upon the server and workstation versions, Kerberos may still default to UDP, which may cause performance problems due to fragmentation. This is particularly noticable where W2K3 and XP are in use, and where there are many nested groups and SID histories to bloat the packets. This manifests itself as a delay in accessing resources across the trust. Debugging ip virtual-reassembly may show maximum fragments or fragmentation buffer being exceeded and some additional tweaking may be required to prevent timeouts and retransmissions within Kerberos.

Sources

I found the following document very useful in getting this to production:
NAT Order of Operation

IPSEC: Tunnel vs Transport Mode

If you go looking for it, there is whole stack of IPSEC documentation out there. It's mostly fairly dense, and tends to concentrate on explaining the somewhat complex operation and configuration details rather than exploring design choices.

One typical scenario is that you find yourself tasked with managing a multisite topology with redundant paths, and over third-party provider networks (private and public). The result is a requirement to implement encryption for all intersite traffic, for which the usual, and often only contender is IPSEC.

Those implementing IPSEC for the first time find that there are a large number of choices to be made, and all of them may seem to be equally important. As a result, the final implementation often bears a strong resemblance to one of the examples which can be found on the Cisco site (with all the subtleties hidden and important decisions pre-made).

One key decision involves the choice of operating mode: Tunnel or Transport.

Typically you find the differences between the two described in a number of ways such as:

• Tunnel mode is used between gateways while Transport mode is used between endstations.
• Tunnel mode is used for pass-through traffic, while Transport mode is used for end-to-end traffic
• Tunnel mode encrypts the whole packet and provides a new header, while Transport mode only encrypts the data (payload).

These descriptions have hints and clues inside them but they don't really tell you why and when you should use them. But once you understand what the basic choice means, IPSEC suddenly becomes a lot more friendly.

Here's the question I think you should be asking:
"Which mode will best support my routing model?"

So, do you use dynamic routing or static routing? This is important because some of the same reasoning you use to justify your routing choice will be the same reasoning you use in making the Tunnel vs Transport choice.

Let's look at the static routing approach:
"I have a simple network and by using static routes I have complete control over what traffic is sent across my links."

So each static route is created at the 'source' of a link, directing traffic to the other end. This is much the same as a typical IPSEC Tunnel-mode link which uses ACLs to define "interesting traffic" at the 'source', to be sent to the other end.

But in order to get your traffic to traverse that IPSEC link, you must have a static route, and you must have a corresponding (crypto) ACL present. Without both in place, the forwarding won't happen. They must be matched by a mirror route/ACL at the other end. So that's four manual entries for each definable flow which must be updated if subnets or paths change.

What about the dynamic routing approach?:
"I have a complex network with multiple paths which I want to be discovered and utilised as needed by the network"

So someone taking this approach doesn't really want to be clumsily routing by ACL, but dynamically with a routing protocol. Trouble is, IPSEC Tunnel mode only handles unicast traffic, which would leave you with BGP as the only usable routing protocol.

You don't really want your routing configuration to have any dependencies on your IPSEC configuration at all. This is where Transport mode comes in:

Create an IPSEC Transport mode link between your pair of site routers and use it to carry only GRE traffic to create a GRE tunnel with its own /30 subnet and addressing, independent of the IPSEC link addressing.

Now, multicast routing protocols such as OSPF and EIGRP will run over the link and take care of all the other traffic.

The only ACL you need to define for IPSEC is one that identifies the peer router for GRE traffic, which won't change even if routing paths and subnet locations do. So that's four manual entries to take care of any number of definable flows, and which don't need to change unless one of the two site routers actually changes its address.

And that's all there is to it. (at a high level).

So in summary:

• Tunnel mode IPSEC forces you to implement "Routing by Crypto-Map", which is ugly and unscalable, but appropriate for links between your external firewall and some other organisation, for instance.
• Transport mode IPSEC (+GRE) frees up the routing design and makes it independent of encryption implementation; it is therefore ideal for any internal links, WAN or LAN.

This is in some ways, counter intuitive: Use Transport mode to carry tunnels and use Tunnel mode to transport raw packets.

PS: Don't forget if you use Transport mode IPSEC with GRE, that there are now two layers of encapsulation and you will need to take extra care with fragmentation and MTU issues. At a minimum you should have path MTU discovery enabled, ICMP unreachables NOT blocked and DF bits copied from the original IP header to the GRE header.