Saturday 4 August 2012

Two-way NAT / PAT on a VPN (Cisco) Stick

Some time ago I was tasked with interfacing to a couple of other multi-site organisations across a large governmental network similar in operation to the Internet. This was an interim measure prior to integrating aspects of the three networks into a single entity, and prior to having any dedicated WAN links in place.

I had to provide connectivity between a variable number of users and servers across all three networks, and with many overlapping IP ranges in place. The idea was to have a flexible enough configuration that I could easily add and change routes at the far end to keep pace with any integration work.

The last requirement was the support of one or more AD trusts between the organisations, with DNS forwarding.

To make it as light a touch a possible for the far-end IT departments, I went for a single interface Cisco router that could be connected directly to a Firewall DMZ on the far end firewall.

The topology essentially looked like this (with extraneous devices stripped away) and Internet substituted for the (private) government network:


At a more useful level, showing physical interfaces and IP addresses:




NAT

Ideally, in order to allow variable numbers of users to cross the NAT boundary in either direction, one would be able to use PAT in both directions. However, this is only available to the “inside” interface.
As a large number of users were likely to be crossing from the remote site, and only a few from the hub site, I had to make the remote physical interface act as “inside” and the tunnel act as “outside”.
This allowed me to use PAT for remote users and dynamic NAT for hub users.
Servers were easily handled by static NAT in both directions

Non-NAT-Compliant Applications

Nowadays, most applications, including to my surprise, Microsoft domain trusts, work quite well across (Cisco IOS) NAT boundaries. I found only one application which didn’t: an old version of HP Openview ServiceDesk, which embeds the source IP of the HPOV server inside the java client for use in a subsequent return connection.
In this particular instance, the server was based at the hub site, and no IP conflict existed at the remote site. I was able to create an identity NAT for the server in the direction of the Hub which worked fine once supporting routes were in place.

MTU issues

Because the remote firewall has not participated in creating the tunnel endpoint, it can’t respond correctly to hub-destined traffic with DF flag set, so we have to ensure that the remote firewall allows ICMP unreachables to be sent from our router to devices on its internal network.

Design Notes

Some basic notes might be required to clarify where all the addresses are coming from:

IPSEC and GRE tunnel end-points

The physical interfaces representing the Hub and Remote endpoints have internal 192.168 addresses and are mapped to NAT addresses on their upstream firewalls. I have used 10.1.1.10 and 10.2.2.10 respectively.

Inter-Org NAT Allocation

Subnet 192.168.198.0/24 is used to Dynamically NAT all Hub users accessing Remote servers
Subnet 192.168.199.0/24 is used to present Hub servers to Remote users.
IP Address 192.168.200.1 is used to PAT all Remote users accessing Hub servers
Subnet 192.168.200.0/24 is used to present Remote servers to Hub users.

Configuration Fragments

The configurations below have been taken from working devices, with some minimal IP address obfuscation:

Hub Distribution Router:
! IKE Phase 1
crypto isakmp policy 1
 encr aes
 authentication pre-share
 group 5

! Pre-shared key for remote site
crypto isakmp key RemoteSiteKey address 10.2.2.10

! IKE Phase 2
crypto ipsec transform-set AES256_SHA_tra esp-aes 256 esp-sha-hmac 
 mode transport

! Crypto ACL for GRE to remote site
ip access-list extended HUB-INTERNET-REMOTE-CRYACL
 remark Tunneled traffic over the Internet to Remote site
 permit gre host 192.168.12.9 host 10.2.2.10

! Crypto MAP entry for remote site
crypto map INTERNET-CM 15 ipsec-isakmp 
 set peer 10.2.2.10
 set transform-set AES256_SHA_tra 
 match address HUB-INTERNET-REMOTE-CRYACL

! Physical interface for termination of all WAN and Internet tunnels
interface GigabitEthernet0/1
 description Connects to Local Firewall inside
 ip address 192.168.12.9 255.255.255.248
 crypto map INTERNET-CM

! Tunnel to remote site
interface Tunnel14200
 description Tunnel over Internet to Remote Site
 ! low bandwidth used (EIGRP) for backup tunnels over Internet
 bandwidth 1000
 ip address 192.168.14.201 255.255.255.252
 ! Maximum starting MTU (1500 - 8(NAT-T) - 53(AES256) - 24(GRE))
 ip mtu 1415
 ! high delay used (EIGRP) for backup tunnels over Internet
 delay 2000
 tunnel source GigabitEthernet0/1
 tunnel destination 10.2.2.10
 ! Tell GRE to copy DF from inner to outer IP header
 tunnel path-mtu-discovery

ip prefix-list EIGRP-SITETUNNELS-OUT-PL description Route adverts to remote sites
ip prefix-list EIGRP-SITETUNNELS-OUT-PL seq 5 permit 0.0.0.0/0
ip prefix-list EIGRP-SITETUNNELS-OUT-PL seq 10 permit 192.168.0.0/16 le 32

router eigrp 192
 passive-interface GigabitEthernet0/1
 network 192.168.12.0
 network 192.168.14.0
 distribute-list prefix EIGRP-SITETUNNELS-OUT-PL out Tunnel14200
 no auto-summary
 no eigrp log-neighbor-changes



Hub Firewall:
PIX Version 7.2(2)

name 10.1.1.10 dist-rt02_INTERNET
name 192.168.12.9 dist-rt02_G01

object-group service NAT-T udp
 description NAT Traversal
 port-object eq 4500

object-group service IPsec_udp udp
 description UDP protocols used by IPsec
 group-object NAT-T
 port-object eq isakmp

object-group network Cisco_Devices
 description Cisco devices' Internet interfaces
 network-object host remote-rt01_INTERNET

interface Ethernet0
 speed 100
 duplex full
 nameif outside
 security-level 0
 ip address 10.1.1.4 255.255.255.0 standby 10.1.1.5 

interface Ethernet1
 speed 100
 duplex full
 nameif inside
 security-level 100
 ip address 192.168.12.12 255.255.255.248 standby 192.168.12.13

route outside 0.0.0.0 0.0.0.0 10.1.1.1 1
route inside 192.168.0.0 255.255.0.0 192.168.12.9 1

! Mapping the routable Tunnel endpoint
static (inside,outside) dist-rt02_INTERNET dist-rt02_G01 netmask 255.255.255.255 

access-list inside-access-in remark Allow ISAKMP & NAT-T to sites using VPN-over-Internet
access-list inside-access-in extended permit udp host dist-rt02_G01 object-group Cisco_Devices object-group IPsec_udp log disable 

access-group inside-access-in in interface inside



Remote Firewall:
PIX Version 6.3(4)

interface ethernet0 100full
interface ethernet1 100full
interface ethernet4 100full

nameif ethernet0 outside security0
nameif ethernet1 inside security100
nameif ethernet4 HUBDMZ security49

ip address outside 10.2.2.4 255.255.255.0
ip address inside 192.168.0.5 255.255.254.0
ip address HUBDMZ 192.168.22.9 255.255.255.248

failover ip address outside 10.2.2.5
failover ip address inside 192.168.0.6
failover ip address HUBDMZ 192.168.22.10

object-group network HUB
  description HUBDMZ network
  network-object 192.168.22.8 255.255.255.248 
  description HUB users on this subnet
  network-object 192.168.198.0 255.255.255.0 
  description HUB servers on this subnet
  network-object 192.168.199.0 255.255.255.0 

! Minimal ACLs to permit traffic flow – not representative!
access-list inside_access_in permit ip any object-group HUB 
access-group inside_access_in in interface inside

access-list hubdmz_access_in permit icmp any any
access-list hubdmz_access_in permit ip host 192.168.22.13 host 10.1.1.10 
access-list hubdmz_access_in permit ip object-group HUB 192.168.0.0 255.255.0.0 
access-group hubdmz_access_in in interface HUBDMZ

access-list outside_access_in permit udp host 10.1.1.10 host 10.2.2.13 eq 4500 
access-list outside_access_in permit udp host 10.1.1.10 host 10.2.2.13 eq isakmp 
access-group outside_access_in in interface outside

! Bypass NAT for incoming HUB traffic (low security to high security)
access-list NO_NAT_HUBDMZ permit ip object-group HUB 192.168.0.0 255.255.0.0 
nat (HUBDMZ) 0 access-list NO_NAT_HUBDMZ

! Mapping the routable Tunnel endpoint
static (HUBDMZ,outside) 10.2.2.10 192.168.22.13 netmask 255.255.255.255 0 0 

! Hub users and servers respectively
route HUBDMZ 192.168.198.0 255.255.255.0 192.168.22.13 1
route HUBDMZ 192.168.199.0 255.255.255.0 192.168.22.13 1



Remote VPN Router:

! example hub hosts with pre(real) and post nat addresses (hub perspective)
ip host hubhost01 192.168.4.10 192.168.199.5
ip host hubhost02 192.168.4.20 192.168.199.6
! example remote hosts with "pre" and "post"(real) nat (hub perspective)
ip host remotehost01 192.168.200.7 192.168.4.50
ip host remotehost02 192.168.200.8 192.168.4.51

! need inspection to activate ALGs
ip inspect name INSPECT_LIST dns
ip inspect name INSPECT_LIST ftp
ip inspect name INSPECT_LIST https
ip inspect name INSPECT_LIST icmp
ip inspect name INSPECT_LIST imap
ip inspect name INSPECT_LIST pop3
ip inspect name INSPECT_LIST esmtp
ip inspect name INSPECT_LIST sqlnet
ip inspect name INSPECT_LIST streamworks
ip inspect name INSPECT_LIST tftp
ip inspect name INSPECT_LIST tcp
ip inspect name INSPECT_LIST udp
ip inspect name INSPECT_LIST vdolive
ip inspect name INSPECT_LIST kerberos
ip inspect name INSPECT_LIST ldap
ip inspect name INSPECT_LIST microsoft-ds

! IKE Phase 1
crypto isakmp policy 1
 encr aes
 authentication pre-share
 group 5

! Pre-shared key for this site
crypto isakmp key RemoteSiteKey address 10.1.1.10

! IKE Phase 2
crypto ipsec transform-set AES256_SHA_tra esp-aes 256 esp-sha-hmac 
 mode transport

! Crypto ACL for GRE to hub site
ip access-list extended REMOTE-INTERNET-HUB-CRYACL
 remark Traffic tunnelled over Internet to HUB
 permit gre host 192.168.22.13 host 10.1.1.10

! Crypto MAP entry for hub site
crypto map INTERNET-CM 2 ipsec-isakmp 
 set peer 10.1.1.10
 set transform-set AES256_SHA_tra 
 match address REMOTE-INTERNET-HUB-CRYACL

! Single physical interface for LAN and VPN traffic
! in/out ACL not included in config
interface FastEthernet0/0
 description Exit to Internet and Remote LAN via Remote DMZ
 ip address 192.168.22.13 255.255.255.248
 no ip redirects
 ip inspect INSPECT_LIST in
 ! Treat the remote network as inside so we can use PAT
 ip nat inside
 ! enabled automatically with NAT config
 ip virtual-reassembly
 duplex full
 speed 100
 no cdp enable
 crypto map INTERNET-CM

interface Loopback0
 description Remote PAT address for overlapping client subnets
 ip address 192.168.200.1 255.255.255.0

interface Tunnel14200
 description Tunnel over Internet to Hub network
 ! low bandwidth used (EIGRP) for backup tunnels over Internet
 bandwidth 1000
 ip address 192.168.14.202 255.255.255.252
 ! Maximum starting MTU (1500-8(NAT-T)-53(AES256)-24(GRE))
 ip mtu 1415
 ! Required to allow PAT in the opposite direction
 ip nat outside
 ! enabled automatically with NAT config
 ip virtual-reassembly
 ! high delay used (EIGRP) for backup tunnels over Internet
 delay 2000
 tunnel source FastEthernet0/0
 tunnel destination 10.1.1.10
 tunnel path-mtu-discovery

router eigrp 192
 passive-interface Loopback0
 network 192.168.14.0
 network 192.168.200.0
 distribute-list prefix EIGRP-TUNNEL-OUT-PL out Tunnel14200
 no auto-summary
 eigrp stub connected

! Floating default route back to the hub over the tunnel
ip route 0.0.0.0 0.0.0.0 192.168.14.201 200

! Example remote site networks - 192.168.4.0 chosen to demonstrate overlaps
ip route 192.168.0.0 255.255.254.0 192.168.22.9
ip route 192.168.4.0 255.255.254.0 192.168.22.9
ip route 192.168.35.0 255.255.254.0 192.168.22.9

! Explicit route for our tunnel destination to avoid recursion
ip route 10.1.1.0 255.255.255.0 192.168.22.9

! We need the flexibility of PAT to be applied to the remote network
ip nat inside source list REMOTE-USERS interface Loopback0 overload

! Which leaves us on the "outside" using dynamic NAT
ip nat pool HUB-POOL 192.168.198.1 192.168.198.254 prefix-length 24
ip nat outside source list HUB-USERS pool HUB-POOL

! Example remote servers - DNS ALG will use these to translate our queries
ip nat inside source static 192.168.4.50 192.168.200.7
ip nat inside source static 192.168.4.51 192.168.200.8

! Example hub servers - DNS ALG will use these to translate their queries
ip nat outside source static 192.168.4.10 192.168.199.5
ip nat outside source static 192.168.4.20 192.168.199.6

! Define which remote subnets hide behind PAT
ip access-list standard REMOTE-USERS
 remark Remote main site
 permit 192.168.0.0 0.0.1.255
 remark Remote secondary site example
 permit 192.168.35.0 0.0.1.255

! Define which hub subnets hide behind Dynamic NAT
ip access-list standard HUB-USERS
 remark Hub IT department
 permit 192.168.32.0 0.0.0.255
 remark Hub main site
 permit 192.168.125.0 0.0.0.255
 remark Hub secondary site example
 permit 192.168.35.0 0.0.0.255

ip prefix-list EIGRP-TUNNEL-OUT-PL description Routes to be advertised from site
ip prefix-list EIGRP-TUNNEL-OUT-PL seq 5 permit 192.168.200.0/24

Postscript

The creation and ongoing support of Microsoft domain trusts across this two-way NAT boundary was reasonably straight forward. There were a couple of issues, neither of which were caused by or really impinged upon the configuration itself, but might be worth mentioning:

1. Problems creating a domain Trust across two-way NAT
I found it useful to ensure that all DNS servers in both domains could see and forward to each other. In one of the organisations I needed to connect to, this was tiresome because they had at least 6 DCs of which 4 were DNS servers. This requires static NAT entries to be configured for each server.
I also found that physical DCs were more reliable  than VMs, in part due to VMware tools not being installed thoughtfully - the Shared Folders option should not be installed as it causes network (RPC) problems. However, you can't chose in advance which DCs will participate on each side, so it becomes useful to be able to mask off the suspect ones by removing their NAT entries.

2. Kerberos-related fragmentation
Depending upon the server and workstation versions, Kerberos may still default to UDP, which may cause performance problems due to fragmentation. This is particularly noticable where W2K3 and XP are in use, and where there are many nested groups and SID histories to bloat the packets. This manifests itself as a delay in accessing resources across the trust. Debugging ip virtual-reassembly may show maximum fragments or fragmentation buffer being exceeded and some additional tweaking may be required to prevent timeouts and retransmissions within Kerberos.

Sources

I found the following document very useful in getting this to production:
NAT Order of Operation