Wednesday, 2 April 2014

When RPF Breaks Traceroute

I came across an interesting little problem recently which was quite fun to unravel....

I work on a hub and spoke network, where most traffic from the spokes follows the default route back to the hubs, except for a few specific destinations which must be reached through a public-network-facing interface.

One day I tried to run a traceroute from one of our site routers, towards one of these specific destinations and found that I couldn't get past the first hop (provider router).


Type escape sequence to abort.
Tracing the route to

  1 4 msec 0 msec 0 msec
  2  *  *  *
  3  *  *  *

Here's the interface config:

interface GigabitEthernet0/0.801
 description Exit to N3 gateway
 encapsulation dot1Q 801
 ip address
 ip access-group N3-ACCESS-IN in
 ip verify unicast reverse-path
 no ip unreachables
 ip inspect SDM_LOW in
 ip inspect SDM_LOW out
 ip nat outside
 ip virtual-reassembly
 no cdp enable
 crypto map N3-CM

So I removed the ACL, then the inspect statements and then the "no ip unreachables", but I still got the same result.

The next hop looked good but I checked my route just to make sure:

fr-rt01#sh ip ro
Routing entry for
  Known via "bgp 65139", distance 200, metric 1, type internal
  Last update from 18:44:40 ago
  Routing Descriptor Blocks:
  *, from, 18:44:40 ago
      Route metric is 1, traffic share count is 1
      AS Hops 0

fr-rt01#sh ip ro
Routing entry for
  Known via "static", distance 1, metric 0
  Routing Descriptor Blocks:
      Route metric is 0, traffic share count is 1

Then clutching at straws, I removed my "ip verify unicast reverse-path", and traceroute worked just like it should...


Type escape sequence to abort.
Tracing the route to

  1 4 msec 0 msec 0 msec
  2 8 msec 8 msec 8 msec
  3 8 msec 8 msec 8 msec
  4 8 msec 8 msec 8 msec
  5 8 msec 8 msec 8 msec
  6 12 msec 8 msec 8 msec

And then I went and read up on traceroute and RPF to try to understand what was going on

When traceroute sends each UDP packet out, it expects to get an ICMP type 11 code 0 (time exceeded) back from each intermediate hop/router, with a source IP address of its exit interface (towards me).

RPF does a reverse lookup of the source address against CEF to check that the receiving interface is one of the best return paths to that address.

Taking the first few addresses in the working traceroute:

fr-rt01#sh ip cef, version 191479, epoch 0
0 packets, 0 bytes
  via, Tunnel13196, 0 dependencies
    next hop, Tunnel13196
    valid adjacency

fr-rt01#sh ip cef, version 191479, epoch 0
0 packets, 0 bytes
  via, Tunnel13196, 0 dependencies
    next hop, Tunnel13196

And there's the problem: without a route to the intermediate addresses, I only have a default route, which leaves by a different interface. So RPF is doing its job properly and breaking my traceroute.

In this case, it seems that the best solution is to tell RPF to ignore my traceroute replies by adding an ACL defining them as exceptions:

fr-rt01#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
fr-rt01(config)#access-list 101 permit icmp any host 11 0
fr-rt01(config)#int gi0/0.801
fr-rt01(config-subif)#ip verify unicast reverse-path 101

Which allows both features to work without side effects:


Type escape sequence to abort.
Tracing the route to

  1 4 msec 0 msec 0 msec
  2 8 msec 8 msec 8 msec
  3 8 msec 8 msec 8 msec
  4 8 msec 8 msec 8 msec
  5 8 msec 8 msec 8 msec
  6 12 msec 8 msec 8 msec
  7  *  *  *

fr-rt01#sh access-list 101
Extended IP access list 101
    10 permit icmp any host ttl-exceeded (15 matches)

Thursday, 6 March 2014

A Flexible RRD Checker for Nagios

I was asked recently to get Nagios to flag *under*utilisation for a bunch of WAN links.

I had been using a shell script written I think by Garry Cook and Israel Brewster, with a number of hacks to add some extra functionality, but I couldn't get this additional mod going without a complete rewrite.

# AUTHOR:   Philip Damian-Grint
# MODIFIED: 6th March 2014
# VERSION:  0.5
#   Nagios Plugin to compare utilisation values from an RRD file with 
#   warning and critical thresholds.
#   Features:
#   1.  Threshold units and RRD units can be individually specified.
#       Thresholds default to Kilobytes/sec and RRD units default to Bytes/sec (MRTG default)
#   2.  RRD filepath can be supplied on the command line or via environment variable
#   3.  Threshold direction can be reversed so that low utilisation can also be checked
#   4.  Time period can be specified in minutes, hours, days or months; a basic mean average
#       is taken over multiple records. Defaults to 10 minutes
#   5.  Multipliers used for unit conversion can be decimal (default) or binary
#   6.  Threshold behaviour can be specified so that only one direction, both directions, 
#       any (default) direction, or the sum of both directions can be checked against the threshold.
#   7.  A maximum age of data threshold can be specified
#   6.  An http link (or any text) can be appended to line 1 output.
# Notes:
#   1.  This has been tested on a Centos 6.4 system with Nagios v4, rrdtool v1.4.8,
#       and Python 2.6.6
#   2.  All errors prior to fetching data or resulting in invalid or suspect data return UNKNOWN.
#   3.  At present, only AVERAGE values are processed
#   4.  Verbose includes a report on number of empty records, latest timestamp, RRD file processed,
#       threshold behaviour and threshold direction
#   Example configuration:
# file: checkcommands.cfg
# # Check 7-day average sum of in and out not below supplied thresholds, 
# #   and insert a link to MRTG at the end of line 1
# #'check_under_util' command definition
# define command{
#    command_name    check_under_util
#    command_line    $USER1$/check_rrd -f /usr/local/mrtg/share/rrd/$ARG1$.rrd -w $ARG2$ -c $ARG3$ -r -p 7days -m sumonly -v -l '<a href=/mrtg/cgi-bin/mrtg-rrd.cgi/$ARG1$.html style=font-size:6pt target=_blank>MRTG</a>'
#    }

import argparse
from argparse import RawTextHelpFormatter
import os
import re
import rrdtool
import sys
import time

class CheckRRD(object):
    '''Structure to store key variables and data'''

    # Nagios states - offsets = return code
    states = ('OK', 'WARNING', 'CRITICAL', 'UNKNOWN')
    # Units for thresholds and data - offsets used to index into multiplier table
    units = ('b', 'B', 'K', 'M', 'G')

    # 5x5 tables to convert data units into threshold units (b,B,K,M,G rows and columns)
    multi_bin = ((1,8,8192,8388608,8589934592),
    multi_dec = ((1,8,8000,8000000,8000000000),

    def __init__(self):
        self.version = '0.5'
        self.output = ''        # Nagios plugin output line 1 = ''          # additional output for line 2 onwards
        self.status = CheckRRD.states.index('OK')   # default to successful return code
        self.empty = 0          # number of empty records found in dataset
        self.stale_secs = False # conditional data age check
        self.verbose = False

def parse_args():
    '''Retrieve and sanity-check script arguments'''

    parser = argparse.ArgumentParser(
            description='RRD Threshold Check Script v{0}'.format(rrdchk.version),
                    + '\n- Warning and critical thresholds are AVERAGE values.'
                    + '\n- Units for stored data and thresholds:'
                    + '\n  "b"=bps, "B"=Bps, "K"=KBps, "M"=MBps, "G"=GBps'
                    + '\n  output BW uses threshold units'
                    + '\n- Threshold behaviour:'
                    + '\n  "inout": both IN and OUT must breach'
                    + '\n  "sum": sum of IN and OUT must breach'
                    + '\n  "inonly"/"outonly": specified threshold must breach'
                    + '\n  "any": either threshold can breach')
    group = parser.add_mutually_exclusive_group(required=True)
    group.add_argument( '-f', dest='rrd_file',
                            help='rrd file-path')
    group.add_argument( '-e', dest='rrd_env',
                            help='rrd environment variable')
    parser.add_argument('-r', dest='direction',
                            help='reverse threshold direction (low check)')
    parser.add_argument('-b', dest='binary',
                            help='use binary (1024) multiples instead of decimal (1000)')
    parser.add_argument('-m', dest='threshold',
                            help='threshold behaviour: (any|inout|inonly|outonly|sumonly)')
    parser.add_argument('-l', dest='embedded_link',
                            help='http link to append to output line 1')
    parser.add_argument('-a', dest='age_check',
                            help='data age threshold in seconds')
    parser.add_argument('-w', dest='warning',
                            help='warning threshold')
    parser.add_argument('-c', dest='critical',
                            help='critical threshold')
    parser.add_argument('-p', dest='period',
                            help='time period: N{minutes|hours|days|months}, default 10minutes')
    parser.add_argument('-d', dest='rrd_units',
                            help='rrd data units (default Bytes/sec)')
    parser.add_argument('-u', dest='thresh_units',
                            help='threshold units (default Kilobytes/sec)')
    parser.add_argument('-v', dest='verbose',
                            help='verbose output')
    args = parser.parse_args()

    # Any arguments?
    if len(sys.argv) == 1:
        return False

    # How much verbosity?
    if args.verbose:
        rrdchk.verbose = True = '\n'

    # Path to RRD file?
    if args.rrd_file:
        rrdchk.rrd_path = args.rrd_file
    elif args.rrd_env:
            rrdchk.rrd_path = os.environ[args.rrd_env]
        except KeyError:
            return bail('Error reading environment variable {0}'.format(args.rrd_env))
    if rrdchk.verbose: += 'RRD file:{0}'.format(rrdchk.rrd_path)

    # Input and output units
    rrdchk.runits = args.rrd_units
    rrdchk.tunits = args.thresh_units

    # Warning and Critical supplied?
        rrdchk.warning = int(args.warning)
        rrdchk.critical = int(args.critical)
    except (TypeError,ValueError):
        return bail('Warning ({0}) and Critical ({1}) thresholds must be positive integers'.format(args.warning, args.critical))

    # Threshold higher or lower?`
    if args.direction:
        rrdchk.opt_eq = '<='
        if rrdchk.verbose:
   += ', checking for LOW threshold'
        rrdchk.opt_eq = '>='

    # Reasonable time period?
    period = re.match(r'([0-9]+)((?:minutes|hours|days|months))',args.period)
    if not period:
        return bail('Invalid time period')
    elif ((int( > 12 and == 'months') or
          (int( > 365 and == 'days') or
          (int( > 8760 and == 'hours') or
          (int( > 381600 and == 'minutes')):
        return bail('Unreasonable time period')
        rrdchk.period = args.period

    # Mandatory thresholds?
    rrdchk.behaviour = args.threshold

    # Binary vs decimal multipliers for calculations?
    if args.binary:
        rrdchk.multipliers = CheckRRD.multi_bin
        rrdchk.multipliers = CheckRRD.multi_dec

    # Record age check required?
    if args.age_check:
            rrdchk.stale_secs = int(args.age_check)
        except (TypeError,ValueError):
            return bail('Data age threshold must be a positive integer, when present')

    # Store embedded link if supplied
    if args.embedded_link:
        rrdchk.href = args.embedded_link
        rrdchk.href = ''

    return True

def fetch_data():
    '''Retrieve traffic samples for requested period'''

    # First check data age
        rrdchk.rrd_info =
    except rrdtool.error,e:
        return bail('Error from {0}'.format(e))

    if rrdchk.stale_secs:
        if int(time.time()) - rrdchk.rrd_info['last_update'] > rrdchk.stale_secs:
            return bail('Data age check failed: latest dataslot {0} more than minutes ago ({1})'.format(
                    time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(rrdchk.rrd_info['last_update']))))

    # Then pull a dataset
         (ds0, ds1),
          rrdchk.dataset) = rrdtool.fetch(rrdchk.rrd_path,
    except rrdtool.error,e:
        return bail('Error from RRDTOOL.fetch: {0}'.format(e))

    if rrdchk.verbose: += ', latest dataslot found: {0}'.format(
                time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(rrdchk.rrd_info['last_update'])))

    return True

def normalise_data():
    '''Convert data to same units as thresholds'''

    # Sum dataset, keep track of empty slots
    in_sum = out_sum = 0
    for (in_slot, out_slot) in rrdchk.dataset:
        if in_slot != None and out_slot != None:
            in_sum += in_slot
            out_sum += out_slot
            rrdchk.empty += 1
    if rrdchk.verbose: += ', found {0} empty records out of {1}'.format(rrdchk.empty, len(rrdchk.dataset))

    # Check for empty dataset
    if rrdchk.empty == len(rrdchk.dataset):
        return bail('No records in the time period contained data')

    # Calculate averages
    rrdchk.in_average = in_sum / (len(rrdchk.dataset)-rrdchk.empty)
    rrdchk.out_average = out_sum / (len(rrdchk.dataset)-rrdchk.empty)

    # Convert averages into threshold units
    rrdchk.in_norm = rrdchk.in_average * rrdchk.multipliers[CheckRRD.units.index(rrdchk.tunits)][CheckRRD.units.index(rrdchk.runits)]
    rrdchk.out_norm = rrdchk.out_average * rrdchk.multipliers[CheckRRD.units.index(rrdchk.tunits)][CheckRRD.units.index(rrdchk.runits)]

    return True

def check_threshold():
    '''Carry out threshold checking on normalised data '''

    in_status = 0
    out_status = 0
    sum_status = 0

    # Calculate all possible statuses
    if eval("rrdchk.in_norm {0} rrdchk.warning".format(rrdchk.opt_eq)):
        in_status = CheckRRD.states.index('WARNING')
    if eval("rrdchk.in_norm {0} rrdchk.critical".format(rrdchk.opt_eq)):
        in_status = CheckRRD.states.index('CRITICAL')
    if eval("rrdchk.out_norm {0} rrdchk.warning".format(rrdchk.opt_eq)):
        out_status = CheckRRD.states.index('WARNING')
    if eval("rrdchk.out_norm {0} rrdchk.critical".format(rrdchk.opt_eq)):
        out_status = CheckRRD.states.index('CRITICAL')
    if eval("(rrdchk.out_norm + rrdchk.in_norm) {0} rrdchk.warning".format(rrdchk.opt_eq)):
        sum_status = CheckRRD.states.index('WARNING')
    if eval("(rrdchk.out_norm + rrdchk.in_norm) {0} rrdchk.critical".format(rrdchk.opt_eq)):
        sum_status = CheckRRD.states.index('CRITICAL')

    # Now determine which will contribute to Nagios output

    # ANY - threshold triggered by either threshold
    # INONLY - threshold only triggered if IN thresholds, OUT not checked
    # OUTONLY - threshold only triggered if OUT thresholds, IN not checked
    # INOUT - threshold only triggered if both IN and OUT threshold
    # SUM - threshold only triggered if the sum of IN and OUT thresholds

    # Check IN, ignore OUT
    if rrdchk.behaviour == 'inonly':
        if rrdchk.verbose:
   += ', IN threshold used, OUT ignored'
        if in_status > rrdchk.status:
            rrdchk.status = in_status
    # Check OUT, ignore IN
    elif rrdchk.behaviour == 'outonly':
        if rrdchk.verbose:
   += ', OUT threshold used, IN ignored'
        if out_status > rrdchk.status:
            rrdchk.status = out_status
    # Check IN AND OUT
    elif rrdchk.behaviour == 'inout':
        if rrdchk.verbose:
   += ', Both IN and OUT thresholds used'
        if (out_status > rrdchk.status and in_status > rrdchk.status):
            if in_status >= out_status:
                rrdchk.status = out_status
                rrdchk.status = in_status
    # Check the sum of IN and OUT
    elif rrdchk.behaviour == 'sumonly':
        if rrdchk.verbose:
   += ', Sum of IN and OUT thresholds used'
        if sum_status > rrdchk.status:
            rrdchk.status = sum_status
    # default either/or case last
        if rrdchk.verbose:
   += ', Either IN or OUT thresholds used'
        if in_status > rrdchk.status:
            rrdchk.status = in_status
        if out_status > rrdchk.status:
            rrdchk.status = out_status

    return True

def bail(msg):
    '''Set status for all processing errors to UNKNOWN'''
    rrdchk.output = 'UNKNOWN - ' + msg
    rrdchk.status = CheckRRD.states.index('UNKNOWN')
    return False

def build_output():
    '''Prepare Nagios-Plugin standard output'''
    rrdchk.output = '{0} - Average BW ({1}) in: {2:.4f}{4}{5}ps, out: {3:.4f}{4}{5}ps {6}'.format(
            ('b' if rrdchk.tunits == 'b' 
                   else '' if rrdchk.tunits == 'B' 
                   else 'B'),

    return True

# MAIN starts here
rrdchk = CheckRRD()

if all(check() for check in (parse_args, fetch_data, normalise_data,check_threshold)):
    rrdchk.output +=

print rrdchk.output

Saturday, 6 April 2013

Traceroute through Cisco PIX / ASA

I recently had to clear and redeploy a PIX firewall to a new location, and realised that I had forgotten some of the subtleties involved in getting management and troubleshooting tools to work properly. So this is more of a note to self....

Windows tracert is fairly straight forward and uses pure ICMP with incrementing TTL values. Linux traceroute with the -I switch works the same way.

The firewall is required to allow the following:
- Echo Request

- Echo Reply
- Time-Exceeded (needed for TTL=0 responses)

Cisco and Linux traceroute by default uses incrementing UDP ports (from 33434) and incrementing TTL values.

The firewall is required to allow the following:
- UDP ports 33434 - 33464

- Time-Exceeded (needed for TTL=0 responses)
- Destination Unreachable (needed for the final hop port-not-found response)

Putting it all together we get a rule set that looks something like this:

object-group icmp-type ICMP-returns
 description Legit ICMP responses
 icmp-object echo-reply
 icmp-object time-exceeded
 icmp-object unreachable

object-group service Cisco_Traceroute_udp udp
 port-object range 33434 33464

access-list outside_access_in extended permit icmp any object-group External_nets object-group ICMP-returns log disable

access-list inside_access_in remark Permit outbound pings
access-list inside_access_in extended permit icmp object-group Internal_nets any echo log disable

access-list inside_access_in remark Permit traceroute from Cisco devices
access-list inside_access_in extended permit udp object-group Internal_nets any object-group Cisco_Traceroute_udp log disable

Obviously, this assumes sensible values for Internal_nets (e.g. and External_nets (i.e. public IP ranges assigned to your external interface)

As an addendum, the Firewall is not (strictly speaking) a router, and therefore in many cases will not decrement the TTL. I have found this unnecessary in most cases, but if needed, can be enabled as follows:

policy-map global_policy
 class class-default
  set connection decrement-ttl

Friday, 12 October 2012

Putty Class in VBScript

We have a fairly busy network, comprising several hundred Cisco devices across some fifty sites, and putty is one of my mainstay tools for updating configs and general troubleshooting.

So when I started looking around for something quick and easy to carry out batched updates, I looked at Putty first. Using Putty for scripted tasks wasn't as easy as I thought it would be, the main problem being access to screen feedback so that I can verify that my commands have had the expected effect.

One solution is to turn on logging and use that as a proxy screen. Here's a VBScript class which includes some basic send and "receive" functionality. Error handling is stripped to a bare minimum to keep the size of the script down here, but hopefully it gives a flavour of what is possible.

Option Explicit
'Name:    Putty class
'Author:  Philip Damian-Grint
'Version: 1.0
'Date:    12th Oct 2012
'  A starter VB class used to drive Putty sessions typically for Cisco
'  devices, allowing sending of commands, and returning screen output
'  to allow the possibility of conditional processing.
'  Putty has a number of logging options; for Cisco vty sessions, only 
'  printable output is required for line-based output processing, but 
'  full session output at least is required where escape sequences
'  need to be captured for screen positioning. (Not demonstrated here)

' Constants

Const EXELOC            = """c:\Program Files\Linux Utilities\PuTTY\putty.exe"""
Const LOG_PRINT         = "1"
Const LOG_SESSION       = "2"
Const MODE_LINE         = 0
Const MODE_CHAR         = 1
Const REGPUTTY          = "HKCU\Software\SimonTatham\PuTTY\Sessions\Default%20Settings\"
Const REGLGFILE         = "HKCU\Software\SimonTatham\PuTTY\Sessions\Default%20Settings\LogFileName"
Const REGLGTYPE         = "HKCU\Software\SimonTatham\PuTTY\Sessions\Default%20Settings\LogType"
Const STATUS_FAILURE    = -1

Class Putty

  Private p_iLastTideMark
  Private p_iMode
  Private p_iStatus
  Private p_iWait
  Private p_oFSO
  Private p_oSession
  Private p_oWShell
  Private p_sEnable
  Private p_sHost
  Private p_sLogName
  Private p_sLogType
  Private p_sPasswd
  Private p_sTempDir
  Private p_sUser


  Private Sub Class_Initialize()
    Set p_oWShell = WScript.CreateObject( "WScript.Shell" )
    Set p_oFSO = WScript.CreateObject( "Scripting.FileSystemObject" ) 
    p_sLogType = LOG_PRINT ' default to printable output
    p_iLastTideMark = 0 ' initial tide mark
    p_iWait = 5         ' default to 5 seconds wait after each command
    p_iMode = MODE_LINE ' default to reading lines
  End Sub

  Private Sub Class_Terminate()
    ResetLog()                       ' Clear our registry settings
    p_oFSO.DeleteFile( p_sLogName )  ' Get rid of the temporary file
    Set p_oWShell = Nothing
    Set p_oFSO = Nothing
    Set p_oSession = Nothing
  End Sub

 'enable() is WO
  Public Property Let enable( sEnable ) : p_sEnable = sEnable : End Property

 'host() is RW
  Public Property Let host( sHost ) : p_sHost = sHost : End Property
  Public Property Get host() : host = p_sHost : End Property

 'logtype() is RW
  Public Property Let logtype( sLogType ) : p_sLogType = sLogType : End Property
  Public Property Get logtype() : logtype = p_sLogType : End Property

 'mode() is RW
  Public Property Let mode( iMode ) : p_iMode = iMode : End Property
  Public Property Get mode() : mode = p_iMode : End Property

 'passwd() is WO
  Public Property Let passwd( sPasswd ) : p_sPasswd = sPasswd : End Property

 'status() is RO
  Public Property Get status() : status = p_iStatus : End Property

 'user() is RW
  Public Property Let user( sUser ) : p_sUser = sUser : End Property
  Public Property Get user() : user = p_sUser : End Property

 'wait() is RW
  Public Property Let wait( iWait ) : p_iWait = iWait : End Property
  Public Property Get wait() : user = p_iWait : End Property

  Private Function EnableLog ' Switch on Putty logging
    EnableLog = -1
    p_sLogName = p_oWShell.ExpandEnvironmentStrings( "%Temp%" ) & _
                "\" & p_oFSO.GetTempName()        
    If IsEmpty( p_oWShell.RegWrite( REGLGFILE, p_sLogName,"REG_SZ" ) ) AND _
       IsEmpty( p_oWShell.RegWrite( REGLGTYPE, p_sLogType, "REG_DWORD" ) ) Then
            EnableLog = 0
    End If
  End Function

  Private Function Quit( sReason ) ' Display message and Exit
    WScript.Echo sReason : WScript.Quit
  End Function

  Private Function ResetLog ' Switch off Putty logging
    p_oWShell.RegDelete( REGPUTTY )
  End Function

  Private Function ReadLog ' Read latest output from Putty log
    Dim oFile : Set oFile = p_oFSO.OpenTextFile( p_sLogName )
    Dim iCount : iCount = 0
    Dim aLogLines(), sLogChars

    Do Until oFile.AtEndOfStream    ' Find our old tide mark
        If iCount < p_iLastTideMark Then
            Redim Preserve aLogLines( iCount - p_iLastTideMark ) 
            aLogLines( iCount - p_iLastTideMark ) = oFile.ReadLine
        End If
       iCount = iCount + 1
    p_iLastTideMark = iCount        ' New tidemark
    ReadLog = aLogLines             ' Return everything since the last tidemark
    Set oFile = Nothing
  End Function

  Private Function SendInput( sInput )  ' find Putty's active window and send keystrokes to it
    WScript.Sleep 3000               ' Or greater if debugging to give time for window switching
        WScript.Sleep 100
    Loop until p_oWShell.AppActivate( p_oSession.ProcessID )    ' Find our session window
    p_oWShell.SendKeys( sInput & "{ENTER}" )        ' Do the deed
  End Function


  Public Function Connect  ' Launch Putty 
    p_iStatus = STATUS_FAILURE          ' assume failure
    If (NOT IsEmpty( p_sUser ) AND _
            NOT IsEmpty( p_sPasswd ) AND _
            NOT IsEmpty( p_sUser ) AND _
            NOT IsEmpty( p_sHost ) ) Then
        If EnableLog <> 0 Then Quit( "Aborting - Can't update registry" )
        On Error Resume Next            ' graceful error handling
        Set p_oSession = p_oWShell.exec( EXELOC & " " & p_sHost & " -l " & _
                        p_sUser & " -pw " & p_sPasswd )
        WScript.Sleep 2000              ' Allow some time to settle down
        If ( ( p_oSession Is Nothing ) OR ( p_oSession.Status <> 0 ) ) Then Exit Function
        On Error Goto 0
        p_iStatus = STATUS_SUCCESS
        Connect = ReadLog()             ' Pass the initial screen back
    End If
  End Function

  Public Function Send( sChars ) ' Send a command and read the output after waiting iWait seconds
    SendInput( sChars )
    WScript.Sleep p_iWait * 1000
    Send = ReadLog()
  End Function

End Class

And to demonstrate the class in use, we take the code above and store it in a file called "classes.vbi", and then pull that file in using an "Include" function to our puttytest.vbs below. 

All this demo does is log onto a cisco device, send a command and logout, relaying any putty screen output to our screen:

Tested with Putty version 0.60 under WIndows XP SP3:

Option Explicit
'Name:    puttytest.vbs
'   Wrapper to test our putty class
'   Run from command line:
'        cscript  puttytest.vbs
'Utility Functions

Function Include ( sFileVBI ) ' include an external vbs/vbi file
    Dim oFSO : Set oFSO = WScript.CreateObject( "Scripting.FileSystemObject" )
    Dim oFile : Set oFile = oFSO.OpenTextFile( sFileVBI )
    ExecuteGlobal oFile.ReadAll()
    oFile.Close : Set oFile = Nothing
    Set oFSO = Nothing
End Function

Function GetUserInfo( sPrompt ) ' prompt for input
    WScript.StdOut.Write( sPrompt )
    GetUserInfo = WScript.StdIn.ReadLine
End Function

Function GetPassword( sPrompt ) ' prompt for hidden input
    Dim oPasswd : Set oPasswd = WScript.CreateObject( "ScriptPW.Password" )
    WScript.StdOut.Write( sPrompt )
    GetPassword = oPasswd.GetPassword()
    Set oPasswd = Nothing
End Function

Function WriteLines( aOut ) ' print array of strings
    Dim sLine : For Each sLine in aOut
        WScript.StdOut.Write( sLine & VbCrLf )
End Function

' Test our Putty Class

Include "classes.vbi"

Dim aOutPut
Dim sLineOut
Dim sTextToSend

Dim oSession : Set oSession = New Putty     ' Create a new instance of our class = GetUserInfo( "Please type hostname: " )  ' Get some basic info
oSession.user = GetUserInfo( "Please type username: " )
oSession.passwd = GetPassword( "Please type password: " )

aOutPut = oSession.Connect                  ' and launch our putty session

If oSession.Status = STATUS_SUCCESS Then

    WriteLines( aOutPut )
    oSession.wait = 3                       ' we can set a timer for each command
    aOutPut = oSession.Send( "show ver" )   ' show version IOS command
    WriteLines( aOutPut )
    aOutPut = oSession.Send( " " )          ' usually runs to 2 screens
    WriteLines( aOutPut )
    aOutPut = oSession.Send( "logout" )     ' close session
    WriteLines( aOutPut )


    WScript.Echo "Failed to launch Putty"
End If

Saturday, 4 August 2012

Two-way NAT / PAT on a VPN (Cisco) Stick

Some time ago I was tasked with interfacing to a couple of other multi-site organisations across a large governmental network similar in operation to the Internet. This was an interim measure prior to integrating aspects of the three networks into a single entity, and prior to having any dedicated WAN links in place.

I had to provide connectivity between a variable number of users and servers across all three networks, and with many overlapping IP ranges in place. The idea was to have a flexible enough configuration that I could easily add and change routes at the far end to keep pace with any integration work.

The last requirement was the support of one or more AD trusts between the organisations, with DNS forwarding.

To make it as light a touch a possible for the far-end IT departments, I went for a single interface Cisco router that could be connected directly to a Firewall DMZ on the far end firewall.

The topology essentially looked like this (with extraneous devices stripped away) and Internet substituted for the (private) government network:

At a more useful level, showing physical interfaces and IP addresses:


Ideally, in order to allow variable numbers of users to cross the NAT boundary in either direction, one would be able to use PAT in both directions. However, this is only available to the “inside” interface.
As a large number of users were likely to be crossing from the remote site, and only a few from the hub site, I had to make the remote physical interface act as “inside” and the tunnel act as “outside”.
This allowed me to use PAT for remote users and dynamic NAT for hub users.
Servers were easily handled by static NAT in both directions

Non-NAT-Compliant Applications

Nowadays, most applications, including to my surprise, Microsoft domain trusts, work quite well across (Cisco IOS) NAT boundaries. I found only one application which didn’t: an old version of HP Openview ServiceDesk, which embeds the source IP of the HPOV server inside the java client for use in a subsequent return connection.
In this particular instance, the server was based at the hub site, and no IP conflict existed at the remote site. I was able to create an identity NAT for the server in the direction of the Hub which worked fine once supporting routes were in place.

MTU issues

Because the remote firewall has not participated in creating the tunnel endpoint, it can’t respond correctly to hub-destined traffic with DF flag set, so we have to ensure that the remote firewall allows ICMP unreachables to be sent from our router to devices on its internal network.

Design Notes

Some basic notes might be required to clarify where all the addresses are coming from:

IPSEC and GRE tunnel end-points

The physical interfaces representing the Hub and Remote endpoints have internal 192.168 addresses and are mapped to NAT addresses on their upstream firewalls. I have used and respectively.

Inter-Org NAT Allocation

Subnet is used to Dynamically NAT all Hub users accessing Remote servers
Subnet is used to present Hub servers to Remote users.
IP Address is used to PAT all Remote users accessing Hub servers
Subnet is used to present Remote servers to Hub users.

Configuration Fragments

The configurations below have been taken from working devices, with some minimal IP address obfuscation:

Hub Distribution Router:
! IKE Phase 1
crypto isakmp policy 1
 encr aes
 authentication pre-share
 group 5

! Pre-shared key for remote site
crypto isakmp key RemoteSiteKey address

! IKE Phase 2
crypto ipsec transform-set AES256_SHA_tra esp-aes 256 esp-sha-hmac 
 mode transport

! Crypto ACL for GRE to remote site
ip access-list extended HUB-INTERNET-REMOTE-CRYACL
 remark Tunneled traffic over the Internet to Remote site
 permit gre host host

! Crypto MAP entry for remote site
crypto map INTERNET-CM 15 ipsec-isakmp 
 set peer
 set transform-set AES256_SHA_tra 

! Physical interface for termination of all WAN and Internet tunnels
interface GigabitEthernet0/1
 description Connects to Local Firewall inside
 ip address
 crypto map INTERNET-CM

! Tunnel to remote site
interface Tunnel14200
 description Tunnel over Internet to Remote Site
 ! low bandwidth used (EIGRP) for backup tunnels over Internet
 bandwidth 1000
 ip address
 ! Maximum starting MTU (1500 - 8(NAT-T) - 53(AES256) - 24(GRE))
 ip mtu 1415
 ! high delay used (EIGRP) for backup tunnels over Internet
 delay 2000
 tunnel source GigabitEthernet0/1
 tunnel destination
 ! Tell GRE to copy DF from inner to outer IP header
 tunnel path-mtu-discovery

ip prefix-list EIGRP-SITETUNNELS-OUT-PL description Route adverts to remote sites
ip prefix-list EIGRP-SITETUNNELS-OUT-PL seq 5 permit
ip prefix-list EIGRP-SITETUNNELS-OUT-PL seq 10 permit le 32

router eigrp 192
 passive-interface GigabitEthernet0/1
 distribute-list prefix EIGRP-SITETUNNELS-OUT-PL out Tunnel14200
 no auto-summary
 no eigrp log-neighbor-changes

Hub Firewall:
PIX Version 7.2(2)

name dist-rt02_INTERNET
name dist-rt02_G01

object-group service NAT-T udp
 description NAT Traversal
 port-object eq 4500

object-group service IPsec_udp udp
 description UDP protocols used by IPsec
 group-object NAT-T
 port-object eq isakmp

object-group network Cisco_Devices
 description Cisco devices' Internet interfaces
 network-object host remote-rt01_INTERNET

interface Ethernet0
 speed 100
 duplex full
 nameif outside
 security-level 0
 ip address standby 

interface Ethernet1
 speed 100
 duplex full
 nameif inside
 security-level 100
 ip address standby

route outside 1
route inside 1

! Mapping the routable Tunnel endpoint
static (inside,outside) dist-rt02_INTERNET dist-rt02_G01 netmask 

access-list inside-access-in remark Allow ISAKMP & NAT-T to sites using VPN-over-Internet
access-list inside-access-in extended permit udp host dist-rt02_G01 object-group Cisco_Devices object-group IPsec_udp log disable 

access-group inside-access-in in interface inside

Remote Firewall:
PIX Version 6.3(4)

interface ethernet0 100full
interface ethernet1 100full
interface ethernet4 100full

nameif ethernet0 outside security0
nameif ethernet1 inside security100
nameif ethernet4 HUBDMZ security49

ip address outside
ip address inside
ip address HUBDMZ

failover ip address outside
failover ip address inside
failover ip address HUBDMZ

object-group network HUB
  description HUBDMZ network
  description HUB users on this subnet
  description HUB servers on this subnet

! Minimal ACLs to permit traffic flow – not representative!
access-list inside_access_in permit ip any object-group HUB 
access-group inside_access_in in interface inside

access-list hubdmz_access_in permit icmp any any
access-list hubdmz_access_in permit ip host host 
access-list hubdmz_access_in permit ip object-group HUB 
access-group hubdmz_access_in in interface HUBDMZ

access-list outside_access_in permit udp host host eq 4500 
access-list outside_access_in permit udp host host eq isakmp 
access-group outside_access_in in interface outside

! Bypass NAT for incoming HUB traffic (low security to high security)
access-list NO_NAT_HUBDMZ permit ip object-group HUB 
nat (HUBDMZ) 0 access-list NO_NAT_HUBDMZ

! Mapping the routable Tunnel endpoint
static (HUBDMZ,outside) netmask 0 0 

! Hub users and servers respectively
route HUBDMZ 1
route HUBDMZ 1

Remote VPN Router:

! example hub hosts with pre(real) and post nat addresses (hub perspective)
ip host hubhost01
ip host hubhost02
! example remote hosts with "pre" and "post"(real) nat (hub perspective)
ip host remotehost01
ip host remotehost02

! need inspection to activate ALGs
ip inspect name INSPECT_LIST dns
ip inspect name INSPECT_LIST ftp
ip inspect name INSPECT_LIST https
ip inspect name INSPECT_LIST icmp
ip inspect name INSPECT_LIST imap
ip inspect name INSPECT_LIST pop3
ip inspect name INSPECT_LIST esmtp
ip inspect name INSPECT_LIST sqlnet
ip inspect name INSPECT_LIST streamworks
ip inspect name INSPECT_LIST tftp
ip inspect name INSPECT_LIST tcp
ip inspect name INSPECT_LIST udp
ip inspect name INSPECT_LIST vdolive
ip inspect name INSPECT_LIST kerberos
ip inspect name INSPECT_LIST ldap
ip inspect name INSPECT_LIST microsoft-ds

! IKE Phase 1
crypto isakmp policy 1
 encr aes
 authentication pre-share
 group 5

! Pre-shared key for this site
crypto isakmp key RemoteSiteKey address

! IKE Phase 2
crypto ipsec transform-set AES256_SHA_tra esp-aes 256 esp-sha-hmac 
 mode transport

! Crypto ACL for GRE to hub site
ip access-list extended REMOTE-INTERNET-HUB-CRYACL
 remark Traffic tunnelled over Internet to HUB
 permit gre host host

! Crypto MAP entry for hub site
crypto map INTERNET-CM 2 ipsec-isakmp 
 set peer
 set transform-set AES256_SHA_tra 

! Single physical interface for LAN and VPN traffic
! in/out ACL not included in config
interface FastEthernet0/0
 description Exit to Internet and Remote LAN via Remote DMZ
 ip address
 no ip redirects
 ip inspect INSPECT_LIST in
 ! Treat the remote network as inside so we can use PAT
 ip nat inside
 ! enabled automatically with NAT config
 ip virtual-reassembly
 duplex full
 speed 100
 no cdp enable
 crypto map INTERNET-CM

interface Loopback0
 description Remote PAT address for overlapping client subnets
 ip address

interface Tunnel14200
 description Tunnel over Internet to Hub network
 ! low bandwidth used (EIGRP) for backup tunnels over Internet
 bandwidth 1000
 ip address
 ! Maximum starting MTU (1500-8(NAT-T)-53(AES256)-24(GRE))
 ip mtu 1415
 ! Required to allow PAT in the opposite direction
 ip nat outside
 ! enabled automatically with NAT config
 ip virtual-reassembly
 ! high delay used (EIGRP) for backup tunnels over Internet
 delay 2000
 tunnel source FastEthernet0/0
 tunnel destination
 tunnel path-mtu-discovery

router eigrp 192
 passive-interface Loopback0
 distribute-list prefix EIGRP-TUNNEL-OUT-PL out Tunnel14200
 no auto-summary
 eigrp stub connected

! Floating default route back to the hub over the tunnel
ip route 200

! Example remote site networks - chosen to demonstrate overlaps
ip route
ip route
ip route

! Explicit route for our tunnel destination to avoid recursion
ip route

! We need the flexibility of PAT to be applied to the remote network
ip nat inside source list REMOTE-USERS interface Loopback0 overload

! Which leaves us on the "outside" using dynamic NAT
ip nat pool HUB-POOL prefix-length 24
ip nat outside source list HUB-USERS pool HUB-POOL

! Example remote servers - DNS ALG will use these to translate our queries
ip nat inside source static
ip nat inside source static

! Example hub servers - DNS ALG will use these to translate their queries
ip nat outside source static
ip nat outside source static

! Define which remote subnets hide behind PAT
ip access-list standard REMOTE-USERS
 remark Remote main site
 remark Remote secondary site example

! Define which hub subnets hide behind Dynamic NAT
ip access-list standard HUB-USERS
 remark Hub IT department
 remark Hub main site
 remark Hub secondary site example

ip prefix-list EIGRP-TUNNEL-OUT-PL description Routes to be advertised from site
ip prefix-list EIGRP-TUNNEL-OUT-PL seq 5 permit


The creation and ongoing support of Microsoft domain trusts across this two-way NAT boundary was reasonably straight forward. There were a couple of issues, neither of which were caused by or really impinged upon the configuration itself, but might be worth mentioning:

1. Problems creating a domain Trust across two-way NAT
I found it useful to ensure that all DNS servers in both domains could see and forward to each other. In one of the organisations I needed to connect to, this was tiresome because they had at least 6 DCs of which 4 were DNS servers. This requires static NAT entries to be configured for each server.
I also found that physical DCs were more reliable  than VMs, in part due to VMware tools not being installed thoughtfully - the Shared Folders option should not be installed as it causes network (RPC) problems. However, you can't chose in advance which DCs will participate on each side, so it becomes useful to be able to mask off the suspect ones by removing their NAT entries.

2. Kerberos-related fragmentation
Depending upon the server and workstation versions, Kerberos may still default to UDP, which may cause performance problems due to fragmentation. This is particularly noticable where W2K3 and XP are in use, and where there are many nested groups and SID histories to bloat the packets. This manifests itself as a delay in accessing resources across the trust. Debugging ip virtual-reassembly may show maximum fragments or fragmentation buffer being exceeded and some additional tweaking may be required to prevent timeouts and retransmissions within Kerberos.


I found the following document very useful in getting this to production:
NAT Order of Operation

Saturday, 4 February 2012

MRTG Log Aggregator

Occasionally, I have needed to provide percentiles on a combined set of interfaces.
This requires a way of adding together samples from a number of log files, even though the sample timestamps might differ from file to file by a few minutes.

Here then is my current hack for doing this. The merged data set is implemented here as a doubly-linked list using nested hashes, not because I make use of these here, but because I lifted it from one of my other log manipulation tools. I will probably return to clean it up as time goes on.

#!/usr/bin/env perl
# AUTHOR:       Philip Damian-Grint
# DESCRIPTION:  Synthesize a new MRTG log file from 2 or more other log files.
#               This utility expects and generates version 2 MRTG log files,
#               (See, based on a 
#               default sampling time of 5 minutes
#               In general there are 600 samples each of 5mins, 30mins, 120mins 
#               and 86400mins. Each dataset is a quintuple:
#               {epoch, in_average, out_average, in_maximum, out_maximum}
#               The file with the newest timestamp is used as a template for generating
#               the output file, processed backwards in time.
#               Samples from the second and further logfiles are combined with the template
#               according to the following rules:
#               1.  Samples from the input logfile which fall between two samples in the
#                   template, are combined into the sample with the higher timestamp
#               2.  Samples are combined using basic addition only
#               Each of the input files are checked for time synchronisation. If the
#               starting times of any of the second and subsequent input files are more 
#               than 5 minutes adrift from the first input file, the utility aborts.
# INPUTS:       Options, Logfile1, Logfile2, ...
#      [--verbose] Logfile1 [, Logfile2, ...]
# OUTPUTS:      Logfile in MRTG format version 2
#               This is written to STDOUT
# NOTES:        1.   It should go without saying that running this against live log files while
#                    MRTG is running will have unpredictable results - copy the logfiles to
#                    a location where they will not be disturbed while being processed.
#               2.  It is possible that due to occasional variations at sample period
#                   boundaries (e.g. 5mins / 30 mins) and between files, some "samples" in the
#                   merged file might combine one or two samples more than expected.
#                   It would be possible to avoid this by say, adding a further field to each hash
#                   record to count and possibly restrict the samples combined from subsequent files.
# HISTORY:      3/2/2012: v1.0 created
#               8/2/2012: v1.1 header detection corrected

use strict;

local $| = 1;                               # Autoflush STDOUT

use Getopt::Long;


# Parameters
my $verbose;

# Working Storage
my @fields;                                 # Holds fields from last record read
my $file_no;                                # Tracks current file being processed
my $inbytes_master;                         # Inbytes counter from the first file
my @keys;                                   # Holds sorted keys for merged dataset
my $outbytes_master;                        # Outbytes counter from the first file
my $prev_time;                              # Remember our previous timestamp
my $record_no;                              # Tracks last record read from current file
my $time_master;                            # First timestamp from first file
my $run_state;                              # Tracks processing phase (first file, subsequent file...)
my %samples;                                # Doubly-linked list representing merged file

# Subroutines
sub record_count {
    print "\r".++$record_no." of ".$file_no;


GetOptions ("verbose" => \$verbose );       # Check for verbosity
$prev_time = 0;                             # Reset previous timestamp copy
$run_state = 'INIT';                        # Reset state
$time_master = 0;                           # Reset starting epoch


# Process All Logfiles
while (<>) {
    chomp();                                # Remove carriage return etc
    @fields = ();                           # Clear our temporary holding area
    @fields = (split);                      # Split up our tuple

    # Start of File Processing    
    if (scalar(@fields) == 3) {             # Check for start of file
        print "\nStart of input file, datestamp: ".(scalar localtime(@fields[0]))."\n" if ($verbose);
        $record_no = 0;                     # Reset record counter

        # First file
        if ($run_state eq 'INIT') {         # If this is our first file
            $time_master = @fields[0];      # Capture the header timestamp
            $inbytes_master = @fields[1];   # Capture the header inbytes
            $outbytes_master = @fields[2];  # Capture the header outbytes
            $run_state = 'FIRST';           # And update our state
            $file_no = 1;                   # Start counting input files

        # Subsequent files
        } else {
            # At the end of the first file (only)
            if ($run_state eq 'FIRST') {
                @keys = reverse sort { $a <=> $b } (keys %samples); # Sort our keys
                $run_state = 'SUBSQ';                               # Note that first file has ended
            # And in all cases
            $file_no++;                     # Count input files
            $inbytes_master += @fields[1];  # Add header inbytes to master
            $outbytes_master += @fields[2]; # Add header outbytes to master
            # Other files must be within 5 minutes of the first
            die("Header timestamp difference > 5 minutes found in file ".$file_no."\n") if (abs($time_master - @fields[0]) > 300);
        &record_count if ($verbose);        # Update our on-screen counter
        $prev_time = @fields[0];            # Take a copy of this timestamp
        next;                               # Now start processing non-header records

    # Check for "all-files" data mangling
    die("\nIncreasing timestamp found in record ".$record_no." of file ".$file_no."\n") if (@fields[0] > $prev_time);
    # First file just populates our template
    if ($run_state eq 'FIRST') {

        # Check for "first-file" data mangling
        die("\nDuplicate timestamp found in record ".$record_no." of file ".$file_no."\n") if (exists ($samples{@fields[0]}));

        # Create a hash entry indexed by datestamp
        $samples{@fields[0]}= {PREV => ($prev_time == @fields[0]) ? undef : $prev_time, NEXT => undef, TUPLE => [@fields[1], @fields[2], @fields[3], @fields[4]]};

        # If not the first item in the list, update the last item's NEXT pointer
        $samples{$prev_time}{NEXT} = @fields[0] if ($record_no > 1);

    # Subsequent files must be merged
    } else {
        foreach (@keys) {
            if ($_ <= @fields[0]) {
                $samples{$_}{TUPLE}[0] += @fields[1];
                $samples{$_}{TUPLE}[1] += @fields[2];
                $samples{$_}{TUPLE}[2] += @fields[3];
                $samples{$_}{TUPLE}[3] += @fields[4];
    $prev_time = @fields[0];                # Take a copy of this timestamp
    &record_count if ($verbose);

# Were we only given one file? @keys only populated on detection of a second file
die("\nError - only one input file supplied\n") unless (@keys);

# Output Merged File

# First our updated header record
print "$time_master $inbytes_master $outbytes_master\n";

# And then our records in reverse order
foreach (@keys) {
    print "$_ $samples{$_}{TUPLE}[0] $samples{$_}{TUPLE}[1] $samples{$_}{TUPLE}[2] $samples{$_}{TUPLE}[3]\n";

Thursday, 19 January 2012

IPSEC: Tunnel vs Transport Mode

If you go looking for it, there is whole stack of IPSEC documentation out there. It's mostly fairly dense, and tends to concentrate on explaining the somewhat complex operation and configuration details rather than exploring design choices.

One typical scenario is that you find yourself tasked with managing a multisite topology with redundant paths, and over third-party provider networks (private and public). The result is a requirement to implement encryption for all intersite traffic, for which the usual, and often only contender is IPSEC.

Those implementing IPSEC for the first time find that there are a large number of choices to be made, and all of them may seem to be equally important. As a result, the final implementation often bears a strong resemblance to one of the examples which can be found on the Cisco site (with all the subtleties hidden and important decisions pre-made).

One key decision involves the choice of operating mode: Tunnel or Transport.

Typically you find the differences between the two described in a number of ways such as:
  • Tunnel mode is used between gateways while Transport mode is used between endstations.
  • Tunnel mode is used for pass-through traffic, while Transport mode is used for end-to-end traffic
  • Tunnel mode encrypts the whole packet and provides a new header, while Transport mode only encrypts the data (payload).

These descriptions have hints and clues inside them but they don't really tell you why and when you should use them. But once you understand what the basic choice means, IPSEC suddenly becomes a lot more friendly.

Here's the question I think you should be asking:
"Which mode will best support my routing model?"

So, do you use dynamic routing or static routing? This is important because some of the same reasoning you use to justify your routing choice will be the same reasoning you use in making the Tunnel vs Transport choice.

Let's look at the static routing approach:
"I have a simple network and by using static routes I have complete control over what traffic is sent across my links."

So each static route is created at the 'source' of a link, directing traffic to the other end. This is much the same as a typical IPSEC Tunnel-mode link which uses ACLs to define "interesting traffic" at the 'source', to be sent to the other end.

But in order to get your traffic to traverse that IPSEC link, you must have a static route, and you must have a corresponding (crypto) ACL present. Without both in place, the forwarding won't happen. They must be matched by a mirror route/ACL at the other end. So that's four manual entries for each definable flow which must be updated if subnets or paths change.

What about the dynamic routing approach?:
"I have a complex network with multiple paths which I want to be discovered and utilised as needed by the network"

So someone taking this approach doesn't really want to be clumsily routing by ACL, but dynamically with a routing protocol. Trouble is, IPSEC Tunnel mode only handles unicast traffic, which would leave you with BGP as the only usable routing protocol.

You don't really want your routing configuration to have any dependencies on your IPSEC configuration at all. This is where Transport mode comes in:

Create an IPSEC Transport mode link between your pair of site routers and use it to carry only GRE traffic to create a GRE tunnel with its own /30 subnet and addressing, independent of the IPSEC link addressing.

Now, multicast routing protocols such as OSPF and EIGRP will run over the link and take care of all the other traffic.

The only ACL you need to define for IPSEC is one that identifies the peer router for GRE traffic, which won't change even if routing paths and subnet locations do. So that's four manual entries to take care of any number of definable flows, and which don't need to change unless one of the two site routers actually changes its address.

And that's all there is to it. (at a high level).

So in summary:
  • Tunnel mode IPSEC forces you to implement "Routing by Crypto-Map", which is ugly and unscalable, but appropriate for links between your external firewall and some other organisation, for instance.
  • Transport mode IPSEC (+GRE) frees up the routing design and makes it independent of encryption implementation; it is therefore ideal for any internal links, WAN or LAN.
This is in some ways, counter intuitive: Use Transport mode to carry tunnels and use Tunnel mode to transport raw packets.

PS: Don't forget if you use Transport mode IPSEC with GRE, that there are now two layers of encapsulation and you will need to take extra care with fragmentation and MTU issues. At a minimum you should have path MTU discovery enabled, ICMP unreachables NOT blocked and DF bits copied from the original IP header to the GRE header.