When RPF Breaks Traceroute

I came across an interesting little problem recently which was quite fun to unravel....

I work on a hub and spoke network, where most traffic from the spokes follows the default route back to the hubs, except for a few specific destinations which must be reached through a public-network-facing interface.

One day I tried to run a traceroute from one of our site routers, towards one of these specific destinations and found that I couldn't get past the first hop (provider router).

fr-rt01#traceroute 155.231.48.203

Type escape sequence to abort.
Tracing the route to 155.231.48.203

  1 10.197.251.1 4 msec 0 msec 0 msec
  2  *  *  *
  3  *  *  *

Here's the interface config:

interface GigabitEthernet0/0.801
 description Exit to N3 gateway
 encapsulation dot1Q 801
 ip address 10.197.251.4 255.255.255.0
 ip access-group N3-ACCESS-IN in
 ip verify unicast reverse-path
 no ip unreachables
 ip inspect SDM_LOW in
 ip inspect SDM_LOW out
 ip nat outside
 ip virtual-reassembly
 no cdp enable
 crypto map N3-CM
end

So I removed the ACL, then the inspect statements and then the "no ip unreachables", but I still got the same result.

The next hop looked good but I checked my route just to make sure:

fr-rt01#sh ip ro 155.231.48.203
Routing entry for 155.231.48.0/24
  Known via "bgp 65139", distance 200, metric 1, type internal
  Last update from 10.139.202.11 18:44:40 ago
  Routing Descriptor Blocks:
  * 10.139.202.11, from 10.139.202.11, 18:44:40 ago
      Route metric is 1, traffic share count is 1
      AS Hops 0

fr-rt01#sh ip ro 10.139.202.11
Routing entry for 10.139.200.0/22
  Known via "static", distance 1, metric 0
  Routing Descriptor Blocks:
  * 10.197.251.1
      Route metric is 0, traffic share count is 1

Then clutching at straws, I removed my "ip verify unicast reverse-path", and traceroute worked just like it should...

fr-rt01#traceroute 155.231.48.203

Type escape sequence to abort.
Tracing the route to 155.231.48.203

  1 10.197.251.1 4 msec 0 msec 0 msec
  2 81.147.220.170 8 msec 8 msec 8 msec
  3 172.16.216.193 8 msec 8 msec 8 msec
  4 217.36.152.160 8 msec 8 msec 8 msec
  5 217.36.152.64 8 msec 8 msec 8 msec
  6 10.100.100.67 12 msec 8 msec 8 msec

And then I went and read up on traceroute and RPF to try to understand what was going on

When traceroute sends each UDP packet out, it expects to get an ICMP type 11 code 0 (time exceeded) back from each intermediate hop/router, with a source IP address of its exit interface (towards me).

RPF does a reverse lookup of the source address against CEF to check that the receiving interface is one of the best return paths to that address.

Taking the first few addresses in the working traceroute:

fr-rt01#sh ip cef 81.147.220.170 
0.0.0.0/0, version 191479, epoch 0
0 packets, 0 bytes
  via 192.168.13.196, Tunnel13196, 0 dependencies
    next hop 192.168.13.196, Tunnel13196
    valid adjacency

fr-rt01#sh ip cef 172.16.216.193
0.0.0.0/0, version 191479, epoch 0
0 packets, 0 bytes
  via 192.168.13.196, Tunnel13196, 0 dependencies
    next hop 192.168.13.196, Tunnel13196

And there's the problem: without a route to the intermediate addresses, I only have a default route, which leaves by a different interface. So RPF is doing its job properly and breaking my traceroute.

In this case, it seems that the best solution is to tell RPF to ignore my traceroute replies by adding an ACL defining them as exceptions:

fr-rt01#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
fr-rt01(config)#access-list 101 permit icmp any host 10.197.251.4 11 0
fr-rt01(config)#int gi0/0.801
fr-rt01(config-subif)#ip verify unicast reverse-path 101
fr-rt01(config-subif)#^Z
fr-rt01#

Which allows both features to work without side effects:

fr-rt01#traceroute 155.231.48.203

Type escape sequence to abort.
Tracing the route to 155.231.48.203

  1 10.197.251.1 4 msec 0 msec 0 msec
  2 81.147.220.170 8 msec 8 msec 8 msec
  3 172.16.216.193 8 msec 8 msec 8 msec
  4 217.36.152.160 8 msec 8 msec 8 msec
  5 217.36.152.64 8 msec 8 msec 8 msec
  6 10.100.100.67 12 msec 8 msec 8 msec
  7  *  *  *

fr-rt01#sh access-list 101
Extended IP access list 101
    10 permit icmp any host 10.197.251.4 ttl-exceeded (15 matches)

Traceroute through Cisco PIX / ASA

I recently had to clear and redeploy a PIX firewall to a new location, and realised that I had forgotten some of the subtleties involved in getting management and troubleshooting tools to work properly. So this is more of a note to self....

Windows tracert is fairly straight forward and uses pure ICMP with incrementing TTL values. Linux traceroute with the -I switch works the same way.

The firewall is required to allow the following:
Outbound
- Echo Request

Inbound
- Echo Reply
- Time-Exceeded (needed for TTL=0 responses)

Cisco and Linux traceroute by default uses incrementing UDP ports (from 33434) and incrementing TTL values.

The firewall is required to allow the following:
Outbound
- UDP ports 33434 - 33464

Inbound
- Time-Exceeded (needed for TTL=0 responses)
- Destination Unreachable (needed for the final hop port-not-found response)

Putting it all together we get a rule set that looks something like this:
 

object-group icmp-type ICMP-returns
 description Legit ICMP responses
 icmp-object echo-reply
 icmp-object time-exceeded
 icmp-object unreachable

object-group service Cisco_Traceroute_udp udp
 port-object range 33434 33464

access-list outside_access_in extended permit icmp any object-group External_nets object-group ICMP-returns log disable

access-list inside_access_in remark Permit outbound pings
access-list inside_access_in extended permit icmp object-group Internal_nets any echo log disable

access-list inside_access_in remark Permit traceroute from Cisco devices
access-list inside_access_in extended permit udp object-group Internal_nets any object-group Cisco_Traceroute_udp log disable

Obviously, this assumes sensible values for Internal_nets (e.g. 192.168.0.0/16) and External_nets (i.e. public IP ranges assigned to your external interface)

As an addendum, the Firewall is not (strictly speaking) a router, and therefore in many cases will not decrement the TTL. I have found this unnecessary in most cases, but if needed, can be enabled as follows:

policy-map global_policy
 class class-default
  set connection decrement-ttl