When RPF Breaks Traceroute

I came across an interesting little problem recently which was quite fun to unravel....

I work on a hub and spoke network, where most traffic from the spokes follows the default route back to the hubs, except for a few specific destinations which must be reached through a public-network-facing interface.

One day I tried to run a traceroute from one of our site routers, towards one of these specific destinations and found that I couldn't get past the first hop (provider router).

fr-rt01#traceroute 155.231.48.203

Type escape sequence to abort.
Tracing the route to 155.231.48.203

  1 10.197.251.1 4 msec 0 msec 0 msec
  2  *  *  *
  3  *  *  *

Here's the interface config:

interface GigabitEthernet0/0.801
 description Exit to N3 gateway
 encapsulation dot1Q 801
 ip address 10.197.251.4 255.255.255.0
 ip access-group N3-ACCESS-IN in
 ip verify unicast reverse-path
 no ip unreachables
 ip inspect SDM_LOW in
 ip inspect SDM_LOW out
 ip nat outside
 ip virtual-reassembly
 no cdp enable
 crypto map N3-CM
end

So I removed the ACL, then the inspect statements and then the "no ip unreachables", but I still got the same result.

The next hop looked good but I checked my route just to make sure:

fr-rt01#sh ip ro 155.231.48.203
Routing entry for 155.231.48.0/24
  Known via "bgp 65139", distance 200, metric 1, type internal
  Last update from 10.139.202.11 18:44:40 ago
  Routing Descriptor Blocks:
  * 10.139.202.11, from 10.139.202.11, 18:44:40 ago
      Route metric is 1, traffic share count is 1
      AS Hops 0

fr-rt01#sh ip ro 10.139.202.11
Routing entry for 10.139.200.0/22
  Known via "static", distance 1, metric 0
  Routing Descriptor Blocks:
  * 10.197.251.1
      Route metric is 0, traffic share count is 1

Then clutching at straws, I removed my "ip verify unicast reverse-path", and traceroute worked just like it should...

fr-rt01#traceroute 155.231.48.203

Type escape sequence to abort.
Tracing the route to 155.231.48.203

  1 10.197.251.1 4 msec 0 msec 0 msec
  2 81.147.220.170 8 msec 8 msec 8 msec
  3 172.16.216.193 8 msec 8 msec 8 msec
  4 217.36.152.160 8 msec 8 msec 8 msec
  5 217.36.152.64 8 msec 8 msec 8 msec
  6 10.100.100.67 12 msec 8 msec 8 msec

And then I went and read up on traceroute and RPF to try to understand what was going on

When traceroute sends each UDP packet out, it expects to get an ICMP type 11 code 0 (time exceeded) back from each intermediate hop/router, with a source IP address of its exit interface (towards me).

RPF does a reverse lookup of the source address against CEF to check that the receiving interface is one of the best return paths to that address.

Taking the first few addresses in the working traceroute:

fr-rt01#sh ip cef 81.147.220.170 
0.0.0.0/0, version 191479, epoch 0
0 packets, 0 bytes
  via 192.168.13.196, Tunnel13196, 0 dependencies
    next hop 192.168.13.196, Tunnel13196
    valid adjacency

fr-rt01#sh ip cef 172.16.216.193
0.0.0.0/0, version 191479, epoch 0
0 packets, 0 bytes
  via 192.168.13.196, Tunnel13196, 0 dependencies
    next hop 192.168.13.196, Tunnel13196

And there's the problem: without a route to the intermediate addresses, I only have a default route, which leaves by a different interface. So RPF is doing its job properly and breaking my traceroute.

In this case, it seems that the best solution is to tell RPF to ignore my traceroute replies by adding an ACL defining them as exceptions:

fr-rt01#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
fr-rt01(config)#access-list 101 permit icmp any host 10.197.251.4 11 0
fr-rt01(config)#int gi0/0.801
fr-rt01(config-subif)#ip verify unicast reverse-path 101
fr-rt01(config-subif)#^Z
fr-rt01#

Which allows both features to work without side effects:

fr-rt01#traceroute 155.231.48.203

Type escape sequence to abort.
Tracing the route to 155.231.48.203

  1 10.197.251.1 4 msec 0 msec 0 msec
  2 81.147.220.170 8 msec 8 msec 8 msec
  3 172.16.216.193 8 msec 8 msec 8 msec
  4 217.36.152.160 8 msec 8 msec 8 msec
  5 217.36.152.64 8 msec 8 msec 8 msec
  6 10.100.100.67 12 msec 8 msec 8 msec
  7  *  *  *

fr-rt01#sh access-list 101
Extended IP access list 101
    10 permit icmp any host 10.197.251.4 ttl-exceeded (15 matches)