The Art of Troubleshooting

Ticket #16
by Dan Shechter Gelles

The Problem:

R1 is configured with redundant bidirectional connection between R3 and R4's Lo0. Its like multihome setup.

R2 is the preferred path between R3 and R4.

R1 is using NAT to allow connectivity, exposing R3 as 4.3.12.3 and 4.3.15.3.

R3 can't reach R4 and vice versa.

Fix the problem without changing configuration on R2,R4 and R5. Keep the solution redundant.

Use telnet to check connectivity. You are allowed to configure one static route.

The Solution:

First let try to telnet from R3 to R4's Lo0

On R3:

R3#telnet 4.3.0.4
Trying 4.3.0.4 ... 
% Destination unreachable; gateway or host down

It looks like we don't have a route to 4.3.0.4. Lets verify this.

On R3:

R3#sh ip route 4.3.0.4
% Subnet not in table
R3#sh ip route 0.0.0.0
% Network not in table

R3 have no specific or default route to R4's Lo0. Lets check if it have an OSPF neighbor relationship with R1.

On R3:

R3#sh ip ospf neighbor 
R3#

R3 got no OSPF neighbors. Lets enable OSPF debug to find out why.

On R3:

R3#deb ip ospf hello 
OSPF hello events debugging is on
R3#
*Oct 29 23:02:51.134: OSPF: Rcv hello from 4.3.0.1 area 13 from FastEthernet0/1 4.3.13.1
*Oct 29 23:02:51.138: OSPF: Mismatched hello parameters from 4.3.13.1
*Oct 29 23:02:51.142: OSPF: Dead R 40 C 40, Hello R 10 C 10  Mask R 255.255.255.248 C 255.255.255.0

We can clearly see what the problem is. R3's network mask is /24 while R1's network mask is /29. We can fix the network mask on R1 or R3. Since we are already talking about R3 then lets fix it there.

On R3:

R3(config)#int f0/1
R3(config-if)#ip addr 4.3.13.3 255.255.255.248
R3(config-if)#
*Oct 29 23:05:18.226: %OSPF-5-ADJCHG: Process 1, Nbr 4.3.0.1 on FastEthernet0/1 from LOADING to FULL, Loading Done

Now that we have an OSPF neighbor relationship with R1, lets check for a route from R1.

On R3:

R3#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route
Gateway of last resort is not set
     4.0.0.0/29 is subnetted, 1 subnets
C       4.3.13.0 is directly connected, FastEthernet0/1

There are no OSPF routes at all on R3. Lets examine R1 and see how it was configured to advertise a path to R4's Lo0.

On R1:

R1#sh ip route 4.3.0.4
% Subnet not in table
R1#sh ip route 0.0.0.0
Routing entry for 0.0.0.0/0, supernet
  Known via "static", distance 1, metric 0 (connected), candidate default path
  Redistributing via ospf 1
  Routing Descriptor Blocks:
  * directly connected, via Serial1/0.15
      Route metric is 0, traffic share count is 1

R1 got no direct route to R4's Lo0, but it got a default route pointing to R5. We should mark this as a problem as the route to R4 should be via R2 or via R5 as a backup, but for now, we will let it go, and concentrate on the OSPF route problem.
So R1 got a static default route. Lets have a look at R1's OSPF configuration to see how it was configured to advertise the default route to R3.

On R1:

R1#show run | section router ospf
router ospf 1
 log-adjacency-changes
 area 13 nssa
 redistribute static

We can see that static route are being redistributed, but this won't redistribute default route. Lets fix that by using default-information-originate OSPF process command. Notice that since area 13 is NSSA, we need to use the area command to send the default route

On R1:

R1(config)#router ospf 1
R1(config-router)#area 13 nssa default-information-originate 

Now lets have a look at R3's routing table and try to telnet R4 again.

On R3:

R3#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route
Gateway of last resort is 4.3.13.1 to network 0.0.0.0
     4.0.0.0/29 is subnetted, 1 subnets
C       4.3.13.0 is directly connected, FastEthernet0/1
O*N2 0.0.0.0/0 [110/1] via 4.3.13.1, 00:00:56, FastEthernet0/1
R3#telnet 4.3.0.4
Trying 4.3.0.4 ... 
% Connection timed out; remote host not responding

We now have a default route, but we can't telnet from R3 to R4's Lo0. Lets verify that R4 got a route back to R3. Remember that R1 is using NAT, so if R1 can telnet to R4 the problem is not with R4.

On R1:

R1#telnet 4.3.0.4
Trying 4.3.0.4 ... Open
R4>who
    Line       User       Host(s)              Idle       Location
   0 con 0                idle                 05:15:28   
*  2 vty 0                idle                 00:00:00 4.3.15.1
  Interface    User               Mode         Idle     Peer Address
R4>exit
[Connection to 4.3.0.4 closed by foreign host]

R1 can reach R4, which means that R4 have a route back to the NAT address of R3. So maybe its NAT to blame

Now lets have a look at the NAT configuration of R1.

On R1:

R1#sh ip nat translations 
Pro Inside global      Inside local       Outside local      Outside global
--- 4.3.13.3           4.3.12.3           ---                ---
R1#sh run | i interface|nat
interface Loopback0
interface FastEthernet0/0
 ip nat inside
interface FastEthernet0/1
interface Serial1/0
interface Serial1/0.12 point-to-point
 ip nat outside
 frame-relay interface-dlci 102   
interface Serial1/0.15 point-to-point
 ip nat outside
 frame-relay interface-dlci 105   
interface Serial1/1
interface Serial1/2
interface Serial1/3
 area 13 nssa default-information-originate
ip nat inside source static 4.3.12.3 4.3.13.3

We can see two problems:

1. There is no NAT mapping for 4.3.15.3.
2. The NAT mapping for 4.3.12.3 is upside down. The inside global and the inside local are swapped.

Lets fix the NAT by swapping the IP addresses in the current NAT statement and add a new NAT statement for 4.3.15.3.

On R1:

R1(config)#no ip nat inside source static 4.3.12.3 4.3.13.3
R1(config)#ip nat inside source static 4.3.13.3 4.3.12.3    
 
R1(config)#ip nat inside source static 4.3.13.3 4.3.15.3 
 
% 4.3.13.3 already mapped (4.3.13.3 -> 4.3.12.3)

Looks like we can't map R3 twice. But there is a way...

On R1:

R1(config)#no ip nat inside source static 4.3.13.3 4.3.12.3 
R1(config)#ip nat inside source static 4.3.13.3 4.3.12.3 extendable 
 
R1(config)#ip nat inside source static 4.3.13.3 4.3.15.3 extendable 
 
R1(config)#do sh ip nat tran
Pro Inside global      Inside local       Outside local      Outside global
--- 4.3.12.3           4.3.13.3           ---                ---
--- 4.3.15.3           4.3.13.3           ---                ---

The extendable NAT keyword allows IOS to map the same inside host to more then out outside addresses. Its doing so by using L4 ports information in the NAT translation table.
Now lets try again to telnet from R1 to R4's Lo0.

On R3:

R3#telnet 4.3.0.4
Trying 4.3.0.4 ... Open
R4>exit
[Connection to 4.3.0.4 closed by foreign host]

Success!? Not yet, remember that we have discovered that R1 is reaching R4's Lo0 via R5 and not via R2. Lets have another look.

On R1:

R1#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route
Gateway of last resort is 0.0.0.0 to network 0.0.0.0
     4.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
C       4.3.0.1/32 is directly connected, Loopback0
C       4.3.15.0/24 is directly connected, Serial1/0.15
C       4.3.13.0/29 is directly connected, FastEthernet0/0
C       4.3.12.0/24 is directly connected, Serial1/0.12
S*   0.0.0.0/0 is directly connected, Serial1/0.15

A static default gateway is going via R5. Lets have a look at R1's static route configuration.

On R1:

R1#sh run | i ip route
ip route 0.0.0.0 0.0.0.0 Serial1/0.12 track 12
ip route 0.0.0.0 0.0.0.0 Serial1/0.15

We can see two problems:

1. The static route via R5 is not configured as a backup route as it has the same AD as the route via R2.
2. The static route via R2 is using object tracking.

Lets fix the static route via R5 by making it a floating static route, and lets drilldown into the track 12 parameter of the static route via R2.

On R1:

R1(config)#no ip route 0.0.0.0 0.0.0.0 Serial1/0.15
R1(config)#ip route 0.0.0.0 0.0.0.0 Serial1/0.15 200   
R1(config)#do sh track
Track 12
  IP SLA 12 state
  State is Down
    1 change, last change 05:45:58
  Latest operation return code: Unknown
  Tracked by:
    STATIC-IP-ROUTING 0
R1(config)#do sh ip sla conf
IP SLAs Infrastructure Engine-II
Entry number: 12
Owner:
Tag:
Type of operation to perform: icmp-echo
Target address/Source address: 4.3.0.4/0.0.0.0
Type Of Service parameter: 0x0
Request size (ARR data portion): 28
Operation timeout (milliseconds): 400
Verify data: No
Vrf Name:
Schedule:
   Operation frequency (seconds): 1  (not considered if randomly scheduled)
   Next Scheduled Start Time: Pending trigger
   Group Scheduled : FALSE
   Randomly Scheduled : FALSE
   Life (seconds): 3600
   Entry Ageout (seconds): never
   Recurring (Starting Everyday): FALSE
   Status of entry (SNMP RowStatus): notInService
Threshold (milliseconds): 5000 (not considered if react RTT is configured)
Distribution Statistics:
   Number of statistic hours kept: 2
   Number of statistic distribution buckets kept: 1
   Statistic distribution interval (milliseconds): 20
History Statistics:
   Number of history Lives kept: 0
   Number of history Buckets kept: 15
   History Filter Type: None
Enhanced History:

We can see that the tracking object is down, and it uses IP SLA feature to monitor the path to R4's Lo using ICMP-echo. We can also see that we have scheduling problem.
Lets have a look at the scheduling configuration.

On R1:

R1#sh run | sec sla
track 12 ip sla 12
ip sla 12
 icmp-echo 4.3.0.4
 timeout 400
 frequency 1

Scheduling is not configured. Lets fix that.

On R1:

R1(config)#ip sla schedule 12 life forever start-time now
*Oct 30 01:01:31.858: %TRACKING-5-STATE: 12 ip sla 12 state Down->Up
R1(config)#do sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route
Gateway of last resort is 0.0.0.0 to network 0.0.0.0
     4.0.0.0/8 is variably subnetted, 4 subnets, 3 masks
C       4.3.0.1/32 is directly connected, Loopback0
C       4.3.15.0/24 is directly connected, Serial1/0.15
C       4.3.13.0/29 is directly connected, FastEthernet0/0
C       4.3.12.0/24 is directly connected, Serial1/0.12
S*   0.0.0.0/0 is directly connected, Serial1/0.12

Success!? Now we have the correct route via R2, but do we really monitor the right thing? Lets bring down the link between R2 and R4 and check R1's routing table.

On R1:

*Oct 30 00:20:10.750: %TRACKING-5-STATE: 12 ip sla 12 state Up->Down
R1(config)#
*Oct 30 00:20:15.754: %TRACKING-5-STATE: 12 ip sla 12 state Down->Up
R1(config)#
*Oct 30 00:20:20.762: %TRACKING-5-STATE: 12 ip sla 12 state Up->Down
R1(config)#
*Oct 30 00:20:25.766: %TRACKING-5-STATE: 12 ip sla 12 state Down->Up
R1(config)#
*Oct 30 00:20:30.770: %TRACKING-5-STATE: 12 ip sla 12 state Up->Down
R1(config)#
*Oct 30 00:20:35.774: %TRACKING-5-STATE: 12 ip sla 12 state Down->Up
R1(config)#
*Oct 30 00:20:40.782: %TRACKING-5-STATE: 12 ip sla 12 state Up->Down

We can see that the tracking state is bouncing. Why? Let have a look at the flow of events:

1. State is UP. Static route is pointing to R2.
2. Link between R2 and R4 is going down.
3. R1 can no longer ping R4's Lo0 via R2. State is DOWN. Static route is pointing to R5.
4. R1 tries to ping R4's Lo0 via R5 and succeed. State is UP. Static route is pointing to R2.
5. Going back to bullet number 3, and so on...

We need R1 to sample R4's F0/0 interface, which is connected to R2. Thats way we can monitor the path to R4's Lo0 via R2. 

On R1:

R1(config)#no ip sla schedule 12 start-time now life forever 
R1(config)#no track 12
R1(config)#no ip sla 12
R1(config)#ip sla 12
R1(config-ip-sla)#icmp-echo 4.3.24.4
R1(config-ip-sla-echo)#timeout 300
R1(config-ip-sla-echo)#frequency 1
R1(config-ip-sla-echo)#ip sla schedule 12 start-time now life forever
R1(config)#track 12 ip sla 12 
R1(config)#ip route 4.3.24.4 255.255.255.255 s1/0.12
R1(config)#
*Oct 30 00:31:01.550: %TRACKING-5-STATE: 12 ip sla 12 state Down->Up
R1(config)#do sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route
Gateway of last resort is 0.0.0.0 to network 0.0.0.0
     4.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
C       4.3.0.1/32 is directly connected, Loopback0
C       4.3.15.0/24 is directly connected, Serial1/0.15
C       4.3.13.0/29 is directly connected, FastEthernet0/0
C       4.3.12.0/24 is directly connected, Serial1/0.12
S       4.3.24.4/32 is directly connected, Serial1/0.12
S*   0.0.0.0/0 is directly connected, Serial1/0.12

Now lets try to bring down the interface between R2 and R4 and see what happens in R1.

On R1:

R1(config)#
*Oct 30 00:33:01.554: %TRACKING-5-STATE: 12 ip sla 12 state Up->Down
R1(config)#do sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
       D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area 
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
       ia - IS-IS inter area, * - candidate default, U - per-user static route
       o - ODR, P - periodic downloaded static route
Gateway of last resort is 0.0.0.0 to network 0.0.0.0
     4.0.0.0/8 is variably subnetted, 5 subnets, 3 masks
C       4.3.0.1/32 is directly connected, Loopback0
C       4.3.15.0/24 is directly connected, Serial1/0.15
C       4.3.13.0/29 is directly connected, FastEthernet0/0
C       4.3.12.0/24 is directly connected, Serial1/0.12
S       4.3.24.4/32 is directly connected, Serial1/0.12
S*   0.0.0.0/0 is directly connected, Serial1/0.15

The tracking object is not bouncing and the route now is via R5. Lets try to telnet from R3 to R4.

On R3:

R3#telnet 4.3.0.4
Trying 4.3.0.4 ...
% Connection timed out; remote host not responding

Its not working. Why? Lets enable debug on R1 to find out why.

On R1:

R1#deb ip nat
IP NAT debugging is on
R1#
*Oct 30 00:37:33.518: NAT*: s=4.3.13.3->4.3.12.3, d=4.3.0.4 [56021]
R1#
*Oct 30 00:37:35.506: NAT*: s=4.3.13.3->4.3.12.3, d=4.3.0.4 [56021]

We can see that R1 is still using NAT address of S1/0.12, which R4 got no route to via R5. What can we do? We can use route map to map the static NAT to depend on a specific interface.

On R1:

R1(config)#route-map rm12
R1(config-route-map)#match interface s1/0.12
R1(config-route-map)#route-map rm15         
R1(config-route-map)#match interface s1/0.15
R1(config-route-map)#exit
R1(config)#no ip nat inside source static 4.3.13.3 4.3.12.3 extendable
R1(config)#no ip nat inside source static 4.3.13.3 4.3.15.3 extendable
R1(config)#ip nat inside source static 4.3.13.3 4.3.12.3 extendable route-map rm12     
 
R1(config)#ip nat inside source static 4.3.13.3 4.3.15.3 extendable route-map rm15     

Now lets try to telnet from R3 to R4 while R2's F0/0 is down, and then try again after bringing up that interface.

On R3:

R3#telnet 4.3.0.4
Trying 4.3.0.4 ... Open
R4>who
    Line       User       Host(s)              Idle       Location
   0 con 0                idle                 00:16:39
*  2 vty 0                idle                 00:00:00 4.3.15.3
  Interface    User               Mode         Idle     Peer Address
R4>exit
[Connection to 4.3.0.4 closed by foreign host]
! now we bring up R2's F0/0 interface.
R3#telnet 4.3.0.4
Trying 4.3.0.4 ... Open
R4>who
    Line       User       Host(s)              Idle       Location
   0 con 0                idle                 00:00:03   
*  2 vty 0                idle                 00:00:00 4.3.12.3
  Interface    User               Mode         Idle     Peer Address
R4>exit
[Connection to 4.3.0.4 closed by foreign host]

Success! Notice how the addresses are changing according to the path.