Content on this page requires a newer version of Adobe Flash Player.

Get Adobe Flash player

July 2010
Home||Table of Contents||2008 Back Issues||2009 Back Issues||2010 Back Issues||Contact the Editor||Contact the Webmaster||

The Art of Troubleshooting

Dan Shechter Gelles


Bookmark and Share

Dan Shechter Gelles

Triple CCIE #13685 (RS, Security, SP). In the last 14 years Dan has worked with enterprise networking and mainframe system programming and is the author of three CCIE workbooks published by leading vendors.

I hope you will enjoy the technical blog (link) and the workbook (link).


Download Dynamips Source files for this lab here.


Ticket #14

The Problem:

R1 can't telnet R6's Lo0. Find and fix the problem.

The Solution:

First, lets try to telnet to R6's Lo0 from R1.

On R1:

R1#telnet 136.85.0.6
Trying 136.85.0.6 ...
% Destination unreachable; gateway or host down

R1#


Indeed R1 can't telnet R6. Lets check for an ICMP connectivity.

On R1:

R1#ping 136.85.0.6

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 136.85.0.6, timeout is 2 seconds:
.....

Success rate is 0 percent (0/5)
R1 can't even ping R6's Lo. Lets check if R1 got a route to R6's Lo0, and if not lets also check for a default route.

On R1:

R1#sh ip route 136.85.0.6
% Subnet not in table
R1#sh ip route 0.0.0.0
% Network not in table


R1 got no route to R6's Lo0 neither it got a default route.

R1 should have learn about R6's Lo0 network from R3 via OSPF. Lets check R1's routing table for any OSPF routes coming from R3.

On R1:

R1#sh ip route ospf
     136.85.0.0/16 is variably subnetted, 3 subnets, 2 masks
O          136.85.0.3/32 [110/2] via 136.85.13.3, 00:44:07, FastEthernet0/0


We do see an OSPF route from R3, which means that OSPF is functional between R1 and R3.

Lets move closer to R6 and check R3's routing table.

On R3:

R3#sh ip route 136.85.0.6
% Subnet not in table
R3#sh ip route 0.0.0.0
% Network not in table


R3 got no route to R6's Lo0. Looking at the diagrams, R3 should have got a route to R6 via iBGP. Lets check R3's BGP table and the BGP status.

On R3:

R3#sh ip bgp

R3#sh ip bgp sum
BGP router identifier 136.85.0.3, local AS number 36
BGP table version is 3, main routing table version 3

Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
136.85.36.6 4 36 49 47 3 0 0 00:46:05 0

We can see that there are no BGP routes coming from R6, and that the BGP session to R6 is up.

Lets go to R6 to check its BGP table and how was it configured to advertise its Lo0 network.

On R6:

R6#sh ip bgp

R6#sh run | sec router bgp
router bgp 36
  no synchronization
  bgp log-neighbor-changes
  network 136.85.0.6 mask 255.255.255.255
  neighbor 136.85.36.3 remote-as 36
  no auto-summary


We can see that R6's BGP table is also empty and that R6 was configured to advertise its network using the network BGP command, however the route is not there. Why?

Lets check R6's routing table and make sure it got a 136.85.0.6/32 route, as maybe the Lo0 interface is down.

On R6:

R6#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
     D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
     N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
     E1 - OSPF external type 1, E2 - OSPF external type 2
     i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
     ia - IS-IS inter area, * - candidate default, U - per-user static route
     o - ODR, P - periodic downloaded static route


Gateway of last resort is not set

     136.85.0.0/24 is subnetted, 2 subnets
C     136.85.0.0 is directly connected, Loopback0
C     136.85.36.0 is directly connected, FastEthernet0/1


We can see that Lo0 is up, but its a /24 route and not /32 route. Let verify that with the configuration and fix it.

On R6:

R6#sh run int lo 0
Building configuration...


Current configuration : 64 bytes

!

interface Loopback0
ip address 136.85.0.6 255.255.255.0
end
R6(config)#int lo 0
R6(config-if)#ip address 136.85.0.6 255.255.255.255
R6(config-if)#do sh ip bgp
BGP table version is 4, local router ID is 136.85.0.6
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal, r
          RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete


Network Next Hop Metric LocPrf Weight Path
*> 136.85.0.6/32 0.0.0.0 0 32768 1


Now R6 got its Lo0 in its BGP table. Lets check if R3 got the route to R6''s Lo0 too.

On R3:

R3#sh ip route 136.85.0.6
Routing entry for 136.85.0.6/32
Known via "bgp 36", distance 200, metric 0, type internal
Last update from 136.85.36.6 00:08:27 ago
Routing Descriptor Blocks:
* 136.85.36.6, from 136.85.36.6, 00:08:27 ago
Route metric is 0, traffic share count is 1
AS Hops 0


Now that R3 got an iBGP route to R6's Lo, lets check if R1 got a route too.

On R1:

R1#sh ip route 136.85.0.6
% Subnet not in table


R1 still got no route to R6's Lo0. Lets have a look at how R3 should have sent the iBGP route to R1 via OSPF.

On R3:

R3#sh run | sec router ospf
router ospf 1
log-adjacency-changes
default-information originate route-map rmOriginate


The only thing configured on R3 is to send the default route the OSPF network only if the route-map matches any installed route. Lets see what the route-map is trying to match.

On R3:

R3#sh route-map
route-map rmOriginate, permit, sequence 10
     Match clauses:
          ip address (access-lists): 10
     Set clauses:
     Policy routing matches: 0 packets, 0 bytes
R3#sh ip access-list 10
Standard IP access list 10
     10 permit 136.85.0.6


The route map looks OK. It matches ACL #10, and ACL #10 matches the host route for R6's Lo0. However, a route-map, used by default-information-originate, will never match an iBGP route!

What shell we do? Watch the magic....

On R3:

R3(config)#int tun 99 *Jul 8 20:20:50.014: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel99, changed state to down
R3(config-if)#tun so lo 0
R3(config-if)#tun dest 136.85.0.6
*Jul 8 20:21:06.910: %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel99, changed state to up
R3(config-if)#ip addr 136.85.0.99 255.255.255.254
R3(config-if)#ip prefix-list pl10 permit 136.85.0.99/31
R3(config)#route-map rmOriginate
R3(config-route-map)#no mat ip addr 10
R3(config-route-map)#mat ip add prefix pl10


What have we done here? Here are the steps to victory:
•Add a tunnel interface
•Set the tunnel destination to R6's Lo0. As long as R3 will learn R6's Lo0 route the tunnel interface will be up
•Set a dummy IP address for the tunnel interface
•Configure the route-map to match the tunnel dummy IP address


So as long as R3 got R6's Lo0 route, the Tunnel interface will be up and its address will be matched by the route-map.


Now lets check R1's routing table and see if we have a default route.

On R1:

R1#sh ip route 0.0.0.0
Routing entry for 0.0.0.0/0, supernet
Known via "ospf 1", distance 110, metric 1, candidate default path
Tag 1, type extern 2, forward metric 1
Last update from 136.85.13.3 on FastEthernet0/0, 00:04:29 ago
Routing Descriptor Blocks:
     * 136.85.13.3, from 136.85.0.3, 00:04:29 ago, via FastEthernet0/0
          Route metric is 1, traffic share count is 1
          Route tag 1


Now that R1 got a route to R6's Lo0, lets try to ping to R6's Lo0.

On R1:

R1#ping 136.85.0.6

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 136.85.0.6, timeout is 2 seconds:
.
Success rate is 0 percent (0/1)


Lets check and see if R3 can ping R6's Lo0.

On R3:

R3#ping 136.85.0.6

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 136.85.0.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/3/4 ms


R3 is able to ping to R6. Let check if R6 got a route back to R1.

On R6:

R6#sh ip route
Codes: C - connected, S - static, R - RIP, M - mobile, B - BGP
      D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
      N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
      E1 - OSPF external type 1, E2 - OSPF external type 2
      i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
      ia - IS-IS inter area, * - candidate default, U - per-user static route
      o - ODR, P - periodic downloaded static route

Gateway of last resort is not set

      136.85.0.0/16 is variably subnetted, 2 subnets, 2 masks
C      136.85.0.6/32 is directly connected, Loopback0
C      136.85.36.0/24 is directly connected, FastEthernet0/1


As we can see there is no route back to VLAN13, where R1 is connected to R3. Let configure R3 to advertise that network to BGP.

On R3:

R3(config)#router bgp 36
R3(config-router)#net 136.85.13.0 mask 255.255.255.0


Now lets try again to ping from R1 to R6's Lo0.

On R1:

R1#ping 136.85.0.6

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 136.85.0.6, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/23/84 ms


Now lets try to telnet from R1 to R6's Lo0.

On R1:

R1#telnet 136.85.0.6
Trying 136.85.0.6 ...
% Connection timed out; remote host not responding


We can ping but we can't telnet from R1. Lets check if R6 is setup for telnet by trying to telnet from R6 to itself.

On R6:

R6#telnet 136.85.0.6
Trying 136.85.0.6 ...
% Connection timed out; remote host not responding


R6 can't even telnet to itself. Lets start verifying configuration.

On R6:

R6#sh run | sec line vty
line vty 0 5
  password cisco
  login


R6 is configured for telnet, so what is wrong? Lets check these new 12.4T features...

On R6:

R6#sh run | sec control
control-plane host
  management-interface FastEthernet0/0 allow ftp http https ssh tftp snmp beep telnet tl1 control-plane


R6 is configured to only allow telnet session from it's F0/0 interface. But R1 connection should come form F0/1 interface. Lets correct that and try again.

On R6:

R6(config)#control-plane host
R6(config-cp-host)#management-interface FastEthernet0/1 allow

On R1:

R1#telnet 136.85.0.6
Trying 136.85.0.6 ...
% Connection timed out; remote host not responding

R1#telnet 136.85.0.6 Trying 136.85.0.6 ... Open

User Access Verification

Password:

R6>


Return to the top of 'The Art of Troubleshooting'. |»«| Send Feedback |»«| Contact the Editor |»«| Contact the Webmaster