The Art of Troubleshooting
Dan Shechter Gelles
Dan Shechter Gelles
Triple CCIE #13685 (RS, Security, SP). In the last 14 years Dan has worked with enterprise networking and mainframe system programming and is the author of three CCIE workbooks published by leading vendors.
I hope you will enjoy the technical blog (link) and the workbook (link).
Download Dynamips Source files for this lab here.
Ticket #13
The Problem:
R3 was configured to send to BB1 only routes originated in AS3. So only R3 can ping BB1 from its Lo0. However, its not working, as no router can ping BB1 from it's Lo0. Find and fix the problem.
BB1's F0/0 IP address is 37.45.213.21.
The Solution:
First, lets try to ping to BB1 from R3's Lo0.
On R3:
R3#ping 37.45.213.21 source loopback 0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 37.45.213.21, timeout is 2 seconds:
Packet sent with a source address of 37.46.0.3
.....
Success rate is 0 percent (0/5)
Indeed, R3 can't ping to BB1 from its loopback0.
Looking at the network diagrams, we can see that BB1 should have learned R3's Lo0 route via eBGP. Lets check R3 to see if it had formed a BGP neighbor relationship with BB1.
On R3:
R3#sh ip bgp neighbors 37.45.213.21
BGP neighbor is 37.45.213.21, remote AS 21, external link
BGP version 4, remote router ID 0.0.0.0
BGP state = Active
Last read 02:53:01, last write 02:53:01, hold time is 180, keepalive interval is 60 seconds
Message statistics:
| InQ depth is 0; | |||
| OutQ depth is 0 | |||
| Sent | Rcvd | ||
| Opens: | 4 | 4 | |
| Notifications: | 2 | 0 | |
| Updates: | 11 | 0 | |
| Keppalives: | 210 | 231 | |
| Route Refresh: | 0 | 0 | |
| Total: | 227 | 217 |
For address family: IPv4 Unicast
BGP table version 3, neighbor version 0/0
Output queue size : 0
Index 1, Offset 0, Mask 0x2
1 update-group member
Outbound path policy configured
Outgoing update AS path filter list is 1
| Sent: | Rcvd: | ||
| Prefix activity: | --- | --- | |
| Prefixes Current: | 0 | 0 | |
| Prefixes Total: | 0 | 0 | |
| Implicit Withdraw: | 0 | 0 | |
| Exlicit Withdraw: | 0 | 0 | |
| Used as bestpath: | n/a | 0 | |
| Used as multipath | n/a | 0 |
| Outbound | Inbound | ||
| Local Policy Denied Prefixes: | --- | --- | |
| Total | 0 | 0 |
Address tracking is enabled, the RIB does have a route to 37.45.213.21
Connections established 2; dropped 2
Last reset 02:53:08, due to User reset
Transport(tcp) path-mtu-discovery is enabled
TCP session must be opened passively
No active TCP connection
We can learn several things from the command output:
- The connection is in "Active" state, which means that there is no BGP connection between R3 and BB1.
- There is no TCP connection.
- TCP connection is in passive mode.
We can now suspect that the connection between R3 and BB1 is configured to be passive on both R3 and BB1. Lets first check the BGP configuration of R3.
On R3:
R3#sh run | sec router bgp
router bgp 3
no synchronization
bgp log-neighbor-changes
redistribute connected route-map rmC2B
neighbor 37.45.36.6 remote-as 6
neighbor 37.45.213.21 remote-as 21
neighbor 37.45.213.21 transport connection-mode passive
neighbor 37.45.213.21 filter-list 1 out
auto-summary
Indeed, we can see that the connection mode is passive. But how can we know that its also passive on BB1? The easiest way is to change the connection mode of R3 to BB1 and see if the connection is coming up.
On R3:
R3(config)#router bgp 3
R3(config-router)#no neighbor 37.45.213.21 transport connection-mode passive
R3(config-router)#do clear ip bgp *
R3(config-router)#
*Jun 20 16:46:54.254: %BGP-5-ADJCHANGE: neighbor 37.45.36.6 Down User reset
R3(config-router)#
*Jun 20 16:46:55.650: %BGP-5-ADJCHANGE: neighbor 37.45.213.21 Up
R3(config-router)#
*Jun 20 16:46:58.706: %BGP-5-ADJCHANGE: neighbor 37.45.36.6 Up
Now the BGP session between R3 and BB1 is up and running. Lets try again to ping from R3's Lo0 to BB1.
On R3:
R3#ping 37.45.213.21 source lo0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 37.45.213.21, timeout is 2 seconds:
Packet sent with a source address of 37.46.0.3
.....
Success rate is 0 percent (0/5)
Its not working, yet. Lets see what routes are advertised to BB1.
On R3:
R3#sh ip bgp neighbors 37.45.213.21 advertised-routes
Total number of prefixes 0
R3 is not sending any routes to BB1. Does R3 got any BGP routes in it own BGP table?
On R3:
R3#sh ip bgp
BGP table version is 3, local router ID is 37.46.0.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
| Network | Next Hop | Metric | LocPrf | Weight | Path |
| *> 37.0.0.0 | 0.0.0.0 | 0 | 32768 | ? | |
| *> 37.46.0.6/32 | 37.45.36.6 | 0 | 0 | 6 i |
We can see the R3's BGP table is not empty.
We were told that R3 was configured with filtering policy to send to BB1 only routes originated in AS3. Lets examine how it was configured.
On R3:
R3#sh run | sec router bgp
router bgp 3
no synchronization
bgp log-neighbor-changes
redistribute connected route-map rmC2B
neighbor 37.45.36.6 remote-as 6
neighbor 37.45.213.21 remote-as 21
neighbor 37.45.213.21 filter-list 1 out
auto-summary
R3#sh run | i as-
ip as-path access-list 1 permit ^3$
R3 was configured with an as-path ACL. And looking at the ACL, we can see a common error. The as-path ACL is filtering entries from the BGP table to be sent to BB1. In R3's BGP table, the AS-PATH of locally originated routes is empty. Its not "3".
Lets fix the as-path ACL.
On R3:
R3(config)#no ip as-path access-list 1 permit ^3$
R3(config)#ip as-path access-list 1 permit ^$
R3(config)#do clear ip bgp 21 out
R3(config)#do sh ip bgp neighbors 37.45.213.21 advertised-routes
BGP table version is 3, local router ID is 37.46.0.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
| Network | Next Hop | Metric | LocPrf | Weight | Path |
| *> 37.0.0.0 | 0.0.0.0 | 0 | 32768 | ? |
Total number of prefixes 1
Now we can see a route being sent to BB1. Lets try again to ping from R3.
On R3:
R3#ping 37.45.213.21 so lo 0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 37.45.213.21, timeout is 2 seconds:
Packet sent with a source address of 37.46.0.3
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/4 ms
Success! R3 can ping BB1 from it's Lo0 interface. Now lets verify that R6 can't do the same.
On R6:
R6#ping 37.45.213.21 source loopback 0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 37.45.213.21, timeout is 2 seconds:
Packet sent with a source address of 37.46.0.6
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 8/8/8 ms
Success?? How BB1 knows about R6's Lo0 route? Does R3 sends non AS3 routes to BB1? Lets check this again.
On R3:
R3#sh ip bgp neighbors 37.45.213.21 advertised-routes
BGP table version is 3, local router ID is 37.46.0.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
| Network | Next Hop | Metric | LocPrf | Weight | Path |
| *> 37.0.0.0 | 0.0.0.0 | 0 | 32768 | ? |
Total number of prefixes 1
R3 does not send routes from AS6, but it does sends a strange route. It looks like a /8 route, which also covers R6's Lo0 route. It smells like summarized route. Let check the configuration for any summarization.
On R3:
R3#sh run | sec router bgp
router bgp 3
no synchronization
bgp log-neighbor-changes
redistribute connected route-map rmC2B
neighbor 37.45.36.6 remote-as 6
neighbor 37.45.213.21 remote-as 21
neighbor 37.45.213.21 filter-list 1 out
auto-summary
We have found a fossil, named "auto-summary", which summarizes redistributed routes when crossing major network boundaries. Lets remove it and try again to ping from R3 and from R6.
On R3:
R3(config)#router bgp 3
R3(config-router)#no auto-summary
R3(config-router)#do clear ip bgp *
R3(config-router)#
*Jun 20 19:47:43.805: %BGP-5-ADJCHANGE: neighbor 37.45.36.6 Down User reset
*Jun 20 19:47:43.813: %BGP-5-ADJCHANGE: neighbor 37.45.213.21 Down User reset
R3(config-router)#
*Jun 20 19:47:46.953: %BGP-5-ADJCHANGE: neighbor 37.45.36.6 Up
*Jun 20 19:47:46.957: %BGP-5-ADJCHANGE: neighbor 37.45.213.21 Up
R3(config-router)#do sh ip bgp neighbors 37.45.213.21 advertised-routes
BGP table version is 3, local router ID is 37.46.0.3
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal,
r RIB-failure, S Stale
Origin codes: i - IGP, e - EGP, ? - incomplete
| Network | Next Hop | Metric | LocPrf | Weight | Path |
| *> 37.46.0.3/32 | 0.0.0.0 | 0 | 32768 | ? |
Total number of prefixes 1
R3(config-router)#do ping 37.45.213.21 so lo
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 37.45.213.21, timeout is 2 seconds:
Packet sent with a source address of 37.46.0.3
!!!!.
Success rate is 80 percent (4/5), round-trip min/avg/max = 4/5/8 ms
On R6:
R6#ping 37.45.213.21 source loopback 0
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 37.45.213.21, timeout is 2 seconds:
Packet sent with a source address of 37.46.0.6
.....
Success rate is 0 percent (0/5)
Success! We can see that R3 is sending only its Lo0 route to BB1, and that only R3 can ping BB1 from its Lo0.
