Learn how to troubleshoot
Ruhann
The new focus is of the R&S v4 exam is troubleshooting.
Although few people like change, this particular change should be easy enough to be embrace if one understands the reasons why Cisco added this new section. I personally believe it is for two main reasons.
Market demand initiated this change. Previously to become an R&S CCIE you needed to apply vanilla configs to the routers, and provided it was done without error, in the correct order, and by avoiding the question pitfalls, you had a pretty good chance of passing. But how is this relevant in the workplace, where 70% of an engineer’s time is spend fixed something that was previously working. In the lab, if a candidate encountered a similar problem, quite often the related config would just be removed and then be reapplied. But in production, this is almost never an option. You must be able to find the one line that is missing or miss configured, causing the issue.
The second driver was to separate guys that studied to pass the lab, from guys that studied the technologies. This I believe, is what catches so many guys off guard, and causing them to fail. I am referring to guys that used the non recommended but shorter approach: “learn-by-labbing”. Using this method to prepare often creates holes in a candidate’s knowledge. Why? Because the candidate only studied what he encountered and configured during the labs from the different vendors. Often these individuals would not know the states a protocol like OSPF would go through to establish an adjacency. Previously knowing it was not needed, as long as it came up.
After all the CCIE track was designed to educate engineers in becoming proficient in the professional workspace. Thus relevance and efficiency are important factors of the CCIE curriculum.
However, a big mistake I see guys make is treating the new troubleshooting section as a separate and new topic to study. Fundamentally it is not a new topic. Instead it focuses a little bit more on what an individual should have studied already. Theory!!
If you do not know and understand the theory, studying to troubleshoot is like buying a race car before you can drive. I see many chasing after troubleshooting material. What is the difference between a theory book, and troubleshooting book? Not that much. Troubleshooting books, just like theory books, explain the ins and outs of the protocols first, then the configuration, before looking at common problems. Troubleshooting books usually just have more examples and configuration outputs.
What am I suggestion then?
Study the theory books, and actually study them well. Learn the phases and states of the protocols and understand how the protocols work. Then do technology based labs to understand how to put it all together on the command line.
But what about troubleshooting? If you did the previously mentioned step efficiently, you have nothing new to study. But I’ll show you how to test your troubleshooting knowledge. Take a protocol, and categorize what you think could go wrong with BGP. Then take each category, and list the items you already know is required for that category to be operational. Turn this list into questions, and you have a working troubleshooting methodology. You can go one step further by adding the command to answer the questions.
- Categorizing possible OSPF issues:
- Frame-Relay problems
- OSPF Adjacencies
- Authentication issues
- Link-state database problems
- Area wide problems
- Expanding on the second point.
- There must be reachability between the neighbors.
- Certain values in the hellos must match between the neighbors, like Area-ID, authentication, Timers, MTU, Options and network mask on some link types.
- Only certain OSPF network types are compatible.
- Protocol 89 must be allowed between the neighbors.
- OSPF does not perform a adjacency with a secondary IP address.
For OSPF neighbors to become adjacent, what is required?
Turning these point into questions, knowing what output to expect with what command will create a very detailed troubleshooting guide.
Here is an example walkthrough of a post I did on troubleshooting BGP:
Troubleshooting BGP session start-up problems
1- Are you seeing the expected neighbors in a NON ‘idle’ or ‘active’ state?
#sh ip bgp summary
2- Is a sourced telnet to the neighbor address working?
#telnet {peer-ip} 179 /source {src-int-ip}
3- Confirm if the configuration is correct and matching to neighbors’ configuration?
#sh run | b router bgp
4- If eBGP, is the neighbor directly connected? (Should be 1 hop in the trace)
4.1- If not directly connected is multihop configured?
#trace {peer-ip} source {src-int-ip}
#sh run | i {peer-ip}.*ebgp-multihop
5- Is there IP reachability to neighbor?
#ping {peer-ip} source {src-int-ip}
6- Are the underlying routing inplace between neighbors?
#sh ip route {peer-ip}
7- If the obvious checks don’t elude to the issue, enable debugging to analyse the TCP session setup.
#debug ip tcp transactions
7.1- If the TCP-SYN packet is not answered with a SYN-ACK packet and times out?
7.1.1- Look for ACL’s blocking TCP-179.
#sh ip interface | i line|list
7.2- If the TCP-SYN packet is answered with a RST packet, it verifies reachability, but the neighbor is not willing to grant the connection attempt.
7.2.1- Does the neighbor have BGP configured or BGP “neighbor shutdown”?
#sh run | i {peer-ip}.*shutdown
7.2.2- Does the outgoing interface IP match the peers “neighbor” statement?
#sh run | i neighbor.*{peer-ip}
7.2.3- If not is the correct source interfaces specified?
#sh run | i {peer-ip}.*update-source
7.3- If the 3-way TCP handshake completes but the router drops the session shortly after causing the neighbor to oscillate between idle and active, check the BGP parameters.
7.3.1- Confirm the AS numbers between the neighbors are correct
#sh run | i router bgp|remote-as
7.3.2- If using confederations, double check AS numbers.
#sh run | i router bgp|remote-as
7.3.3- Is MD5 password authentication configured correctly?
#sh run | i neighbor.*password
Troubleshooting BGP Route Selection Issues
1- Are locally originated routes appearing in the BGP table?
#sh ip bgp
1.1- If auto-summary is enabled, is at least one subnet of the major network present in the RIB?
#sh run | i router bgp|summary
#sh ip route {prefix} longer-prefixes
1.2- If auto-summary is disabled, is there a exact prefix match in the RIB?
#sh run | i router bgp|no summary
#sh ip route | i {prefix}/{mask}
1.3- Is there a distribute-list configured blocking prefixes?
#sh run | i distribute-list
2- Is there a aggregate configured that is not advertised?
#sh run | i aggregate
2.1- Is there a more specific network of the aggregate in the BGP table?
#sh ip bgp {prefix}/{mask} longer-prefixes
3- Is a prefix in the BGP table not getting advertised to a iBGP neighbor?
#sh ip bgp {prefix} 'Yields no result'
3.1- Was the prefix learned via iBGP? BGP split horizon? (Look for ‘i’ routes)
#sh ip bgp {prefix} | i _i|>i
4- Are you receiving any prefixes from the neighbor? (Look at ‘PfxRcd’)
#sh ip bgp summary | i {peer-ip}
4.1- Is the neighbor sending any routes? (This is issued on the neighbor router)
#sh ip bgp neighbor {peer-ip} advertised routes
4.2- Are the prefixes showing BEFORE any filters were applied? (For this step “soft-reconfig” must have been enabled)
#sh ip bgp neighbor {peer-ip} received-routes
4.3- Are the prefixes showing AFTER the filters were applied?
#sh ip bgp neighbor {peer-ip} routes
4.3.1- Are any prefix-filters configured denying the prefixes?
#sh run | i {peer-ip}.*prefix-list
4.3.2- Are any AS-path filters configured denying the prefixes?
#sh run | i {peer-ip}.*filter-list
4.3.3- If a route-map is configured
#sh run | i {peer-ip}.*route-map
4.3.3.1- The routes must be explicitly permitted to be accepted/used.
#sh route-map {name}
4.3.3.2- Are the prefixes explicitly denied
#sh route-map {name}
4.4- Was the BGP session cleared after changes to filters and route-maps? (BOTH SIDES)
#clear ip bgp * in
5- The prefix is in the bgp table, but not in the RIB
#sh ip bgp | i {prefix}
5.1- Is the BGP next-hop reachable?
#sh ip route {bgp-next-hop}
5.2- Is the prefix selected as the best route? (Indicated with ‘>’)
#sh ip bgp | i {prefix}.
5.2.1- If not, verify the BGP attributes are correct.
#sh ip bgp {prefix}
5.3- Prefix is selected as best, but not entered into RIB? Synchronization issue!
#sh run | i no synch
5.4- If the prefix is listed in the BGP with the options:
5.4.1- ‘r’ means a lower admin distance route is used and entered in the RIB.
#sh ip bgp | i ^r.*{prefix}
5.4.2- ’s’ means specific routes suppressed by aggregation are not advertised.
#sh ip bgp | i ^s.*{prefix}
5.4.3- ‘S’ stale routes marked during a graceful restart is not advertised.
#sh ip bgp | i ^S.*{prefix}
5.4.4- ‘d’ means the route is dampened, due to flapping violations.
#sh ip bgp | i ^d.*{prefix}
5.5- Are any communities attached to the prefix causing problems?
#sh ip bgp {prefix}
5.6- Is the expected communities being received? Sending communities enabled?
#sh run | i neighbor.*send-community
Feel free to visit my blog at blog.ru.co.za, to get many much more information.
ruhann