Saved by the Configuration Repository!
Technical : Terry Slattery
We recently had a near-disaster occur at a customer. A 6500 10/100 ethernet blade had died and was replaced. When the replacement blade was installed, the configuration for its interfaces disappeared from the configuration! The techs couldn't figure out what happened. They saved the current config as part of their normal operations, but now the old configs were missing from both the running and saved configurations. With the interface configs now missing, they called for assistance from the network engineering group.
If you want a good thought exercise, stop reading now, try to figure out what happened, then read the solution below.
The person handling the call from the technicians took a quick look and saw that both the running and saved configurations were now missing for all interfaces on that blade. He immediately thought of the NetMRI configuration repository and found the configuration archive going back many months. Looking through the saved configs, he found the most recent config that had the affected interfaces. The config was quickly exported to his laptop and he pasted the configs into the 6500.
To his surprise, the configs didn't immediately work. (Have you figured out why?)
The 10/100 blade had been replaced with a 10/100/1000 blade. Not a problem, it would handle all the interface speeds and had more modern QoS support. A win all the way around, right?
The new blade's interfaces were GigabitEthernet while the old interfaces were FastEthernet. The interface names in the original config didn't match the new blade, so they were not set. When the techs saved the running config, it over-wrote the saved copy. The only other backup config was in NetMRI. When the engineer handling the call saw the configuration problem, it was easy for him to edit the interface configs to change the name and load them into the 6500. Problem solved!
There's another critical component to this story. It is important to have a configuration repository that automatically discovers devices and collects their configurations. It is easy to forget to add a new device to the configuration collector, especially if the organization doesn't have strong procedures in place for deployment of new network devices. One of the things that I like about NetMRI is that it automatically discovers new devices and will automatically begin collecting data, including the configurations (provided that it has the proper access credentials). Another factor is for the configuration collector to save all the configurations that it collects. It doesn't take much disk space to save every copy of a configuration throughout the history of a device. Automatic functions like those in this example make it easier to run a network.
-Terry
BIO
Terry Slattery, CCIE #1026, is a senior network engineer with decades of experience in the internetworking industry. Prior to joining Chesapeake NetCraftsmen as a full time consultant, Terry was the founder and CTO of Netcordia, and inventor of NetMRI, a suite of network management products. Terry started Netcordia as a consulting company in 2000 and transitioned to a network management product company in 2003.
During the consulting days, he used his network design and implementation skills to lead a team in the design and implementation of a high availability network at a brokerage clearing house. Terry is the former President and founder of Chesapeake Computer Consultants, Inc., a networking and computer systems training and consulting company.
He co-invented and patented the vLab(tm) internet-based remote lab system. He is co-author of the McGraw Hill text Advanced IP Routing in Cisco Networks. Terry led the team that developed the current Cisco IOS user interface under contract to Cisco Systems. Terry is experienced in the design and installation of large TCP/IP based networks and is a successful network protocol instructor. He is the second Cisco Certified Internetworking Expert (CCIE) #1026 and the first outside of Cisco. He enjoys membership on the Vanderbilt University Engineering School’s Industrial Advisory Board and the IEEE.