Interesting day today, that it was. Well, I guess I should say that the weekend was interesting as we had some bad weather – and just by some random chance we lost a CoreWAN router and a Cisco 3750 Stack member. So lets just say that I spent my morning recovering the WAN router by moving the circuits to another router and recovering a 3750 switch. Since moving WAN circuits around is no big deal, lets talk about recovering a Cisco 3750 stack member switch.
I will assume you have identified the switch that is dead – no lights is a good indication – and have received your replacement switch from Cisco already. The first thing you should do is independently boot the new switch and check the IOS version and make sure that it is either the same or compatible. For me, I prefer to make sure they are the same before adding a switch to the stack as I never trust “compatible” version of switch code. If you have ever upgraded a 3750 stack, you know it can take some time for the stack to come back – over 10 minutes is not unheard of.
So what do you do if you lose a switch in your stack, and have the right replacement? Amazingly enough, just swap it out and the stack will work as normal. Now, don’t we always wish life was that simple!
So what happens if the switch they send you has the wrong code on it? You can just use TFTP to copy the image over to the switch. Just configure Vlan 1 with an IP, configure an IP on your laptop on the same subnet and run a cable between the laptop and switch and then copy the code. Most of us know how to TFTP, so I will skip the “how to”.
Now, what if it is really bad – the IOS image is corrupt and you can only boot to the switch: prompt? Ahh, the joys of recovering a corrupt image with x-modem. For that, I will show you how to upload the image so you are not waiting hours to copy an image at 9600 baud. And for those of us who are saying, use tdftdnld, that is not available on the 3750s that I have – trust me I looked!
So, lets start by consoling in and connecting at 9600 baud (default Cisco console connection)
Now we are at the switch: prompt.
switch:
The first command we need to run is to initialize the flash.
switch: flash_init
Initializing Flash…
…The flash is already initialized.
Setting console baud rate to 9600…
switch:
Now lets take a look at the directory:
switch: dir
List of filesystems currently registered:
flash[0]: (read-write)
xmodem[1]: (read-only)
null[2]: (read-write)
bs[3]: (read-only)
switch:
Whoops, forgot to specify flash. Lets try that again:
switch: dir flash:
Directory of flash:/
15997952 bytes available (1024 bytes used)
switch:
Hmm, nothing in flash here – must have been eresaesd. lets format it just in case.
switch: format flash:
Are you sure you want to format “flash:” (all data will be lost) (y/n)?y
flashfs[0]: 0 files, 1 directories
flashfs[0]: 0 orphaned files, 0 orphaned directories
flashfs[0]: Total bytes: 15998976
flashfs[0]: Bytes used: 1024
flashfs[0]: Bytes available: 15997952
flashfs[0]: flashfs fsck took 8 seconds.
Setting console baud rate to 9600…
Filesystem “flash:” formatted
switch:
Now that it is formated we need to get an image there. If we copy an 8 meg image at 9600 baud, we are talking about 2 hours of time. There has to be a faster way – and there is.
Lets set the baud rate to 115200, that should cut the transfer time down to about 16 minutes (yeah, not quick but faster then 9600!)
switch: set BAUD 115200
ÿ–
You will get some strange characters, what you need to do now is disconnect from the serial interface, change the speed to 115200, and then reconnect
There we go, all connected. Now lets get that image up there.
To copy the image, just issue the command:
copy xmode:<image.bin> flash:<image.bin>
switch: copy xmodem:c3750-ipservicesk9-mz.122-25.SEE4.bin flash:c3750-ipservicesk9-mz.122-25.SEE4
Begin the Xmodem or Xmodem-1K transfer now…
CCC
Start your x-modem terminal transfer with the image, it will take some time to copy, my transfer showed a time remaing of about 19 minutes, much nicer then 2 hours!
Once the transfer is complete, the switch will copy the image from its buffer to the flash file system. You will see a bunch of dots across the screen
100% 7852 KB 6 KB/s 00:19:28 0 Errors..
…..<bunch of dots when complete>………………………………………….
File “xmodem:c3750-ipservicesk9-mz.122-25.SEE4.bin” successfully copied to “flash:c3750-ipservicesk9-mz.122-25.SEE4.bin”
switch:
Lets look at the flash:
switch: dir flash:
Directory of flash:/
2 -rwx 8041088 <date> c3750-ipservicesk9-mz.122-25.SEE4.bin
7956480 bytes available (8042496 bytes used)
switch:
Nice, we now have an imge. Lets see if we can get it to boot!
switch: boot flash:c3750-ipservicesk9-mz.122-25.SEE4.bin
Loading “flash:c3750-ipservicesk9-mz.122-25.SEE4.bin”…@@@@@@@@@
There we go, we are loading the image!
Cisco IOS Software, C3750 Software (C3750-IPSERVICESK9-M), Version 12.2(25)SEE4, RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2007 by Cisco Systems, Inc.
Compiled Mon 16-Jul-07 03:24 by myl
Would you like to terminate autoinstall? [yes]:
Back in business!
Now lets add the switch back into the stack, being sure to cable it just like it was cabled before failure.
Once it has been cabled back up and powered on, we should watch the console to make sure it comes back correctly.
I was not able to console in due to stack location, so most of the rest of this post is grabbed from logs, but you would still see these same messages on the console.
3750Stack# sh log
Aug 15 14:03:33.120 UTC: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 5 has changed to state UP
Aug 15 14:03:33.867 UTC: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 1 has changed to state UP
There we go, stack ports are back up
3750Stack# sh switch
Current
Switch# Role Mac Address Priority State
——————————————————–
*1 Master 0018.186b.1234 15 Ready
2 Member 0018.18cd.1235 1 Ready
3 Member 0018.1834.1236 1 Ready
4 Member 0018.186b.1237 1 Ready
5 Member 0018.1879.1238 1 Ready
6 Member 0000.0000.0000 0 Provisioned
Switch is in a provisioned state, so we are still booting.
3750Stack# sh switch
Current
Switch# Role Mac Address Priority State
——————————————————–
*1 Master 0018.186b.1234 15 Ready
2 Member 0018.18cd.1235 1 Ready
3 Member 0018.1834.1236 1 Ready
4 Member 0018.186b.1237 1 Ready
5 Member 0018.1879.1238 1 Ready
6 Member 001a.e324.1239 1 Initializing
Good deal, we are now initiailizing – it sees the replacement as Switch 6
3750Stack#
Aug 15 14:03:42.591 UTC: %STACKMGR-4-SWITCH_ADDED: Switch 6 has been ADDED to the stack
Aug 15 14:03:42.633 UTC: %STACKMGR-4-SWITCH_ADDED: Switch 6 has been ADDED to the stack (3750Stack#-2)
Aug 15 14:03:42.641 UTC: %STACKMGR-4-SWITCH_ADDED: Switch 6 has been ADDED to the stack (3750Stack#-3)
Aug 15 14:03:42.649 UTC: %STACKMGR-4-SWITCH_ADDED: Switch 6 has been ADDED to the stack (3750Stack#-5)
Aug 15 14:03:42.666 UTC: %STACKMGR-4-SWITCH_ADDED: Switch 6 has been ADDED to the stack (3750Stack#-4)
3750Stack#
There we go, the switch is added back to the stack
3750Stack#
Aug 15 14:04:01.088 UTC: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 1 Switch 6 has changed to state UP
Aug 15 14:04:01.088 UTC: %STACKMGR-4-STACK_LINK_CHANGE: Stack Port 2 Switch 6 has changed to state UP
00:00:21: %STACKMGR-4-SWITCH_ADDED: Switch 1 has been ADDED to the stack (3750Stack#-6)
00:00:21: %STACKMGR-4-SWITCH_ADDED: Switch 2 has been ADDED to the stack (3750Stack#-6)
00:00:21: %STACKMGR-4-SWITCH_ADDED: Switch 3 has been ADDED to the stack (3750Stack#-6)
00:00:21: %STACKMGR-4-SWITCH_ADDED: Switch 4 has been ADDED to the stack (3750Stack#-6)
00:00:21: %STACKMGR-4-SWITCH_ADDED: Switch 5 has been ADDED to the stack (3750Stack#-6)
00:00:21: %STACKMGR-4-SWITCH_ADDED: Switch 6 has been ADDED to the stack (3750Stack#-6)
3750Stack#
Aug 15 14:04:05.056 UTC: %LINK-3-UPDOWN: Interface GigabitEthernet6/0/1, changed state to up
3750Stack#
Aug 15 14:04:06.323 UTC: %LINK-3-UPDOWN: Interface FastEthernet6/0/13, changed state to up
3750Stack#
Looks like the interfaces came back, nice!
Lets take a look at the switch stack and makre sure we are good:
3750Stack# sh switch
Current
Switch# Role Mac Address Priority State
——————————————————–
*1 Master 0018.186b.1234 15 Ready
2 Member 0018.18cd.1235 1 Ready
3 Member 0018.1834.1236 1 Ready
4 Member 0018.186b.1237 1 Ready
5 Member 0018.1879.1238 1 Ready
6 Member 001a.e324.1239 1 Ready
3750Stack#
All members are ready and we should be passing traffic again!
Cisco 3750 Stack Member Recovery
15 Monday Aug 2011