Tags
crash, hidden, hidden commands, reload, test crash, watchdog
I was working on some stuff the other day in the lab and needed to demonstrate how to tell why a router reloaded. Did it reload by the reload command? SNMP string? or some random Error (not withstanding Solar Flares). Well, the first two are easy to demonstrate, but the third one is a little difficult. Or is it…
As we all know there are some hidden commands out there – a popular one is CSIM (Call Simulator). It is used to generate a call from a Cisco router to test a dial-peer and trunk configuration. If you are on Cisco router, you can type
(quick housekeeping)
Router/Switch Output
Commands
Notes
PSTNGateway.PA# csim start 6105551212
csim: called number = 6105551212, loop count = 1 ping count = 0
csim err csimDisconnected recvd DISC cid(1170604)
csim: loop = 1, failed = 1
csim: call attempted = 1, setup failed = 1, tone failed = 0
PSTNGateway.PA#
That command, CSIM, will tell the router to dial 6105551212 and place a call. They usually returned with a failed state, but I can assure you my phone rang (I changed the number in the post)
OK, I digress a bit there but I wanted to provide a hidden command that would not be destructive that you can look at.
So, how does one show a fellow engineer what a crash looks like, well TEST CRASH is the command. And yes, it is a hidden command that will crash the router.
The version of IOS I am running is 12.4(24)T3 for this lab and you can see that the router was reloaded.
Rack1R1# sh ver
Cisco IOS Software, 3800 Software (C3845-ENTSERVICESK9-M), Version 12.4(24)T3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Tue 23-Mar-10 11:22 by prod_rel_team
ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)
Rack1R1 uptime is 4 days, 5 hours, 15 minutes
System returned to ROM by reload at 16:41:07 UTC Mon Jun 27 2011
System image file is “flash:c3845-advipservicesk9-mz.124-17b.bin”
To show that this is a hidden command, I will enter TEST ?
Rack1R1# test ?
Virtual-Template Virtual Template interface
aaa AAA Authentication, Authorization and Accounting
appletalk APPLETALK diagnostic code
arp_retry Test ARP Retry
atm ATM test commands
cac test the l2 cac functionality
call Call test commands
cch323 cch323 tests
cef Cisco Express Forwarding test command
cfmpal CFM test commands
cns CNS agents
corrupt-nvram Corrupt NVRAM Magic
cpu-performance Performance Measurement
[— SNIP —]
Rack1R1#test
As you can see, the option for Crash is not listed, it won’t even auto complete if you tried.
So, how does one use this command? Simple, just type test crash and you will be prompted with a menu.
Rack1R1# test crash
WARNING: Command selections marked with ‘(crash router)’ will crash
router when issued. However a selection ‘C’ will need to
be issued IMMEDIATELY before these selections to enable them.
Type the number for the selected crash:
————————————–
1 (crash router) Bus Error, due to invalid address access
2 (crash router) Bus Error, due to parity error in Main memory
3 (crash router) Bus Error, due to parity error in I/O memory
4 (crash router) Address Error, due to fetching code from odd address
5 (crash router) Jump to zero
6 (crash router) Software forced crash
7 (crash router) Illegal read of address zero
8 (crash router) Divide by zero
9 (crash router) Corrupt memory
C Enable crash router selection marked with (crash router)
R (crash router) User enter read bus error address
U (crash router) User enter write bus error address
W (crash router) Software watchdog timeout (*** Watch Dog Timeout ***)
w (crash router) Process watchdog timeout (SYS-2-WATCHDOG)
d Disable crashinfo collection
e Enable crashinfo collection
i Display contents of current crashinfo flash file
m Write crashinfo on crashinfo RAM
n Change crashinfo flash file name
q Exit crash menu
s Save crashinfo to current crashinfo flash file
c Close current crashinfo flash file
t Write crashinfo on console TTY
x Exit crash menu
?
The ? mark is the prompt that you are placed at. Any option that has the word (crash router) show, will require you to enter C first and then that option.
To demonstrate, lets do a Watch Dog Timeout error and crash the router:
Rack1R1# test crash
WARNING: Command selections marked with ‘(crash router)’ will crash
router when issued. However a selection ‘C’ will need to
be issued IMMEDIATELY before these selections to enable them.
Type the number for the selected crash:
————————————–
1 (crash router) Bus Error, due to invalid address access
2 (crash router) Bus Error, due to parity error in Main memory
3 (crash router) Bus Error, due to parity error in I/O memory
4 (crash router) Address Error, due to fetching code from odd address
5 (crash router) Jump to zero
6 (crash router) Software forced crash
7 (crash router) Illegal read of address zero
8 (crash router) Divide by zero
9 (crash router) Corrupt memory
C Enable crash router selection marked with (crash router)
R (crash router) User enter read bus error address
U (crash router) User enter write bus error address
W (crash router) Software watchdog timeout (*** Watch Dog Timeout ***)
w (crash router) Process watchdog timeout (SYS-2-WATCHDOG)
d Disable crashinfo collection
e Enable crashinfo collection
i Display contents of current crashinfo flash file
m Write crashinfo on crashinfo RAM
n Change crashinfo flash file name
q Exit crash menu
s Save crashinfo to current crashinfo flash file
c Close current crashinfo flash file
t Write crashinfo on console TTY
x Exit crash menu
? C
I enter the C to confirm that I do want to Crash the router
Type the number for the selected crash:
————————————–
1 (crash router) Bus Error, due to invalid address access
2 (crash router) Bus Error, due to parity error in Main memory
3 (crash router) Bus Error, due to parity error in I/O memory
4 (crash router) Address Error, due to fetching code from odd address
5 (crash router) Jump to zero
6 (crash router) Software forced crash
7 (crash router) Illegal read of address zero
8 (crash router) Divide by zero
9 (crash router) Corrupt memory
C Enable crash router selection marked with (crash router)
R (crash router) User enter read bus error address
U (crash router) User enter write bus error address
W (crash router) Software watchdog timeout (*** Watch Dog Timeout ***)
w (crash router) Process watchdog timeout (SYS-2-WATCHDOG)
d Disable crashinfo collection
e Enable crashinfo collection
i Display contents of current crashinfo flash file
m Write crashinfo on crashinfo RAM
n Change crashinfo flash file name
q Exit crash menu
s Save crashinfo to current crashinfo flash file
c Close current crashinfo flash file
t Write crashinfo on console TTY
x Exit crash menu
? W
As soon as I Enter the W (hit return), the router causes a watchdog timeout and the router crashes and reloads.
Causing a software watchdog timeout
*** Watch Dog Timeout NMI ***
PC = 0x60e00e5c, SP = 0x68cc4608
CPU Registers:
ZERO = 0x00000000, AT = 0x66d70000, V0 = 0x00010000, V1 = 0x43190eb3
A0 = 0x000003e8, A1 = 0x000003e8, A2 = 0x0000028a, A3 = 0x00000001
T0 = 0x00080000, T1 = 0x000186a0, T2 = 0x00000000, T3 = 0x00002a21
T4 = 0x4318e492, T5 = 0x00000000, T6 = 0x00000000, T7 = 0x00000000
S0 = 0x00000b90, S1 = 0x0000000b, S2 = 0x00000001, S3 = 0x66460000
S4 = 0x00000017, S5 = 0x68cc4638, S6 = 0x00000000, S7 = 0x63fe0000
T8 = 0x00000003, T9 = 0x000000b2, K0 = <N/A>, K1 = <N/A>
GP = 0x6685f780, SP = 0x68cc4608, S8 = 0x63fe0000, RA = 0x60e00cec
SREG = 0x3448ff04, BADVADDR = 0xcf9bfef3, CAUSE = 0x00008800
EPC = 0x60e00e5c, ERR_EPC = 0x60e00e5c
c3845 platform with 262144 Kbytes of main memory
Main memory is configured to 72/0(dimm 0/1) bit mode with ECC enabled
Now if I do a show ver, it shows System returned to ROM by watchdog timer expired. No mention that it was done via a command, so almost no way to tell what caused the router to reload.
Rack1R1# sh ver
Cisco IOS Software, 3800 Software (C3845-ENTSERVICESK9-M), Version 12.4(24)T3, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2010 by Cisco Systems, Inc.
Compiled Tue 23-Mar-10 11:22 by prod_rel_team
ROM: System Bootstrap, Version 12.4(13r)T, RELEASE SOFTWARE (fc1)
Rack1R1 uptime is 3 minutes
System returned to ROM by watchdog timer expired
System image file is “flash:c3845-entservicesk9-mz.124-24.T3.bin”
What would you use this command for beside training and showing people what different crashes look like, do not know. But, just wanted to share with you that it is there.
Also, I do not know if your command will be logged anywhere, so there may be no way of knowing if it was entered. Since this router was in an isolated lab, I do not have the ability to run the logging to a Syslog or check an ACS server for commands executed. One would assume that the ACS server would have a log of the command being run, so keep that in mind.
…but… if you need to crash a router for some reason, this might be better then doing a debug all 🙂
Curious Yellow said:
Some of my work is doing enterprise network testing. The most disruptive tests in terms of outage time are when switches are reloaded. It would be interesting to see if crashing a switch produces the same results. Thanks for sharing this!
Oleg said:
The crashinfo will clearly say it was a test crash.