Had a small problem today that was interesting, but not unique. I have seen this problem before, but realized that maybe most people might not know how to fix it. I think the normal approach that people take is a reload of the device, not always a good thing to do on a production network in the middle of the day.
So the depicts is a VTY session that is unclearable via the normal clear command route. When you do a who, you see that someone is connected to VTY 0 for an extended period of time:
Switch1#who
Line User Host(s) Idle Location
1 vty 0 idle 6w4d HackerHost.virtua.com.br
* 2 vty 1 jfry idle 00:00:00 192.168.0.1
Interface User Mode Idle Peer Address
Switch1#
They are not logged in as there is no User listed, but you can see they have been idle for 6 weeks and 4 days.
So the first think that you would do would be to clear the session and do a who again to make sure they are gone.
Switch1#clear line vty 0 [confirm]y [OK] Switch1#who Line User Host(s) Idle Location 1 vty 0 idle 6w4d HackerHost.virtua.com.br * 2 vty 1 jfry idle 00:00:00 192.168.0.1 Interface User Mode Idle Peer Address Switch1#
Odd, they are still there. Well, maybe its VTY 1 and not 0, lets clear that one instead:
Switch1#clear line vty 1
% Not allowed to clear current line
Switch1#who
Line User Host(s) Idle Location
1 vty 0 idle 6w4d HackerHost.virtua.com.br
* 2 vty 1 jfry idle 00:00:00 192.168.0.1
Interface User Mode Idle Peer Address
Switch1#
Nope, that is my line – just like I thought. Ok, so now what?
Since we know this is a TCP based session to the switch, we should be able to gather some TCP information. This will include their remote ip (10.0.13.37) ports (56667) as well as all our information 192.168.100.254 port 23 as you can see via the show tcp vty 0 command.
Switch1#sh tcp vty 0
tty1, virtual tty from host HackerHost.HackerDom.com
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Local host: 192.168.100.254, Local port: 23
Foreign host: 10.0.13.37, Foreign port: 56667
Enqueued packets for retransmit: 1, input: 0 mis-ordered: 1 (0 bytes)
Event Timers (current time is 0xBD23926C4):
Timer Starts Wakeups Next
Retrans 1 0 0x0
TimeWait 0 0 0x0
AckHold 0 0 0x0
SendWnd 66409 66408 0xBD239AA5C
KeepAlive 0 0 0x0
GiveUp 0 0 0x0
PmtuAger 0 0 0x0
DeadWait 0 0 0x0
iss: 1865937424 snduna: 1865937425 sndnxt: 1865937426 sndwnd: 0
irs: 949507069 rcvnxt: 949507070 rcvwnd: 4128 delrcvwnd: 0
SRTT: 247 ms, RTTO: 3727 ms, RTV: 3480 ms, KRTT: 119264 ms
minRTT: 1980 ms, maxRTT: 1980 ms, ACK hold: 200 ms
Flags: passive open, higher precedence, retransmission timeout
timing out
Datagrams (max data segment is 536 bytes):
Rcvd: 30 (out of order: 1), with data: 0, total data bytes: 0
Sent: 3 (retransmit: 66407), with data: 1, total data bytes: 1
So, what does this tell us – just some triage information. At least before we do anything, we can actually gather information of the IP of the remote host. Now to fix the problem.
Welcome to Transmission Control Block commands – ie tcb
What is a TCB you might ask, well I found a good explanation here:
http://puck.nether.net/lists/cisco-nsp/6073.html
Internet Protocol Control Block Structure (inpcb{}): PCB’s are used
at the transport layer by TCP (and UDP) to hold various pieces of
information needed by TCP. They hold: TCP state information, IP address
information, port numbers, IP header prototype and options and a pointer
to the routing table entry for the destination address. PCB’s are
created for a given TCP based server when the server calls listen()
So what does this have to do with our problem of this hung session? Well, since it is a TCP session to the router, there is an associated TCB that we can clear!
To find out the TCB information for the session, we can use the command show tcp brief
Switch1#sh tcp brief
TCB Local Address Foreign Address (state)
02DC0AB0 192.168.253.254..22 192.168.0.1.51917 ESTAB
0155CBF0 192.168.253.254..23 HackerHost.hac..56667 ESTAB
Now that we have the associated TCB session – 0155CBF0 – and we can clear it.
Switch1#clear tcp tcb 0155CBF0 [confirm]y [OK] Switch1#
Now we can make sure the session is gone:
Switch1#who
Line User Host(s) Idle Location
* 2 vty 1 jfry idle 00:00:00 192.168.0.1
Interface User Mode Idle Peer Address
Switch1#
There you go, the session is cleared and the remote connection is now gone!
If you still see the session there, give it a moment to clear – but it will go away!