Had a small problem today that was interesting, but not unique. I have seen this problem before, but realized that maybe most people might not know how to fix it. I think the normal approach that people take is a reload of the device, not always a good thing to do on a production network in the middle of the day.
So the depicts is a VTY session that is unclearable via the normal clear command route. When you do a who, you see that someone is connected to VTY 0 for an extended period of time:
Switch1#who
Line User Host(s) Idle Location
1 vty 0 idle 6w4d HackerHost.virtua.com.br
* 2 vty 1 jfry idle 00:00:00 192.168.0.1
Interface User Mode Idle Peer Address
Switch1#
They are not logged in as there is no User listed, but you can see they have been idle for 6 weeks and 4 days.
So the first think that you would do would be to clear the session and do a who again to make sure they are gone.
Switch1#clear line vty 0 [confirm]y [OK] Switch1#who Line User Host(s) Idle Location 1 vty 0 idle 6w4d HackerHost.virtua.com.br * 2 vty 1 jfry idle 00:00:00 192.168.0.1 Interface User Mode Idle Peer Address Switch1#
Odd, they are still there. Well, maybe its VTY 1 and not 0, lets clear that one instead:
Switch1#clear line vty 1
% Not allowed to clear current line
Switch1#who
Line User Host(s) Idle Location
1 vty 0 idle 6w4d HackerHost.virtua.com.br
* 2 vty 1 jfry idle 00:00:00 192.168.0.1
Interface User Mode Idle Peer Address
Switch1#
Nope, that is my line – just like I thought. Ok, so now what?
Since we know this is a TCP based session to the switch, we should be able to gather some TCP information. This will include their remote ip (10.0.13.37) ports (56667) as well as all our information 192.168.100.254 port 23 as you can see via the show tcp vty 0 command.
Switch1#sh tcp vty 0
tty1, virtual tty from host HackerHost.HackerDom.com
Connection state is ESTAB, I/O status: 1, unread input bytes: 0
Local host: 192.168.100.254, Local port: 23
Foreign host: 10.0.13.37, Foreign port: 56667
Enqueued packets for retransmit: 1, input: 0 mis-ordered: 1 (0 bytes)
Event Timers (current time is 0xBD23926C4):
Timer Starts Wakeups Next
Retrans 1 0 0x0
TimeWait 0 0 0x0
AckHold 0 0 0x0
SendWnd 66409 66408 0xBD239AA5C
KeepAlive 0 0 0x0
GiveUp 0 0 0x0
PmtuAger 0 0 0x0
DeadWait 0 0 0x0
iss: 1865937424 snduna: 1865937425 sndnxt: 1865937426 sndwnd: 0
irs: 949507069 rcvnxt: 949507070 rcvwnd: 4128 delrcvwnd: 0
SRTT: 247 ms, RTTO: 3727 ms, RTV: 3480 ms, KRTT: 119264 ms
minRTT: 1980 ms, maxRTT: 1980 ms, ACK hold: 200 ms
Flags: passive open, higher precedence, retransmission timeout
timing out
Datagrams (max data segment is 536 bytes):
Rcvd: 30 (out of order: 1), with data: 0, total data bytes: 0
Sent: 3 (retransmit: 66407), with data: 1, total data bytes: 1
So, what does this tell us – just some triage information. At least before we do anything, we can actually gather information of the IP of the remote host. Now to fix the problem.
Welcome to Transmission Control Block commands – ie tcb
What is a TCB you might ask, well I found a good explanation here:
http://puck.nether.net/lists/cisco-nsp/6073.html
Internet Protocol Control Block Structure (inpcb{}): PCB’s are used
at the transport layer by TCP (and UDP) to hold various pieces of
information needed by TCP. They hold: TCP state information, IP address
information, port numbers, IP header prototype and options and a pointer
to the routing table entry for the destination address. PCB’s are
created for a given TCP based server when the server calls listen()
So what does this have to do with our problem of this hung session? Well, since it is a TCP session to the router, there is an associated TCB that we can clear!
To find out the TCB information for the session, we can use the command show tcp brief
Switch1#sh tcp brief
TCB Local Address Foreign Address (state)
02DC0AB0 192.168.253.254..22 192.168.0.1.51917 ESTAB
0155CBF0 192.168.253.254..23 HackerHost.hac..56667 ESTAB
Now that we have the associated TCB session – 0155CBF0 – and we can clear it.
Switch1#clear tcp tcb 0155CBF0 [confirm]y [OK] Switch1#
Now we can make sure the session is gone:
Switch1#who
Line User Host(s) Idle Location
* 2 vty 1 jfry idle 00:00:00 192.168.0.1
Interface User Mode Idle Peer Address
Switch1#
There you go, the session is cleared and the remote connection is now gone!
If you still see the session there, give it a moment to clear – but it will go away!
Mike McPhee said:
Wow, this is pretty slick. Of course, now that you have shared this trick, I am destined to never see this bug until I forget what you have shared. Either way, thank you!
Mike McPhee said:
Wow, this is pretty slick. Of course, now that you have shared this trick, I am destined to never see this bug until I forget what you have shared. Either way, thank you!
Strebzilla said:
Could this be prevented in the first place with the ‘service tcp-keepalives-in’ command?
Jeff said:
That command was there – the TCP session was just hung somehow.
Strebzilla said:
Ah okay. Good info.
Strebzilla said:
Could this be prevented in the first place with the ‘service tcp-keepalives-in’ command?
jjfry18018 said:
That command was there – the TCP session was just hung somehow.
Strebzilla said:
Ah okay. Good info.
igor_rodri said:
Really interesting post. This has been very good to learn some basics of connections to routers.
igor_rodri said:
Really interesting post. This has been very good to learn some basics of connections to routers.
Joe said:
I had this problem… tried the above, and it didn’t work. I got the same result as clear line vty
3850#who
Line User Host(s) Idle Location
2 vty 0 idle 00:00:03 192.168.1.2
* 7 vty 5 hoo1854 idle 00:00:00 10.251.122.226
Interface User Mode Idle Peer Address
BCHS-MAIN-MDF-3850#sh tcp brief
TCB Local Address Foreign Address (state)
3C4F0E20 192.168.1.1.23 192.168.1.2.41395 TIMEWAIT
3F3F79C0 192.168.1.1.23 192.168.1.2.42789 ESTAB
32BA141C 10.250.133.3.22 167.251.122.226.3691 ESTAB
10656:178 10.250.133.3.16113 167.251.108.58.51654 ESTAB
I can clear lines or clear tcp tcb and the connection on vty 0 keeps coming back… any ideas?
fryadmin said:
Have you tried to block the IP to make sure they are not re-establishing the session after you clear it?
Jatinder Bawa said:
very good blog thanks for the help dear
Balamurali Ganeshapandy said:
Wow… so good…thanks…
Girish Ts said:
Dear Sir, I am facing this issue and when I try to run the Clear tcp vty # or clear line # the particular session is not cleared also I cannot see the users on the show TCP brief command as well