/images/logo.png
A Network Blog by a Network Engineer

Gestalt IT’s first datacenter-focused Field Day event

Well I got the official notice today that I will be attending a Gestalt IT Tech Field Day event on February 10th and 11th in 2011 in San Jose, CA.  I feel honored as well as humbled to be invited to this event.  It is a great chance to meet new people, learn from their experiences, listen to their war stories, as well as the chance to see what the vendors have to say.
What is Tech Field Day, you ask?  Well – to explain it best I am going to quote it from Gestalt’s website:

New Cisco Compact Switches!

Well, just the other day Cisco announced some new Compact switches that should replace the Cisco 2940(yeah, I know that they were EOL and replaced with 2960-8)  and 3560-8 series switches.  The product information can be found here – Link.  The new switches are the Cisco 3560-C and 2960-C series.
These really have my attention from an Enterprise perspective.  I cannot tell you how many conference rooms we have that have either the 2940 or the 3560-8 switches mounted under the table.  Those switches work great, except for the fact that we need to also run a power-cord to them.  I cannot tell you how many times our NOC has contacted us with regards to a switch being reset due to power-on and it turning out someone kicked the power cord.  We run the Cisco 3560-8 at places where we need PoE ports  for phones and such, and the 2940/2960 series are used where no PoE is required.
So what is so intriguing to me about the new 2960-C switches? Well, the biggest thing that I noticed is that you can now get switches that will be powered via PoE and  PoE+ ports instead of an external power cord. This is feature is called PD PSE -Powered Devices (PD) and Power Sourcing Equipment (PSE) and is in available it WS-C2960CPD series of switches.  The way that it works is that you connect the dedicated copper uplinks on the 2960CPD switch to a PoE or PoE+ capable switch, the switch then senses the device requesting the power and provides the power necessary to the device.  This is the 802.3af (PoE) and 802.3at (PoE+) standards and works just like an IP phone does connected to a switch. One thing to note on the PS/PSE 2960C is that you can also get an external power supply to power the device.  This is a nice feature if you do not have PoE capable devices today but plan to deploy them at a later date. Below is an image taken from Cisco’s website that shows the PD/PSE switches on the left and the non PD/PSE switches below

Sorry…

With the holidays and such, and now that I am sick – I have ignored this blog for a little bit. I promise I will get something good up soon. Need to focus on those VRF labs and get them published soon.
I promise…

Cisco ASR and LACP

I am currently working on a new site deployment utilizing some Cisco ASR1002 routers.  When we typically design a location we cross-connect the two WAN routers via a cross-over cable; this allows us to have the capability to continue to transit traffic over the connected WAN routers and WAN links if need to take down the Core at the site for maintenance.
If you look at the image below you will see three Sites – A – B – C.  Traffic that is destined to Site B from Site A will transit the connected WAN link.  So what happens if that WAN link goes down, it will transit through Site C instead.  By having the WAN routers connected via a cross-over cable, the traffic will stay at the end and not transverse the internal LAN of the site.

Ok, the background information is now out of the way. Lets continue.
Typically we use a single gigabit ethernet cross-over cable between the routers, due to hardware limitations. Well, with the ASR series we now have more interfaces that we can use for this connectivity.  What I have also found is that the ASR and IOS-XE support LACP and etherchannels.  This is a wonderful thing as it actually solves some of the problems of a single cross-over cable between routers.
So, how does one configure LACP on the ASR 1002?  It is done the same way as you would on the Nexus for L3 etherchannel. Below are the configuration commands as well as the ways to check.
For this example, we are going to create Port-Channel 12 on the router and assign interfaces G0/0/2 and G0/0/3 to the channel-group
Router#conf t
Router(config)#interface GigabitEthernet0/0/2
Router(config-if)# channel-group 12 mode active
Router(config-if)# no shut
Router(config-if)# exit
Router(config)#interface GigabitEthernet0/0/3
Router(config-if)# channel-group 12 mode active
Router(config-if)# no shut
Router(config-if)# exit
Now we can configure an IP address on the port-channel interface:
Router(config)#interface Port-channel12
Router(config-if)# ip address 10.1.1.1 255.255.255.252
T0 check the port-channel:
Router#sh int port-channel 12
Port-channel12 is up, line protocol is up
Hardware is GEChannel, address is c471.fe0c.55cb (bia c471.fe0c.55cb)
Internet address is 10.98.255.9/30
MTU 1500 bytes, BW 2000000 Kbit/sec, DLY 10 usec,
You can also look at the LACP information:
Router#sh lacp 12 neighbor
Flags:  S – Device is requesting Slow LACPDUs
F – Device is requesting Fast LACPDUs
A – Device is in Active mode       P – Device is in Passive mode
Channel group 12 neighbors
Partner’s information:
Partner Partner   LACP Partner  Partner   Partner  Partner     Partner
Port      Flags   State     Port Priority Admin Key Oper Key Port Number Port State
Gi0/0/2   SA      bndl      32768         0x0       0xC      0x3         0x3D
Gi0/0/3   SA      bndl      32768         0x0       0xC      0x4         0x3D
Router#
Now lets ping to test:
Router#p 10.1.1.2 t 1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 10.1.1.1, timeout is 1 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Router#
If you noticed I typed t 1 – that means use a 1 second timeout.  I did that in case, for some reason, it does not PING successfully I only have to wait half the normal time of 2 seconds per ping.  This might not seem like a big deal, but if you use this on some LAB test – you can save some time and frustration. 🙂

Clearing hung TCP session on a Cisco router

While going through some of my old notes I came across some notes from a few years ago on hung TCP sessions on some of our routers. These hung sessions where due to the latency differences on a Multi-link T1 connection that was in place.  Below are the notes (generalized IP and Names) on how to detect and reset these connections.It is my hope for you to never have to use these commands, but since there is little out there on them I figured I would share.

Type 7 password decryption via IOS router (? bonus)

I have seen a few posts out there about this as well as links to sites that will decrypt a Cisco type 7 password.  Since I am trying to share what I know I figured I might as well throw my hat into the ring and include this information here.
If you have a type 7 enable password such as : enable password 7 00331A0A087D071F012F7F5B1B0F0C011754 and want to decrpyt it, all you need is another router that you can get to enable (priv 15) access on and configuration mode.
Enter configuration mode and create a key-chain. For the key-string, tell it type 7 and cut-and-past the encrypted key
Rack1R6(config)#key chain DecrpytThis
Rack1R6(config-keychain)#key 1
Rack1R6(config-keychain-key)#key-string 7 00331A0A087D071F012F7F5B1B0F0C011754
Then all you have to do from a command prompt is issue show key chain
Rack1R6#sh key chain
Key-chain DecrpytThis:
key 1 — text “WillFlynnSurvive?
accept lifetime (always valid) – (always valid) [valid now]
send lifetime (always valid) – (always valid) [valid now]
Rack1R6#
As you can see here, the password is WillFlynnSurvive?
This works for any Type 7 password, including users.  Below I have done a show run | inc user and they added Key 2 with he user type 7 password.
Rack1R6#sh run | inc user
username Flynn privilege 15 password 7 0478071303245F5D
Rack1R6#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Rack1R6(config)#key chain DecryptThis
Rack1R6(config-keychain)#key 2
Rack1R6(config-keychain-key)#key-string 7 0478071303245F5D
Rack1R6(config-keychain-key)#^Z
Rack1R6#sh key chain
Key-chain DecrpytThis:
key 1 — text “WillFlynnSurvive?”
accept lifetime (always valid) – (always valid) [valid now]
send lifetime (always valid) – (always valid) [valid now]
Key-chain DecryptThis:
key 2 — text “Cluless
accept lifetime (always valid) – (always valid) [valid now]
send lifetime (always valid) – (always valid) [valid now]
Rack1R6#
As you can see, the password for Key 2 ( the user one we just added ) is Cluless
 
Now, I am not sure if you noticed but for the first example the password is WillFlynnSurvive? – yes, that is a question mark in the password.  If you want to use a question mark as part of your password, you need to enter ctrl-v prior to typing the question mark in the password or the IOS will think you are looking for help. Old dirty CCIE lab prep tricks there.
 
 

Cisco Embedded Event Manager and its usefullness for TAC

A few years ago I had a TAC case open with Cisco.  The case was centered around an unusually high CPU condition on a Cisco 3800ISR series router.  This router was only routing traffic between the Ethernet interfaces, doing some SNAT (stateful NAT), and basic BGP routing – so the CPU should not have been in the 90%+ range.  Well, TAC wanted me to run some commands every 60 minutes 24×7 for a few days to see what was going on.  Now we all now that that is not an easy thing to do, so what I decided to do was grab an existing EEM script from Cisco’s website and modify it to provide all the information that the TAC engineer wanted.
Needless to say, the case was solved about 2 days later when we upgraded the code per their suggestion because of the identification of the bug. 🙂  I am guessing that having a router send you the logs and command output every hour either provided them the information they needed, or more likely, the engineer realized that  – well, you know.
So, for this post I have decided to post the EEM configuration as well as the EEM script in case you ever need to do something similar.  If I can save you some time, I am all for that.  I still use this script today – albeit in a different manner.  I now use it to get “Daily Health Checks” from the routers in the field.  Every day at noon they send me standard output commands so that we can check on them without having to log into each and every one of them.
So, without further typing – here is the script broken down: (to download this script, you can click here – HealthCheck_BLOG.tcl
Router Configuration:
First thing we will need to do is configure the Event Manager variables on the router:
The variables that we will use are:

Cisco IP Router DHCP Server and Client

Configuring DHCP on routers is not something that is normally
done in a production network.  Typically in a production network
DHCP servers are either Windows, Linux, or some other vendors
implementation.  Occasionally you will need to configure a Cisco
device to provided DHCP for initial purposes - say a new location
build-out, a lab network, or even just perhaps for your home
networking.
For this post, I will show you how to:
 - Configure your route to be a DHCP Server
 - Configure your router to be a client
 - Configure DHCP Reservations for a specific client
 - Release / Renew your DHCP lease
 - Configure DHCP options (150 for example)
Here is the network information we are going to use:
 Network   - 192.168.1.0/24
 Gateway   - 192.168.1.1
 DNS       - 192.168.1.2 and 192.168.1.3
 Domain    - FryGuyLab.Net
 Cisco UCM - 192.168.1.5 and 192.168.1.6
 WINS      - 192.168.1.2
 Node Type - Hybrid

So, first lets setup the DHCP Server router (R2)
Rack1R2#conf t
Enter configuration commands, one per line.  End with CNTL/Z.
Rack1R2(config)#ip dhcp pool DHCP_POOL_1
Rack1R2(dhcp-config)#network 192.168.1.0 /24
Rack1R2(dhcp-config)#default-router 192.168.1.1
Rack1R2(dhcp-config)#dns-server 192.168.1.2 192.168.1.3
Rack1R2(dhcp-config)#domain-name FryGuyBlog.Net
Rack1R2(dhcp-config)#option 150 ip 192.168.1.5 192.168.1.6
Rack1R2(dhcp-config)#netbios-node-type h-node
Rack1R2(dhcp-config)#netbios-name-server 192.168.1.2
Rack1R2(dhcp-config)#exit

Now, don't forget to exclude addresses if necessary. 
Here will exclude 192.168.1.1 - 192.168.1.10
Rack1R2(config)#ip dhcp excluded-address 192.168.1.1 192.168.1.10
Rack1R2(config)#
Now we can go over to R1 and configure DHCP and then enable
the G0/0 interface on the router.
Rack1R1#show run int g0/0
Building configuration...
Current configuration : 103 bytes
!
interface GigabitEthernet0/0
 no ip address
 shutdown
 duplex auto
 speed auto
 media-type rj45
end

Now we can configure the interface:
Rack1R1(config)#int g0/0
Rack1R1(config-if)#ip address dhcp
Rack1R1(config-if)#no shut
Rack1R1(config-if)#exit
Rack1R1(config)#exit
Rack1R1#
*Nov 15 18:37:55.302: %SYS-5-CONFIG_I: Configured from console
 by console
*Nov 15 18:37:56.102: %LINK-5-CHANGED: Interface GigabitEthernet0/0,
changed state to reset
Rack1R1#
*Nov 15 18:37:58.866: %LINK-3-UPDOWN: Interface GigabitEthernet0/0,
changed state to up
*Nov 15 18:37:59.866: %LINEPROTO-5-UPDOWN: Line protocol on Interface
GigabitEthernet0/0, changed state to up
Rack1R1#
*Nov 15 18:38:05.490: %DHCP-6-ADDRESS_ASSIGN: Interface GigabitEthernet0/0
 assigned DHCP address 192.168.1.13, mask 255.255.255.0, hostname Rack1R1
As you can see, we received a DHCP address of 192.168.1.13/24 from the
server. To check the DHCP Lease information on the client, issue the
show dhcp lease command
Rack1R1#show dhcp lease
Temp IP addr: 192.168.1.13  for peer on Interface: GigabitEthernet0/0
Temp  sub net mask: 255.255.255.0
 DHCP Lease server: 192.168.1.1, state: 3 Bound
 DHCP transaction id: 859
 Lease: 86400 secs,  Renewal: 43200 secs,  Rebind: 75600 secs
Temp default-gateway addr: 192.168.1.1
 Next timer fires after: 11:58:27
 Retry count: 0   Client-ID: cisco-001c.f6e6.6a90-Gi0/0
 Client-ID hex dump: 636973636F2D303031632E663665362E
 366139302D4769302F30
 Hostname: Rack1R1

Now to check the connectivity, lets PING the other router:
Rack1R1#ping 192.168.1.1

Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms
Rack1R1#
To check the active leases on the server, issue the sh ip dhcp
binding command:
Rack1R2#sh ip dhcp binding
Bindings from all pools not associated with VRF:
IP address          Client-ID/              Lease expiration        Type
 Hardware address/
 User name
192.168.1.13        0063.6973.636f.2d30.    Nov 16 2010 06:25 PM    Automatic
                    3031.632e.6636.6536.
                    2e36.6139.302d.4769.
                    302f.30
Rack1R2#
Now for reservations. 
We will configure the DHCP server (R2) to assign 192.168.1.254/24
 to R3's F0/0 interface.
On the DHCP Server (R2) - configure a special DHCP pool just
for that client:
Rack1R2#conf t
Rack1R2(config)#ip dhcp pool R3
Rack1R2(config)#host 192.168.1.254 255.255.255.0
Rack1R2(config)#client-identifier 0100.0bfd.5661.e0

To find out the Cleint Identifier to use, on R3 look at the Fa0/0
MAC address and then pre-pend 01 to the mac address. 
(see IETF RFC1700 Hardware table below)
Rack1R3#sh int f0/0
FastEthernet0/0 is up, line protocol is up
 Hardware is AmdFE, address is 000b.fd56.61e0
On R3, configure the Fa0/0 interface as follows:
Rack1R3(config)#interface FastEthernet0/0
Rack1R3(config)#ip address dhcp client-id FastEthernet0/0
Rack1R3(config)#no shut
Rack1R3(config)#exit
Rack1R3#
You will then see:
*Mar 13 23:29:28.142: %DHCP-6-ADDRESS_ASSIGN: Interface FastEthernet0/0
assigned DHCP address 192.168.1.254, mask 255.255.255.0, hostname Rack1R3
And on R2 you will see under the DHCP binding:
Bindings from all pools not associated with VRF:
IP address          Client-ID/              Lease expiration        Type
 Hardware address/
 User name
192.168.1.13        0063.6973.636f.2d30.    Nov 16 2010 06:25 PM    Automatic
 3031.632e.6636.6536.
 2e36.6139.302d.4769.
 302f.30
192.168.1.254       0100.0bfd.5661.e0       Infinite                Manual
Rack1R2#

IETF Hardware Table from RFC1700
http://www.ietf.org/rfc/rfc1700.txt
Number Hardware Type (hrd)                           References
------ -----------------------------------           ----------
 1 Ethernet (10Mb)                                    [JBP]
 2 Experimental Ethernet (3Mb)                        [JBP]
 3 Amateur Radio AX.25                                [PXK]
 4 Proteon ProNET Token Ring                          [JBP]
 5 Chaos                                              [GXP]
 6 IEEE 802 Networks                                  [JBP]
 7 ARCNET                                             [JBP]
 8 Hyperchannel                                       [JBP]
 9 Lanstar                                             [TU]
 10 Autonet Short Address                             [MXB1]
 11 LocalTalk                                         [JKR1]
 12 LocalNet (IBM PCNet or SYTEK LocalNET)             [JXM]
 13 Ultra link                                        [RXD2]
 14 SMDS                                              [GXC1]
 15 Frame Relay                                        [AGM]
 16 Asynchronous Transmission Mode (ATM)              [JXB2]
 17 HDLC                                               [JBP]
 18 Fibre Channel                            [Yakov Rekhter]
 19 Asynchronous Transmission Mode (ATM)      [Mark Laubach]
 20 Serial Line                                        [JBP]
 21 Asynchronous Transmission Mode (ATM)              [MXB1]
Now, if you need to force release an IP address on a client, you can issue the command:
Rack1R1# release dhcp g0/0

If you need to renew the address, you can use the command
Rack1R1# renew dhcp g0/0

IP Expert CCIE SP Class last week

Well, finally had a chance to take a Cisco CCIE Service Provider class last week – and I am very glad I did.  I know I was not ready for the lab when I took it the other month, and it was my hope that this class would help me fill in the pieces.  I have to say, I feel much better about what I know and what I need to brush up on.
Many of the more complex topics are now much easier to understand and configure. The way that Marko explained and diagramed the concepts was wonderful.  He knows his stuff, and what really helped is that he has some real-world experience in deploying these technologies.  He is able to articulate them into a real-world deployment, so understanding the concept (Multicast for example) makes much more sense when you can relate it to something you understand.  It was a loooooong week of class with few breaks.
Each day was about 12 hours long – some lecture some labbing.  The lecture was not “power-point waterboarding” but more of interactive communication and understanding.  He would take a topic, explain it in lecture, and then actually configure it on the screen.  We broke configs, tweaked configs, and even reinterpreted things in order to see if we could get them work in alternate ways (when the possibility existed).  After he would show us the topic and configs, it was then up to us to do our labs for a few hours.
At first these labs looked easy, but you quikly realized that they where not.  You had to really read and understand what they where asking.  What is nice about that is that is how the real test is rumored to be.  At first it might look easy, but you need to understand it to realize what they are asking.  What was also nice about this was that it really helped you learn and understand a topic and topology.  You needed to understand the interactions between the technologies as to not break something you have already done.
The final two labs you get to do are 8-hours (yeah right) labs. These labs are close to what you can expect on the real lab  in the ways of complexity and such, so they are a good guage at your understanding of the topics.  I did not finish my first lab during class and, unfortunately, had to skip out on the second one (Friday).  I would have loved to spend the day there doing the lab but instead chose to get home to family.  Since I have the lab workbook from class, I will so do the second lab one this week and see what happens.
All in all I am very glad I was able to attend the class, I learned alot, but best of all – met some great people who also have the same goal to pass the SP lab.