Archives for posts with tag: QOS

DMVPN | Phase 3 | IPsec | VRF | Per-Tunnel QoS

Covering the configuration, confirmation and troubleshooting of DMVPN Phase 3 with IPsec and Per-Tunnel QoS. The main reason behind this is to help out my friend who is implementing a production design similar to this.

Below is the high level topology used for this lab. Its a fairly simple diagram but gives you the idea, the devil is in the detail (config)


For the dynamic routing protocol, I have chosen EIGRP, you could use BGP if you want a hyper-scale design. Phase 3 DMVPN is chosen simply to enable spoke-to-spoke communication and maintain a default route to the spokes. Phase 3 allows this by using redirect messages / shortcut routing in NHRP.

The purpose of this blog is not to explain DMVPN, there are plenty of resources for that.

Here is the config for the HUB.

vrf definition internet
address-family ipv4
ip route vrf internet [internet next-hop IP]
interface Tunnel0
ip address
no ip redirects
ip mtu 1400
ip nhrp authentication cisco
ip nhrp map multicast dynamic
ip nhrp map group Spoke-5 service-policy output DMVPN_5000kbps
ip nhrp map group Spoke-1 service-policy output DMVPN_1500kbps
ip nhrp map group Spoke-2 service-policy output DMVPN_10000kbps
ip nhrp network-id 123
ip nhrp redirect
ip tcp adjust-mss 1360
qos pre-classify
tunnel source Ethernet0/0
tunnel mode gre multipoint
tunnel key 123
tunnel vrf internet
tunnel protection ipsec profile DMVPN
router eigrp DMVPN
address-family ipv4 unicast autonomous-system 123
af-interface Tunnel0
no split-horizon
topology base
crypto keyring dmvpn vrf internet
pre-shared-key address key topsecret!
crypto isakmp policy 100
encr aes 256
authentication pre-share
group 5
crypto ipsec transform-set DMVPN esp-aes esp-sha-hmac
crypto ipsec profile DMVPN
set transform-set DMVPN


QOS – On the hub tunnel interface you can set the QoS policies to map to a group name, then on the spoke you can set the command to have the tunnel subscribe to a particular group.

The QoS polices need to be setup in a specific way with parent and child policies, this way you can be specific in which spoke gets which shaper settings. Create a policy with your required settings see example below.

policy-map DMVPN
class icmp
police 8000
class telnet
set precedence 4
class www
police 80000
set precedence 6


Then create your shaping policy and attach the child policy like below.

policy-map DMVPN_5000kbps
class class-default
shape average 5000000
service-policy DMVPN


Front Door VRF – The reason for this is to allow a default route for internet access and also allow a default route for LAN traffic. The Internet facing interface is in its own VRF and routes from this are not part of the global routing table.  We simply tell the tunnel to use the VRF for its NBMA routing with a simple command “tunnel vrf [name of VRF]“. Once this is enabled we can have a default route for LAN traffic via the tunnel and a default route for internet traffic.

A great write up by Denise Fishburn can be found here.

IPsec – Using the front door VRF, we have to adjust the crypto keyring slightly to apply the VRF to the key and address.

Confirmation – From the HUB one command can tell us all we need to know from a DMVPN perspective. It shows the tunnel interface and number of peers,  DMVPN peers and the QoS profiles they are subscribed to and shows the IPsec details.

HUB#show dmvpn detail


Interface Tunnel0 is up/up, Addr. is, VRF “”
Tunnel Src./Dest. addr:, Tunnel VRF “internet”
Protocol/Transport: “multi-GRE/IP”, Protect “DMVPN”
Interface State Control: Disabled
nhrp event-publisher : Disabled
Type:Hub, Total NBMA Peers (v4/v6): 3
# Ent Peer NBMA Addr Peer Tunnel Add State UpDn Tm Attrb Target Network
—– ————— ————— —– ——– —– —————–
1 UP 02:54:32 D
NHRP group: Spoke-1
Output QoS service-policy applied: DMVPN_1500kbps
1 UP 02:54:32 D
NHRP group: Spoke-2
Output QoS service-policy applied: DMVPN_100kbps
1 UP 02:54:32 D
NHRP group: Spoke-5
Output QoS service-policy applied: DMVPN_5000kbps
Crypto Session Details:
Interface: Tunnel0
Session: [0xF384C6B8]
IKEv1 SA: local remote Active
Capabilities:(none) connid:1005 lifetime:21:05:26
Crypto Session Status: UP-ACTIVE
fvrf: internet, Phase1_id:
IPSEC FLOW: permit 47 host host
Active SAs: 2, origin: crypto map
Inbound: #pkts dec’ed 2635 drop 0 life (KB/Sec) 4150500/3474
Outbound: #pkts enc’ed 2655 drop 0 life (KB/Sec) 4150500/3474
Outbound SPI : 0x B4E148D, transform : esp-aes esp-sha-hmac
Socket State: Open


Here is the config for one of the spokes. The crypto config is identical to the HUB so omitted for brevity.

vrf definition internet
address-family ipv4
ip route vrf internet [internet next-hop ip]
interface Tunnel0
ip address
no ip redirects
ip mtu 1400
ip nhrp authentication cisco
ip nhrp group Spoke-1
ip nhrp map [NBMA – HUB]
ip nhrp map multicast [NBMA – HUB]
ip nhrp network-id 123
ip nhrp nhs
ip nhrp shortcut
ip tcp adjust-mss 1360
qos pre-classify
tunnel source Ethernet0/0
tunnel mode gre multipoint
tunnel key 123
tunnel vrf internet
tunnel protection ipsec profile DMVPN
router eigrp DMVPN
address-family ipv4 unicast autonomous-system 123
topology base
eigrp stub connected summary


Confirmation on the spoke is similar to the hub using the “show dmvpn detail” command lists everything you need to know from a DMVPN and IPsec perspective.

Just to confirm everything is OK lets run some commands from the spoke

Spoke1# ping sou lo 2
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to, timeout is 2 seconds:
Packet sent with a source address of
Success rate is 100 percent (5/5), round-trip min/avg/max = 9/15/22 ms
Spoke1#traceroute sou lo 2
Type escape sequence to abort.
Tracing the route to
VRF info: (vrf in name/id, vrf out name/id)
1 13 msec 9 msec 8 msec
Spoke1#sh ip nhrp shortcut via
Tunnel0 created 00:00:26, expire 00:01:13
Type: dynamic, Flags: router used rib
NBMA address:
Spoke1#sh ip route nhrp
H [250/1] via, 00:00:31, Tunnel0
Spoke1#sh ip route eigrp
Gateway of last resort is to network
D* [90/76800640] via, 03:07:27, Tunnel0
Spoke1#sh ip eigrp ne
EIGRP-IPv4 VR(DMVPN) Address-Family Neighbors for AS(123)
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
0 Tu0 13 03:08:04 23 1398 0 30
Spoke1#show ip cef
nexthop Tunnel0


We can see spoke to spoke ping is ok, traceroute is direct to spoke, NHRP has a shortcut, the routing table has an NHRP route, and a default route (Global Routing Table) with one EIGRP neighbor. All looks good.

Troubleshoot – The best command to use is the “debug dmvpn detail all” this will debug crypto and nhrp.  It can be a lot to look at so perhaps you might want to use the”debug crypto isakmp“or “debug crypto ipsec” individually, depends on which part is failing.



It’s worth making some notes on this subject to clear up a few misconceptions I found online.

Firstly I setup a lab to confirm what i’m about to say, the reason we use this command is to allow QoS to correctly classify or view the packets based on the original header, if the packet is encapsulated  it’s  treated the same as any other encapsulated packet. The original header and its QoS value is now unknown to the forwarding device.

So by enabling this command we can apply the classification before the encapsulation or tunneling happens.

Apply the command “qos pre-classify” to a tunnel interface, a crypto-map or a virtual tunnel interface.

Classification based on layer3 and layer4 information is the exact reason we would consider using this feature, classification based on TOS or DSCP values do not need to use this feature, that’s because of TOS byte preservation inherently built into IPSEC.

Once you’ve added the pre-classify command, apply a service-policy to the physical interface outbound, then all IP packets will be classified pre-encapsulation on any tunnels egressing that physical interface. In other words you will see hits on the policy-map individual classes.

A really great resource I found online is this QoS values calculator check it out………….


The last item on our list is the service-policy, this is the part where we apply our policy-map to an interface and specify the direction for the policy.

To help you visualise how the command is used and which direction see the picture below.


QOS is being applied to traffic coming from inside our network going outbound to the WAN.

Your interface will look something like this.

interface FastEthernet0/0
 description WAN_interface
 bandwidth 20000
 ip address
 ip nbar protocol-discovery
 service-policy output QOS_Egress

To confirm the QOS is working you would use the command “show policy-map interface fastethernet 0/0” this will give you a whole bunch of information, see below for an example of how this might look.

Service-policy output: QOS_Egress

queue stats for all priority classes:

queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 331150787/112001702019

Class-map: voice_traffic (match-all)
266055005 packets, 57655712122 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match: ip dscp ef (46)
Priority: 7% (1400 kbps), burst bytes 35000, b/w exceed drops: 0
QoS Set
dscp ef
Packets marked 266055059

Class-map: voip_signal (match-any)
78149834 packets, 7487646866 bytes
5 minute offered rate 0 bps, drop rate 0 bps
Match:  dscp cs3 (24) af31 (26)
78149834 packets, 7487646866 bytes
5 minute rate 0 bps
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/0/0
(pkts output/bytes output) 78149832/7487645664
bandwidth 3% (600 kbps)
QoS Set
dscp af31
Packets marked 78149834

Class-map: class-default (match-any)
9073254895 packets, 3302249852254 bytes
5 minute offered rate 998000 bps, drop rate 0 bps
Match: any
queue limit 64 packets
(queue depth/total drops/no-buffer drops) 0/16751970/0
(pkts output/bytes output) 569664615/3424829483864
QoS Set
dscp default
Packets marked 9071570170
shape (average) cir 16000000, bc 160000, be 160000
target shape rate 16000000

As you can see it gives lots of information on how your QOS is performing, the default-class shows some drops that means its had to drop some packets when the link was congested and in fact means your QOS policy is doing its job if the drops were excessive you would increase the values or decrease whichever suits your network.

If you just want to view an indivdual class then you would use the command “show policy-map interface fastethernet 0/0 output class web_traffic” for example.

When using the “service-policy input QOS_Egress” you will receive an error stating that you cannot apply this to the interface, this is becuase CBWFQ is a mechanism for traffic queuing in the outbound direction, never for the inbound direction.

I hope my 3 part blog has gone some way to helping you understand QOS.


My last blog covered class-map’s, which were used to identify traffic to our device, the next part of QOS covers what the device does with the traffic after its been identified. This is called a policy-map.

So you create your policy with the following command

R1#conf t
R1#(config)policy-map QOS_Egress

Next you need to reference each class-map then tell the device what to do with that traffic. Here’s an example

class voice_traffic
priority percent 7
set ip dscp ef
class voip_signal
bandwidth percent 3
set ip dscp af31
class SQL_App
bandwidth 20000
set ip dscp 26
class telnet_traffic
bandwidth percent 3
set ip dscp af21
class citrix_traffic
bandwidth percent 10
set ip dscp af41
class my_desktop
bandwidth percent 40
set ip dscp 18
class web_traffic
police rate percent 15
set ip dscp 10
class class-default
set ip dscp default
shape average percent 80

I’ve used some different actions on my policy map, “priority” and “bandwidth” are worth some discussion.

The idea with priority is just as it sounds, that’s your most important traffic class hence why it’s used for voice, so in my policy-map I’ve reserved 7% bandwidth, on a 20mbps circuit that would mean when the link is saturated, any voice traffic upto 1400kbps would be given priority over all other traffic. The difference that priority has over bandwidth is built-in policer, provides low latency, manages delay and jitter.

For your environment you may need more or less than 7% depending on what codecs you use between your IP phones.

Bandwidth defines the minimum bandwidth guarantee to that class, this can be specified in percentage of the interface bandwidth or in kbps. With CBWFQ the excess bandwidth is divided up amongst the remaining classes if it’s not already being used by the default-class.

Traffic shaping can be applied to the default-class as a way of ensuring your link is never fully utilized by unimportant traffic, what it does in this case is only ever allows maximum outbound traffic for the default-class to reach 80% of the interface rate.

Police means the maximum rate that is always applied to that class, the difference between police and shape is that police will drop packets and shape will buffer them, policing the traffic is alot more complicated than what I’ve explained, this is just simple terms.

DSCP values are set on my policy-map for  inbound traffic to my WAN, so in advance I would speak with my WAN provider and break up all the different DSCP values into classes like gold, silver, bronze then once the WAN provider receives a layer 3 packet with a DSCP value assigned it can decide how to process the packet when the link is congested.

There are lots of variations you can use with policy-maps, you can use nested policy-maps aswell, but this is just a brief overview of what you can achieve with QOS.

Remember you need to have end to end QOS, that means determining trust boundaries. For example from IP phone to IP Phone would be end to end QOS, if you don’t mark the packets coming from your IP Phone then the router will classify them as default traffic, by default your IP phone will send voice traffic marked as EF, You need to connect it to a switchport that at the very least has the command “trust cos


Quality Of Service is one of my favourite subjects the more I learn about it, the more I realise what i don’t know!

So the idea with QOS is you can classify different types of traffic using a class-map, then you can create a policy-map and specify what to do with that traffic, Next you apply the service-policy to an interface either inbound or outbound.

QOS really comes into its own in a  VoIP system, when your WAN link bandwidth is saturated you need to make sure that your VoIP traffic skips to the front of the queue giving the user the clear and uninterrupted call they expect. Remember that QOS only works when its deployed end to end.

My assumption here is you know what DSCP values are.

So lets look at a class-map this is the first part of our configuration, what a class-map does is identify traffic to your router, traffic can be identified at layer 3, layer 4 or layer 7.

This class-map matches DSCP EF (expedited forwarding) which is what voice traffic is marked, this is the most important and should be given top priority. I’ve included is voice_signal traffic which is marked as cs3 or af31, next matches applications, this is used in conjunction with NBAR, below we’ve matched some well known ones SQL, Telnet & Citrix. I’ve thrown in a class-map that can match ip address’s specified in access-list 10 and an extended access-list for matching tcp traffic.

Your router will parse each line sequentially, this is the same with your policy-map.

class-map match-all voice_traffic
 match ip dscp ef
class-map match-any voip_signal
 match ip dscp cs3 af31
class-map match-any SQL_App
match protocol sqlserver
class-map match-all telnet_traffic
  match protocol telnet
class-map match-all citrix_traffic
  match protocol citrix
class-map match-any my_desktop
 match ip access-group 10
class-map match any web_traffic
 match ip access-group 101
access-list 10 permit  
access-list 101 permit tcp any any eq www

Now you have your class-map your router can now identify traffic that passes through it.

So part 1 of this blog has covered identifying traffic to your router next we will cover policy-maps which decide what the router does with the traffic, so the class-map identifies the traffic and the policy-map defines the action for that traffic.

See you soon for my next part!


Network Based Application Recognition is an IOS feature which can recognize applications, traditionlly routers will look at traffic in layer 3 but with NBAR routers can see traffic in layer 4, 5, 6 & 7.

From what i gather online it looks like NBAR was released around 2005 IOS 12.0

I wont go too deep into how it recognises them other than to say the routers uses a sort of list also known as Packet Description Language Module (PDLM) which is updated regularly by Cisco.

Now i hear you ask …. how can i use and benefit from NBAR?

NETFLOW……. you can use the command “ip nbar protocol-discovery” on an interface that you want to monitor netflow traffic on, with this enabled the router will send the flow host information of the application traffic types i.e. telnet, SQL, HTTP etc. there is other configuration for netlfow which is outside this conversation….. perhaps that will be my next blog.

QOS……. this is my favourite use for NBAR, with this you can write QOS polices based on traffic type so for example you might want to give certain types of traffic a higher priority you would create a normal class-map just like this.

#class-map test
#match protocol xxxxxx

Then apply that to your policy map with whatever QOS settings you require.
#policy-map test
#class test
#priority percent 15

SECURITY ……. This is probably the most impressive use for NBAR although i have yet to implement it anywhere, but it may suit your needs? What you can do is again create your class-map and policy-map which match your protocols so for example we want to block all telnet traffic.

#class-map nbar-block
#match protocol telnet
#policy-map nbar-block-telnet
#class nbar-block
#set ip dscp 2

Next you apply the policy to your WAN interface

#int gig 0/0
#service-policy input nbar-block-telnet

This will mark the traffic inbound to your router, next you create an ACL to deny the marked traffic.

#ip access-list 101 deny ip any any dscp 2
#ip access-list 101 permit ip any any

Finally we apply this to the outbound LAN interface, so imagine your router is now a firewall (inbound & outbound) so technically this is outbound from the routers point of view.

#int gig 0/1
#ip access-group 101 out

Setup some configs with NBAR in your lab and see how you could use it.

stay safe