SD-WAN vs MPLS – Part 2

Continuing on with this mini-blog series of SD-WAN vs MPLS! This time I want to talk about…

Public Cloud and local internet breakouts


Traditionally for enterprises, all offices and locations would be connected together forming the private WAN. Internet bound traffic is then either routed to the internet through a central point (such as a DC), or a breakout from within the MPLS network. For many years this was optimum, a reliable managed service, security for internet traffic as it would be routed through a central proxy – perfect? Not so.


Each year, more and more service move to the public cloud. Enterprises are moving to PaaS, SaaS and even IaaS. This is a key point and benefit for moving to SD-WAN.


As we have covered, rather than having a single (or in some cases, dual) connected managed MPLS circuits, we can consume consumer/business grade broadband, direct internet access, at a significantly lower cost.

SD-WAN appliances can recognise based on application, or internet service – this means that the device can constantly measure the performance of the available links, and choose the best link for a particular application or service, providing a much better user experience.



From the below, our SD-WAN appliances has 3 links available: 2 x Direct Internet Access and a 4G connection.

Our user (who has an ethernet cable running from his laptop, into the communications room – do not try this at home the office), is consuming  applications within the various public clouds (green) and also some privately hosted applications in the DC (blue).For each of the 3 internet connections, the SD-WAN appliance is constantly measuring the latency and performance. It then selects the best quality link for each application, in real time! If one of those connections degraded, it would simply not be utilised. No need for the user to call the service desk and report ‘it is slow’ anymore!


“But, Adam, what about security? I get it, SD-WAN is great but do I really want to be providing users with unrestricted access to the internet?”

This ties in with the security posture of the enterprise. There are ways around this, endpoint security clients and cloud proxies, such as zScaler. As well as next-generation/UTM firewalls running as a web proxy/URL filter at the edge.


There are a number of SD-WAN appliances on the market today that are: routers, next-gen firewalls and SD-WAN – in a single box.

SD-WAN vs MPLS – Part 1

Hello again, continuing on from my previous blog post: here,  I want to continue the discussion of SD-WAN vs MPLS.


Firstly, going back to a point from the first post since I have had questions on it – cost!

Putting it simply, it is considerably cheaper to deploy an SD-WAN solution over MPLS because you can run a branch or remote site using commodity broadband links.

Rather than high cost links with SLAs, you can choose multiple, cheaper (no SLA) ‘business broadband’ circuits from different providers, on the basis that you already have a sub-second failover should one of them fail – no more single point of failure! Moving away from an expensive ‘primary’ circuit and a cheaper ‘backup’ circuit.

1 circuit with 500Mbps 99.99% uptime SLA, or 4 circuits with 150Mbps each, no SLA?

Of course there are drawbacks here, operationally and administratively – managing all those circuits isn’t an easy task either.


Let’s consider one of the key differences, once deployed…

The Topology

An MPLS deployment for branch or remote sites, it is effectively a full mesh private cloud. All sites can communicate directly with all other sites.

There are great resources here on RouteMyPacket to give you a low level understanding of MPLS, and deploying it yourself – here!


Let’s take the image below, each of these sites can communicate directly to each other – a full mesh.



Now the SD-WAN topology, typically will look like the below, Hub and Spoke – all the sites must connect to each other through the DC.



Remember that SD-WAN is underpinned by IPSec VPNs. Configuring a full mesh of IPSec VPNs between all of your sites would not be advised – there are some technologies to overcome this challenge though. Look into DMVPN or ADVPN, this combined with SD-WAN could achieve a partial mesh.



As SDN becomes more and more prominent, many SD-WAN solutions will offer an Overlay Controller to orchestrate and control the WAN. This would closely align, logically, to the full mesh/MPLS topology above. This does vary between vendors, but more of these controllers are emerging.


In summary, I wrote this post to explain a benefit, but also to challenge it too. SD-WAN and MPLS are two very different technologies. They both have benefits and drawbacks – they do share a common use-case, hence the debate goes on!



MPLS is still ruling the world of core networks and there really is no other protocol out there that can challenge its undisputed status as THE ultimate tunneling protocol for service providers and large enterprises.

So here are five reasons why MPLS is so irresistibly attractive when you are designing your core network infrastructure as the protocol of choice.

Supports innovation

MPLS has revolutionised itself overtime and has been able to answer the needs of growing networks with multiple protocol stacks with little hiccups. It has been able to support the needs of modern networks with simple tweaks and updates where it has managed to tag along all types of new traffic without any change to its underlying framework. This in itself has been a huge game changer and one of the key reasons why it rules service provider networks with no real challenger in sight.

A protocol that can adapt to changing needs with minimal cost is good for business and that is exactly what MPLS has been to service providers and other types of businesses that have made use of MPLS deployment in their networks.


MPLS is very easy to scale and add more nodes as and when needed provided the underlying singling protocols have been carefully picked and configured to handle the size of label information database supported by nodes in your network.

Feature richness

Originally designed to support networks with multi-protocol environment, MPLS can work with pretty much any protocol out there, be it Ethernet, Frame Relay or even future protocols yet to be introduced. The reason this is possible is predominantly in the power it provides when hiding protocol data behind its label stack. Anything really can be stashed behind the label stack and sent to an MPLS destination end point with minimal difference in operation.

Traffic engineering 

Traffic engineering in itself is one of the big subjects when deploying large networks. In summary, traffic engineering features allow you to take any path for a destination end point which the underlying routing protocol (IGP) did not fathom to be the best path. You can use MPLS traffic engineering features to completely ignore your IGP’s calculation for the best path and deliver traffic via the alternate path still based on the calculations made by the IGP. This coupled with the ease in switching traffic from one path to another has made MPLS the protocol of choice for networks where large traffic engineering decisions are required on the fly. LDP as we know can only be traffic engineered by fiddling with the IGP metrics but RSVP-TE brings it’s own subset of amazing traffic engineering features. That said, SPRING or Segment Routing is another MPLS protocol which is up and coming and provides more promise for the future giving us the best of both LDP and RSVP in one protocol. How well it performs is all dependent on how it is received in the coming years, but the future looks promising!


MPLS provides credible mechanisms to read into delay sensitivity being suffered by traffic. This allows traffic to be course corrected in due time and with minimal headache. Moreover, informed metrics are good for business and the return on investment on MPLS is usually always higher. In other words, the protocol pays for itself in time.

For more technical information related to MPLS’s deployment in service provider networks, refer to my post here.

Video: MPLS & BGP Simplified


Over the years, whilst designing and managing large core MPLS networks, I have found it frustratingly hard to explain MPLS to fellow engineers, sometimes it would lead me to wanting to jump out of the window!

It can be hugely frustrating to be able to understand something and not be able to explain it in the same way.

But… what can you do? MPLS is that complex subject and when the likes of MP-BGP and IGP protocols such as IS-IS come into play, it becomes an absolute mess to simplify.

In fact, for many years, I just left myself believing that there is no easy way to explain this. It’s only when I was designing a MPLS network for a customer this year that I stumbled upon this idea to explain MPLS in the form of a triangle with IGP as the base and BGP and MPLS hanging on in the form of a dependency.

The result was a simple conversation with this logic would make life easier for engineers trying to understand and make sense of a MPLS network running BGP with an IGP such as IS-IS or OSPF.

I then decided to make a video to share it with you so that you can use this logic too in order to understand core MPLS networks in the same way and also use this to simply large core networks whenever you need to, be it job interviews or at work or wherever!

I am so excited to share my latest video simplifying MPLS and its relationship with BGP along with the frustrating question as to why the heck do we need an IGP?

Please checkout the video below and the only thing I ask in return is to please subscribe to my youtube channel so that you can regularly see informative content that I put out there for FREE 🙂




Apart from general traffic engineering via link colouring and modifying strict path for a colourless LSP, we can have a unique use case where we would like to send l3vpn traffic for a specific customer via a specific LSP not preferred by general traffic.

rsvp mpls

For example, all traffic from PE5 to PE1 uses the gold coloured LSP, but we want a specific customer vrf to use another LSP which might take the path PE5-PE6-PE4-PE2-PE1. This usecase can arise due to multiple reasons, sometimes, its the cost of certain links which are premium and you would like to send specific traffic for specific customers that pay premium rates through those links only.

To achieve this, we need to configure RSVP to ensure all traffic from PE5 Customer A L3VPN utilises a new LSP built between PE5 and PE1 for this purpose, separate from the gold LSP.

The process for this involves the following:

  • Assign a fake address to PE1 and advertise it as the next hop along with the route target for Customer A L3VPN.
  • At PE5, signal a new LSP that terminates at PE1 with the FAKE next-hop address assigned and advertised by PE1.
  • Modify the import policy for Customer A L3VPN to import the route target with the fake next-hop only.

Once the above is configured, we will be able to see Customer A send all it’s traffic via the newly signalled LSP.

Route table for l3vpn customer_A before l3vpn traffic engineering:

[email protected]# run show route logical-system PE5 table customer_A
customer_A.inet.0: 17 destinations, 31 routes (17 active, 0 holddown, 0 hidden)
+ = Active Route, -- = Last Active, * = Both *[BGP/170] 6d 01:27:08, localpref 100, from
AS path: 65300 13000 I, validation-state: unverified
> to via lt-0/0/10.11, label-switched-path PE5-to-PE2-Gold
to via lt-0/0/10.10, label-switched-path PE5-to-PE2-Gold *[BGP/170] 6d 03:08:13, localpref 100, from
[BGP/170] 6d 01:42:12, localpref 100, from
AS path: 65300 13000 I, validation-state: unverified
> to via lt-0/0/10.11, label-switched-path PE5-to-PE1-Gold
to via lt-0/0/10.10, label-switched-path PE5-to-PE1-Gold

Route table for l3vpn customer_A after l3vpn traffic engineering:

[email protected]# run show route logical-system PE5 table customer_A
customer_A.inet.0: 17 destinations, 31 routes (17 active, 0 holddown, 12 hidden)
+ = Active Route, -- = Last Active, * = Both *[BGP/170] 00:00:01, localpref 100, from
AS path: 65300 13000 I, validation-state: unverified
> to via lt-0/0/10.11, label-switched-path X-SERVICE-TO-PE1

With this implementation, all other L3VPNs on PE5 will continue to use the standard gold lsp, whereas l3vpn customer_A would use X-SERVICE-TO-PE1 LSP.

Configuration on PE5:

Configuration on PE1:


ARP requests and EVPN table population in leaf/spine data centre fabric


The advantage of using EVPN is the ability to extend layer 2 over a layer 3 fabric without the use of a SDN controller. This way we intend to get the flexibility of hosting any host belonging to any subnet behind any leaf switch and still form part of the same network by overlaying its layer 2 information over IP.

Refer to the topology below:



In a standard layer 2 deployment, we would have to span the vlan Customer Server A and Customer Server B are part of to ensure they are part of the same broadcast domain.

However, in the new leaf/spine based IP fabric deployment, we are going to advertise the layer 2 reachability information via EVPN over BGP.

We are going to achieve this by encapsulating the vlan customers are part of into a VNI which will be an additional tag that would be placed on top of the standard frame. The only thing we have to ensure is both servers on different leaf switches, whatever vlan they are part of, should be bridged to the same VNI. Vlans are only locally unique to the leaf switch.

Once the necessary configuration has been applied, the traffic work flow would be as follows:

Customer Server A would send an arp request to leaf05 which would be a broadcast request with a ff.ff.ff.ff type destination mac address. This broadcast packet when received by leaf05 would be sent to all local ports that are part of the same vlan by means of traditional layer 2 broadcast and all other leaf switches that have a valid VTEP configured over IP. A VTEP is a virtual tunnel IP which is the same as the loop back address. All layer 2 overlay traffic is sent and received between VTEPs over IP. The only difference here on wards would be that the broadcast packet would be forwarded over ip multicast instead to other VTEPs.

In this example, we have already established BGP as the underlying IP fabric so all VTEPs, i.e. leaf switches loopback addresses are already present in the routing table of each leaf switch.

The broadcast packet would then be encapsulated with the associated VNI tag and forwarded to all other VTEPs. Once received by other leaf switches, if the VNI is locally configured on a leaf switch, it would strip the vni tag and forward the frame to all ports that are associated to that VNI.

The ARP request was intended for Customer Server B which had IP configured on it. Therefore, every host that receives that ARP request would discard it apart from Customer Server B. Customer Server B would formulate an ARP reply based on the destination mac address of Customer Server A which was found in the ARP request. This reply would be forwarded to leaf08 which would already know the destination mac address of Customer Server A as it made an entry of the mac address and associated it with the ip address along with the next hop address of leaf05 it was advertised from.

The packet is then forwarded to leaf05’s VTEP address and leaf05 would perform the same function upon receiving the packet as leaf09 did and forward the frame to Customer Server A.

This way, our mac address overlay table is populated with the mac addresses of Customer Server A and Customer Server B on leaf05 and leaf09. Moreover, all leaf switches that received the broadcast ARP packet initially sent would now know the mac address of Customer Server A and the next hop associated to it.

Cumulus configuration snippets:

Following is a snippet of physcial interface vlan and vlan bridging along with the vni config over a cumulus linux switch:


Mac addresses learnt via evpn from other leaf switches:

[email protected]:mgmt-vrf:~$ net sh evpn mac vni
<1-16777215> : An integer from 1 to 16777215
all : Run a command on all connected hosts
[email protected]:mgmt-vrf:~$ net sh evpn mac vni all
VNI 10200 #MACs (local and remote) 5
MAC Type Intf/Remote VTEP VLAN
00:03:00:11:11:07 remote
00:03:00:11:11:06 remote
00:03:00:11:11:08 remote
00:03:00:11:11:05 local swp1 200
44:38:39:00:02:1c local vlan200 200

A quick look at the global evpn table yields the ip addresses associated with those mac addresses and the AS path to reach the VTEPs on the destination leaf switches:

[email protected]:mgmt-vrf:~$ net sh bgp l2vpn evpn route
BGP table version is 5, local router ID is
Status codes: s suppressed, d damped, h history, * valid, > best, i -- internal
Origin codes: i -- IGP, e -- EGP, ? -- incomplete
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
*> [2]:[0]:[0]:[48]:[00:03:00:11:11:06]:[32]:[] 0 4200003000 4200001005 i
* [2]:[0]:[0]:[48]:[00:03:00:11:11:06]:[32]:[] 0 4200003000 4200001005 i
* [2]:[0]:[0]:[48]:[00:03:00:11:11:06]:[32]:[] 0 4200003000 4200001005 i
* [2]:[0]:[0]:[48]:[00:03:00:11:11:06]:[32]:[]
*> [2]:[0]:[0]:[48]:[00:03:00:11:11:07]:[32]:[] 0 4200003000 4200001006 i
* [2]:[0]:[0]:[48]:[00:03:00:11:11:07]:[32]:[] 0 4200003000 4200001006 i
* [2]:[0]:[0]:[48]:[00:03:00:11:11:07]:[32]:[] 0 4200003000 4200001006 i
* [2]:[0]:[0]:[48]:[00:03:00:11:11:07]:[32]:[] 0 4200003000 4200001006 i
* [2]:[0]:[0]:[48]:[00:03:00:11:11:08]:[32]:[] 0 4200003000 4200001007 i
*> [2]:[0]:[0]:[48]:[00:03:00:11:11:08]:[32]:[] 0 4200003000 4200001007 i
* [2]:[0]:[0]:[48]:[00:03:00:11:11:08]:[32]:[] 0 4200003000 4200001007 i
* [2]:[0]:[0]:[48]:[00:03:00:11:11:08]:[32]:[] 0 4200003000 4200001007 i

Pinging from Server behind leaf05 to server behind leaf08:

[email protected]:~$ ping
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=64 time=5.06 ms
64 bytes from icmp_seq=2 ttl=64 time=3.30 ms
--- ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 3.304/4.182/5.061/0.880 ms

Basics of a leaf/spine based IP Fabric architecture


Refer to the topology below:


The basic summary of the architecture is as follows:


All leafs are expected to form a BGP relationship with all spines (max 4) and each leaf will have a spine between them and any other leaf.

If using cumulus linux based switches, this BGP relationship would be based on BGP un-numbered.


Ethernet VPN would be responsible for sharing mac addresses between the leafs via the spines. This way each leaf will have the full mac view of all leafs to than proceed and emulate layer 2 over layer 3 via the use of VXLANs.

The underline mechanism is straight forward, i.e. use MP-BGP to advertise all mac addresses from a leaf to all spines and than redistribute from spines to all other leafs.


VXLAN is just the name of the protocol, in reality, it is VNIs that create unique tunnels over the IP fabric. VNIs are responsible for inter-connecting all access ports from any leaf to all other access ports with the same vlan on all other leafs.

This way we would achieve layer 2 transportation over layer 3 with the added flexibility of stretching any vlan across all leafs, assigning multiple vlans to the same VNI and keeping the same broadcast domain (will become multicast over IP) whilst getting rid of STP and making use of all available ports


This in a nutshell is the basic workflow of the modern data centre where layer 2 is over layed over IP with the use of VXLANs.

What is EVPN?


In a Leaf/Spine topology, Ethernet VPN or EVPN is predominantly used to carry mac addresses between leaf switches. These mac addresses belong to all servers attached to leaf switches.

Since bgp unnumbered runs between all leaf, spine and exit switches, all we really do is specify an l2vpn evpn based address family under BGP and proceed to advertise all layer 2 information from each vni into it. We can also choose to reduce this to specific vni’s if we prefer but in our design we intend to advertise all vni’s on each leaf switch so that the layer 2 reachability information is passed and forwarded to each member of the bgp fabric, i.e. other leaf switches, spines and exits.

This way, all subsequent network switches than have an evpn table can refer to it in order to forward frames they receive that want to go out to a destination mac address that lives on another leaf switch in the topology.

Cumulus configuration example to advertise all vni’s into evpn:

The above configuration snippet is defined under bgp which enables evpn and advertises all mac addresses learnt locally to bgp peers which in the case of a leaf switch would be all spine switches it is peered with (usually four).

A look at the leaf’s evpn table yields the following:

[email protected]:mgmt-vrf:~$ net sh bgp l2vpn evpn route
BGP table version is 5, local router ID is
Status codes: s suppressed, d damped, h history, * valid, > best, i -- internal
Origin codes: i -- IGP, e -- EGP, ? -- incomplete
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]:[IPlen]:[IP]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]
EVPN type-5 prefix: [5]:[ESI]:[EthTag]:[IPlen]:[IP]
Network Next Hop Metric LocPrf Weight Path
* [2]:[0]:[0]:[48]:[00:03:00:11:11:05]:[32]:[] 0 4200003000 4200001004 i
*> [2]:[0]:[0]:[48]:[00:03:00:11:11:05]:[32]:[] 0 4200003000 4200001004 i
* [2]:[0]:[0]:[48]:[00:03:00:11:11:05]:[32]:[] 0 4200003000 4200001004 i
* [2]:[0]:[0]:[48]:[00:03:00:11:11:05]:[32]:[] 0 4200003000 4200001004 i
[2]:[0]:[0]:[48]:[00:03:00:11:11:08]:[32]:[] 0 4200003000 4200001007 i
*> [2]:[0]:[0]:[48]:[00:03:00:11:11:08]:[32]:[] 0 4200003000 4200001007 i
* [2]:[0]:[0]:[48]:[00:03:00:11:11:08]:[32]:[] 0 4200003000 4200001007 i
* [2]:[0]:[0]:[48]:[00:03:00:11:11:08]:[32]:[]

In the example above, you can see that leaf01 learns about the mac addresses of the destination ip and via EVPN. It also learns about the next-hop address which is for and for

These next-hop addresses don’t belong to directly connected neighbours, however, a quick look at the global routing table helps us resolve the next-hop address:

[email protected]:mgmt-vrf:~$ net sh route
show ip route
Codes: K -- kernel route, C -- connected, S -- static, R -- RIP,
O -- OSPF, I -- IS-IS, B -- BGP, E -- EIGRP, N -- NHRP,
T -- Table, v -- VNC, V -- VNC-Direct, A -- Babel, D -- SHARP,
F -- PBR,
> -- selected route, * -- FIB route
C>* is directly connected, lo, 1d01h38m
B>* [20/0] via fe80::4638:39ff:fe00:28b, swp52, 1d01h38m
* via fe80::4638:39ff:fe00:4bb, swp49, 1d01h38m
* via fe80::4638:39ff:fe00:3bf, swp50, 1d01h38m
* via fe80::4638:39ff:fe00:3b, swp51, 1d01h38m
B>* [20/0] via fe80::4638:39ff:fe00:28b, swp52, 1d01h38m
* via fe80::4638:39ff:fe00:4bb, swp49, 1d01h38m
* via fe80::4638:39ff:fe00:3bf, swp50, 1d01h38m
* via fe80::4638:39ff:fe00:3b, swp51, 1d01h38m
B>* [20/0] via fe80::4638:39ff:fe00:28b, swp52, 1d01h38m
* via fe80::4638:39ff:fe00:4bb, swp49, 1d01h38m
* via fe80::4638:39ff:fe00:3bf, swp50, 1d01h38m
* via fe80::4638:39ff:fe00:3b, swp51, 1d01h38m
B>* [20/0] via fe80::4638:39ff:fe00:28b, swp52, 1d01h38m
* via fe80::4638:39ff:fe00:4bb, swp49, 1d01h38m
* via fe80::4638:39ff:fe00:3bf, swp50, 1d01h38m
* via fe80::4638:39ff:fe00:3b, swp51, 1d01h38m
B>* [20/0] via fe80::4638:39ff:fe00:28b, swp52, 1d01h38m
* via fe80::4638:39ff:fe00:4bb, swp49, 1d01h38m
* via fe80::4638:39ff:fe00:3bf, swp50, 1d01h38m
* via fe80::4638:39ff:fe00:3b, swp51, 1d01h38m
B>* [20/0] via fe80::4638:39ff:fe00:28b, swp52, 1d01h38m
* via fe80::4638:39ff:fe00:4bb, swp49, 1d01h38m
* via fe80::4638:39ff:fe00:3bf, swp50, 1d01h38m
* via fe80::4638:39ff:fe00:3b, swp51, 1d01h38m
B>* [20/0] via fe80::4638:39ff:fe00:28b, swp52, 1d01h38m
* via fe80::4638:39ff:fe00:4bb, swp49, 1d01h38m
* via fe80::4638:39ff:fe00:3bf, swp50, 1d01h38m
* via fe80::4638:39ff:fe00:3b, swp51, 1d01h38m

Any intended packet that wants to reach or would be forwarded to these ipv6 neighbours which are the spine switches. The spine switches would then forward the packet to the leaf switches advertising these mac addresses in similar fashion.

This in a nutshell is how we overlay layer 2 over layer 3 using EVPN.


SD-WAN – What is it?


SD-WAN. What is it? Should I care? Is it a standard? Why would I want to software define my WAN?! Let’s get to it…


You would have seen or heard SD-WAN if you work in IT, it is a hot topic and honestly, it has many definitions which can be confusing. SD-WAN is not a standard, so vendors and providers use it as a marketing term. I want to demystify the marketing with this post.


SD-WAN is an evolution in branch connectivity. It is intelligence at the edge of the network to leverage multiple links, load balance and monitor the performance of each, as well as application awareness. The technology is built into appliances, which there are many! Ok, yes, that was basically marketing, let me break it down.


If you think about your WAN currently, it is likely a mix of direct internet access (DIA), ADSL, 4G, MPLS, IPSsec VPNs… there are so many ways to connect! You may also have different edge devices too, a router, firewall, or from a switch. You will also have a process to handle a connectivity failure at a site. One of the challenges with office connectivity, or building a WAN, is resilience. There are many ways to achieve this, both physically and logically. A backup circuit, IP SLA, or a floating static route, some of these solutions are complex and over-engineered.


One of the features of SD-WAN overcomes this problem. Utilising all of the links available at a site, with real-time monitoring of each and steering traffic down the ‘best’ for specific types of traffic. For example, which of my available links is the best for voice traffic? Which has the lowest latency to a destination? and so on… the devices are constantly probing the links to determine the best – insight into the quality of the link. It challenges circuit failover as all the links are in use, but you can  achieve sub-second failover for your IPSsec VPN.


SD-WAN appliances are also application/layer 7 aware. This means you have even more control to steer traffic, which of my available links is the best, lowest latency to Office365? As more and more applications move to the cloud, you realise why SD-WAN has so much traction.


However, arguably the main reason why SD-WAN is such a hot topic, is cost. You can significantly reduce WAN spend by deploying an SD-WAN solution, because instead of paying for expensive links, with SLAs, you can replace them with commodity/consumer links. This is also the reason it is compared with MPLS, they are very different offerings but ultimately, you will be able to connect and office for less with SD-WAN, when compared with MPLS.


I hope that has helped you understand SD-WAN, a very brief overview! I will write a follow up. We will deep dive into SD-WAN vs MPLS at a lower level, I’ll also go through SD-WAN deployments/topologies, as well as the appliance offerings!

BGP Unnumbered


Successful Leaf/Spine deployment is entirely dependent on BGP which forms the underlying foundation of the IP based fabric as our intention is to provide layer 2 as an overlay service running on top of layer 3.

To run BGP unnumbered, the following configuration should be enough on cumulus leaf switches:

Great thing about BGP un-numbered as the name suggests is you don’t need to specify ip addresses and you just have to specify the interface you want BGP un-numbered to run and as long as the remote end has similar configuration, the sessions should come up.

BGP unnumbered configuration on cumulus spines:

Since the spines peer with the leafs and exits via bgp unumbered, you can see all corresponding interfaces are part of the BGP unnumbered configuration.

The result is successful bgp peering with all leafs and exits seamlessly as seen below:

[email protected]:mgmt-vrf:~$ net sh bgp summary
show bgp ipv4 unicast summary
BGP router identifier, local AS number 4200003000 vrf-id 0
BGP table version 25
RIB entries 37, using 5624 bytes of memory
Peers 18, using 347 KiB of memory
Peer groups 1, using 64 bytes of memory
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
leaf01(swp1) 4 4200001000 30445 30380 0 0 0 1d01h14m 1
leaf02(swp2) 4 4200001001 30398 30380 0 0 0 1d01h14m 1
leaf03(swp3) 4 4200001002 30398 30380 0 0 0 1d01h14m 1
leaf04(swp4) 4 4200001003 30398 30380 0 0 0 1d01h14m 1
leaf05(swp5) 4 4200001004 30480 30380 0 0 0 1d01h14m 1
leaf06(swp6) 4 4200001005 30461 30380 0 0 0 1d01h14m 1
leaf07(swp7) 4 4200001006 30462 30380 0 0 0 1d01h14m 1
leaf08(swp8) 4 4200001007 30463 30380 0 0 0 1d01h14m 1
leaf09(swp9) 4 4200001008 30398 30380 0 0 0 1d01h14m 1
leaf10(swp10) 4 4200001009 30398 30380 0 0 0 1d01h14m 1
leaf11(swp11) 4 4200001010 30398 30380 0 0 0 1d01h14m 1
leaf12(swp12) 4 4200001011 30398 30380 0 0 0 1d01h14m 1
leaf13(swp13) 4 4200001012 30398 30380 0 0 0 1d01h14m 1
leaf14(swp14) 4 4200001013 30398 30380 0 0 0 1d01h14m 1
leaf15(swp15) 4 4200001014 30398 30379 0 0 0 1d01h14m 1
leaf16(swp16) 4 4200001015 30398 30379 0 0 0 1d01h14m 1
exit01(swp25) 4 4200502000 30583 30442 0 0 0 21:00:58 1
exit02(swp26) 4 4200502000 30398 30410 0 0 0 18:42:05 1
Total number of neighbors 18

BGP unumbered is one of the best tools I have come across in deploying an IP fabric. It totally removes the need for manual ip allocation/configuration as it relies on ipv6 link local addresses for point to point addressing.

Once you use Ansible to configure your leafs and spines, bgp unnumbered will bring up bgp as if you were running ospf in the fabric instead of bgp. This is largely thanks to static switchport configuration inside the bgp config.