MPLS and its deployment in Internet Service Provider (ISP) Networks

Multi Protocol Label Switching (MPLS) is a mechanism to route packets based on labels rather than traditional route lookups based on IP address. It gets the name multiprotocol because it can run over many protocols such as Frame Relay, ATM, IP etc.

Before I delve further into the details of MPLS it is important to understand why do service providers and some large enterprises use MPLS?

A typical service provide network would consist of many ‘edge’ routers. These edge routers than proceed and connect to the core routers. This way all edge routers can communicate with each other via the core routers instead of connecting to each other in a full mesh. As the network of a service provider grows, so does the complexity and layers between the core and edge routers.

Larger ISPs have multiple edge layers where the routers at the ultimate edge connect to customers and are called ‘provider edge’ routers. The second layer of edge routers connected to these provider edge routers are another layer of edge routers which then communicate between each other via core routers in a simply multi-tier top down model where all edge routers hang off the core routers and all the provider edge routers hang off the edge routers.

Nevertheless, the basic communication model remains the same. That is, run IGP (Interior gateway protocol) such as IS-IS or OSPF to form infrastructure communication between all routers from core to all edges and then simply run BGP on top between all routers to share routes learned from all edge routers into the core so that all routers can then send packets to the edge router that injected a specific destination route into network.

This means when a packet that intends to reach the internet route 8.8.8.8 enters the core network via an edge router, every router in the path including that edge router performs route lookup to decide where to forward the packet to. This would result in the first edge router performing route lookup and then forwarding the packet to the core router which would then perform another route lookup and forward the packet to the edge router that learned and advertised the route to network 8.8.8.0/24 via it’s eBGP neighbour.

As the network grows, more routers find themselves in the path performing route lookups. All these route lookups significantly add to the latency the packet suffers before reaching its destination. As route recursion forces the router to check every entry in the routing table with the longest possible match until it finds the best match before forwarding the packet, there is simply no way to avoid this induced latency in a basic IP based model.

This is where MPLS comes into play!

Instead of performing route lookups at every hop, the router simply swaps labels and forwards the packet. Instead of route recursion, the lookups are performed at a layer 2 level. This drastically reduces the time it takes from receiving a packet to forwarding it to the next hop so much so that in classic MPLS design, the core routers which usually end up processing all traffic between edge routers get away with not running BGP at all and simply relies on swapping labels for all traffic at all times!

Apart from low latency, MPLS infrastructure allows ISPs to bundle many Virtual Private Network (VPN) services which they can offer to their customers such as , L2VPNs, L3VPNs, VPLS etc.

Typical MPLS deployment:

MPLS works by establishing label switched paths called LSPs throughout the network over the existing IP connectivity, usually an IGP such as IS-IS or OSPF.

Once LSPs are established between two end routers traffic can simply take the intended path determined by the LSP and each router in the path simply swaps the label on the MPLS header and forwards it to the next router in the LSP. Each router in the path already knows what LSP a given packet is traversing through and what label to swap it with as it holds this information in its MPLS table which was populated when the LSP was initially created.

Once LSPs are created, you can then run BGP throughout the network and for every route received with the next hop address of any edge router, BGP will install the route in the routing table with the label reference for the correct LSP to use to send the traffic to the edge router which would be the next hop for that route.

BGP can be run before running MPLS as well but in that instance, BGP would simply install all routes received from an edge router into the routing table after checking the routing table to determine if the next hop associated with the route is accessible. If the next-hop is reachable, it installs the route in the routing table. The only difference when MPLS is running on the router is that instead of installing a route with the next hop entry derived using IGP, BGP looks into the MPLS routing table and determines if a LSP exists to the next hop address associated with the route. If it finds one, it prefers that over a route that exists in the routing table populated via the IGP. The MPLS routing table tells BGP what label to push and which directly connected router to send the packet to for the loopback address of the edge router. Thus, BGP installs the destination route learned from the edge router with the next hop address of the next router in the path instead of the ultimate next hop the route was advertised with. This installed next hop associated with the route is the address of the router next in line in the LSP path all the way to the edge router which advertised the destination route to begin with. Along with this next-hop address is the label number to be pushed onto the packet before sending.

All this might be a bit confusing to understand first hand so allow me to explain this with an example:

Edge Router A -> Core Router B-> Core Router C -> Edge Router D -> 1.1.1.0/30 -> External Router E (8.8.8.0/24)

Suppose Router E advertises route 8.8.8.0/24 to router D which is the edge router in our network. When the route is received by router D, it is received with the next-hop address that exists on Router E, i.e. 1.1.1.1. If router D advertises this route as it is to Router A, Router A’s BGP process will mark this route as unusable because it wouldn’t know how to reach 1.1.1.1 unless 1.1.1.0/30 is advertised into IGP, which is not the recommend. Therefore, Router D would have to invoke the next-hop self command and then advertise the route to Router A with the next hop address of 10.1.1.1 (loopback address on Router D) which router A can reach as it knows this route via IGP. Once router A receives the route 8.8.8.0/24 from router D, since it can reach the next hop address of 10.1.1.1, the route would get installed in the routing table with the next hop of 10.1.1.1.

However, if MPLS was working on the network there would be a LSP present in the MPLS routing table with the destination address of 10.1.1.1 with PUSH label 111511 (for example). BGP by default, would now prefer this over the route to the next hop 10.1.1.1 (to get to 8.8.8.0/24) learnt via IGP. Therefore, instead of the entry in the routing table with the next-hop of 10.1.1.1 for the route 8.8.8.0/24, it would add an entry into the routing table for 8.8.8.0/24 with the next hop of 10.2.2.2 instead as that is the loopback address on core router B which is the next router in line of the LSP. This way, the router would now send all traffic intended for 8.8.8.0/24 to the next-hop of 10.2.2.2 with label 111511. The next router in path which is core router B would know upon receiving label 111511 it is supposed to simply swap it with another label 111611. This way the packet would reach router D which initially advertised the route 8.8.8.0/24 to Router A. The label would have already been removed by Core Router C and Edge Router D can simply perform route recursion on its routing table for route 8.8.8.0/24 to forward the packet to External Router E where 8.8.8.0/24 is directly connected.

Therefore, the BGP route in Router A’s routing table although points to 10.2.2.2 as the next-hop instead of 10.1.1.1, it subsequently would say on the route which label to push onto the packet. This information has been taken from the MPLS routing table which had label 111511 push function associated to the LSP to 10.1.1.1. So label 111511 with next-hop address 10.2.2.2 actually forces the packet to use the LSP to 10.1.1.1.

This way all traffic intended to 8.8.8.0/24 network would utilise the LSP created to go to Router D. It’s important to note that all traffic whose destination address is 10.1.1.1, i.e. ping/management traffic to Router D would not use the LSP. That traffic would simply use the path figured out by the IGP. LSPs will be used for all traffic that take the BGP path to a given route, in this case, 8.8.8.0/24. This is the default function on Juniper routers. Most vendors prefer this and so limit the IGP from accessing the MPLS routing table by default.

This means that if BGP isn’t configured and MPLS is running over IGP, no route would utilise the fast LSPs established by MPLS unless some configuration is invoked to force the IGP to utilise LSPs as well and change the default mechanism.

MPLS router states:

A router running MPLS can be in three states for any LSP.

– Ingress state when the router pushes a label in
– Transit state when the router simply swaps the label
– Egress state where the LSP (label switched path) ends

In Junos, we have inet.0 table which is the IP unicast routing table, inet.3 which is the MPLS routing table and mpls.0 which is the routing table used by transit and egress routers (if no penultimate hop popping is in place) to swap labels.

Difference between inet.3 and mpls.0:

Inet.3 is the MPLS routing table, this table is useful specifically for BGP when it is trying to resolve a specific next hop address for a route it has received from a neighbour. This is where all information of all LSPs to all destinations are listed. The inet.3 routing table is the MPLS routing table and all the LSPs listed here have a label associated next to them that is to be pushed onto the packet to send traffic via that LSP. Other information includes the IP address of the neighbouring router the packet is to be sent to after the PUSH process.

On the other hand, mpls.0 is where all labels are listed and this table is a label lookup tool. Mpls.0 tells the router what to do when it receives a packet with a label already pushed onto it. It is the corresponding entry next to the label in mpls.0 table that tells the router to either SWAP the label for another one and send it via an interface on the router or POP if it’s the second to last or the last router (if Penultimate Hop Popping is not active).

If a router receives a packet with a label pushed already onto it, then that means the router itself is a transit router or the destination router where the LSP ends (if Penultimate Hop Popping is not active).

The ingress router doesn’t use mpls.0 as it is simply interested in pushing a label onto the packet which it achieves using the MPLS routing table since the packet is not part of a LSP yet. However, as explained before, MPLS routing table is not actively used to perform label lookup. For an IP packet, the router still uses the inet.0 or the main routing table , it is BGP that picks up the next hop information from this table and associates that with the route it injects in the global routing table which then corresponds to the LSP from the MPLS routing table, i.e. inet.3 in Junos.

However, the transit routers make full use of the mpls.0 table as when they receive a packet with a label already pushed on it, they look through the mpls.0 table and find the corresponding label to swap it with along with the next hop to send it out to.

The mpls.0 table contains all entries to the LIB (label information base) as it is where the router figures out which received label to be swapped by which label and forwarded across.

LIB is stored in the control plane and it’s forwarding plane variant which is the forwarding table based on labels is called LFIB or Label forwarding information base.

Label Switched Path (LSP) example:

An LSP from Router A to Router D might look like below:

Router A ->100010 -> Router B -> 100020 -> Router C -> POP -> Router D

Router A defines the label 100010 to be pushed onto the packet if the destination is Router D, Router B’s configuration simply guides it to take label 100010 and swap it with 100020 and send the packet to the next hop. Router C’s configuration simply guides it to take 10020 and pop it and send the remainder IP packet via a next-hop until it reached Router D where it simply performs a route lookup to forward the packet.

Penultimate Hop Popping (PHP):

Usually the label is popped at second to last hop and simply sent as a normal packet also known as Penultimate hop popping (PHP). This is achieved during the LSP formation when the destination router of a LSP sends the return packet to the neighbouring router with a value of 3 which indicates popping of the label before sending. This way the last router simply receives a packet without a label and performs the necessary function (route lookup if it’s an IP packet) required to forward that packet instead of adding latency by looking up in the MPLS table and realising it in itself is the destination for the LSP and then removing the label and then moving ahead with route recursion.
PHP is the default behaviour in Junos but you can configure and remove PHP whereby destination router for the LSP POPs the label.

A word on PE routers:

In MPLS, you will hear the word Provider Edge router a lot. This is basically the edge router that connects to the customer. The customer traffic comes into the Provider edge (PE) router where it takes a LSP to reach another PE router and since the label has already been popped using PHP, the PE router than simply proceeds with route recursion to send the traffic to the destination.

This model of LSPs between PE routers to PE routers allows ISPs to provide many VPN based services to their customers as any type of traffic can easily be routed to any PE router using a LSP and all routers in the path simply swap labels. This results in PE routers seeing each other as next-hop routers for Multi-protocol BGP defined services and the core routers appear invisible to the configured service.

This means, the ISP can now offer the customer to connect all their sites to the ISP’s PE routers at different locations and simply expand their layer 2 switched network using MPLS and the result would be the customer would end up seeing mac address tables from Office A to Office B! This is implementation of VPLS using MPLS.

OR, the Customer might simply want a point to point link from an office connected to PE router A to an office connected to PE router B. The ISP can simply create L2VPN cross connect circuits which are vlan based circuits where the ISP tags all customer traffic inside a vlan and make PE router A communicate with PE router B via LSP to deliver customer traffic from site A to site B. At the PE router directly connected to the destination customer site, the ISP simply removes the vlan it encapsulated the traffic into and forwards it to the customer.

OR, the ISP can provide a VPN solution to the customer where the customer can share all office routes with each office by simply connecting every office to the PE router, again utilizing LSPs, the ISP can offer this specific type of service where it is aware of all customer routes and keeps them segregated in Virtual Route Forwarders (VRFs) configured on each PE router for the customer to only see all his routes at each office for all other offices via the ISP, also called L3VPNs or IPVPNs.
This service has the added advantage of providing the customer with a VPN without the use of overhead encapsulation using protocols such as IPSEC VPNs over the internet.

MPLS opens the door for ISPs to create tunnels across their network from a PE router to another PE router using LSPs. LSPs, however, are also used to deliver other type of in-house traffic within the ISP that is non-customer traffic. This is where multiple layers of edge routers come in to play. The edge routers behind the provider edge routers have LSPs to the rest of the network and these enable the ISP to then run other in house services, connections to transit networks for global route sharing etc all with a robust MPLS network to carry traffic at extreme low latency.

Summary:

IGP’s install paths to reach the destination into the routing table and all traffic destined to that destination simply uses the IGP installed path even if you have a label installed by default.
By default, in junos, IGPs are not going to use the LSP created to reach the edge routers unless you explicitly configure them to.

MPLS creates LSPs to all routers in the network. If configured, traffic can then take the path to the destination router using the LSP. However, default behaviour means it doesn’t.

The mpls.0 table contains all entries to the LIB (label information base) as it is the LIB where the router figures out which received label to be swapped by which label and forwarded across.

LIB is stored in the control plane and it’s forwarding plane variant which is the forwarding table based on labels is called LFIB or label forwarding information base.

Problem is how do you tell traffic to use the LSP instead of path defined by IGP?

It all changes when you configure BGP and let BGP do its magic of next-hop resolution upon receiving a route from an IBGP neighbour.

Only BGP is allowed by default to look in to inet.3 by default, which is the MPLS routing table.

If BGP finds a path for the next-hop in inet.3, it will prefer that over a similar path learned via IGP to the destination router.

BGP would then install the route received by the neighbour into inet.0 routing table with next hop labelled as the IP address of the immediate next-hop in the LSP along with a PUSH label ID.

Traffic to destination for the received route would now use the LSP and reach the edge router using MPLS. After the label is removed at PHP, the packet would be forwarded based on IP lookup.

The label PUSH function is performed only at the egress router.

The label POP function is performed either at the penultimate hop router or the egress router.

The label SWAP function is performed by all the transit routers in the path.

Only the ingress and egress routers know what traffic is actually being carried on the path as the transit LSRs never examine the packet, they simply swap labels and send them across.

All labels in a LSP have local significance. They are unique only on the transite router or Label switched Router (LSR) itself. The same label number can exist on two routers for two different LSPs.

You can configure static MPLS LSPs or use signalling protocols such as RSVP or LDP to dynamically learn LSPs.

Got questions? Leave a comment! Let’s chat.

Rafay Rasool is a Network Specialist with over 8 years of experience designing, configuring and implementing core network solutions based predominantly but not limited to Juniper Routers, Switches and Firewalls along with other vendors such as Cisco, Huawei, Siemens, Aerohive, Ringmaster, Pulse etc for Internet Service Provider and Enterprise Networks.

Rafay is an avid supporter of network automation and likes to code and automate networking solutions.