L3VPNs using MP-BGP over MPLS

L2vpns are great for services where the ISPs don’t require reading any customer information. However, if the product offered by the ISP requires the ISP to handle customer routes to connect multiple customer sites, L3VPNs come into play.
Instead of a customer buying multiple leased lines and creating their own network from office A to office B to Office C, they can simply connect all offices to the ISP’s Provider Edge (PE) routers and let the ISP share all routes between offices. This way the customer can off load it’s routing onto the ISP and simply focus on managing other key business components.

The ISP achieves this VPN route sharing using L3VPNs which essentially run over the ISP’s core network.

The Customer Edge (CE) router shares the routes with the PE router which has the interface facing the CE router configured under a VRF. A VRF is a virtual route forwarder which creates a table separate from the primary routing table on the Provider Edge (PE) router. The customer’s routes are received using a routing protocol such as RIP, OSPF, BGP or even static and stored in the VRF.

The same process takes place at the remote site where the Customer Edge (CE) router connects to the ISP’s PE router and shares its routes onto a VRF specifically created for the customer by the ISP.

Once the two VRFs are created, Multi-Protocol BGP (MP-BGP) shares the routes in the VRF table to all PEs in the ISP with a Route Target (RT). However, only PE’s that have a VRF configured with the same RT receive the route and install it in the VRF table.

Route Targets (RT) and Route Distinguishers (RD) are configured when the VRF is initially configured on the PE router. These are required to be unique throughout the ISP’s core MPLS network as they are used to differentiate and distinguish one customer’s routes from another.

The VRF creates a VPN-v4 label which has the route distinguisher and the route target – both 8bit fields inside. The VPN-v4 label has the routes from the VRF which are received from the customer and these routes are then shared by BGP to all neighbours. The customer simply advertises ipv4 routes to the PE router. When the PE router decides to advertise them to other routers in the ISP’s network, it creates a VPN-v4 field and attaches the RT and RD associated with that VRF. The customer’s interface resides inside the VRF hence why it’s IPv4 routes appear in the VRF table.

The PE routers at the other end of the network where a VRF is configured with the same route target and route distinguisher accepts the routes and after removing the route distinguisher and route target vpn-v4 field, installs the routes as IPv4 routes in the routing table of the VRF. From here any routing protocol such as RIP, OSPF, BGP or even static routing protocol can be used to share the routes between the PE and CE device.

Similarly, as customer adds remote sites in different locations where different PE routers already exist, all the ISP has to do is create another VRF instance on the new PE router the customer has connected a new office site to with the same RD and RT that belongs to the same customer. This usually initiates a route refresh for BGP to then re-advertise routes again as previously this PE router had rejected the advertised routes with the customer’s RT because the VRF wasn’t configured on it. Each customer’s CE router at each location can now see all routes for all remote sites.

This way multiple customers can have the same private routes such as 10.0.0.0/8 shared across the ISP but they would all appear in different VRFs and if configured properly will always be distinguished by the ISP as routes belonging to a specific customer. Due to the correct use of route distinguishers, customer traffic will always stay segregated even when the IPv4 address for a given destination is the same as RD would be different in the VPNv4 field when advertised vi MP-BGP.

A word on RD and RT:

Route distinguisher (RD) distinguishes multiple routes that appear to be the same but belong to different VRFs. RD creates a unique VPNv4 address that is then shared across the MP-BGP mesh. A RD is a value that is attached to the vpn-v4 route and since its different for each customer, distinguishes one customer routes from other customers.

It’s important to remember that routes shared between VRFs are shared using MP-BGP and the PE routers before installing routes into VRFs need a mechanism to distinguish routes for Customer A from Customer B.

Route distinguishers distinguish individual routes when the PE router receives routes from another PE router via MP-BGP and tries to make sense of the received routes that look similar but belong to different customers.

Similarly, Route target is the 64 bit BGP community used to tag prefixes. It tells the PE router which prefix to import into any specific VRF. This insures that customer A, whose VRFs across all PEs use route target 65223:22 will only receive routes tagged and advertised using MP-BGP with community 65223:22.

Once the routes are shared and installed in the correct routing tables across all VRFs, customer traffic intended for any site simply reaches the PE router and then the PE router installs a stacked label of VPN-v4 before pushing a MPLS label belonging to the LSP onto the packet that terminates at the remote end PE router.

The packet is then sent across the MPLS backbone with all transit LSRs simply swapping labels until the packet reaches the egress PE router from where the corresponding customer VRF table is then used to forward the traffic to the remote site’s CE router until traffic reaches its destination.

All routes between all VRFs configured on all PE routers are shared using MP-BGP running over IGP. On the other hand, traffic for all destinations utilises LSPs to reach the destination PE router.

How does MP-BGP work?

MP-BGP is the mechanism that allows BGP to support more than 15 different BGP address families. L3VPNs run using MP-BGP. All these address families are exchanged between BGP neighbours over a single BGP session in parallel.

This way, large service providers usually have a full mesh IBGP network in the core that supports MP-BGP along with route reflectors at the edge of their core network for all PE devices running MP-BGP over this fabric which allows application of L3VPNs on top.

Any address family other than IPv4 over BGP can only work if MP-BGP is configured.

Got questions? Leave a comment! Let’s chat.

Rafay Rasool is a Network Specialist with over 10 years of experience designing, configuring and implementing core network solutions based predominantly but not limited to Juniper Routers, Switches and Firewalls along with other vendors such as Cisco, Huawei, Siemens, Aerohive, Ringmaster, Pulse etc for Internet Service Provider and Enterprise Networks.

Rafay is an avid supporter of network automation and likes to code and automate networking solutions.