Understand VXLAN Concepts

0:01(upbeat jingle) So you as a data center professional, or a candidate who's going for the JNCIP Data Center Certification Exam, you looked at the blueprint of this course outline or the JNCIP Data Center Exam blueprint, and you saw EVPN-VXLAN and you circled it, because you know, based on some experience, or word of mouth, or based on things that you've already seen here at CBT Nuggets, that really what makes a data center a data center, really the protocols that make it work the way that it does is EVPN-VXLAN.

0:35This is really the big topic, and this is really the reason why you go for this certification exam, and the reason why you study it here on CBT Nuggets. Well, here's the kicker. I'm gonna tell you right now that the reason you're really going here is because you want to answer these questions. What makes EVPN-VXLAN so special? Why is it dominant in the data center and not necessarily in other environments? How do you lay out an EVPN-VXLAN environment, and how do you actually configure it, deploy it, validate it, and troubleshoot it, and then have something really cool up and running?

1:07There's a lot of moving parts, a lot of components to EVPN-VXLAN. And to answer those questions, like "what makes it so great?" is really to answer a few questions. Why is VXLAN special? Why is EVPN special? And why is it that when they're combined together, they make something special for the data center? So what we really need to do is we need to break apart our big question into smaller questions, starting with what makes VXLAN so special. And that's what this set of videos is all about, really diving into VXLAN, understanding the key terms, how it functions, how it can be designed, and how it's evolved throughout the years.

1:43Now, in the next set of videos, we're gonna be focusing in on EVPN, and at that point you'll be able to bring it all together. We'll be able to talk about why the combination of EVPN-VXLAN is a special one for the data center before we actually jump on the CLI and get it configured. So for now, what we're going to do is we're gonna start talking about VXLAN, really introducing the point of VXLAN. What are the key terms? How does it really work? How can you design a layout, different layouts for VXLAN? And how has it evolved throughout the years? So get ready, because this is gonna be some big stuff, but it's so much fun.

2:15I'll see you there.

0:01(bright music) So, what is the point of VXLAN? Why did we create it? What makes it so special? What problem does it solve? VXLAN at the end of the day is a layer 2 VPN, and you hear a lot about layer 2 VPNs when you go into the service provider space. But when you go into the service provider space, you don't hear a lot about VXLAN. Instead, you're hearing about things like MPLS or different forms of MPLS combinations with BGP. So in this video, what we're gonna do is we're gonna talk about how VXLAN really works, or at least at a very fundamental level, and understanding why VXLAN was created and what problems does it solve.

0:39So let's get going talking about the point of VXLAN. I'll see you there. So what is the point of VXLAN? To answer that question is really to answer the question, why did we need VXLAN in the first place, or what was the problem with our traditionally designed layer 2 enterprise and data center infrastructures? And we already answered that question actually, way back when, when we talked about the data center foundations and we introduced EVPN-VXLAN. But to refresh ourselves a little bit and make sure that we're on the same page as to why we need VXLAN, we need to reexamine this just a little bit.

1:12Now, the big thing here is when we deployed layer 2 absolutely everywhere, so in this case, I would have things like 802.1Q trunk ports this way, this way, this way, this way, this way, this way, blah, blah, blah, blah, you get the idea. All of these are 802.1Q trunk ports. What do we know happens then? We have trunk ports or switches interconnecting to switches. We then have to have loop prevention mechanisms because of the very nature of how broadcast traffic works. And to do that, we run something like Spanning Tree Protocol. And in order to prevent these broadcast storms and these switching loops from occurring, we end up blocking links.

1:46We just stop traffic from ever making its way back into our switch by blocking 50% roughly of the available links, or maybe more, maybe less depending on your environment, but roughly 50% of the available links in this particular topology such that this traffic could never actually return back. And that's fine up 'til a point. That's fine for an enterprise, that's fine for an access layer switch. But in the data center, we need extraordinarily high throughput for our very expensive, extraordinarily expensive data center switches and routers in order to actually handle extraordinarily powerful servers and the constant stream of requests and services that they're trying to provide back to our enterprise, or more importantly, back to our tenants.

2:34Since one of the revenue generating streams that many different public clouds or colocation systems is, is that they have a data center. Now they're just gonna lease out some of it to tenants so that they can come along, spin up their own services and leverage our already existing equipment and are already existing throughput and bandwidth and everything like that. So we can't afford to block half of our links or limit half of our topology, but we also can't afford to have switch loops because then nothing works, right?

3:04I mean, that's a huge problem because it just eats up our CPU until all of our devices are pegged out and then rebooting and trying to start over again. So what we ended up doing, as you can see here in the diagram, is we put /31 routed interfaces absolutely positively everywhere. So all of these devices, they no longer have broadcast domains that span from one switch to the next. That's kind of the idea, is if you wanna get off of this switch, you have to be routed off of your subnet.

3:34So that's actually a really cool thing, because when we deployed this, when we actually deploy this with things like IGPs, like OSPF or IS-IS, now by default we have loop-free topologies. IS-IS and OSPF are loop-free because of their link state databases and the nature of link state protocols. Link state protocols all agree on every single possible route in the topology. Therefore, they know how to get, they all agree on the fastest way to get from point A to point B. It's as if all of these switches have the exact same roadmap and we all agree based on traffic patterns that this is the fastest way to get there because we all have the same roads and the same paths.

4:15If, however, any of these devices didn't have the same roads, we could actually create a potential loop, but that's one of the protection mechanisms of link state protocols, is they're typically loop-free because they all have the exact same consistent database. And the second thing is, is because we all agree on the exact same paths and we know they're loop-free, if there is a potential to use more than one path to reach a destination, we can now do load balancing or what they call equal-cost multi-pathing. So we actually moved away from limiting our links and limiting our bandwidth towards actually load balancing across it.

4:51This was a huge step for us by moving into /31s and layer 3 absolutely everywhere. But the challenge then becomes in the data center is the fact that most traffic in a data center actually moves east to west, and the very technologies themselves are that we have servers that are very powerful that we now break down into little bitty mini servers in the forms of virtual machines. That's kind of the whole virtualization thing that you've probably heard of at this point, with ESXi and Hyper-V and some other device like Citrix XenServer and things like that.

5:27So now we in a data center, let me hide Knox for a second, We in a data center deploy servers absolutely everywhere, and then we deploy little virtual machines, or our tenants deploy little virtual machines. Here, let me draw a tenant's pink virtual machine over here. And this is all well and good because these servers can be in direct communication with each other. For instance, this server right here can communicate over layer 3 infrastructure down to this server and they can then say, like, "Hey, Mr. Web server over here, "that web server is actually a host "that is running four load balanced web VMs.

6:08"They're all servicing my website." And the problem that could happen here is that this web server could say, "Oh my goodness, my CPU percentage is at 95%," or, "My RAM percentage is at 95%." We're really, really overutilized, while this server over here is just chilling with a CPU percentage of 5% and maybe a RAM percentage of 3%, just make something up. So this device says, "Hey, can I send you some of my workload?

6:38"Can I send you some of my VMs, "and then we could coordinate the load balancing "of these VMs between each other." And they say, "Sure." So, two of these VMs immediately migrate over here, and this is all well and good, except for these VMs were part of this subnet over here, and now they can't reach this subnet because our broadcast domain stops right here because we have a layer 3 domain in the core. If this VM right here has the IP address, 1921681.5, and this VM over here has 1921681.7, how do they communicate directly to each other?

7:22Well, since they're on the same subnet, they would be trying to generate frames, right? They would be trying to generate frames and communicate on layer 2. But when the frame comes into this switch, what then happens? It can't be sent over a trunk port over here the way you would normally expect. It doesn't communicate over the same VLAN because we have layer 3 all throughout the course separating our broadcast domain, and this is the challenge that we have when we introduce /31s everywhere. In a data center, these VMs are moving left to right absolutely all the time in rapid-fire secession.

7:56These host machines, these big servers, are constantly in communication trying to figure out how can we balance our load out so that we're not overutilized and cooking our hardware or anything like that. So these VMs are migrating back and forth all the time, but they need to maintain connectivity within their own subnet, within their own virtual networks. So the need became that we needed to get this device at 192168.15's frames over here onto this device so that they can communicate directly to each other.

8:28Enter VXLAN to the rescue. VXLAN is all about creating a tunnel, a logical tunnel from point A to point B. The whole idea is that the frame could make its way into the local switch, and the switch would say, "Oh, you know what? "This frame is destined for a VM over here "off of that switch, so what I'm gonna do "is I'm gonna take this original frame, "I'm gonna box it in like this, and I'm gonna sandwich it "inside of my own layer 3 packet "destined for this tunnel endpoint."

9:05So the packet will be routed through the spine, over here down to the switch, and that switch will know to de-encapsulate it by stripping off the layer 3 header, and then it receives a frame that it can punt right down here to the server. Now, how does this actually work? There's a lot more to it than just that, and that's what this set of videos and the next set of videos is really all about. Right now we're just trying to conceptually understand what the point of VXLAN is and why we need it. The real thing that you should take away from this video is that we have broadcast domains for all of our virtual machines on these servers, but these broadcast domains are broken up by a layer 3 core.

9:47This is what we're end up going to end up calling the underlay or the underlay portion of our fabric. And we need to get these frames over our underlay in such a way that the frame can be carried over to the remote destination, and then sent out towards the broadcast domain that it needs to belong in. And it does this by just tunneling a frame inside of an IP packet. It's actually kind of a straightforward concept when you really think about it. It makes a lot of sense. It's no different from how we tunnel a layer 3 packet inside of a layer 3 packet when we deploy a site-to-site VPN, right?

10:23We're just ascending our inside IP packet that's inside of our network over the public internet. The internet is just routing the outer IP packet, and that's kind of the same thing we're doing here. Our core is just routing our outer IP packet until it reaches the tunnel endpoint, which knows to strip it off, and it's being left with a frame that it can forward down directly towards its host. Now, like I said, there's more to it than that, but it's just understanding the concept, and that's really what matters right now, is that we are just going to be tunneling traffic throughout a broadcast domain over a layer 3 core.

10:58That is the point of VXLAN. Now, in the next video, we're gonna examine why aren't we also talking about EVPN, since the whole point of the data center topology is EVPN-VXLAN, right? So that's what we're gonna talk about next. In the meantime, I hope this has been informative for you, and I'd like to thank you for viewing.

0:00[MUSIC PLAYING] Wait a minute, Knox. I came here to learn about EVPN VXLAN, why aren't we talking about EVPN VXLAN? Why aren't we talking about EVPN? In this video, we're going to do is fundamentally break our mind apart for a second, divorce ourselves from the concept of EVPN and VXLAN. And reconsider the fact that EVPN and VXLAN are two completely separate protocols that just so happen to be able to work together. So let's take a moment to actually examine what it means to break apart the data plane from the control plane in this video.

0:36I'll see you there. So to me personally, this is honestly a big topic to kind of wrap your head around. Because lots of times, up at least until this point, throughout your associate journey, your specialist journey, you deploy a protocol, and it just works. But the interesting thing about VXLAN is it, more than likely, will not work on just its own. It's going to need some help getting a frame from point A to point B. And what we're really talking about here is the fact that this solution of EVPN VXLAN is actually two completely separated protocols.

1:10Two completely separated solutions that just so happen to be able to interact with each other. Think about it another way. Remember in the previous example, let me kind of clear up my screen. Let me clear Knox here for a second. If I brought this server down here and we migrated a VM, right? How did this switch, right here, know that it needs to build a tunnel over here to this switch? And how did it know that when a frame comes in from this VM destined to this VM-- or I should say this VM, how did it know that it should go over the tunnel?

1:46This is kind of the tricky part is that VXLAN by itself is not designed to do that. It's not designed to know this. It relies on EVPN in order to figure that out. And I've kind of mentioned this already a few times, I'm going to mention it again. EVPN is nothing more than iBGP, that's really it. It's just iBGP. I know it seems scary because it's got E and a VPN, and you're like, oh, it's going to be like some mega-layer VPN. No, it's just iBGP. Instead of carrying layer 3 prefixes, it's carrying MAC addresses.

2:18And this is really how it's going to work. If I were to give you a crash course on EVPN, when this VM sends in its frame for the first time into the switch, this switch does exactly what all switches do. How does a switch make a forwarding decision? How does a switch actually decide where frames should be sent? When frames come into a device, the first thing a switch does is it learns the source MAC address. In this case, this switch is going to say, OK, I've just learned this VM's MAC address on this interface.

2:52So whenever I receive return traffic, or traffic destined to that MAC address, I know to point it out this interface. Now, the next thing it does is it says OK, do I know where the destination MAC address go? And if it doesn't know it, it floods it out all the interfaces in the same VLAN, except the one that turned it on, right? We know this, or it sends it out the trunk boards until it gets to the destination and we learn about a response traffic, right? That's how this switch normally works. Well this is similar. This is similar. At the point when we've got EVPN up and running, remember, we really just have iBGP up and running.

3:27So we form iBGP neighbor relationships to all of our peers. And, when we learn about a VM's MAC address, what do? We do we advertise it to all of our iBGP peers. We say, hey, I've got this MAC address. You can reach this MAC address off of me. Route a packet to me. So the alternative is when this virtual machine over here on the left sends its frame up to this switch, it learns about the MAC address and then advertises it back over here.

4:00Now, MAC address to MAC address communication is now facilitated. Unicast MAC address communication is now facilitated. Because both of these devices have-- both of these switches have learned they're locally connected MAC addresses and then advertised them towards each other. So now we can tunnel these MAC addresses over here. When we receive a frame destined for the MAC address over here in the bottom left corner, we know that we need to tunnel it to this switch, who advertised it to me in the first place.

4:31OK, this all makes sense. We understand how the MAC addresses are now learned. But the big thing to understand is there are actually many different ways that you could go about learning these different types of technology. EVPN is just one control plane that can be used by VXLAN. LISP is another control plane that can be used by VXLAN. PIM is another control plane that can be used by VXLAN. Do you see where I'm going with this? We don't have to use EVPN.

5:04We just choose to use EVPN in the data center. Now why? We're going to talk about that when it comes time to actually talk about EVPN. For now, you need to understand the concept that VXLAN is its own protocol. It's its own data plane. It can handle figuring out how to build tunnels from one endpoint to the next endpoint and sending frames over those tunnels. But it can't actually learn MAC addresses. It's not about advertising and learning MAC addresses at all.

5:34It relies on, OK, hey EVPN, did you learn where I should send this MAC address? Oh, you did? OK, thanks for that. I'll go ahead and handle sending it from here. VXLAN is just 50% of the equation. It's just a data plane. And like I say, it is a solution that is primarily used in data centers. It is not primarily used in service providers who, often, are trying to figure out the exact same thing. How do we deploy a Layer 2 VPN? How do we tunnel frames across our backbone?

6:06They end up using MPLS instead of VXLAN. Why? Because not all of their traffic is east to west. They require using protocols like RSVP to perform very specific traffic engineering calculations when it comes to things like bandwidth reservations. I mean, think about a service provider at the end of the day, what do they sell? They sell bandwidth. I mean, you go in your home, what did you do? You bought a connection to a service provider. But specifically, you bought a connection with a certain speed, right? 100 megs up or 10 megs up, 100 megs down, or synchronous fiber, or whatever you did.

6:41You bought it. And you were guaranteed a bandwidth reservation. That's kind of the idea here, is with MPLS, service providers have the ability to actually perform traffic engineering, such that when they promise a customer they're going to get this much bandwidth, they can use MPLS to actually reserve that bandwidth throughout the path. So that way, they can actually deploy a Layer 2 VPN, like EVPN MPLS and say, OK, you're going to get this much bandwidth when you go from this site to that site. So this is why they went with MPLS instead of VXLAN.

7:13Now we went with VXLAN because it's so stinking fast and scalable for east to west traffic. And it responds very quickly when it's coupled with EVPN. So this is the big thing right now that you got to wrap your head around, is we're only focused on the data plane right now. But at least you got an idea as to how it's going to know how to make forwarding decisions before we actually dig deep into how EVPN operate. So this is separating the control and the data plane. I hope this has been informative for you. And I'd like to thank you for viewing.

0:01(cheerful music) Now, one of the cool things about VXLAN is there's really not many new key terms to bring to the table when it's time to learn VXLAN. There's really only a couple, but they're important key terms. And they really introduce the concepts and technologies about how VXLAN works. So while this video is focused on introducing you to the key terms like a VNI and a VTEP, what we're really talking about here is, how does a VNI work and how does a VTEP work?

0:33So in this video, that's exactly what we're going to cover. What is a VNI and a VTEP and how do they really work in order to bring VXLAN to life? Let's see. Let's get going. All right, usually when I say like, let's just learn some terms and some definitions, my eyes kind of roll because that's gonna be boring content. It's reminds me of like English class when you had index cards that were just words and definitions. That's not gonna be the case in this video. When we introduce key terms with VXLAN, this is really digging into how VXLAN works and one of the biggest selling points for why you want to go with VXLAN.

1:06The other cool part is there's really not many key terms to VXLAN. Let's talk about the first one. When we have these devices, these switches here, my leaf switches, and they wanna build tunnels to each other, the big thing that we call this, we call these switches now VTEPs. VTEPs stands for Virtual Tunnel Endpoint. Pretty straightforward. It makes a whole lot of sense. This is where a tunnel starts and stops. It's an endpoint for where tunnel can be a accumulator or where they can go.

1:38And we end up forming VTEPs, or I should say tunnels to every single one of my potential partners here. So it kind of becomes a full mesh of tunnels between my VTEPs. The other interesting part is VTEP neighbor ships, VTEP tunnels are always done based on loop back addresses. This actually has a pretty big design implementation or consideration when it comes time to implement VTEPs and VXLAN in your environment because now we have to know how to get from one loop back address to the next in the most optimal manner.

2:14Of course, we also have equal-cost multi-pathing in the mix. Did you notice this about the actual spine leaf design itself, is that spines never actually connect to spines and leaves never actually connect to leaves? But the other interesting thing is, is that every leaf connects to every spine and every spine connects to every leaf. Therefore, destinations are always only two hops away. So I can go one hop here, then down here. I can go one hop here, then down here.

2:44I can go one hop here, then back over here. Destinations are always two hops away in a spine leaf environment. So therefore, no matter how our tunnels end up laying out, we can get a frame from one end of our data center all the way to the other end of our data center in exactly two hops. And that's a big design consideration. So really the big thing that you need to know, first of all, is that when we talk about these switches, we actually refer to them as VTEPs. This is where the switch is going to be a tunnel endpoint.

3:15And all of our VTEPs typically form a full mesh as long as they need to, as long as they need to tunnel traffic towards each other. Now, how do they know when they need to? This comes up into the next big talking point, and that is VNI, Virtual Networks and Virtual Network Identifiers. Pretend for a moment that we have this blue client right here. We are a public cloud. And we now have someone who's come along, they've given us their credit card and they wanna start spinning up resources in their environment.

3:47So they spin up a couple VMs. What is a big thing that those VMs do? They communicate on a network. And they want to be able to declare that they wanna use VLAN 10 as their network to communicate on. Well, no big deal so far. Except, maybe there is a big deal. What happens now when a pink client comes along and they spin up VMs and they want to use VLAN 10? Well, by default you would think that they actually couldn't because then they would have overlapping broadcast domains.

4:18For instance, if I had a VM here and a blue VM here, and we build our tunnel like so, so far so good. As this VM goes to communicate over the tunnel, it'll get de-encapsulated and the device will go, oh, VLAN 10, but wait a minute, it's got two VLAN 10's. So that doesn't really work well, does it? So you would then think, okay, well then we're limited to the amount of VLANs that we have. We can never have overlapping VLANs, right. Well, the other big problem with that is that there are only 4,094 usable VLANs because VLANs are a 12 bit number.

4:56So now effectively what I'm saying is the maximum amount of customers we could have then is 4,094 customers. Would that work for AWS? Would that work for Azure? No. And these customers would only have to want one VLAN. That doesn't work that way. The cool thing that VXLAN brings to the table, they use a 24 bit identifier to identify their customer's VLANs. This is probably one of the cooler things. So we can say for customer pink, use a VNI or a VLAN identifier, a Virtual Network Identifier of something like 65,322, make something up.

5:36And we say, blue customers VLAN 10 is directly mapped to this VNI. And then for our pink customer, we could say, use a VNI of something like 131,013, make something up. And we say, pink customers VLAN 10 gets mapped to this VNI. So when we advertise our Mac addresses, we also say, hey, I learned this Mac address on this VNI. So now when the frame goes over the tunnel, this VTEP can receive the frame encapsulated with a packet that also identifies the virtual network that it belongs to, like 131,013.

6:17And this device can go, oh, 131,013, that's specifically associated with my pink client. Therefore, I'll de-encapsulate the frame and send it out towards the interfaces that face my pink client. So this is the huge, huge win about VXLAN, is the fact that it can now support millions upon millions of virtual networks. And that means our customers can now have overlapping VLANs because we can use the VNI to uniquely identify our customers VLAN.

6:50And that way we can send the traffic layer three throughout the entire topology. And we just say, this is specifically associated, this frame it's specifically associated with this VNI. So when you receive this frame, send it out towards all of the devices that are connected towards this VNI in that VLAN. Now, I do wanna specify that it is usually or sometimes manually mapping this VLAN to a specific VNI. But there are ways that you can automate that process too. There are discovery and advertisement methods that you should know exists.

7:21And that way when you get into the expert level tier, those are the things that you'll need to know. But when it comes time to configure it, we're just gonna do a basic manual mapping of a VLAN to a VNI and you'll be kind of surprised at how easy it is. So this has been introducing the two key terms for VXLAN, VTEPs and VNIs. I hope this has been informative for you. And I'd like to thank you for viewing.

0:06Now, the temptation is to say that VXLAN comes in many different shapes and sizes. But in a data center, it only comes in two shapes and sizes. The options that we have when a data center is deploying EVPN-VXLAN are really down to spine-leaf architectures or spine-only architectures. So that's what we're going to cover in this video is what are the differences between these two and when would you actually use them. Here's the spoiler. You're pretty much almost always going to use spine-leaf architecture. Spine-only really comes in for a transitional period when you're moving towards EVPN-VXLAN.

0:38So what are the businesses that actually use this and what are the use cases and how is it actually laid out? That's what we're talking about in this video. Let's get going. Now, when we talk about VXLAN, especially EVPN-VXLAN, in the data center, we are almost exclusively talking about deploying it in the spine-leaf model. But you need to know that there are a couple different styles of EVPN VXLAN that can actually be deployed, and that's spine-leaf or spine-only EVPN-VXLAN.

1:10And this actually has a very large implication about the flavors and behaviors of EVPN itself. Now, this won't make a whole lot of sense yet. You have to actually dive deep into EVPN to understand the impacts that it has. But I wanted to set this up now. I wanted to set this up now because this is, at the end of the day, a deployment designed for VXLAN. So let's start off by talking about EVPN-VXLAN in spine-leaf. We understand some of the moving components now.

1:41We understand VTEPs and VNIs. And when this traffic comes in, we have slash 31s absolutely everywhere throughout our core. So we're going to be routing traffic as it moves over a tunnel. Here, let me draw a little tunnel here. Over a tunnel, let's say this tunnel right here. So as a frame comes in, we'll encapsulate it in the packet, and it'll be routed up into a spine and then back down towards one of the other leaves. That's another VTEP. So in the spine-leaf deployment, the VTEPs exist on the leaf tier.

2:17This is the big thing. And the spines really do nothing more than facilitating very fast routing between the leaves. And the traffic still moves, primarily east to west. That's the idea here is that really the brains of the operation are actually the leaves themselves. We also remember this is where external networks connect in, and this is also where like the internet will connect in. It'll all connect in on the leaf tier, and these routes will all be propagated this way. Now, the actual deployment of the underlay and the overlay, we're not there yet.

2:48But just know right now in the spine leaf deployment, this is the gist of it here is that our VTEP and our routing decisions themselves and how we tunnel traffic, that all occurs on the leaf layer. But for traditional enterprises and traditional data centers that are currently running layer 2 everywhere, what are you going to do if you want to deploy EVPN-VXLAN? Are you literally going to just take a weekend and log in to all of your devices and deploy layer 3 absolutely everywhere? If you're a large enterprise or a medium to large data center, there's absolutely no feasible way.

3:23What you would have to do is you would have to transition into EVPN-VXLAN by first deploying EVPN-VXLAN on your spines. So instead of having slash 31 everywhere, mark that out. You're still going to have 802.1q trunks all the way up to your spines. But then this is where things get wild. You're spines are now going to be the VTEPs. And we will now interconnect our spines so that the tunnel looks like this.

3:54The tunnels now exist between the spines. And of course, you're going to have more than two spines. You're going to have multiple spines like so, and they'll tunnel each other like this. And they'll be the ones who will share, OK, well, I've just learned about this MAC address off of this link. So if you want to reach this MAC address, you can reach me here. And if they want to tunnel the traffic through the VTEPs, they'll be able to do it this way. That way you'll now start propagating the learning of MAC addresses throughout EVPN-VXLAN environment even though you're going to be trunking this traffic up towards the spines.

4:28Now you're thinking to yourself, wait a minute. 802.1q, that means spanning tree protocol. That's correct. That's why our spines now are deployed with multi-chassis link aggregations down to the actual leaves themselves. And that deployment itself, right there deploying an MC lag between a spine and a leaf, in EVPN is a massive, massive, massive consideration. And when we get into EVPN-VXLAN, you'll understand why we typically don't do spine-only deployments in a data center.

5:05Now, this design model right here-- I want to stress this-- where you have multihoming, that's really what this is called. This switch is now multi-homed to EVPN-VXLAN. It now has a link up here as well as a link up here. It sees this as one LACP link. We know this because we know MC lag. This device is now multihome. This really actually comes up in the service provider world. Because they'll have customer equipment, like a customer switch, that is now multi-homed to our two provider equipments to provide redundancy.

5:36So you deploy MC lag this way. And then when they want to have a layer 2 VPN and they send frames in, then this becomes an EVPN consideration. And that's really where you see this in the EVPN world, and we're going to talk a lot more about this. But you should know that in the EVPN-VXLAN deployment, doing spine only is going to have a major implication as we deploy MC lag between spines and leads. But for the most part, we deploy spine-leaf because it just keeps things nice and clean with layer 3 everywhere and our VTEP, instead of being at the spine layer, they're deployed at the leaf layer.

6:15So this is understanding the two different deployment models for EVPN-VXLAN in a data center. You've got spine-leaf, which is almost deployed like 95% of the time, and spine-only which is primarily used as a transition towards spine-leaf to start edging away from layer 2 links absolutely everywhere. So that's been understanding the two design models. I hope this has been informative for you, and I'd like to thank you for viewing.

0:01(soft music) So quite frankly, if I'm being honest, there's not a whole lot to actually inspecting a VXLAN frame as it's being traversed across a Layer 3 infrastructure. If we were to inspect the actual encapsulated frame, and look at what's going on, it looks kind of exactly like what you would expect. However, you do have to consider how much overhead is eaten up when we actually use EVPN and VXLAN. So in this video, what we're gonna do is we're gonna decipher the VXLAN packet just a little bit more.

0:33I'll see you there. Now, I challenge anyone to find a better source of information on a protocol than the RFC itself. This is one of the best ways that we can actually inspect what is actually carried in a VXLAN header and how much overhead is it really eating up. And if I jump down into section 5 of RFC 7348, let me actually zoom in a little bit here. That way we can see big text. Let me see, make sure I'm not in the way, okay, cool. So we've zoomed in quite a bit. Let me zip down. Okay, look, right here, the VXLAN header that we're adding into the mix is an 8-byte field, that is 64 bits that we've tacked on for the VXLAN header.

1:14Recall for a second that what are we really talking about though? We have our original payload, our original frame. If we think about the way a frame looks, we've got a source MAC, a destination MAC, and then we have a source IP and a destination IP, that's the Layer 3 portion. We have a source port and a destination port, and then we have the payload, right? The data that comes after that. Well, we're taking this, and we're sandwiching it inside of another packet. So what we're doing is right here, we're shoving in a VXLAN specific header to be used by VTEPs.

1:51Then we have our source and destination ports. The destination port for VXLAN is a well-known UDP port. That is UDP port 4789, that's how I remember it, I just like "7, 8, 9". It's just easy to remember that way. So we've got a source port that's generated by a device, and then a destination port for UDP 4789. Then we've got our Layer 3 header for the routing throughout the core. And then the next hop MAC addresses, we've got the Layer 2 header, you get the idea here. So we've taken our original device, we've shoved in 64 bits potentially for the VXLAN header, then taken up that additional space for the ports, the IP addresses, and the MAC addresses to be used to move this traffic throughout the core.

2:37So this is already some interesting things to really consider, but let's inspect the VXLAN header a little bit closer. I'm gonna actually zip down to the diagram, instead of just reading that boring info to you, and we'll jump down to the VXLAN header. Well, what do you see here? We see a bunch of "reserves", and then a "reserved" and then dead smack in the center, the VNI. Ah, now it makes a whole lot of sense. When we send traffic to destination port 4789, we then say, oh wait, the device that received it says, "Wait a minute, this is coming in on UDP 4789."

3:12"Therefore, it must be VXLAN traffic," "because that's the application specific port." "Let me see, before I de-encapsulate this," "which virtual network," "therefore, which of my customer's VLANs" "is this really associated with?" So it looks in the VNI, and then says, "Ah, I know, this is for, you know," "my Pink customer's VLAN 10." "Let me strip off all of that unnecessary information" "and boom, what am I left with?" The original payload, the original frame that exists right there. So we see the original inner ethernet header.

3:44There could be some additional info, like we see we could keep VLAN tag information if we wanted to keep that VLAN tag information. And then the actual payload, the actual payload itself, followed by lastly, a frame check sequence. So inspecting this actual, this actual payload, what changes with the VXLAN header, and how does it actually transmit throughout the topology? I think is really important. So for recapping it from the top, we've got our outer ethernet header. This is going to be, if we understand the OSI model, and how traffic moves throughout the network.

4:14The outer ethernet header is my next hop MAC address, right? That's my destination next hop MAC address that's gonna be as it traverses through the core. Then I scroll down, we've got the outer IPv4 address. This is really gonna be the destination IP address of the VTEP device. Then the outer UDP header. This is the VXLAN destination of UDP 4789, the VXLAN header, really that's the VNI, so we know what network it goes to. And then, the original inner ethernet header and the payload itself. So now we've deciphered the VXLAN packet, and we actually know how traffic moves throughout the topology, and how it's received by a VTEP.

4:50And it knows how to de-encapsulate it, and which interfaces and which broadcast domains it needs to send the original frame out of. I hope this has been informative for you, and I'd like to thank you for viewing.

0:00(bright music) One of the interesting things that's easy to overlook is when you focus in on VXLAN, what are you really focused in on? You're focused in on taking a frame and moving it across an infrastructure so that it can communicate in the exact same broadcast domain no matter where these devices live within our data center. But what happens when they don't want to communicate within the broadcast domain? I mean, these VMs, they spin up, they have a default gateway, right?

0:31And they wanna be routed out to the internet sometimes. How then do we handle this scenario when we've deployed EVPN-VXLAN? How do we actually handle routing when the devices are configured to actually tunnel frames? That's what this video is all about, understanding our layer 2 and our layer 3 gateways in the various designs that can be deployed of EVPN-VXLAN. Let's get going. Now, here's an interesting one that you, like I say, may have been easily overlooked when we're talking about VXLAN, because at the end of the day, VXLAN's real purpose in life is to make sure that we're tunneling frames from one broadcast domain across our data center to reach the same broadcast domain.

1:11The broadcast domain has now been broken up by layer 3 links, but we can still reach it because that's the whole purpose. This actually introduces the first key term. If we've got this blue device right here, my blue devices are in VLAN 20, and they're associated with the VNI of 4920. So we've got VLAN 20 existing all over the place, and we agree that this is for VNI 4920. We now build a tunnel to our remote VTEP, and we agree, "Hey, we're looking for MAC addresses "in VNI 4920, so if you learn of any MAC addresses "that are associated with VNI 4920, send them my way."

1:51So EVPN steps in, it does its thing. We share MAC addresses, and now all of a sudden we've got communication for the broadcast domain associated with VLAN 20 or VNI 4920. That makes this device, this VTEP, called a layer 2 gateway. We can now extend layer 2 communications across our data center because these devices are now layer 2 gateways. So that's honestly pretty straightforward. We understand that why this would be called a layer 2 gateway.

2:21It's just 'cause we're extending layer 2 communications. But what about layer 3? How does layer 3 work at all? And this is actually a huge talking point about EVPN and EVPN-VXLAN. Well, the first things first, what you need to know about this, this is one of the most incredible and cool design concepts that there are, is that these VTEPs, these switches on our leads, they will serve as the default gateway for our client devices, our host servers or virtual machines or any of the devices that are communicating into this.

2:57Now, how does this really work? This is actually pretty cool. Let's say VLAN 20 here, my blue VLAN associated with VNI 4920, let's say it exists in the 10.1.1.0/24 subnet. So we wanna give this device a gateway IP address of 10.1.1.1. No problem. Now this device can communicate into the switch and reach its default gateway. But then what happens? Let's say we now wanna route traffic from VLAN 20 to VLAN 30?

3:30Let's say this is, make something up like a Windows 10 computer, a host computer, and it needs to be routed over here towards this file share. Makes sense. This is a pretty typical need in our environment. How does that then work? Honestly, if you think about the stuff that we covered in the previous video when we talked about how a VXLAN packet is actually formed, it makes a whole lot of sense. All we're gonna do is we still use VTEPs and we still use the same tunnel, but instead we just say the destination VNI is now 4930.

4:06So this device will communicate and send a packet to its default gateway, and it says, "Look, I'm trying to get over here into this pink subnet," which, make something up. It could be something like 10.1.2.0/24, and the device will say, "Okay, well, since I know "about VNI 4930, I'm gonna go ahead "and encapsulate your frame and send it over via VNI 4930, "and that way it'll go out towards the pink customer "and the pink file share."

4:37And the file share will reply with a very similar traffic pattern. When it comes time to reply, this switch will send it with a VNI of 4920, and it'll come back towards this switch like so. That then means that these devices are now layer 3 gateway, but there's a very critical thing that you need to understand about data centers, and it all comes down to virtual machine mobility. So it's not just as simple as just a default gateway. What we actually have to deploy to make this work is an anycast gateway.

5:10Let me clear up the screen a little bit and I'll show you what I mean. One of the biggest things that makes a data center a data center is the fact that what we're really doing at the end of the day is spinning up a whole bunch of little virtual machines, and these server hosts can migrate these virtual machines back and forth all the time, and they absolutely do. But think about this. If these virtual machines had the default gateway of 10.1.1.1, and that was existing here, what happens when this virtual machine migrates all the way over here towards this switch up here?

5:44Well, without any sort of special configurations or anything really going on here, we would have to jump onto the virtual machines, reconfigure the default gateway to now be the IP address of the new VTEP, the new switch, and then it would have to ARP for that new IP address, find out the MAC address, and then default gateway communications could resume. What we can do, however, though, is we can actually configure these switches to tell them, "Hey, you are going to be a VTEP, "and you are also going to be an anycast gateway."

6:14So we will tell all of these VTEPs that you will all support the exact same virtual IP address and the exact same MAC address. Here, I'll write that as abbreviated. That way when a virtual machine migrates from one host to the next, the default gateway remains the exact same and the ARP resolution remains the exact same, and therefore connectivity during a migration never really gets lost. You may lose one or two packets, but that's about it.

6:46Most of the time you don't lose anything at all. In fact, I was doing things like Plex servers back in the day, and I would actually migrate a Plex server from one VM to the next while streaming a video, and I never actually lost any packets or frames. I was able to continue streaming videos from the Plex server while it was migrating back and forth. That's how good these things are at handing off from one machine to the next, as long as the network is able to support it. So now we understand what a layer 2 and a layer 3 gateway is in the context of EVPN-VXLAN during the spine-leaf deployment, but we also understand a major concept, that we will be deploying anycast gateways on these leaf nodes.

7:27That way when these migrations take place, none of the traffic actually gets lost. So I hope this has been informative for you, and I'd like to thank you for viewing.

0:01(gentle music) Now, the last topic that we need to talk about with VXLAN is more about the evolution of VXLAN's control plane. When VXLAN was implemented as an RFC, it was ratified in an RFC, it was deployed without a control plane. They call this controllerless VXLAN. And instead, what it did is it used multicast to broadcast effectively all of these MAC addresses throughout the topology until the remote devices learned the MAC addresses and then could build shortest path trees to each other.

0:35So in this video, we're gonna do is we're going to explore how this multicast deployment of VXLAN worked without a control plane. And that's actually gonna help us as we transition into the EVPN discussion in the next set of videos. So let's take a look at how multicast worked for supporting VXLAN back in the old days. I'll see you there. Now, this is not something that you need to know a whole lot of information about for real-world deployments. This is not something you're gonna encounter very much in today's production data center networks. However, you do need to know this for the JNCIP-DC exam, as well as it's just kind of fun information to have.

1:12If you ever find yourself in an impromptu networking history trivia battle, you've got this arrow in your quiver. So let me clear my head and just kind of tell you what we're talking about here. When VXLAN and the RFC was designed and deployed, there was no mention of how EVPN or BGP could be used as the control plane. Instead, what it wanted to do was let's just use multicast and handle the distribution of traffic on our own.

1:42And this is specifically going to be using PIM sparse mode. Now, if you need to pause and circle back and go watch the multicast traffic, I'd encourage you to do that because the expectation at this point is that you do know at least fundamentally how PIM sparse mode and rendezvous points really work. So how does it work? We've got our VTEPs here, this is a spine leaf architecture, so our VTEPs are going to be located on our leaves, and we have a broadcast domain, actually there's a word B there, we have a broadcast domain, VLAN 20, which we're mapping to VNI 4920.

2:15So the frame comes in on this device right here, and this device learns the MAC address. How then does it relay this MAC address information to all of the other devices that are interested in commuting on, or communicating on VNI 4920? What we really did is we associated VNI 4920 with a multicast group. So we'll say, let's just call this 224.1.5.30, make something up.

2:47So we would then stand to our rendezvous point, which is sitting here in the middle of our network, we would then send a star,G, saying we would like to join the group of 224.1.5.30 and the flip side, the exact same thing would happen over here for VNI 4920. The frame comes in and we have mapped that to 224.1.5.30, and this device will send a star,G to 224.1.5.30 Now, because we're also sourcing traffic from 224.1.5.30, we can send our S,Gs over here to the rendezvous point who will then forward it on to the devices whoever requested a star,G, and the same thing happens in the other direction.

3:41And this is where the magic really starts to occur. When traffic that has been sourced from this device right here makes its way in and we start advertising an S,G as well as a star,G and then it makes its way over to this VTEP, this is where it gets magical. The VXLAN frame now looks like this. We have the original frame, we still have the VXLAN header which identifies the VNI of 4920 but now the outer packet is 224.1.5.30, right?

4:13Well, because we know that this is associated with VNI 4920 and because we now see the source IP address and because we can now learn the incoming MAC address, we now know how to build our VTEP tunnel and send direct communications over the tunnel that way. So this is how controllerless VXLAN was always working. Instead of sending traffic directly towards a unicast IP prefix, we associated a VNI with a multicast group, and all of the devices that joined the group based on communicating to the rendezvous point and originating a star,G join request, they now receive multicast traffic that comes inbound.

5:00But when they receive that multicast traffic, they immediately see the VTEP source IP address, the VNI, and they learn the MAC address that's coming inbound that way. So this is the way that multicast and PIM could have been used to actually deploy VXLAN back in the day. What we find out is that that's not nearly as scalable as using EVPM, but we'll talk about that more in the next set of videos. For now, you understand how the original RFC can called for controllerless VXLAN using multicast. It all originates the exact same way that multicast handles it, we go to a rendezvous point and then we build shortest path trees.

5:36In this case, we're building a shortest path tree towards a specific VTEP for a specific MAC address we learned on a specific VNI. I hope this has been informative for you and I'd like to thank you for viewing.

0:00(soft music) So now we have a pretty good grip as to what VXLAN brings to the table and how it actually operates. We learned the key terms, we learned the designs and layouts and how all traffic would flow throughout the topology. And then we even talked a little bit about the evolution of VXLAN and how it interacted with a control plane or more or less, without a control plane by using multicast. So now that we understand VXLAN, we can transition a little bit to talk about the other half of the topic, which is EVPN.

0:34At that point, we'll have our heads wrapped around EVPN VXLAN pretty well. I hope this has been informative for you, and I'd like to thank you for viewing.