Overlay networks explained: underlay, tunnels, VXLAN, LISP and when to use them

Two networks in one

An underlay is the physical network: switches, routers and links running an ordinary routing protocol such as OSPF, IS-IS or BGP. Its only job is to get IP packets reliably from one point to another. An overlay is a logical network built on top by encapsulation: a device at the edge wraps the original packet or frame inside a new header addressed to another edge device, the underlay carries it, and the far end unwraps it.

Because the underlay only sees the outer headers, the overlay can use any addresses and segments it likes, and you can change the overlay without touching the physical network. This decoupling is the main reason overlays are so popular in data centres, campus fabrics and WANs.

Common overlay technologies

VLANs are sometimes listed alongside these because they also create virtual networks on shared hardware, but they work by tagging frames rather than tunnelling them, and they stop at the first router.

VXLAN in a little more detail

Virtual Extensible LAN (VXLAN), defined in RFC 7348, carries Layer 2 frames across a routed Layer 3 network. The devices that wrap and unwrap traffic are VXLAN tunnel endpoints (VTEPs), usually the leaf switches of a spine-leaf fabric. Each virtual segment has a 24-bit VXLAN network identifier (VNI), giving around 16 million segments compared with about 4,000 VLANs. The outer packet uses UDP destination port 4789.

VTEPs need to learn which hosts sit behind which other VTEP. Early designs used flooding through multicast; modern fabrics use BGP EVPN as a control plane, so VTEPs advertise MAC and IP addresses to each other like routes.

interface nve1
 source-interface loopback1
 host-reachability protocol bgp
 member vni 10100
  ingress-replication protocol bgp

This NX-OS style snippet shows the idea: the VTEP interface, its loopback source, and BGP learning for one VNI. Exact syntax differs between platforms.

Benefits and costs

• Benefits: decoupling (change the virtual network without recabling), agility (new segments in minutes, often through a controller), scale (millions of segments, any-to-any reach across a routed fabric with no spanning tree) and mobility (a virtual machine keeps its address when it moves).
• Overhead: every encapsulation adds bytes. VXLAN adds about 50 bytes, so the underlay MTU must be raised (data centres often use 9,000-byte jumbo frames) or packets will be fragmented or dropped.
• Complexity: there are now two networks to design, document and monitor, plus a control plane such as EVPN or LISP.
• Troubleshooting: a capture on the underlay shows outer headers only; you need tools that understand the overlay, and you must check both layers when something fails.

Real-world example: two hospital data halls

A fictional hospital trust had two data halls on the same site. Clinical applications needed some servers to sit in the same Layer 2 segment in both halls for clustering, and the old design stretched VLANs with spanning tree, which caused an outage when a loop formed during maintenance. The network team rebuilt both halls as one routed spine-leaf underlay running OSPF, and added a VXLAN overlay with a BGP EVPN control plane. The clustered servers kept their shared segment, but now as a VNI carried across routed links with no spanning tree between the halls. The team raised the underlay MTU to 9,216 bytes before migration, after a test showed large database transfers failing at the default 1,500.