IPsec explained: how site-to-site VPNs keep traffic safe

What IPsec does

The internet is a shared, untrusted network. When a head office needs to reach a branch office, sending traffic in plain form would let anyone along the path read or change it. IPsec (Internet Protocol Security) solves this by protecting each IP packet. It gives you:

• Confidentiality: packets are encrypted, so an eavesdropper sees only scrambled data.
• Integrity: any change to a packet in transit is detected and the packet is dropped.
• Authentication: each side proves who it is before any data flows.
• Anti-replay: captured packets can't be sent again later to trick the receiver.

These map directly to the CIA triad: IPsec is one of the most common ways networks deliver confidentiality and integrity in real life.

Where you meet IPsec at work

• Site-to-site VPNs: two routers or firewalls build a permanent tunnel between offices. Users don't notice it; traffic to the other site just works.
• Remote-access VPNs: a laptop builds a tunnel to the office firewall (many products also use SSL/TLS VPNs for this).
• Cloud connections: cloud providers offer VPN gateways so an office network can reach cloud servers over IPsec.

The building blocks

Tunnel mode vs transport mode

ESP can protect a packet in two ways:

• Tunnel mode encrypts the whole original packet and wraps it in a brand new IP header between the two VPN gateways. The original private addresses (for example 10.1.1.10 to 10.2.1.20) are hidden inside. This is the normal mode for site-to-site VPNs.
• Transport mode keeps the original IP header and protects only the payload. It's used when the two endpoints are the hosts themselves, or when IPsec protects another tunnel such as GRE.

Original packet:    [ IP 10.1.1.10 -> 10.2.1.20 | TCP | data ]

Tunnel mode (ESP):  [ New IP 198.51.100.1 -> 203.0.113.2 | ESP | IP 10.1.1.10 -> 10.2.1.20 | TCP | data | ESP trailer + ICV ]
                                                               |<-------------- encrypted -------------->|

Transport mode:     [ IP 10.1.1.10 -> 10.2.1.20 | ESP | TCP | data | ESP trailer + ICV ]

How a tunnel comes up

A tunnel is built in two stages. Older documentation calls them phase 1 and phase 2 (IKEv1); IKEv2 does the same job with fewer messages.

1. Interesting traffic arrives. A user at the branch sends a packet to the head office. The router sees it matches the VPN traffic list and starts IKE.
2. Phase 1: build a secure management channel. The peers agree an encryption and integrity algorithm, run a Diffie-Hellman exchange to create a shared secret without ever sending it, and authenticate each other with a pre-shared key or certificates. The result is the IKE SA.
3. Phase 2: build the data tunnel. Protected by the IKE SA, the peers agree how to protect user traffic (for example ESP with AES-256 and SHA-256) and which networks it covers. The result is a pair of IPsec SAs.
4. Data flows. Packets are encrypted with ESP and sent to the peer. Before the SA lifetime runs out, the peers quietly create new keys (rekey) so the tunnel stays up.

IKEv2 is the modern choice: it uses fewer messages, has built-in dead-peer detection and NAT traversal, and supports more authentication options. Use it unless the other side only supports IKEv1.

What both sides must agree on

Most failed VPNs come from a mismatch in one of these settings. Write them down and share them with the engineer at the other end before you start.

IPsec, NAT and firewalls

If a NAT device sits between the peers, plain ESP has a problem: it has no port numbers, so NAT can't track it. NAT traversal (NAT-T) fixes this by wrapping ESP inside UDP port 4500. Peers detect NAT automatically during IKE and switch over.

On any firewall in the path, allow these between the two peer addresses only:

• UDP 500 for IKE
• UDP 4500 for NAT traversal
• IP protocol 50 (ESP) when there's no NAT

Also remember NAT exemption: if the router translates internal traffic to its public address before encrypting it, VPN traffic no longer matches the tunnel. Exclude site-to-site traffic from your internet NAT rule.

Example: site-to-site tunnel on a Cisco router

Head office LAN 10.1.0.0/16, public address 198.51.100.1. Branch LAN 10.2.0.0/16, public address 203.0.113.2. This is the head-office side using IKEv2; the branch is the mirror image.

! 1. IKEv2: how to protect the management channel
crypto ikev2 proposal FMT-PROP
 encryption aes-cbc-256
 integrity sha256
 group 19
crypto ikev2 policy FMT-POL
 proposal FMT-PROP

! 2. Who the peer is and how we authenticate
crypto ikev2 keyring FMT-KEYS
 peer BRANCH
  address 203.0.113.2
  pre-shared-key Use-A-Long-Random-Key-Here
crypto ikev2 profile FMT-PROFILE
 match identity remote address 203.0.113.2 255.255.255.255
 authentication remote pre-share
 authentication local pre-share
 keyring local FMT-KEYS

! 3. How to protect user traffic (phase 2)
crypto ipsec transform-set FMT-TS esp-aes 256 esp-sha256-hmac
 mode tunnel

! 4. Which traffic goes into the tunnel
ip access-list extended VPN-TRAFFIC
 permit ip 10.1.0.0 0.0.255.255 10.2.0.0 0.0.255.255

! 5. Tie it together and apply it to the internet interface
crypto map FMT-MAP 10 ipsec-isakmp
 set peer 203.0.113.2
 set transform-set FMT-TS
 set ikev2-profile FMT-PROFILE
 set pfs group19
 match address VPN-TRAFFIC
interface GigabitEthernet0/0
 crypto map FMT-MAP

Many teams now prefer route-based VPNs (a virtual tunnel interface plus routes, or a dynamic routing protocol) instead of crypto maps. They are easier to scale and to monitor, but the IKE and IPsec settings above stay the same.

Verify and troubleshoot

show crypto ikev2 sa          ! is the IKE SA up? (state READY)
show crypto ipsec sa          ! are packets being encrypted and decrypted?
show crypto session           ! quick summary of every tunnel
debug crypto ikev2            ! detailed negotiation, use with care in production

Work through it in order, just like the OSI model:

1. Can the peers reach each other? Ping the peer's public address from the router. If not, it's a routing or ISP problem, not IPsec.
2. Is IKE getting through? If the IKE SA never appears, check that UDP 500 and 4500 are allowed and that both sides point to the right peer address.
3. Does IKE fail to authenticate? Check the pre-shared key (watch for typos and trailing spaces) or the certificate and its date.
4. Does phase 2 fail? Compare the transform sets and make sure the traffic lists are exact mirrors: 10.1.0.0/16 to 10.2.0.0/16 on one side, 10.2.0.0/16 to 10.1.0.0/16 on the other.
5. Tunnel up but no traffic? In show crypto ipsec sa, if encaps rises but decaps stays at zero, the other side isn't sending back. Check its routes, its NAT exemption and its firewall rules. If neither counter rises, the traffic isn't reaching the tunnel: check routing and NAT exemption on your side.

Most common real-world causes: mismatched traffic lists, a missing NAT exemption, a blocked UDP 4500 and a mistyped pre-shared key. Check these four first and you'll solve most "the VPN is down" tickets.