Flamingo Raises $4.5M Seed Round · Led by Vertex Ventures, bringing total funding to $6.7M to build the open, AI-native layer for IT and security.
An IPsec VPN is an encrypted tunnel built from two protocols: IKE agrees on keys and settings, ESP encrypts the packets. This guide covers tunnel vs transport mode, IKEv1 vs IKEv2, NAT traversal, the IPsec vs TLS VPN vs WireGuard trade-off, and the phase 1 and phase 2 failures behind most broken site-to-site tunnels.
Two offices need to share a file server, and the internet between them is a public road. Something has to wrap every packet so nobody along the way can read it or change it, and both ends have to agree on how. That something is usually IPsec, and this guide explains what an IPsec VPN is, how its parts fit together, and where it breaks.
The internet moves data in small envelopes called packets. Each packet has an address label on the outside and the contents inside. By default anyone who handles the packet on its way can read the contents, copy them or swap them for something else.
IPsec is a set of rules that puts a lock on those packets. It does three jobs. It encrypts the contents so a reader in the middle sees noise. It adds a checksum built from a secret key, so any change in transit is caught at the other end. And it proves each end is who it says it is before any of that starts.
The name stands for Internet Protocol Security, and the important word is Internet Protocol. IPsec works at the IP layer, the same layer as the address label itself. That is why it can protect any traffic at all, a file share, a phone call, a database connection, without the applications knowing it is there. A TLS connection, by contrast, protects one application's conversation at a time.
An IPsec VPN is what you get when you use those rules to join two networks, or a laptop and a network, as if they shared a cable. The IETF architecture document that defines the whole family is RFC 4301 from 2005. The pieces inside it are older, but that is the version every current firewall follows.
If the lock-and-key idea is new, PowerCert's short animation walks through it before the detail below.
Every IPsec connection has two separate jobs, and two separate protocols do them. Mixing them up is the root of a lot of confused troubleshooting.
The first job is agreement. Both ends need the same encryption method, the same integrity method, the same keys and the same idea of which traffic belongs in the tunnel. Internet Key Exchange, IKE, handles that conversation. It runs over UDP port 500, it authenticates the peers with a pre-shared key or a certificate, and it uses a Diffie-Hellman exchange so both sides can compute the same secret key without ever sending it across the wire.
The second job is carrying the traffic. Once IKE has done its work, Encapsulating Security Payload, ESP, takes each data packet, encrypts it, adds an integrity check and sends it on. ESP is its own IP protocol, number 50, not TCP or UDP, which matters the moment a firewall or a home router sits in the path.
The result of a successful IKE negotiation is called a security association, or SA. Think of an SA as a signed agreement: these two peers, these algorithms, these keys, this traffic, for this long. IKE builds one SA for itself, the control channel, and then one or more child SAs for the data. Child SAs are one-way, so a working tunnel always has a pair, one in each direction.
The algorithm names you pick inside those agreements, AES, SHA-256, ChaCha20, come from the same toolbox every encrypted protocol draws on. Our explainer on what a cipher is covers how they work and why key length matters.
IPsec originally shipped with two ways to protect traffic. Authentication Header, AH, protocol 51, proves a packet came from the right peer and was not changed. It does not encrypt anything. ESP does both: it encrypts and it proves integrity.
AH had one trick ESP did not. It also protected parts of the outer IP header, including the addresses. That sounded useful in 1998. In practice it means any device that rewrites the address on the way, which is every NAT router on earth, breaks the AH check. A home connection, a hotel Wi-Fi, a carrier-grade NAT on a cellular link: AH fails on all of them.
NIST's guide to IPsec VPNs, SP 800-77 Revision 1 from June 2020, settles the question in one line: "AH has been obsoleted and should not be implemented or deployed." Where encryption is not wanted, the guide points to ESP with null encryption or AES-GMAC instead. So when you configure a tunnel today, the protocol choice is ESP, and the remaining decisions are about modes and algorithms.
ESP can wrap a packet in two ways, and the choice decides what an observer on the internet can still see.
Transport mode protects the contents and leaves the original IP header in place. The source and destination addresses stay visible. It suits a conversation between two specific hosts that can each talk IPsec themselves, such as two servers protecting a replication link, or L2TP over IPsec on an older remote-access setup.
Tunnel mode wraps the entire original packet, header and all, inside a new packet. The outer header carries the addresses of the two gateways. The inner header, with the real source and destination inside each office, is encrypted along with everything else. An observer sees two firewalls talking, nothing more.
Site-to-site VPNs use tunnel mode, because the gateways are protecting traffic for whole networks behind them, not for themselves. That also means the inner addresses on both sides have to be distinct, a point that comes back later as one of the classic failure tickets. If the address ranges themselves are unfamiliar, the guide to public vs private IP ranges covers which blocks each office is likely using.
IKE has two versions in service, and the documentation for each uses different words for the same steps. Here is the plain version of both.
IKEv1 works in two phases. Phase 1 builds a protected control channel between the two gateways. It either uses main mode, six messages that hide the peers' identities, or aggressive mode, three messages that are faster and leak the identity in the clear. Phase 2, also called quick mode, runs inside that channel and builds the child SAs that carry data. When a vendor screen says "phase 1 up, phase 2 down", it means the gateways trust each other but cannot agree on what to send.
IKEv2, defined in RFC 7296 in 2014, collapses most of that. Four messages in two exchanges, IKE_SA_INIT and IKE_AUTH, set up the control channel and the first child SA together. More child SAs come later with a small CREATE_CHILD_SA exchange. It also has dead peer detection, NAT traversal and support for EAP authentication built into the standard rather than bolted on as vendor extensions.
The words phase 1 and phase 2 survive in every firewall menu, even on IKEv2. Read phase 1 as "the IKE SA" and phase 2 as "the child SAs", and the menus line up with the standard.
NIST's guidance is blunt about which to use: "IKEv1 should not be used for new deployments, and existing deployments using IKEv1 should be converted to IKEv2 when possible." For a walkthrough of what each IKEv1 message carries on a Cisco box, Deploy and Defend's tutorial goes message by message.
IPsec gets used in two shapes, and they are configured very differently even when the protocol underneath is the same.
A site-to-site VPN joins two networks through their gateways. The firewall in office A and the firewall in office B form one tunnel, and every device behind either firewall can reach the other side without installing anything. Branch offices, a link to a cloud network in AWS or Azure, and connections to a supplier's network all look like this. The two gateways usually authenticate each other with certificates or a pre-shared key, and the tunnel stays up permanently.
A remote-access VPN joins one device to a network. The laptop runs a client, Windows' built-in one or a vendor app, and connects to a gateway at the office. Each user authenticates separately, ideally with a certificate or EAP plus MFA rather than a shared secret. NIST's guide is explicit that for remote access, "EAP-TLS or machine certificate authentication should be used."
Both shapes are possible on a Windows server too, with RRAS terminating IKEv2 for laptops. Our guide to running a Windows VPN server covers that build and the cases where a firewall or a Zero Trust product is the better choice.
IPsec was designed for an internet where every device had a public address. NAT, the trick that lets a whole office share one address, arrived shortly after and broke two assumptions at once.
ESP has no port numbers. A NAT router tracks connections by address and port, so it has nothing to map an incoming ESP packet back to the right device behind it. And if the IKE exchange includes the original addresses in its checks, the rewrite makes them fail.
NAT traversal, NAT-T, fixes both. During the IKE exchange, each side sends hashes of the addresses it thinks it has. If the other side sees different ones, a NAT is in the path, and both peers switch to wrapping ESP inside UDP on port 4500, which RFC 3948 defines. UDP has ports, so the NAT router can track it like any other flow.
That gives the short list of what a firewall in front of an IPsec gateway has to allow:
| Traffic | What it carries | When it is used |
|---|---|---|
| UDP 500 | IKE negotiation | Always, at the start |
| UDP 4500 | IKE and ESP after NAT-T switches | Whenever a NAT sits between the peers |
| IP protocol 50 (ESP) | Encrypted data | Only when there is no NAT in the path |
Protocol 50 is not a port. A rule that opens "port 50" opens nothing useful, which is a common reason a tunnel negotiates and then carries no data. The TCP vs UDP explainer covers why UDP is the carrier here.
UDP is also why a stateful firewall in front of the gateway needs explicit inbound rules for the peer, rather than trusting it to recognise return traffic on its own.
IPsec is not the only way to build a VPN, and for some jobs it is not the best one. The three common choices differ in where they sit, how they negotiate and how much there is to configure.
SSL VPNs, more accurately TLS VPNs, run over TCP or UDP 443, the same port as every website. They get through hotel and airport networks that block everything else, and most firewall vendors ship one with a client. WireGuard, merged into the Linux kernel in version 5.6 in March 2020, takes the opposite approach to IPsec: one fixed set of modern algorithms, no negotiation, a small codebase and public keys configured like SSH keys.
| IPsec (IKEv2) | SSL/TLS VPN | WireGuard | |
|---|---|---|---|
| Layer | IP, network layer | Usually above transport, often TCP 443 | IP, over UDP |
| Ports | UDP 500 and 4500, or ESP | 443, sometimes UDP too | One UDP port of your choosing |
| Negotiation | Many proposals, both sides must match | Handled by TLS | None; algorithms are fixed |
| Interoperability | Any firewall vendor to any other | Vendor client to vendor gateway | Needs WireGuard on both ends |
| Native clients | Windows, macOS, iOS, Android | Usually a vendor app | Apps on all platforms; kernel on Linux |
| Typical fit | Site-to-site between different vendors, cloud VPN gateways | Remote users on hostile networks | Small teams, homelabs, overlay networks |
| Main risk | Misconfigured proposals; unpatched gateways | Gateway CVEs on the TLS front end | Key distribution at scale |
The column that settles most site-to-site decisions is interoperability. When the other end is AWS, Azure, Google Cloud or a supplier's FortiGate, IPsec is the language every one of them speaks. WireGuard is excellent between devices you control on both ends, and TLS VPNs earn their keep for laptops on networks that block UDP.
Every IPsec tunnel failure shows up at one of two points: the IKE SA will not form, or it forms and the child SAs will not. Knowing which one narrows the cause to a short list.
| Symptom | Where it fails | Usual cause | First check |
|---|---|---|---|
| No response from the peer | Phase 1 | UDP 500 or 4500 blocked, wrong peer IP | Packet capture on UDP 500 at both edges |
| "No proposal chosen" | Phase 1 or 2 | Encryption, integrity, DH group or PRF differ | Compare the proposals side by side, line by line |
| Authentication failed | Phase 1 | Pre-shared key typo, wrong certificate or ID | Peer ID type and value on both ends |
| Phase 1 up, phase 2 down | Phase 2 | Traffic selectors (local and remote subnets) do not mirror | Each side's local must equal the other's remote |
| Tunnel drops every hour | Rekey | PFS group or lifetime mismatch | PFS on or off, same group, same lifetime on both |
| Up, but only one subnet works | Phase 2 | One SA per selector pair expected by one vendor | Split or combine selectors to match the peer |
| Up, no traffic either way | Data path | Overlapping subnets, missing route or NAT rule | Routes and NAT exemptions for the inner ranges |
| Stays "up" after the peer died | DPD | Dead peer detection disabled or too slow | Enable DPD with a sensible interval |
The selector row is the one that eats afternoons. In this February 2025 thread, an admin found hundreds of phase 2 subnets on one FortiGate tunnel producing log noise every second. The debug output showed the peer and the local side proposing different subnets, and the replies were unanimous: make each side's local the other's remote, and use supernets or a route-based tunnel rather than hundreds of selectors.
An hourly drop does not always mean a lifetime mismatch. In the thread below, a pfSense to SonicWall tunnel came up cleanly with two phase 2 entries, then lost one of them about 45 minutes later, right around the 3,600-second phase 2 lifetime. The lifetimes matched. The fix was a pfSense option that splits multiple phase 2 entries into separate SAs, because the SonicWall only accepts one traffic selector pair per SA. Replies pointed out the same mismatch appears between ASA, FortiGate and other vendors depending on how each side bundles selectors.
Overlapping subnets come last on the list and first in the time they cost. If both offices use 192.168.1.0/24, no routing table can tell which side a packet belongs to. The fix is NAT inside the tunnel on one side, or renumbering, and the lasting answer is to give every new site its own range from day one, which the small office network design checklist plans for.
The worst version of this ticket is a call with a supplier's engineer, reading proposals aloud line by line.
The algorithm menu on a firewall still lists options that were weak a decade ago, usually for compatibility with old peers. NIST SP 800-77 Revision 1 draws the line clearly, and it still holds: "3DES, MD5, SHA-1, and DH Groups 2 and 5 should not be used. Instead, AES-CBC with HMAC-SHA-2 or AES-GCM with either DH group 14 or an ECDH group (19, 20, or 21) should be used."
AES-GCM does encryption and integrity in one pass, and the guide notes it is often offloaded to hardware, which makes it the fast choice on most current firewalls. On authentication, NIST prefers certificates: pre-shared keys are acceptable when two vendors cannot agree on a certificate method, but "within an administrative domain, PSKs should not be used."
Here is the checklist to apply to every tunnel you own:
Running that list across every client's firewalls is a documentation job as much as a configuration one. If laptops carry IKEv2 profiles too, a script that runs Get-VpnConnection across a client's devices with the output collected is a quick way to find the ones still set to a weak cipher, and that is the kind of fleet job OpenFrame's script runner handles.
A tunnel with perfect algorithms is still only as safe as the box at each end. VPN gateways sit on the internet by design, they hold the keys, and they sit in front of the network they protect. That makes them one of the most targeted pieces of equipment an organization owns.
The pattern repeats across vendors. On September 25, 2025, CISA issued Emergency Directive 25-03, ordering US federal agencies to inventory their Cisco ASA and Firepower devices, collect forensic data and patch two vulnerabilities, CVE-2025-20333 and CVE-2025-20362, that attackers were chaining against the VPN web services. Fortinet, Ivanti, Palo Alto Networks and SonicWall gateways all had their own entries on CISA's Known Exploited Vulnerabilities list across 2024 and 2025.
None of those bugs were in the IPsec protocol itself. They were in the software around it, the management pages and the web portals. That is the point worth keeping: the protocol choices above protect the traffic, and patching protects the box. Subscribe to each vendor's security advisories, keep management interfaces off the internet, and treat a critical firmware update on an edge device as a same-week job.
An IPsec VPN is two protocols working together: IKE agrees on keys and settings, ESP encrypts the packets. Tunnel mode wraps whole packets for site-to-site links, NAT traversal moves everything to UDP 4500 when a router rewrites addresses, and IKEv2 with AES-GCM, SHA-2 and DH group 14 or higher is the configuration NIST points to. When a tunnel breaks, ask which stage failed first, then check the proposal, the selectors and the lifetimes on both ends, in that order.
For the remote-access side of the same problem, the Windows VPN server guide is the next read.
If the tunnel is joining segmented networks, the VLAN explainer covers how those segments are built on each side.
Content Marketing Lead
Ohayo! I run content, SEO, social, and community at Flamingo. Before IT, I worked as a correspondent for Ukraine's Public Broadcasting Company and have a Master's in journalism.
Still deciding?
Ask Mingo answers from our docs and customer stories.
Still deciding?
Ask Mingo answers from our docs and customer stories.
