VPN Protocol Breakdown: VPNs Behind the Scenes

“Security and privacy” is one of the most important sections in our reviews. If you’ve read them — and we humbly recommend that you do — you know that we cover the protocols VPNs offer, security features, such as having a killswitch, and the privacy policy that’s in place.

All those topics make up VPN security, which is important to getting a broad perspective on how well you’re protected. That said, the VPN protocol is the chief security concern and knowing which of the built-in open source and proprietary options to use can be difficult.

In this VPN protocol breakdown, we’re going to clear up the confusion. There are many networking protocols, so we’ve done the research and condensed our list to the VPN protocols you’re likely to see in your application. Before getting to the specifics, though, we have to define what a VPN protocol is.

What Is a VPN Protocol?

A virtual private network secures and anonymizes your internet connection by connecting you to a remote server before launching websites. The connection to that server is encrypted, too, meaning none of your web-based requests can be seen by the outside world.

The type and level of encryption is determined by the security protocol. Depending on the protocol you use, you’ll connect to the VPN over different ports and with varying levels of security.

Though encryption type is the main difference between protocols, it affects other aspects of using a VPN, too. For example, a more sophisticated level of encryption will protect your connection more, but it won’t be as fast as a protocol that’s using less secure encryption.

On a practical level, the choice in protocol comes down to how you want to balance security and speed. VPN protocols are a form of network protocol, meaning they bring together the requirements for establishing a connection between two devices. That includes security and speed.

Unfortunately, like most networking topics, it’s not that simple. Platform support is important, as well as where and when the encryption happens in the communication chain. Those differences are why the best VPN providers include multiple protocol options.

Given how easy VPNs are to use, though, it’s unlikely you’ll notice or care to change your protocol. Even so, we’re going to run through the common protocols available to you and the different use cases for them. In doing so, our hope is that you’ll learn why to use OpenVPN when configuring a VPN for a router and IKEv2 when using one on your phone.

Common VPN Protocols

Below you’ll find the most common protocols we’ve come across. There are a few more exotic variants around, of course, but most VPN providers will use a combination of these.

OpenVPN

OpenVPN is a popular protocol to use because it’s open source and free. Some providers, such as AirVPN and e-VPN (read our AirVPN review and e-VPN review), have built their services around it. It’s over 15 years old and has a community surrounding it that is constantly scanning the source files for security vulnerabilities, making it one of the most secure options available.

It can use two transport protocols: TCP or UDP. The Transmission Control Protocol is most common. Your machine sends a packet, then waits for confirmation before sending another, making for a more reliable connection.

That benefits reliability, but not speed. Because each packet has to wait for confirmation, using TCP adds overhead to the network connection. That’s where the User Datagram Protocol comes in. It continues to send packets without confirmation, making for a faster, if less reliable, connection.

As far as encryption goes, OpenVPN is top-notch. It uses the OpenSSL library, meaning it has access to all the ciphers there. It also uses a custom security protocol based on SSL/TLS that provides up to 256-bit encryption.

256-bit encryption isn’t required, though. Some providers, such as Private Internet Access, default to 128-bit encryption, as you can read in our PIA review. Using a smaller key size usually allows for a faster connection, but that comes at the cost of security.

That said, even the fastest VPN providers use a 256-bit key, which shows why OpenVPN is so popular. Beyond the many other reasons for a provider to offer it, OpenVPN has the best balance of security and speed.

Because of its open-source nature, it shows up in custom protocols from some VPN providers, as well. VyprVPN’s Chameleon protocol scrambles OpenVPN packets and Astrill’s StealthVPN does mostly the same thing (read our VyprVPN review and Astrill review).

VyprVPN and Astrill have developed their protocols to bypass censorship in China. OpenVPN, though highly secure, doesn’t do anything special to hide from deep packet inspection. VyprVPN ranked in our best VPN services for China guide because its Chameleon protocol can scramble the OpenVPN packets being sent.

Another benefit of OpenVPN is that it can be adapted to almost any platform. ExpressVPN, for example, allows you to use OpenVPN on your router, as you can see in our ExpressVPN review. It uses custom firmware that includes a pre-configured version of OpenVPN, enabling you to secure traffic traveling to your router and the internet.

WireGuard

A newcomer on the scene, WireGuard solves a lot of the problems present with OpenVPN. You can read our full what is Wireguard piece, but we’ll get into the basics here. Unlike most other protocols, WireGuard’s code is incredibly lean. That makes for faster connections with far less problems. However, there are still some concerns about the privacy of WireGuard. 

It’s a free, open source tunneling protocol that’s based around ChaCha20 instead of AES-256 like OpenVPN. The differences between the two ciphers are minimal when it comes to security. The TLS 1.3 standard even uses ChaCha20 as a backup cipher, as well as a way to encrypt connections on devices without AES hardware acceleration. 

WireGuard is able to achieve such high speeds with adequate encryption because of its slim code and connection process. The process is similar to how a SSH connection is established. For those among you with remote computing prowess, you know that means the server needs a static IP and needs to store your local IP for authentication. 

Clearly, that’s a problem for privacy. However, multiple VPN services have developed systems to deal with the privacy angle. NordVPN approaches the problem with a double NAT that can assign each user a dynamic IP, while Mullvad continually dumps the IPs stored in server memory (read our Mullvad review and NordVPN review). 

Outside of speed benefits, WireGuard’s streamlined code base brings security benefits, too. For reference, OpenVPN is around half a million lines of code, while WireGuard is under 10,000. Simply put, there’s less attack surface area with WireGuard. There are far fewer chances for a rogue line of code to present a security vulnerability. 

It’s also much easier for security researchers to audit the code, and because it’s open source, that’s possible. WireGuard is very likely the future of VPNs, though only a handful of services support it now. 

Layer 2 Tunnel Protocol

The Layer 2 Tunnel Protocol is a tunneling protocol that allows data to move from one network to another. Unlike OpenVPN, L2TP is strictly a tunneling protocol. It doesn’t provide encryption on its own. Because of that, L2TP is often paired with an encryption protocol to provide security.

It was created in 1999 and based on two older tunneling protocols called L2F and PPTP. We’ll talk about the latter in a later section. Though a new version of the protocol, known as L2TPv3, was introduced in 2005 to add security features, L2TP has mostly stayed the same.

L2TP uses two types of packets: control packets and data packets. The control packets deal with establishing a connection and opening the tunnel between you and the server you’re accessing. Because that is the core function of the tunneling protocol, L2TP has reliability features, such as packet confirmation, tied to control packets.

Data packets don’t have such features. L2TP sends packets within a UDP datagram, meaning they aren’t verified as they’re being sent. That makes for a faster, but less reliable, connection.

The problem with L2TP on its own is that the packets you’re sending aren’t encrypted. They’re encapsulated, but there isn’t a cryptographic algorithm to conceal the data. Because of that, you’ll most likely find L2TP paired with IPSec in your VPN client.

IPSec provides encryption, encapsulating the already-encapsulated packet as it goes through the L2TP tunnel. That means the source and destination IP addresses are encrypted in the IPSec packet, creating a secure VPN connection.

As far as encryption goes, IPSec offers a few options, including HMAC with an appropriate hashing algorithm, TripleDES-CBC, AES-CBC and AES-GCM. Some VPN providers, such as TorGuard (read our TorGuard review), allow you to change the cipher used,but you’ll mostly find L2TP/IPSec secured with AES 128-bit or 256-bit.

L2TP/IPSec is considered secure, but some security experts have doubts because IPSec was developed, in part, by the U.S. National Security Agency. Even so, it’s normally a worse choice than OpenVPN. The port L2TP uses is easily blocked by firewalls, so you’ll have a tough time getting around censorship unless you use a VPN that supports port forwarding.

Secure Socket Tunneling Protocol

The Secure Socket Tunneling Protocol is a proprietary Microsoft technology that was developed for Windows Vista. Though it’s a Microsoft-developed protocol, SSTP can also be used on Linux. That said, it isn’t supported on macOS and likely never will be. Read our best VPN for Mac if you’re on Team Apple.

Like OpenVPN, SSTP allows point-to-point traffic to pass through an SSL/TLS channel. Because of that, it has the same pros and cons of using such a system. For instance, it uses SSL/TLS over TCP port 443, making it excellent at passing through most firewalls because the traffic appears normal.

The problem with that, which is the same problem with using TCP on OpenVPN, is that you’re vulnerable to TCP meltdown. TCP must wait for confirmation before sending a packet back. It has built-in features to detect and attempt to solve problems if a packet hasn’t been confirmed.

In such a case, a TCP packet in one layer may try to solve a problem, causing the packet in the layer above it to overcompensate. When that happens, the performance of a TCP connection falls significantly. That can be avoided with OpenVPN by using UDP instead. With SSTP, the problem is unavoidable.

Though SSTP is available in some VPN applications, it’s rarely used. It’s better at getting around firewalls than L2TP, but so is OpenVPN. The trouble with SSTP is that it’s not as configurable as OpenVPN, so it’s more susceptible to problems, such as TCP meltdown. OpenVPN provides all the upsides of SSTP without the drawbacks.

Internet Key Exchange Version 2

Internet Key Exchange is a protocol that was developed by Microsoft and Cisco in 1998. Technically, it’s not a VPN protocol. IKE is used to set up a security association in the IPSec protocol suite. The security association includes attributes such as the cipher and traffic encryption key.

Even so, it’s often treated as a VPN protocol, called IKEv2, which is simply the second version of IKE, or IKEv2/IPSec. Unlike L2TP/IPSec, which just uses IPSec for encryption, IKE uses IPSec to transport data.

As far as security goes, it’s as good as L2TP or SSTP, assuming you trust Microsoft. It can support multiple versions of AES and you’ll most likely find it paired with a 128-bit or 256-bit key in your VPN application.

It’s not just another option, though. IKEv2 is usually the fastest protocol VPNs offer.

IKE uses UDP packets and begins creating the security association after the first few packets are sent. The security association is then transferred to the IPSec stack, which causes it to start intercepting relevant IP packets and encrypting or decrypting them as appropriate.

Because of that, IKE is good at reconnecting after a connection has dropped. On a wired or WiFi connection, that is less of a concern because they are generally static and stable. For mobile devices, though, IKE is much more enticing.

3G and 4G LTE networks are constantly shifting as your phone or tablet moves with you. You may drop from 4G LTE to 3G or temporarily lose connection. Because IKE is quick to reconnect, it’s an ideal choice on mobile devices. IKEv2 has even been built in to BlackBerry devices.

Point-to-Point Tunneling Protocol

The Point-to-Point Tunneling Protocol is a dated and unsecure tunneling protocol that shouldn’t be used if you’re concerned about security. Despite that, some VPN providers still include it in their applications. For most users, it should simply be ignored.

The best use case for PPTP is accessing a corporate building’s internal network externally, which is why VPNs were developed in the first place. PPTP doesn’t specify encryption. Rather, it relies on the point-to-point protocol to implement security features.

Because of the lower form of encryption, PPTP is fast. It’s almost the same speed as your normal internet connection. In a personal use case, it’s about as secure as your normal internet connection, too. That’s why we only recommend using PPTP if you’re doing something you can’t do without a VPN, such as accessing an external network.

Don’t expect that connection to be safe, though. There are many tools for cracking PPTP tunnels, some of which can simply extract the key from the authentication method and others that can find the key within a few hours using a brute-force attack.

Plus, the NSA has been known to actively spy on PPTP networks because of the weak security. Unless you have a specific reason to use it, we recommend avoiding PPTP, even if it’s an option for it in your VPN application (for more detail, check out our PPTP vs OpenVPN article).

Final Thoughts

Our hope is that you can go into your VPN application with more confidence now that you know the difference between the protocols. For most users, OpenVPN  is the best option because it provides top-level security and configurability out of the box. Plus, its open-source nature allows you to download configuration files and further tweak it to your liking.

The other options have their strengths, but they also have weaknesses. SSTP solves the firewall problem but can fall victim to TCP meltdown. L2TP is fast and stable, but easily blocked. The only exception would be IKEv2, which, though arguably inferior to OpenVPN, has a lot of upside for mobile users.

Do you feel more confident in your knowledge of VPN protocols? Let us know how that’s changed your VPN usage in the comments below and, as always, thanks for reading.