IPsec Simplified

What is IPsec? What network security does IPsec provide?
How do I set up IPsec? A very simple explanation

IPsec is an extension of the Internet Protocol (IP) to secure network communication through cryptography. Its primary goals are data confidentiality, data integrity, and host authentication. The combination of integrity and authentication provides non-repudiation. IPsec also detects replay attacks.

It is a modification of the IP implementation within the TCP/IP protocol suite, and so it means a modification to the kernel of modern operating systems.

Table of Contents

The following attempts to be a "just enough" explanation. If you want all the details, see the official documents:

How do you pronounce its name? I've heard all of these:


Architecture

IPsec provides network communication security between hosts. User issues like authenticating a human as some user, restricting access to data by users, and so on, are outside the scope of IPsec. Those important security tasks are left to the operating systems run on the hosts.

Two hosts interested in securely communicating with each other (and pardon the anthropomorphism, but it makes this story much easier to tell and understand) must establish a Security Association (SA) with each other. The details can be tremendously complicated, but the SA comes down to the following. See my "Just Enough Crypto" page for explanations of ciphers, hashes, and HMACs:

The SA is negotiated by the Internet Key Exchange (IKE) protocol. There is a quick burst of IKE traffic before two hosts start communicating with IPsec. IKE simplifies to these major steps:

  1. The hosts authenticate to each other using cryptography. This could be done with shared secrets: The problem with that approach is that it does not scale. You can set up shared secrets between a few machines. But since every machine-to-machine pair needs a key, and the number of keys grows as the square of the number of machines, that does not scale to a large organization. Set up digital certificates instead: After a round of that in each direction, each host is confident in the other host's identity.
  2. The two hosts negotiate a mutually acceptable combination of confidentiality (or not) and integrity plus authentication (or not). This is usually the complete CIA triple, but some host pairs may be configured to only use some.
  3. The two hosts negotiate a mutually acceptable set of algorithms and keys. The algorithms will generally be the strongest cipher and hash supported by both ends. The keys will be agreed upon using Diffie-Hellman key negotiation.

The SA information is then passed to the IPsec module of the kernel.

Once the SA has been established, IKE is used from time to time to negotiate new session keys. You want to use a key only for a limited time and amount of data. IKE can quickly re-run Diffie-Hellman to agree on new keys. Any delay caused by periodic re-keying will tend to disappear into the background noise of routine network latency jitter.

Once the kernel has an SA for another host, it will enforce its use. Every packet sent to that host will be modified according to the SA. Every packet apparently from that host will be dropped if it cannot be handled according to the SA. That means that if one host of an IPsec pair suddenly quit using IPsec and simply sent IP datagrams, the other host would drop all those apparently bogus datagrams.

Authentication Header is an IPsec extension to IP to provide data integrity, source host authentication, and protection against replay attacks.

Encapsulating Security Payload is an IPsec extension to IP to provide data confidentiality, data integrity, source host authentication, and protection against replay attacks.

Authentication Header (AH)

Authentication Header inserts an extension to the original IP header. The AH header includes the Security Parameters Index (SPI), which is effectively an index number indicating which SA is used. It also includes the integrity check value (ICV), basically a hashed message authentication code of the payload, the non-changing IP header fields, and appropriate key.


ORIGINAL, what would have been sent:
+------------+-----------------------------+
|            |                             |
| IP header  |  Payload                    |
|            |                             |
+------------+-----------------------------+

WITH AH:
+------------+-----+-----------------------------+
|            |     |                             |
| IP header  | AH  |  Payload                    |
|            |     |                             |
+------------+-----+-----------------------------+
   ^          ^ ^ ^
   |          | | |
   |          | | +--- "Next header" field indicates the
   |          | |     original encapsulated Protocol field.
   |          | |
   |          | +--- ICV = hash(Key, Payload, some IP header fields)
   |          |
   |          +--- SPI = indication of which SA to apply
   |
   +--- "Protocol" field changed to 51 to indicate AH.

Encapsulating Security Payload (ESP)

I have no idea why the term is Encapsulating Security Payload when Encrypting Security Payload would better emphasize the main point....

ESP inserts an extension to the original IP header, and adds an authentication block to the end of the datagram. The ESP header includes the Security Parameters Index (SPI), which is effectively an index number indicating which SA is used. It also includes a sequence number, a number that should increase with each IPsec datagram sent. This allows a system to recognize and ignore duplicate received IPsec datagrams.

A "Next header" field is added to the end of the encrypted payload, along with padding to make the datagram an integer multiple of 32 bits. Then a final authentication data block contains the ICV, an HMAC of the payload, the non-changing IP header fields, and appropriate key.


ORIGINAL, what would have been sent:
+------------+----------------------------------+
|            |                                  |
| IP header  | Payload                          |
|            |                                  |
+------------+----------------------------------+

WITH ESP:
+------------+-----+----------------------------------+-+------+
|            |     |                                  | |      |
| IP header  | ESP | Encrypted payload                |N| Auth |
|            |     | (ciphertext of original payload) |H| Data |
+------------+-----+----------------------------------+-+------+
   ^           ^ ^                                     ^    ^
   |           | |                                     |    |
   |           | +--- Sequence number allows detection |    |
   |           |      of replay attempts               |    |
   |           |                                       |    |
   |           +--- SPI = indication of which SA       |    |
   |                      to apply                     |    |
   |                                                   |    |
   |                "Next header" field indicates the  |    |
   |                original encapsulated Protocol ----+    |
   |                                                        |
   |                ICV = hash(Key, Sequence, Payload) -----+
   |
   +--- "Protocol" field changed to 50 to indicate ESP.

Transport Mode and Tunnel Mode

Transport Mode encrypts only the payload, the IP header itself is left alone. This is used for host-to-host communications. The data contents are protected, but anyone observing network traffic can see traffic flow patterns.


ORIGINAL, what would have been sent:
+--------------+----------------------------------+
| IP header    | Payload                          |
| host1->host2 |                                  |
+--------------+----------------------------------+

PROTECTED with ESP in Transport Mode:
+--------------+-----+----------------------------------+-+------+
| IP header    | ESP | Encrypted payload                |N| Auth |
| host1->host2 |     | (ciphertext of original payload) |H| Data |
+--------------+-----+----------------------------------+-+------+

Tunnel Mode is used to encrypt the entire original IP datagram, both payload and original header, and then encapsulate that inside a new IP header. You create a Virtual Private Network (VPN) between two remote sites by doing Tunnel Mode IPsec at the gateways.

Below is a simple example. Our organization has two sites, "Site 1" and "Site 2", each with its own simple LAN and IPsec gateway.


      (--------------------------------)
      (                                )
      (          The Internet          )
      (                                )
      (---+------------------------+---)
          |                        |
        +-+-+                    +-+-+
        |GW1|                    |GW2|
        +-+-+                    +-+-+
 (--------+---)                (---+--------)
 (            )                (            )
 ( Site 1 LAN )                ( Site 2 LAN )
 (            )                (            )
 (--+---------)                (---------+--)
    |                                    |
  HostA                                HostB

HostA sends a packet to HostB. The hosts see the IPsec gateways GW1 and GW2 as nothing more then routers. The packet traveling across the Site 1 LAN looks like this:


+--------------+----------------------------------+
| IP header    | Payload                          |
| HostA->HostB |                                  |
+--------------+----------------------------------+

If GW1 and GW2 do not not have an SA established, they set one up via IKE when the packet bound for the Site 2 LAN reaches GW1. Gateway GW1 then encrypts that entire datagram and wraps it inside an IPsec header for delivery across the Internet to gateway GW2. If you captured the packet as it moved across the Internet, you would see this:


+-----------+-----+-------------------------------------------------+-+----+
| IP header | ESP | /  /  /  /  / ciphertext version  /  /  /  /  / |N|Auth|
| GW1->GW2  |     |  /  /  /  /  of original datagram  /  /  /  /  /|H|Data|
+-----------+-----+-------------------------------------------------+-+----+

That datagram is received at gateway GW2 and handled as per the SA specified in the SPI field of the ESP header. It is decrypted with the appropriate cipher and key, and the resulting datagram, identical to the original, is dropped onto the Site 2 LAN.

Maybe HostA and HostB shared an SA. If that were the case, the payload of the datagrams cross the two LANs would be ciphertext. There's certainly no harm in encrypting data that happens to already be ciphertext. And this would provide defense in depth and protect against hostile insiders at both sites.

How To Set Up Host Keys and Certificates

Do these steps on the CA host:

  1. Generate a new certificate signing request. At the prompt Common Name put the new hostname, otherwise the defaults should be fine:
    # cd /etc/ssl
    # /etc/pki/tls/misc/CA.pl -newreq
  2. Sign the new certificate:
    # /etc/pki/tls/misc/CA.pl -sign
  3. Decrypt the private key to cleartext, as the IPsec racoon daemon can't very well decrypt it on its own at boot time:
    # openssl rsa -in newkey.pem -out newkey.pem
    # chmod 400 newkey.pem
  4. Rename the files:
    # mv newkey.pem newhostname_key.pem
    # mv newcert.pem newhostname_cert.pem
    # mv newreq.pem newhostname_req.pem
  5. Install the appropriate key pair, digital certificate, and public key of the trusted signing authority on each host. Those will be the three files named newhostname_* plus the CA public key cacert.pem. Copy them to the IPsec host and install them in the correct location. On a Linux machine with the racoon daemon managing the SAs, those files go in the directory /etc/racoon/certs/.

How To Configure IPsec on Linux

Assuming that you have generated and distributed certificates as shown above, and you want to turn on IPsec between the two hosts 192.168.0.1 and 10.1.2.3:

Create a policy in /etc/setkey.conf on each host:
Put the following in /etc/setkey.conf on host 192.168.0.1:

# Out our 192.168.0.0/24 connection to 10.1.2.3:
spdadd 192.168.0.0/24 10.1.2.3/32 any -P out ipsec
	esp/transport//require
	ah/transport//require;
# In from 10.1.2.3 on our 192.168.0.0/24 connection:
spdadd 10.1.2.3/32 192.168.0.0.0/24 any -P in ipsec
	esp/transport//require
	ah/transport//require; 

Then put the converse in /etc/setkey.conf on host 10.1.2.3:

# Out our 10.1.2.0/24 connection to 192.168.0.1:
spdadd 10.1.2.0/24 192.168.0.1/32 any -P out ipsec
	esp/transport//require
	ah/transport//require;
# In from 192.168.0.1 on our 10.1.2.0/24 connection:
spdadd 192.168.0.1/32 10.1.2.0/24 any -P in ipsec
	esp/transport//require
	ah/transport//require; 

Configure the racoon daemon to run IKE on each host:
Modify /etc/raccoon/racoon.conf to include two stanzas like this, replacing thishostname with the local host name in each case:

sainfo anonymous
{
	lifetime time 1 hour ;
	encryption_algorithm 3des, blowfish 448, rijndael, aes ;
	authentication_algorithm hmac_sha1, hmac_md5 ;
	compression_algorithm deflate ;
}

remote anonymous
{
	exchange_mode main;
	doi ipsec_doi;
	situation identity_only;
	my_identifier asn1dn;
	# Change these host names to match yours!
	certificate_type x509 "thishostname_cert.pem" "thishostname_key.pem";
	verify_cert off;

	nonce_size 16;
	initial_contact on;
	proposal_check obey;    # obey, strict, or claim

	proposal {
		encryption_algorithm aes;
		hash_algorithm sha1;
		authentication_method rsasig;
		dh_group 2;
	}
} 

Start racoon and configure it to start after each boot:

# /etc/init.d/racoon start
# chkconfig racoon on 

Opportunistic Encryption

Many people very reasonably say that it makes no sense to arrange a method for whispering secrets to strangers. Data encryption implies data sensitivity, and so you should only send sensitive data to a host of known identity. Well, sure, but....

There is an interesting thing called opportunistic encryption. This is an attempt to establish encrypted communication with any host before falling back to plaintext communication if needed. This means you can routinely encrypt some of your network traffic without setting up any Security Associations, just by asking your operating system to attempt opportunistic encryption and then communicating with other hosts supporting the same thing.

I have taught networking and information security to people from various U.S. Government Three-Letter Agencies. The topic of opportunistic encryption takes them through these stages:

  1. Polite attempts to correct my error, surely I don't really mean what I seem to be saying.
  2. Derision at the foolish people who would even consider such a silly thing as whispering secrets to strangers.
  3. Utter horror when they are finally convinced that people really do use this for things like routine web browsing, IRC, electronic mail, peer-to-peer file sharing, and more.

Yes, this would allow the opportunistic encryption subset of Internet users to communicate with each other while frustrating attempts to routinely eavesdrop on them. See the Wikipedia page for pointers to the details for doing this on Linux and routers with Openswan and other FreeS/WAN derivatives, on Windows, to protect e-mail and VoIP, and more.


Back to the Networking page

To the Network Security page


Home Page Unix/Linux TCP/IP Infosec Travel Radio Site Map Contact
Use /bin/vi! Manipulate images with ImageMagick! Hosted on OpenBSD
Hosted on Apache Valid XHTML 1.1! Valid CSS!
© Bob Cromwell Jan 2009. Created with /bin/vi and ImageMagick, hosted on OpenBSD with Apache.    Root password available here