How IPSEC Works
IPSEC or IP Security is IETF‘s proposal and is touted as the most secure option for building VPNs. Bundled with IPv6, IPSEC is a tunneling protocol for IP that provides per-packet, end-to-end or segment-by-segment protection. It’s highly flexible, supporting a wide variety cryptographic algorithms and allows bundling and nesting of its component protocols. IPSEC also has a very efficient key management and exchange procedure.
Components
Authentication Header (AH)
AH is used for connection-less integrity and authentication of IP packets. It also protects from replay attacks, every packet increases the sequence number by one and numbers are never reused. If it ever runs out of sequence numbers before renegotiating a new security association it triggers a renegotiation.
Authentication Header:
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}
Next Header Code Header Length Reserved Security Parameters Index (SPI) Sequence Number Authentication Data (Message Digest)
Legend:
Next Header Code (8-bits): Identifies what the next head is if there is one.
Header Length(8-bits): The length of the AH.
Reserved(16-bits): Not currently used.
SPI(32-bits): Arbitrary value which is used (together with the destination IP address) to identify the security association of the receiving party.
Sequence Number (32-bits): An increasing sequence number (incremented by 1 for every packet sent) to prevent replay attacks.
Authentication Data(Multiple of 32-bits): A message digest to authenticate the payload.
Encapsulating Security Payload (ESP)
ESP provides origin authenticity, integrity the same as the AH, but ESP also provides confidentiality protection of packets. ESP however is what makes IPSEC useful for VPNs. In tunneling mode ESP take the original packet, encrypts it, and inserts it into a new packet with source and destination IP addresses corresponding to the beginning and end of the tunnel.
ESP Header & Trailer:
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-6qw1{background-color:#c0c0c0;text-align:center;vertical-align:top}
Next Header Code Header Length Reserved Security Parameters Index (SPI) Sequence Number Encrypted Payload Authentication Data (Message Digest)
Legend:
Next Header Code (8-bits): Identifies what the next head is if there is one.
Header Length(8-bits): The length of the AH.
Reserved(16-bits): Not currently used.
SPI(32-bits): Arbitrary value which is used (together with the destination IP address) to identify the security association of the receiving party.
Sequence Number (32-bits): An increasing sequence number (incremented by 1 for every packet sent) to prevent replay attacks.
Encrypted Payload: the data or packet being encrypted.
Authentication Data(Multiple of 32-bits): A message digest to authenticate the payload.
Everything before the encrypted payload is the ESP header, the trailer is the Authentication data.
Internet Key Exchange (IKE)
IKE is much more involved than the last 2 components. It’s used for securely negotiating encryption methods and exchanging the keys needed.
This protocol is broken into 2 phases.
Phase I:
Overview:
Phase I has 6 messages:
Security Association (SA) messages are sent and negotiated in messages 1 and 2.
Diffie-Hellman exchanges are done in messages 3 and 4. A master key SKEYID is established.
Digital Signatures and certificates are exchanged in message 5 and 6. They are encrypted using SKEYID. The 2 nodes authenticate each other using these.
Message 1: A -> B
Node A sends a packet containing proposals for what encryption and authentication to use.
Each proposal contains:
Encryption Algorithm
Encryption Key length
Authentication algorithm
Authentication Key Length
Key Lifetime
SA lifetime
Cookie containing the hash of (A’s secret + IP of A + timestamp)
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
Proposal 1 Proposal 2 … Proposal n Cookie A
Message 2: B -> A
Node B sends a message back to A containing the subset of proposals that B accepts along with a Cookie B = hash (B’s secret + B’s IP + timestamp)
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
Proposal 1 Proposal 2 … Proposal n Cookie A Cookie B
Message 3: A -> B
Node A sends a message to B with information for a Diffie-Hellman Exchange. This exchange is for generating the SKEYID.
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
(N1)pubB P, g, TA
(N1)pubB: random number encrypted with B’s public key.
P: prime number
g: generator number.
TA: number generated by A in Diffie-Hellman protocol.
Message 4: B -> A
Node A sends a message to B with information for a Diffie-Hellman Exchange. This exchange is for generating the SKEYID.
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
(N2)pubA P, g, TB
(N2)pubA: random number encrypted with A’s public key.
P: prime number
g: generator number.
TB: number generated by B in Diffie-Hellman protocol.
Message 5: A -> B
A sends it’s digital signature and certificate encrypted by SKEYID. B will decrypt these with SKEYID and uses them to authenticate A.
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
A’s digital signature A’s Certificate
Message 6: B -> A
B sends it’s digital signature and certificate encrypted by SKEYID. A will decrypt these with SKEYID and uses them to authenticate B.
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
B’s digital signature B’s Certificate
Phase II:
Overview:
Phase II has 3 messages. Using encrypted packets protected by digital signatures, they do another Diffie-hellman exchange. This generates the secret session key. Data is transfered using the secret session key. Keys are then refreshed every few minutes in Phase II.
All messages until the session key is generated are encrypted with the SKEYID.
Message 1: A -> B
A sends a hash, a random number N3, and the Diffie-Hellman values P, g, TA.
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
Hash1 n3 P, g, TA
Hash1 = hash(SKEYIDa || N3 || p, g, TA)
B opens message 1, gets P, g, TA and verifies Hash1.
Message 2: B -> A
B sends a hash, a random number N4 and TB.
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
Hash2 n4 TB
Hash2 = hash(SKEYIDa || N4 || A (IP address) || B (IP address))
A opens message 2, gets TB and verifies Hash2.
Message 3: A -> B
A sends just a Hash3 = (SKEYIDa || N3 || N4 || A || B)
and the final secret session key is = hash(SKEYIDa || Diffie-Hellman Secret || N3 || N4).
All further communications are encrypted with this key until the key expries and they generate a new one using phase 2.
Using for Tunneling
Authentication Header
Packet:
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
Source Destination Payload
Transport mode:
In transport mode a AH is simply added to insure that the packet cannot be replayed or tampered with. It doesn’t actually provide any encryption.
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
Source Destination Authentication Header Payload
Tunnel Mode:
In tunnel mode the start and end of the tunnel (essentially a new temporary source and destination) are added and the original source and destination are moved into the packet to be used at the end of the tunnel.
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
Tunnel Start Tunnel End Authentication Header Source Destination Payload
Encapsulating Security Payload
This is where confidentiality enters the equation. With ESP the payload is encrypted, meaning that when using transport the payload is encrypted and when using tunneling both the original source and destination are encrypted along with the payload. The encryption is determined using IKE. Packet:
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
Source Destination Payload
Transport Mode:
Transport simply takes the original payload, encrypts it, and adds a ESP header and trailer to the packet.
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
Source Destination ESP header Payload (now encrypted) ESP trailer
Tunnel Mode:
When in tunnel model an entirely new packet is created and the original packet become the encrypted payload. This is why this Protocol is so secure for VPN usage. Once the packet enters the tunnel you cannot determine anything about it other than the beginning and end of the tunnel. This means a 3rd party can’t even tell what server you were communicating with while your packet is in the tunnel.
.tg {border-collapse:collapse;border-spacing:0;}.tg td{font-family:Arial, sans-serif;font-size:14px;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg th{font-family:Arial, sans-serif;font-size:14px;font-weight:normal;padding:10px 5px;border-style:solid;border-width:1px;overflow:hidden;word-break:normal;}.tg .tg-baqh{text-align:center;vertical-align:top}.tg .tg-yw4l{vertical-align:top}
Source Destination ESP header Payload (now encrypted) ESP trailer
How IPSEC Works was originally published on Cole Talks













