Copying/renaming, node comments, and other little known features of the VyOS CLI

I promised not to write about either IPsec or NAT this time, so we'll discuss something else: the little known features of the VyOS CLI. Many people only ever use set/delete and commit, but there's more to it, and those features can save quite a bit of time.

The edit level (never write long node paths again)

You might have noticed that after every command, the CLI outputs a mysterious "[edit]" line. This is a side effect of the system that allows editing the config at any level.

By default, you are at the top level, so you have to specify the full path, such as "set firewall name Foo rule 10 action accept". However, to avoid writing or pasting long paths, you can set the edit level to any node with the "edit" command, such as "edit firewall name Foo". Once you are at some level, you can use relative node paths, such as "set rule 10 action accept" in this case.

To move between levels, you can use the "up" command to move one level up, or the "top" command to instantly move back to the top level.

Look at this session transcript:

dmbaturin@reki# edit firewall name Foo
[edit firewall name Foo]

dmbaturin@reki# set rule 10 protocol tcp
[edit firewall name Foo]

dmbaturin@reki# edit rule 10
[edit firewall name Foo rule 10]

dmbaturin@reki# set destination port 22
[edit firewall name Foo rule 10]

dmbaturin@reki# up
[edit firewall name Foo]

dmbaturin@reki# set rule 10 description "Allow SSH"
[edit firewall name Foo]

dmbaturin@reki# top
[edit]

Setting up GRE/IPsec behind NAT

In the previous posts of this series we've discussed setting up "plain" IPsec tunnels from behind NAT.

The transparency of the plain IPsec, however, is more often a curse than a blessing. Truly transparent IPsec is only possible between publicly routed networks, and the tunnel mode creates a strange mix of the two approaches: you do not have a network interface associated with the tunnel, but the setup is not free of routing issues either, and it's often hard to test whether the tunnel actually works or not from the router itself.

GRE/IPsec (or IPIP/IPsec, or anything else) offers a convenient solution: for all intents and purposes it's a normal network interface and makes it look like the networks are connected with a wire. You can easily ping the other side, use the interface for firewall and QoS rulesets, and setup dynamic routing protocols in a straightforward way. However, NAT creates a unique challenge for this setup.

The canonical and the simplest GRE/IPsec setup looks like this:

interfaces {
  tunnel tun0 {
    address 10.0.0.2/29
    local-ip 192.0.2.10
    remote-ip 203.0.113.20
    encapsulation gre
  }
}
vpn {
  ipsec {
    site-to-site {
      peer 203.0.113.20 {
        tunnel 1 {
          protocol gre
        }
        local-address 192.0.2.10

It creates a policy that encrypts any GRE packets sent to 203.0.113.20. Of course it's not going to work with NAT because the remote side is not directly routable.

Let's see how we can get around it. Suppose you are setting up a tunnel between routers called East and West. The way to get around it is pretty simple even if not exactly intuitive and boils down to this:

  1. Setup an additional address on a loopback or dummy interface on each router, e.g. 10.10.0.1/32 on the East and 10.10.0.2/32 on the West.
  2. Setup GRE tunnels that are using 10.10.0.1 and .2 as local-ip and remote-ip respectively.
  3. Setup an IPsec tunnels that uses 10.10.0.1 and .2 as local-prefix and remote-prefix respectively.

This way when traffic is sent through the GRE tunnel on the East, the GRE packets will use 10.10.0.1 as a source address, which will match the IPsec policy. Since 10.10.0.2/32 is specified as the remote-prefix of the tunnel, the IPsec process will setup a kernel route to it, and the GRE packets will reach the other side.

Let's look at the config:

interfaces {
  dummy dum0 {
    address 10.10.0.1/32
  }
  tunnel tun0 {
    address 10.0.0.1/29
    local-ip 10.10.0.1
    remote-ip 10.10.0.2
    encapsulation gre
  }
}
vpn {
  ipsec {
    site-to-site {
      peer @west {
        connection-type respond
        tunnel 1 {
          local {
            prefix 10.10.0.1/32
          }
          remote {
            prefix 10.10.0.2/32
          }

This approach also has a property that may make it useful even in publicly routed networks if you are going to use the GRE tunnel for sensitive but unencrypted traffic (I've seen that in legacy applications): unlike the canonical setup, GRE tunnel stops working when the IPsec SA goes down because the remote end becomes unreachable. The canonical setup will continue to work even without IPsec and may expose the GRE traffic to eavesdropping and MitM attacks.

This concludes the series of posts about IPsec and NAT. Next Friday I'll find something else to write about. ;)

How to setup an IPsec connection between two NATed peers: using id's and RSA keys

In the previous post from this series, we've discussed setting up an IPsec tunnel from a NATed router to a non-NATed one. The key point is that in the presence of NAT, the non-NATed side cannot identify the NATed peer by its public address, so a manually configured id is required.

What if both peers are NATed though? Suppose you are setting up a tunnel between two EC2 instances. They are both NATed, and this creates its own unique challenges: neither of them know their public addresses or can identify their peers by their public address. So, we need to solve two problems.

In this post, we'll setup a tunnel between two routers, let's call them "east" and "west". The "east" router will be the initiator, and "west" will be the responder.

Why IPsec behind 1:1 NAT is so problematic and what you can do about it

Not so long ago the only scenario when the issues with IPsec and NAT could arise was a remote access setup, while routers invariably had real public addresses and router to router IPsec operators were incredibly unlikely to run into those issues. A router behind NAT was seen as a pathological case.

I believe it's still a pathological case, but as platforms such as Amazon EC2 that use one to one NAT to simplify IP address management rose in popularity and people started setting up their routers there, it became the new normal. This new reality became a constant source of confusion for beginner network admins and people who simply are not VPN specialists. Attempts to setup a VPN as if there's no NAT, with peer external IP addresses and pre-shared keys as it's commonly done, just fail to work.

Sometimes I'm asked why is that so, if the NAT in question is 1:1, and no other protocol just refuses to work behind it without reconfiguration. The reasons are inside the IPsec protocols themselves, and 1:1 NAT is not much better in this regard than any other kind (even though it's somewhat better than the situation when there are multiple clients behind NAT).

IPsec pre-dates NAT by over a decade, and was explicitly designed with end to end connectivity in mind. It was also designed as a way to add built-in security to the IP protocol stack rather than a new protocol within the stack, so it made heavy use of those underlying assumptions. That's why workarounds had to be added when the assumption of end to end connectivity became incorrect.

Let's look into the details and then see how IPsec can be configured to work around those issues. Incidentally, the solutions are equally applicable to machines with dynamic addresses as well as those behind a 1:1 NAT, since the root cause of the issues is not knowing beforehand what your outgoing address will be next time.

AH and ESP

We all (hopefully) remember that there are three criteria of information security: availability, confidentiality, and integrity. Availability (when required) in the IP is provided by reliable transmission protocols such as TCP and SCTP. IPsec was supposed to provide the other two.

AH (Authentication Header) was designed to ensure packet integrity. For that, it calculates a hash function from the entire packet with all its headers, including source and destination addresses. Since NAT changes the addresses, it invalidates the checksum and NATed packets would be considered corrupted. So, AH is unconditionally incompatible with NAT.

ESP only protects the payload of the packet (whether it's some protocol data, or an entire tunnelled packet), so it, luckily, can work behind NAT. The UDP encapsulation of ESP packets was introduced to allow the traffic to pass through incorrect NAT implementations that may discard non-TCP or UDP protocols, and for correct implementations this should be irrelevant (I've seen a number of SOHO routers that couldn't handle typical L2TP/IPsec connections despite that UDP encapsulation — but that's another story).

Before we can start sending ESP packets, we need to establish a "connection" (security association) though. In practice, most IPsec connections are negotiated via the IKE protocol rather than statically configured, and that's where IKE issues come into play.

IKE and pre-shared keys

To begin with, original IKE was specified to use UDP/500 for both source and destination port. Since NAT may change the source port, the specification had to be adjusted to allow other source ports. Even more issues arise when there are multiple clients behind NAT and traffic has to be multiplexed, but we'll limit the discussion to 1:1 NAT where canonical IKE might work. Let's assume the IKE exchange has gone that far.

The vast majority of IPsec setups in the wild use pre-shared keys. I've configured connections to vendors and partners on behalf of my employers and customers countless times, and I believe I've seen an x.509-based setup once or twice. In a setup with pre-shared keys, routers need to know which keys to use for which peers. How do they know? Suppose you've configured a tunnel with peer 192.0.2.10 and key "qwerty". The other side has 172.16.0.1 address NATed to 192.0.2.10. Will it be the source address of the peer that will be used for the key lookup? Not quite.

In the IKE/ISAKMP exchange, there are identifiers. It's the pairs of identifiers and pre-shared keys that must be unique to reliably tell one peer from another and use correct keys for every peer. There are several types of identifiers specified by the ISAKMP protocol: IP (v4 or v6) address, subnet, FQDN, user and FQDN pair (user@example.com), and raw byte string.

By default, most implementations (including StrongSWAN in VyOS) will use the IP address of the outgoing interface for the identifier, and it will be embedded in the IKE packet. If the host is behind NAT, that address is a private address, 172.16.0.1. When the packet passes through the NAT, the payload will obviously remain unchanged, and when it arrives, the responder will try to find the pair of 172.16.0.1:qwerty in its configuration rather than 192.0.2.1:qwerty that is has configured, and will send NO_PROPOSAL_CHOSEN to the NATed peer.

The solutions

So, how do you get around those issues? Whether IKE is configured to use NAT detection or not, you'll get to use peer identification methods that do not rely on known and fixed IP addresses.

The simplest approach is to use a manually configured peer identifier. This approach will work when one side is NATed and the other is not. The only thing you need to remember is that VyOS (right now at least) requires that FQDN identifiers are prepended with the "@" character. They also do not need to match any real domain name of your machine. You also need to set the local-address on the NATed side to "any".

Here's an example:

Non-NATed side

vpn {
  ipsec {
    site-to-site {
       peer @foo {
         authentication {
             mode pre-shared-secret
             pre-shared-secret qwerty
         }
         default-esp-group Foo
         ike-group Foo
         local-address 203.0.113.1
         tunnel 1 {
             local {
                 prefix 10.103.0.0/24
             }
             remote {
                 prefix 10.104.0.0/24
             }
         }
     }
 } 

NATed side

  ipsec {
    site-to-site {
         peer 203.0.113.1 {
             authentication {
                 id @foo
                 mode pre-shared-secret
                 pre-shared-secret qwerty
             }
             default-esp-group Foo
             ike-group Foo
             local-address any
             tunnel 1 {
                 local {
                     prefix 10.104.0.0/24
                 }
                 remote {
                     prefix 10.103.0.0/24
                 }
             }
         }
     }
 }