When VyOS CLI isn't enough

Sometimes a particular configuration option is supported by the software that VyOS uses, but the CLI does not expose it. Since VyOS is open source, you can always fix that, but sometimes you need it by tomorrow, and there's simply no time to do it.

In a number of places, we've left an escape hatch for it that allows bypassing the CLI and including a raw snippet into the generated config. Of course, you give up the sanity checks of the CLI and take full responsibility for the correctness of the resulting config, but sometimes it's necessary.


In openvpn, we have an option called "openvpn-option". You can pass any options to OpenVPN process with it, but note that in the current versions, it has to follow the command line rather than config file option, i.e. prepend it with "--". See this example:

set interfaces openvpn vtun10 openvpn-option "--connect-freq 10 60"

Note that the "push" option em is supported. I see OpenVPN configs with openvpn-option heavily overused once in a while — before including an option, make sure what you need to do is really not supported.

DHCP server

In the DHCP server, there is not one, but too escape hatches. One is the "subnet-parameters" option under "subnet". Another one is a "global-parameters" under "shared-network-name".

See an example:

set service dhcp-server shared-network-name LAN subnet subnet-parameters "ping-timeout 5;"

Since dhcpd.conf syntax is more complex than just a list of options, it's important to make sure that generated config will be valid. It's easy to make your DHCP server stop loading and spend some time reading the log to see what is wrong, so be careful here.

Note that these options are not supported by the DHCPv6 server. Anyone thinks we should support it?

Dynamic DNS

In dynamic DNS, you can use the generic HTTP method if your provider and protocol is not supported.

set service dns dynamic interface eth0 use-web url http://dyndns.example.com/?update

Since no one can possibly support all providers, I believe it will remain a necessary option forever.

When all else fails: the postconfig script

If something is not supported and doesn't have a handy escape hatch, you still can implement it with the postconfig script. That script is found at /config/scripts/vyatta-postconfig-bootup.script and runs after config.boot loading is complete, so it's particularly conductive to manipulating things like raw iptables rules.

VyOS doesn't delete or overwrite anything in the global netfilter tables after boot, so it's safe to put your commands there, for example "/sbin/iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu" for global MSS clamping.

You still need to be careful not to conflict with any of the rules inserted by VyOS though, in general, so always make sure to check what exactly VyOS generates before using the postconfig script.

Looking forward

VyOS 1.2.0 brings improvements to the postconfig script execution and adds some more escape hatches — stay tuned for updates.

DNS forwarding in VyOS

A lot of small networks do not have their own DNS server, but it's not always desirable to just leave hosts to use an external third-party server either, that's why we've had DNS forwarding in VyOS for a long time and are going to keep it there for the foreseeable future.

Experienced VyOS users already know all about it, but we should post something for newcomers too, shouldn't we?

Configuring DNS forwarding is very simple. Assuming you have "system name-server" set, all you need to do to simply forward requests from hosts behind eth0 to it is "set service dns forwarding listen-on eth0". Repeat for every interfaces where you have clients and you are done.

There are some knobs for telling the service to use or not use specific DNS servers though:

set service dns forwarding listen-on eth0

# Use name servers from "system name-server"
set service dns forwarding system

# Use servers received from DHCP on eth1 (typically an ISP interface)
set service dns forwarding dhcp eth1

# Use a hardcoded name server
set service dns forwarding name-server

You can also specify cache size:

set service dns forwarding cache-size 1000

One of the less known features is the option to use different name servers for different domains. It can be used for a quick and dirty split-horizon DNS, or simply for using an internal server just for internal domains rather than recursive queries:

set service dns forwarding domain mycompany.local server
set service dns forwarding domain mycompany.example.com server

And that's all to it. DNS forwarding is not a big feature — useful doesn't always equal complex.

Loopback and the dummies

"There is no place like" the old saying goes. While the loopback interface is most often seen as the interface where the address is assigned by default and where the network is routed, and just a way for programs on the same host to communicate over the network without actual network, it has uses in networked context as well.

Before we talk about those use cases, we need to discuss interfaces themselves. In some OSes, such as Cisco IOS, and many BSD derivatives, it is possible to create multiple loopbacks. Linux kernel (and thus VyOS) historically allowed only one loopback (named "lo"), and this behaviour has become too traditional and relied upon to change overnight, so to implement multiple loopback, a new interface type called "dummy" was added. Dummy interfaces are functionally identical to loopbacks so the difference is mostly aesthetic.

This is how to setup a dummy interface: "set interfaces dummy dum0 address ...". If your problem does not require independent interfaces, you can also just add another address to the loopback.

So, why would one want to use a loopback/dummy interface instead of assigning another address to a physical NIC?

Case 1: tunnel endpoints

We have already talked about GRE/IPsec behind NAT and/or with dynamic addresses. Since GRE requires fixed local and remote endpoint  addresses to work, and in a setup where dynamic addresses or NAT is involved you do not have fixed addresses, the trick is to use a pair of addresses made up specially for this purpose as GRE endpoints.

Case 2: management addresses

Suppose you have a router A with two NICs, connected to networks B and C that are connected to each other, so that if any of the links fails, the network as a whole is still operational. However, if you choose either NIC A or NIC B address as a management address, it may become inaccessible if one of the NICs fail, forcing you to manually fall back to the other address.

To prevent this situations, people often assign a dedicated management address to a loopback, create a DNS record for it, and advertise that address to all other routers so that as long as there is at least one path to that router that works, they do not need to worry about addresses of physical NICs to SSH to the router, and are free to change those addresses without having to update the DNS or memorize the new address.

Case 3: iBGP peer addresses

Since iBGP uses the same autonomous system number for all routers, it loses the ability to use AS path for path selection and loop detection. This means to keep the network loop-free, one has to setup it as a full mesh, or use a route reflector.

If we use addresses of physical NICs for session endpoints, we run into the same problem as in the previous use case: a session goes down with the link even if there are other valid paths. A possible solution is to select dedicated addresses for iBGP sessions, assign them to loopbacks, and advertise them to all other routers through a link-state protocol such as OSPF.

Your use case?

If you know other cases when a network setup can be improved by using loopback or dummy interfaces, let us know!

Naming of the nightly builds

Historically, we used to use "999.$timestamp" version numbers for development builds, including nightly builds. In our build scripts termninology, a development build is any build that is started without doing "./configure --build-type=release --version=1.2.0" or similar (before the build script rewrite that was "./configure --with-release-build" and you also needed to put a version string in livecd/version or somethinf like that). In short, most builds in existence had that nondescript 999 version. That's how it was before the fork and we just didn't change that.

However, that approach is rather problematic. The 999 version doesn't tell anything about the branch it's built from or the nearest release, so one can only guess from the timestamp what it might be, and even that is not reliable. With introduction of a rolling release that will exist alongside the stable releases, this gets even more problematic, so something needs to be done about it.

We decided to change the format to "$release-rolling+$timestamp", like "1.2.0-rolling+201804060100". I have some hesitations about the "+", so if people think it should rather be "-", we can change it.

If you visit https://downloads.vyos.io/?dir=rolling/current/amd64 , you can see the new naming scheme in action. Let us know if you experience any problems with it!

VyOS to leverage the blockchain technology

Wild success of the blockchain technology on the mass market in the past couple of years is truly remarkable and it allowed many startups to raise substantial capitals. To keep up with those developments, we've been working on vyCoin, a cryptocurrency that will revolutionize the way people think about their network peering.

People will be able to mine vyCoin by routing other people's traffic and pay for traffic routing with it. Through smart contracts, it will be possible to charge vyCoins for routing other people's traffic with it as well. The use cases are endless, from facilitating peering arrangements at Internet exchanges to allowing people to earn something by participating in distributed private networks and ad hoc wireless.

To speed up the adoption, 50% of vyCoins come pre-mined and will be available to early adopters via an ICO. The vyCoin client will be integrated in the VyOS release images along with a kernel module for packet tracking so it will automatically start mining when you route traffic of other networks. The client will be sending 50% of the mined coins back to us as a service fee.

The ICO date will be announced later, meanwhile you can watch the vyCoin introduction video on Youtube.

Take a third option: site to site OpenVPN

I've written a long series of post about setting up IPsec VPNs between NATed machines. As you've already seen, with some creative configuration it's possible, but is it always worth the sacrifice? Sometimes performance requirements, or lack of support for anything else on the other side make it necessary, but if the other side is also a VyOS, or another open source system, there's an alternative.

While OpenVPN is usually associated with road warrior VPN setups, it is not limited to it. It does have a site to site option and it's very quick and easy to setup. For some strange reason, that option is neglected by just about everyone who otherwise supports OpenVPN: in OpenWRT it's possible to setup through custom config options, while in Mikrotik RouterOS it's not possible to setup at all. In VyOS we have an explicit option for it. OPNsense also supports site to site OpenVPN out of the box (I thought it doesn't, but the authors corrected me).

The advantages are that it takes very few commands to get a tunnel to work, and that it will work in any network where you can forward a single port, even is both sides are behind double NAT. The downside is performance: squeezing even 100mbit/s of encrypted traffic out of it can be impossible, typical iperf figures are 10-20 mbit/s. For many use cases that performance is more than enough, though if you plan to use the tunnel for storage replication or another high-traffic job, that option is definitely not for you and you'll have to resort to IPsec.

Ongoing improvements in project

Hi everyone,

While developers are working on the 1.2.0 release candidate, myself I’ve been looking into ways to improve the social and commercial aspects of project.

I’m happy to have established a positive relationship with a few companies, many of whom are themselves open source project developers and are ready to support other open source project.

The following open source tools will be added to our toolchain:

-  SonarQube by SonarSource is a source code analyzer that will help us improve code quality of the 1.2.0 and subsequent releases.

It’s odd that we didn’t know about it before, but now that we know of it, I hope it will change the project for the better. It uses an open core model and SonarCloud SaaS offering is available with no cost for open sources projects.

-  Discourse is a great discussion board software that will replace our old mybb forum. The old forum is now in read only mode to allow migration of the old topics to Discourse, and you give the new forum a try at https://forum.vyos.io. Discourse kindly offered us a discount for managed hosting via their open source program. The domain will be eventually changes to discourse.vyos.io (as per their discounted hosting plan requirements), but we’ll setup a redirect from both old domains when it’s ready.

On the commercial side, we are happy to announce that we now a part of two important Technology Alliances:

VMware Technology Alliance Program - Access Level will allow us to validate VyOS 1.2 OVA as a VMware Ready solution, which is also the first step towards the vCloud NFV certification. We hope it will help VyOS gain trust of the enterprise users and increase its adoption in that market.

Nutanix Elevate Technology Alliance Partner Program opens doors to the Nutanix platform for us. One of our goals from the beginning has been to bring VyOS to as many virtual platforms as possible, and it already runs on VMware, Hyper-V, Xen, and KVM, so Nutanix AHV support is a logical continuation of that effort, and we will also be listed on the Nutanix Marketplace to make it easy to deploy for Nutanix users.

I suppose these steps will help VyOS move forward and become a more successful project!

How to use AS path matching in your BGP policies

AS path is one of the most fundamental attributes of a (e)BGP advertisments. Its length is the first parameter in the best path selection algorithm (shorter is better), and it's also the sole mechanism of loop detection (if an AS is seen twice, there's a loop). However, despite the important role it plays behind the scenes, it's rather underutilized in routing policies.

A lot of time when prefix-list or specific route-map rule options such as next-hop can do, route filtering and modification based on AS path can do it better.

Let's see how to use it.

Firewall groups today and tomorrow

Substantial work has been done by Marian Tudosoiu to bring IPv6 firewall groups to the current implementation of firewall configuration scripts even before we give it a complete rewrite. It's already merged into the current branch and is expected to be included in the 1.2.0-rc1 release. Now it's probably a good time to make a post about using firewall groups for those who haven't used them yet.

Of course there's still a lot of work to be done, such as integrating groups into NAT, which likely does require a complete rewrite to be feasible.

The concept is simple enough: instead of creating multiple rules that only differ in one address or port number, you create a group with all those addresses and ports, and reference it in a rule.

VyOS has three group types: address groups, network groups, and port groups. In 1.1.8 they can only be used with IPv4 firewall rulesets, including "policy route" rules.

Let's create some groups:
set firewall group port-group ManagementPorts port 22
set firewall group port-group ManagementPorts port 23
set firewall group port-group ManagementPorts port 443

set firewall group address-group Servers address
set firewall group address-group Servers address
set firewall group address-group Servers address

set firewall group network-group TrustedNets network
set firewall group network-group TrustedNets network
set firewall group network-group TrustedNets network

Now we can create a ruleset that uses them. Let's make a rule that references nothing but groups:

set firewall name DMZ-In rule 10 action accept
set firewall name DMZ-In rule 10 protocol tcp
set firewall name DMZ-In rule 10 source group network-group TrustedNets
set firewall name DMZ-In rule 10 destination group port-group ManagementPorts
set firewall name DMZ-In rule 10 destination group address-group Servers

An important part is that you can modify groups on the fly without updating any rules.

As you can see, groups is a simple concept that can be learnt in minutes. Once they are in IPv6 and NAT, their use will be very similar.

The night of living dead protocols: RIPv2

RIP's name seems to have anticipated its ultimate fate. It used to stand for Routing Information Protocol before newer and better protocols killed it. Still, most routers in the world still support it even though few people seriously consider using it, thus making it an undead protocol.

This is mainly for compatibility reasons. After all, if you have an old box at a remote location, connected to a 128kbps ISDN line or worse, that is working fine and is impractical to replace, but supports nothing but RIP, what can you do? Likewise, some modern small routers by Netgear or D-link only have RIP, so if you can't just replace it, there's no other choice.

Besides, RIP remains a valuable teaching tool since it's conceptually simple, and understanding its limitations can help one understand the new routing protocols and their strengths.

What's bad about RIP?

There are some very good reasons to choose something else than RIP if you can. It's one of the oldest protocols in existence, and it was designed at the time when neither modern route selection techniques existed, nor the size of networks was big enough to warrant those techniques. When RIP reached its scalability limits, it was impossible to retrofit those techniques into it, so it was replaced rather than upgraded. RIPv2, the latest version, replaced broadcast advertisments with multicast and added support for CIDR, but that's about it — the fundamental design problems are, well, fundamental.

The biggest issue is that in RIP, routers are only aware of themselves and their immediate peers. They are completely blind to the rest of  the network. Link-state and path-vector protocols such as OSPF and eBGP are aware of the full topology (concrete or abstract), and can either reduce the full network graph to loop-free tree or immediately detect route advertisments as loop-inducing respectively.

All information that RIP advertisments include is the network, next hop, and an integer metric value. No router has any idea how its peers are connected to one another, so there is no way to detect loops before they form.

RIP includes a number of solutions to this problem, and they all have a limiting effect on its scalability.

Split horizon

It's simple — do not advertise routes received from peers back to them. It prevents the trivial loop when peers are trying to route traffic to networks they learnt about from someone else through you, but it cannot prevent wider loops. At least it has no effect on convergence time and doesn't create scalability limits either.

Counting to infinity

Before the other mechanisms were developed, this one was the only measure for detecting unreachable or looping routes. To make sure a route that is not actually reachable will be eventually recognized as such, it was decided to choose a maximum value that represents "infinite" (unreachable) metric. Since this process already can take quite some time, the value had to be small. In RIP, the infinite metric was set to 16. This means if a network has paths longer than 15 hops, the protocol just stops working.

Reverse poisoning

Even with 16 as infinite metric, the process of counting to infinity can be slow. The next idea was to not just wait for unreachable routes to become known as unreachable naturally, but actively advertise them to your peers as unreachable if a peer that was advertising them goes down.

This still is not a complete solution because if a router first receives  an unreachability advertisment from a router who's aware of the true situation, but later receives an update from a router that is still not aware of it, it will start using the second false advertisment.

Hold timer

The ultimate solution is to ignore any advertisment for a network whose metric has recently increased for some time, to avoid receiving updates from routers who do not know the real situation yet. This time shouldn't be too small, it should be similar to the time it takes for updates to propagate through the entire network. A common default value is 180 seconds. That is, to prevent convergence issues completely, it may take a network of just a few routers three minutes to converge.


So, suppose you are aware of all issues, but want or need to configure RIP nonetheless. It's pretty simple. First, enable RIP on all interfaces where you want to send and receive advertisments:

set protocols rip interface eth0
set protocols rip interface eth1

Networks that are configured on those interfaces will become a part of the RIP table automatically. If you want to advertise networks that are on other interfaces, you need to add them explicitly:

set protocols rip network

You can also use "redistribute" and "default-information originate" commands just like in all other routing protocols.

If everything is right, you will see something like this on the neighbor router:

vyos@vyos# run show ip rip 
Codes: R - RIP, C - connected, S - Static, O - OSPF, B - BGP
      (n) - normal, (s) - static, (d) - default, (r) - redistribute,
      (i) - interface

     Network            Next Hop         Metric From            Tag Time
R(n)         2     0 02:55
C(i)               1 self              0

If you remove the interface or the network statement, you will see the unreachable metric 16 for the duration of the hold timer, and only when the timer expires it will disappear from your table complely:

vyos@vyos# run show ip rip 
Codes: R - RIP, C - connected, S - Static, O - OSPF, B - BGP
      (n) - normal, (s) - static, (d) - default, (r) - redistribute,
      (i) - interface

     Network            Next Hop         Metric From            Tag Time
R(n)        16     0 01:57