1.1.8 release is available for download

1.1.8, the major minor release, is available for download from https://downloads.vyos.io/?dir=release/1.1.8 (mirrors are syncing up).

It breaks the semantic versioning convention, while the version number implies a bugfix-only release, it actually includes a number of new features. This is because 1.2.0 number is already assigned to the Jessie-based release that is still in beta, but not including those features that have been in the codebase for a while and a few of them have already been in production for some users would feel quite wrong, especially considering the long delay between the releases. Overall it's pretty close in scope to the original 1.2.0 release plan before Debian Squeeze was EOLd and we had to switch the effort to getting rid of the legacy that was keeping us from moving to a newer base distro.

You can find the full changelog here.

The release is available for both 64-bit and 32-bit machines. The i586-virt flavour, however, was discontinued since a) according to web server logs and user comments, there is no demand for it, unlike a release for 32-bit physical machines b) hypervisors capable of running on 32-bit hardware went extinct years ago. The current 32-bit image is built with paravirtual drivers for KVM/Xen, VMware, and Hyper-V, but without PAE, so you shouldn't have any problem running it on small x86 boards and testing it on virtual machines.

We've also made a 64-bit OVA that works with VMware and VirtualBox.

Security

Multiple vulnerabilities in OpenSSL, dnsmasq, and hostapd were patched, including the recently found remote code execution in dnsmasq.

Bugfixes

Some notable bugs that were fixed include:

  • Protocol negation in NAT not working correctly (it had exactly opposite effect and made the rule match the negated protocol instead)
  • Inability to reconfigure L2TPv3 interface tunnel and session ID after interface creation
  • GRUB not getting installed on RAID1 members
  • Lack of USB autosuspend causing excessive CPU load in KVM guests
  • VTI interfaces not coming back after tunnel reset
  • Cluster failing to start on boot if network links take too long to get up

New features

User/password authentication for OpenVPN client mode

A number of VPN providers (and some corporate VPNs) require that you use user/password authentication and do not support x.509-only authentication. Now this is supported by VyOS:

set interfaces openvpn vtun0 authentication username jrandomhacker
set interfaces openvpn vtun0 authentication password qwerty
set interfaces openvpn vtun0 tls ca-cert-file /config/auth/ca.crt
set interfaces openvpn vtun0 mode client
set interfaces openvpn vtun0 remote-host 192.0.2.1

Bridged OpenVPN servers no longer require subnet settings

Before this release, OpenVPN would always require subnet settings, so if one wanted to setup an L2 OpenVPN bridged to another interface, they'd have to specify a mock subnet. Not anymore, now if the device-type is set to "tap" and bridge-group is configured, subnet settings are not required.

New OpenVPN options exposed in the CLI

A few OpenVPN options that formerly would have to be configured through openvpn-option are now available in the CLI:

set interfaces openvpn vtun0 use-lzo-compression
set interfaces openvpn vtun0 keepalive interval 10
set interfaces openvpn vtun0 keepalive failure-count 5

Point to point VXLAN tunnels are now supported

In earlier releases, it was only possible to create multicast, point to multipoint VXLAN interfaces. Now the option to create point to point interfaces is also available:
set interfaces vxlan vxlan0 address 10.1.1.1/24
set interfaces vxlan vxlan0 remote 203.0.113.50
set interfaces vxlan vxlan0 vni 10

AS-override option for BGP

The as-override option that is often used as an alternative to allow-as-in is now available in the CLI:

set protocols bgp 64512 neighbor 192.0.2.10 as-override

as-path-exclude option for route-maps

The option for removing selected ASNs from AS paths is available now:
set policy route-map Foo rule 10 action permit
set policy route-map Foo rule 10 set as-path-exclude 64600

Buffer size option for NetFlow/sFlow

The default buffer size was often insufficient for high-traffic installations, which caused pmacct to crash. Now it is possible to specify the buffer size option:
set system flow-accounting buffer-size 512 # megabytes
There are a few more options for NetFlow: source address (can be either IPv4 or IPv6) and maximum number of concurrenct flows (on high traffic machines setting it too low can cause netflow data loss):
set system flow-accounting netflow source-ip 192.0.2.55
set system flow-accounting netflow max-flows 2097152

VLAN QoS mapping options

It is now possible to specify VLAN QoS values:
set interfaces ethernet eth0 vif 42 egress-qos 1:6
set interfaces ethernet eth0 vif 42 ingress-qos 1:6

Ability to set custom sysctl options

There are lots of sysctl options in the Linux kernel and it would be impractical to expose them all in the CLI, since most of them only need to be modified under special circumstances. Now you can set a custom option is you need to:
set system sysctl custom $key value $value

Custom client ID for DHCPv6

It  is now possible to specify custom client ID for DHCPv6 client:
set interfaces ethernet eth0 dhcpv6-options duid foobar

Ethernet offload options

Under "set interfaces ethernet ethX offload-options" you can find a number of options that control NIC offload.

Syslog level "all"

Now you can specify options for the *.* syslog pattern, for example:

set system syslog global facility all level notice

Unresolved or partially resolved issues

Latest ixgbe driver updates are not included in this release.

The issue with VyOS losing parts of its BGP config when update-source is set to an address belonging to a dynamic interface such as VTI and the interface takes long to get up and acquire its address was resolved in its literal wording, but it's not guaranteed that the BGP session will get up on its own in this case. It's recommended to set the update-source to an address of an interface available right away on boot, for example, a loopback or dummy interface.

The issue with changing the routing table number in PBR rules is not yet resolved. The recommended workaround is to delete the rule and re-create it with the new table number, e.g. by copying its commands from 'run show configuration commands | match "policy route "'.

Acknowledgements

I would like to say thanks to everyone who contributed and made this release possible, namely: Kim Hagen, Alex Harpin, Yuya Kusakabe, Yuriy Andamasov, Ray Soucy, Nikolay Krasnoyarski, Jason Hendry, Kevin Blackham, kouak, upa, Logan Attwood, Panagiotis Moustafellos, Thomas Courbon, and Ildar Ibragimov (hope I didn't forget anyone).

A note on the downloads server

The original packages.vyos.net server is still having IO performance problems and won't handle a traffic spike associated with release well. We've setup the https://downloads.vyos.io server on our new host specially for release images and will later migrate the rest of the old server including package repositories and the rsync setup.

A VyOS 1.2.0-alpha image with FRR instead of Quagga is available for testing (and we've found a GPL violation in VyOS)

FreeRangeRouting

Now that 1.1.8 release candidate is out and is (hopefully) being tested by community members, we can get back to building the future of VyOS.

It's been obvious that Quagga needs a replacement, and, since we've been using a Quagga fork inherited from Vyatta Core that includes features that never made it to the mainline quagga, even more so. The mainline Quagga still doesn't have usable commands for configuring multiple routing tables, nor they seem to actively accept patches that would be OS-specific.

The options were discussed many times and so far it seems FreeRangeRouting is the best option. It's a fork of Quagga that is being actively developed, actively accepts contributions, and already includes a number of features that Quagga lacks, such as support for network namespaces, PIM-SM, working IS-IS and more. There's also work being done on non-disruptive config reload.

While FRR is more or less a drop-in replacement for Quagga, it's not identical, and many CLI adjustments will be needed to make VyOS work with it. It needs a lot of testing. For this I've built a custom image that has vyatta-quagga replaced with FRR.

You can download the image here: http://dev.packages.vyos.net/tmp/vyos-1.2.0-alpha-frr-test.iso Please test it and report any issues in the routing protocols configuration you find. It's obviously experimental and you shouldn't use it in real routers, the best way to test is to load your production configs into test virtual machines.

GPL violation

Now to the GPL violation we've found. That violation in fact has been there for over five years and no one noticed it! Then again, it's relatively indirect and subtle.

Quagga is licensed under GPL (and so is FRR). In Vyatta/VyOS, Quagga has been built with SNMP support, so it links with net-snmp. In turn, net-snmp is built with SSL support and links with OpenSSL. This is where the problem is, OpenSSL is licensed under a four-clause BSD license that is not compatible with GPL.

Sadly, there is no easy way out, so it will take some time to fix this violation. The options are:

  • Build Quagga/FRR without SNMP support, which means routing protocol data will not be available through SNMP
  • Build net-snmp without SSL support, which means SNMPv3 will stop working
  • Patch net-snmp to support another cryptographic library that is GPL-compatible

The third option is hardest to implement, but it's the most appealing of all since all functionality will be preserved. We'd like to hear your suggestions regarding the libraries that would be license compatible and that can co-exist with OpenSSL.

1.1.8-rc2 release (fixes the problem with VLAN interfaces)

A new image is available for testing: http://dev.packages.vyos.net/iso/testing/vyos-1.1.8-rc2-amd64.iso 

If you are using VLAN interfaces, avoid the rc1 one and install the rc2 right away. Thanks to Samir we have identified a bug in VLAN interface configuration scripts (https://phabricator.vyos.net/T435) that prevented them from loading correctly. It was caused by an overlooked syntax error in the new options for ingress/egress VLAN QoS options.

Oddly, the current branch had a correct version of that patch, so 1.2.0 builds are not affected.


1.1.8-rc1 release is available for testing

The long overdue 1.1.8 release candidate is available for download from http://dev.packages.vyos.net/iso/testing/vyos-1.1.8-rc1-amd64.iso

While a number of people have already been running 1.2.0 nightly builds in production, we do acknowledge there are people who are not in position to install updates that are not completely stable, and recently discovered vulnerabilities in dnsmasq that potentially allow remote code execution are impossible to ignore (unlike many older vulnerabilities that are only locally or aren't practical to exploint).

It's stable for all practical purposes, but since it includes pretty big updates and a few new features, I suppose it's better to go through the release candidate phase. If in say a week no one finds any issues.

The release is only available for 64-bit machines at the moment. We can provide it for 32-bit, but we are wondering if anyone still wants it, when even small boards have 64-bit CPUs.

You can read the full changelog here: https://wiki.vyos.net/wiki/1.1.8_changelog_proposal 

Among package updates, there are openssl 1.0.2l and dnsmasq 2.72. Since squeeze is long EOL, the OpenSSL update required re-compiling everything that depends on OpenSSL ourselves, which took longer than we hoped.

Among VyOS fixes and features, there are user/password authentication for OpenVPN, as-override option for BGP neighbors, as-path-exclude option for route-map rules, tweakable pipe (buffer) size for netflow/sflow (too small hardcoded value could cause pmacct crash on high traffic routers), peer-to-peer VXLAN interfaces, and multiple fixes for bugs of varying severity, such as overly high CPU load on KVM guests or protocol negation in NAT rules not working.

A lot of features from 1.2.0 are not backportable due to big code changes and dependencies on way newer software versions than 1.1.x could provide, so features for cherry-picking had to be carefully chosen and even that needed quite a bit of merge conflict resolution. Quite a few of those were meant for the ill-fated "lithium" release that was supposed to be named 1.2.0 and be the last squeeze-based release, but then squeeze EOL'd, then serious life circumstances forced Alex Harpin to put all his VyOS work on hold thus leaving the maintainers team even more understaffed, and then the company we started to fund VyOS development through commercial support and services had a hard time when it almost reached the point of bankruptcy and dissolution (and, since it's self-funded, its founders almost reached the point of personal bankruptcy along with it), so by the time we could get things back on track a feature release based on squeeze wouldn't be feasible, especially considering how much we had to change to make the old codebase run on jessie. In a sense, it's a lithium that could have been, at least partially, rather than a straight maintenance release with nothing but bugfixes.
But, many of those features spent so much time in the limbo without making it into a release called stable that we felt compelled to include at least some of them.

I would like to say thanks to everyone who contributed and made this release possible, namely: Kim Hagen, Alex Harpin, Yuya Kusakabe, Yuriy Andamasov, Ray Soucy, Nikolay Krasnoyarski, Jason Hendry, Kevin Blackham, kouak, upa, Logan Attwood, Panagiotis Moustafellos, Thomas Courbon, and Ildar Ibragimov (hope I didn't forget anyone).





A book on VyOS in German is available

Our community member Markus Stubbig wrote a VyOS book in German that is now available for purchase from Amazon or bob.de.

Markus says there are no definite plans for an English version yet because he's not confident about his translation skills and will need help. If you have those skills and want to offer your services, we can connect you with Markus.

We don't have copies of the book yet (and none of the VyOS maintainers are fluent in German either, though I can read it a little), we cannot provide any kind review of the book yet, but we have no reasons to doubt Markus' expertise. If you get the book and write a review, we may publish it in the blog.


Update on the AWS SSH key fetching issue

We have fixed the issue with key fetching and submitted the updated AMI for review. It passed the automated scan, but manual review and deployment to the marketplace will take some time.

The new AMI also includes updates for dnsmasq security vulnerabilities that will be included in 1.1.8. If you want to install those updates on 1.1.7 by hand, you can use these packages: http://dev.packages.vyos.net/tmp/dnsmasq/


Permission denied issues with AWS instances

Quick facts: the issue is caused by an unexpected change in the EC2 system, there is no solution or workaround yet but we are working on it.

In the last week a number of people reported an issue with newly created EC2 instances of VyOS where they could not login to their newly created instance. At first we thought it may be an intermittent fault in the AWS since the AMI has not changes and we could not reproduce the problem ourselves, but the number of reports grew quickly, and our own test instances started showing the problem as well.

Since EC2 instances don't provide any console access, it took us a bit of time to debug. By juggling EBS volumes we finally managed to boot an affected instance with an disk image modified to include our own SSH keys.

The root cause is in our script that checks if the machine is running in EC2. We wanted to produce the AMI from an unmodified image, which required inclusion of the script that checks if the environment is EC2. Executing a script that obtains an SSH key from a remote (even if link-local) address is a security risk since in a less controlled environment an attacker could setup a server that could inject their keys to all VyOS systems.

The key observation was that in EC2, both system-uuid and system-serial-number fields in the DMI data always start with "EC2". We thought this is a good enough condition, and for the few years we've been providing AMIs, it indeed was.

However, Amazon changed it without warning and now the system-uuid may not start with EC2 (serial numbers still do), and VyOS instances stopped executing their key fetching script.

We are working on the 1.1.8 release now, but it will go through an RC phase, while the solution to the AWS issue is needed right now. We'll contact Amazon support to see what are the options, stay tuned.

VyOS development digest #10

At last, there are some news. In the order of immediate importance...

1.1.8 release plan

There have been some uncertainty over this issue and it wasn't clear if we'll be able to make an 1.1.8 release or not with squeeze's death, but recently Kim and I got squeeze builds to work again, and this enables us to finally make one.

What's certain is that bugfixes from 1.2.0 are going to make it there. What's not yet certain is which features we should cherry-pic. OpenVPN user/password auth, for example, is definitely safe and well tested enough to bring it to 1.1.8.

1.2.0 development status

1.1.8, of course, is nothing more than a maintenance release. But, we are way closer to a full feature release now that, especially with the work done by two awesome contributors, namely Christian Poessinger and Jules Taplin. Among recent contrubutions are multiple fixes to IPsec operational and configuration mode (in particular, "show vpn ipsec sa" works properly now), correct deletion of VTI interfaces, and there's also work being done on integrating mDNS repeater.

1.2.0, Python, and code rewrites

This was already discussed in http://blog.vyos.net/vyos-2-dot-0-development-digest-number-7-python-coding-guidelines-for-config-scripts-in-1-dot-2-0-and-config-parser-for-vyconf and http://blog.vyos.net/vyos-2-dot-0-development-digest-number-5-doing-1-dot-2-x-and-2-dot-0-development-in-parallel

By now, the Python library is "beta" rather than "alpha" and it has already been used to rewrite the cron ("set system task-scheduler") scripts by Tania Dzyubenko and me.

The library is now a proper Python package and it's installed as vyos.config module. You can use it for VyOS scripting, as well as code rewrites.

It has also been moved out of the vyatta-cfg package. The package where the new rewritten code goes is https://github.com/vyos/vyos-1x

You can find the rewritten cron script here: https://github.com/vyos/vyos-1x/blob/current/src/conf-mode/vyos-update-crontab.py
As you can see, it's architecturally pretty different from the older scripts. You can find the guideline it's written according to here in the wiki: https://wiki.vyos.net/wiki/Python_coding_guidelines

The architecture boils down to this: all VyOS config reads are confined to one function that converts it into an abstract representation, the rest of the logic is split into separate "verify", "generate", and "apply" stages that, accordingly, verify config correctness, generate configuration files, and apply them to the live system.

I'll re-iterate the reasons for these changes:
  • Testability: if only one place in the code really needs VyOS to work, the rest can be test on developers' workstations and build hosts, by hand as well as with automated unit and integration tests
  • Easier syntax changes: same, redesigned syntax can translate to the same abstract representation or a modified version of it, and there will not be need to weed out hundreds instance of the old syntax all over the script
  • Transactional commits: if the config correctness checking stage is clearly separated, once all scripts are rewritten in this manner, it will be possible to implement commit dry-run and abort commits if an error is detected before any change to the live system is made, thus greatly increasing the system's robustness

Scripts written in this manner will be reusable in VyOS 2.0 once it's ready with little change, thus ensuring more gradual rather than radical rewrite.

2.0 style command definitions in VyOS 1.2.0

If you look into the vyos-1x package, you will notice that there are no command "templates". That's right.

As you remember, the future VyOS 2.0 and its config backend will not be using the old style command "templates" (bunches of directories with node.def files). There is no way to get rid of them in VyOS 1.x, but we still can abstract them away, thus enabling a more gradual rewrite in this area too.

There are multiple problems with those old style templates. They are notoriously hard to navigate even for experienced developers and are a repellent for newcomers. They are equally hard to syntax check and the only real way to find out if they have any chance to work is to install a package on a test VyOS instance and try them by hand.
And last but not least, they allow embedded shell scripts that further spread the logic all over and make debugging even harder than it already is.

New style templates are in XML. Before anyone says "why not JSON", tell me if JSON got a widely accepted schema language and its implementation (I'm aware of some attempts, but...). XML had been machine-verifiable for almost two decades already.
XML interface definitions for VyOS come with a RelaxNG schema (https://github.com/vyos/vyos-1x/blob/current/schema/interface_definition.rng), which is a slightly modified schema from VyConf (https://github.com/vyos/vyconf/blob/master/data/schemata/interface_definition.rnc) so the definitions will be reusable with very minimal changes.

The great thing about schemas is not only that people can know the complete grammar for certain, but also that it can be automatically verified. The scripts/build-command-templates script that converts the XML definitions to old style templates also verifies them against the schema, so a bad definition will cause the package build to fail. I do agree that the format is verbose, but there is no other format now that would allow this. Besides, a specialized XML editor can alleviate the issue with verbosity.

Right now that script is complete enough to produce the templates for cron, but there's still work to be done. For example, it doesn't support the "allowed:" statement used for command completion. Any testing and patches are greatly appreciated!


VyOS 1.2.0 repository re-structuring

In preparation for the new 1.2.0 (jessie-based) beta release, we are re-populating the package repositories. The old repositories are now archived, you still can find then in the /legacy/repos directory on dev.packages.vyos.net

The purpose of this is two-fold. First, the old repo got quite messy, and Debian people (rightfully!) keep reminding us about it, but it would be difficult to do a gradual cleanup. Second, since the CI server has moved, and so did the build hosts, we need to test how well the new procedures are working. And, additionally, it should tell us if we are prepared to restore VyOS from its source should anything happen to the packages.vyos.net server or its contents.

For perhaps a couple of days, there will be no new nightly builds, and you will not be able to build ISOs yourself, unless you change the repo path in ./configure options by hand. Stay tuned.

Phabricator migration

I know you are tired of this already, but... yes, you guessed it right, we are migrating the phabricator again!

This time we are moving it to the host that currently houses the wiki, a VM at OpenITC (thanks, Sean!). This should be the last migration for a long while. We plan to consolidate all web resources on that host: while this is not so good for redundancy, it's easier to manage. Since the blog is not hosted there, and we also have a twitter, we still have out of band channels to notify people about outages and resolution timeframe estimates, should that server ever failed.

Today we'll shutdown the phabricator to migrate all the data on the new host and re-deploy it there (it would be nice to put it in read only mode instead, but it currently doesn't have it, it's planned for future versions). We also need to reconfigure the web server for the new setup, so there may be short periods of downtime for the wiki too.

We'll notify you when migration is complete.