Update on the AWS SSH key fetching issue

We have fixed the issue with key fetching and submitted the updated AMI for review. It passed the automated scan, but manual review and deployment to the marketplace will take some time.

The new AMI also includes updates for dnsmasq security vulnerabilities that will be included in 1.1.8. If you want to install those updates on 1.1.7 by hand, you can use these packages: http://dev.packages.vyos.net/tmp/dnsmasq/


Permission denied issues with AWS instances

Quick facts: the issue is caused by an unexpected change in the EC2 system, there is no solution or workaround yet but we are working on it.

In the last week a number of people reported an issue with newly created EC2 instances of VyOS where they could not login to their newly created instance. At first we thought it may be an intermittent fault in the AWS since the AMI has not changes and we could not reproduce the problem ourselves, but the number of reports grew quickly, and our own test instances started showing the problem as well.

Since EC2 instances don't provide any console access, it took us a bit of time to debug. By juggling EBS volumes we finally managed to boot an affected instance with an disk image modified to include our own SSH keys.

The root cause is in our script that checks if the machine is running in EC2. We wanted to produce the AMI from an unmodified image, which required inclusion of the script that checks if the environment is EC2. Executing a script that obtains an SSH key from a remote (even if link-local) address is a security risk since in a less controlled environment an attacker could setup a server that could inject their keys to all VyOS systems.

The key observation was that in EC2, both system-uuid and system-serial-number fields in the DMI data always start with "EC2". We thought this is a good enough condition, and for the few years we've been providing AMIs, it indeed was.

However, Amazon changed it without warning and now the system-uuid may not start with EC2 (serial numbers still do), and VyOS instances stopped executing their key fetching script.

We are working on the 1.1.8 release now, but it will go through an RC phase, while the solution to the AWS issue is needed right now. We'll contact Amazon support to see what are the options, stay tuned.

VyOS development digest #10

At last, there are some news. In the order of immediate importance...

1.1.8 release plan

There have been some uncertainty over this issue and it wasn't clear if we'll be able to make an 1.1.8 release or not with squeeze's death, but recently Kim and I got squeeze builds to work again, and this enables us to finally make one.

What's certain is that bugfixes from 1.2.0 are going to make it there. What's not yet certain is which features we should cherry-pic. OpenVPN user/password auth, for example, is definitely safe and well tested enough to bring it to 1.1.8.

1.2.0 development status

1.1.8, of course, is nothing more than a maintenance release. But, we are way closer to a full feature release now that, especially with the work done by two awesome contributors, namely Christian Poessinger and Jules Taplin. Among recent contrubutions are multiple fixes to IPsec operational and configuration mode (in particular, "show vpn ipsec sa" works properly now), correct deletion of VTI interfaces, and there's also work being done on integrating mDNS repeater.

1.2.0, Python, and code rewrites

This was already discussed in http://blog.vyos.net/vyos-2-dot-0-development-digest-number-7-python-coding-guidelines-for-config-scripts-in-1-dot-2-0-and-config-parser-for-vyconf and http://blog.vyos.net/vyos-2-dot-0-development-digest-number-5-doing-1-dot-2-x-and-2-dot-0-development-in-parallel

By now, the Python library is "beta" rather than "alpha" and it has already been used to rewrite the cron ("set system task-scheduler") scripts by Tania Dzyubenko and me.

The library is now a proper Python package and it's installed as vyos.config module. You can use it for VyOS scripting, as well as code rewrites.

It has also been moved out of the vyatta-cfg package. The package where the new rewritten code goes is https://github.com/vyos/vyos-1x

You can find the rewritten cron script here: https://github.com/vyos/vyos-1x/blob/current/src/conf-mode/vyos-update-crontab.py
As you can see, it's architecturally pretty different from the older scripts. You can find the guideline it's written according to here in the wiki: https://wiki.vyos.net/wiki/Python_coding_guidelines

The architecture boils down to this: all VyOS config reads are confined to one function that converts it into an abstract representation, the rest of the logic is split into separate "verify", "generate", and "apply" stages that, accordingly, verify config correctness, generate configuration files, and apply them to the live system.

I'll re-iterate the reasons for these changes:
  • Testability: if only one place in the code really needs VyOS to work, the rest can be test on developers' workstations and build hosts, by hand as well as with automated unit and integration tests
  • Easier syntax changes: same, redesigned syntax can translate to the same abstract representation or a modified version of it, and there will not be need to weed out hundreds instance of the old syntax all over the script
  • Transactional commits: if the config correctness checking stage is clearly separated, once all scripts are rewritten in this manner, it will be possible to implement commit dry-run and abort commits if an error is detected before any change to the live system is made, thus greatly increasing the system's robustness

Scripts written in this manner will be reusable in VyOS 2.0 once it's ready with little change, thus ensuring more gradual rather than radical rewrite.

2.0 style command definitions in VyOS 1.2.0

If you look into the vyos-1x package, you will notice that there are no command "templates". That's right.

As you remember, the future VyOS 2.0 and its config backend will not be using the old style command "templates" (bunches of directories with node.def files). There is no way to get rid of them in VyOS 1.x, but we still can abstract them away, thus enabling a more gradual rewrite in this area too.

There are multiple problems with those old style templates. They are notoriously hard to navigate even for experienced developers and are a repellent for newcomers. They are equally hard to syntax check and the only real way to find out if they have any chance to work is to install a package on a test VyOS instance and try them by hand.
And last but not least, they allow embedded shell scripts that further spread the logic all over and make debugging even harder than it already is.

New style templates are in XML. Before anyone says "why not JSON", tell me if JSON got a widely accepted schema language and its implementation (I'm aware of some attempts, but...). XML had been machine-verifiable for almost two decades already.
XML interface definitions for VyOS come with a RelaxNG schema (https://github.com/vyos/vyos-1x/blob/current/schema/interface_definition.rng), which is a slightly modified schema from VyConf (https://github.com/vyos/vyconf/blob/master/data/schemata/interface_definition.rnc) so the definitions will be reusable with very minimal changes.

The great thing about schemas is not only that people can know the complete grammar for certain, but also that it can be automatically verified. The scripts/build-command-templates script that converts the XML definitions to old style templates also verifies them against the schema, so a bad definition will cause the package build to fail. I do agree that the format is verbose, but there is no other format now that would allow this. Besides, a specialized XML editor can alleviate the issue with verbosity.

Right now that script is complete enough to produce the templates for cron, but there's still work to be done. For example, it doesn't support the "allowed:" statement used for command completion. Any testing and patches are greatly appreciated!


Donations and other ways to support VyOS

Hello, community!

We got many requests about how you can donate, we decided open this possibility to those who asked

After all, this is direct support to the project that all you offer, and we constantly need a support of all types.

As was mentioned before, you can contribute in many ways:

But if you would like to contribute via donation you are welcome to do so!

Raised money will be used for project needs like:

  • Documentation development
  • Tutorials and training courses creation
  • Artwork creation
  • Travels of project maintainers to relevant events 
  • Event organization
  • Videos
  • Features development 
  • Popularization of VyOS
  • Servers
  • Lab
  • Software
  • Hardware

Of course, that is not a complete list of needs that project have but most visible.

Thank you!


VyOS 1.2.0 repository re-structuring

In preparation for the new 1.2.0 (jessie-based) beta release, we are re-populating the package repositories. The old repositories are now archived, you still can find then in the /legacy/repos directory on dev.packages.vyos.net

The purpose of this is two-fold. First, the old repo got quite messy, and Debian people (rightfully!) keep reminding us about it, but it would be difficult to do a gradual cleanup. Second, since the CI server has moved, and so did the build hosts, we need to test how well the new procedures are working. And, additionally, it should tell us if we are prepared to restore VyOS from its source should anything happen to the packages.vyos.net server or its contents.

For perhaps a couple of days, there will be no new nightly builds, and you will not be able to build ISOs yourself, unless you change the repo path in ./configure options by hand. Stay tuned.

Phabricator migration

I know you are tired of this already, but... yes, you guessed it right, we are migrating the phabricator again!

This time we are moving it to the host that currently houses the wiki, a VM at OpenITC (thanks, Sean!). This should be the last migration for a long while. We plan to consolidate all web resources on that host: while this is not so good for redundancy, it's easier to manage. Since the blog is not hosted there, and we also have a twitter, we still have out of band channels to notify people about outages and resolution timeframe estimates, should that server ever failed.

Today we'll shutdown the phabricator to migrate all the data on the new host and re-deploy it there (it would be nice to put it in read only mode instead, but it currently doesn't have it, it's planned for future versions). We also need to reconfigure the web server for the new setup, so there may be short periods of downtime for the wiki too.

We'll notify you when migration is complete.