tag:blog.vyos.net,2013:/posts blog.vyos.net 2017-03-28T19:13:12Z VyOS tag:blog.vyos.net,2013:Post/1140890 2017-03-23T02:44:02Z 2017-03-28T19:13:12Z Donations and other ways to support VyOS

Hello, community!

We got many requests about how you can donate, we decided open this possibility to those who asked

After all, this is direct support to the project that all you offer, and we constantly need a support of all types.

As was mentioned before, you can contribute in many ways:

But if you would like to contribute via donation you are welcome to do so!

Raised money will be used for project needs like:

  • Documentation development
  • Tutorials and training courses creation
  • Artwork creation
  • Travels of project maintainers to relevant events 
  • Event organization
  • Videos
  • Features development 
  • Popularization of VyOS
  • Servers
  • Lab
  • Software
  • Hardware

Of course, that is not a complete list of needs that project have but most visible.

Find below most convenient way for donation

If you need invoice, please drop me email or ping me on chat


Thank you!


Bitcoin: 1PpUa61FytNSWhTbcVwZzfoE9u12mQ65Pe

PayPal Subscription

PayPal One time donation
]]>
Yuriy Andamasov
tag:blog.vyos.net,2013:Post/1128826 2017-02-05T18:10:58Z 2017-03-19T04:59:27Z VyOS 1.2.0 repository re-structuring

In preparation for the new 1.2.0 (jessie-based) beta release, we are re-populating the package repositories. The old repositories are now archived, you still can find then in the /legacy/repos directory on dev.packages.vyos.net

The purpose of this is two-fold. First, the old repo got quite messy, and Debian people (rightfully!) keep reminding us about it, but it would be difficult to do a gradual cleanup. Second, since the CI server has moved, and so did the build hosts, we need to test how well the new procedures are working. And, additionally, it should tell us if we are prepared to restore VyOS from its source should anything happen to the packages.vyos.net server or its contents.

For perhaps a couple of days, there will be no new nightly builds, and you will not be able to build ISOs yourself, unless you change the repo path in ./configure options by hand. Stay tuned.

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1127419 2017-01-31T00:08:48Z 2017-01-31T00:08:48Z Phabricator migration is over

Hello there!

We finished migration of phabricator, 

and also used this opportunity to add SSL to forum and wiki, so now all resources using a secure connection.

Enjoy!


]]>
Yuriy Andamasov
tag:blog.vyos.net,2013:Post/1126761 2017-01-28T01:28:11Z 2017-01-28T01:28:11Z Phabricator migration

I know you are tired of this already, but... yes, you guessed it right, we are migrating the phabricator again!

This time we are moving it to the host that currently houses the wiki, a VM at OpenITC (thanks, Sean!). This should be the last migration for a long while. We plan to consolidate all web resources on that host: while this is not so good for redundancy, it's easier to manage. Since the blog is not hosted there, and we also have a twitter, we still have out of band channels to notify people about outages and resolution timeframe estimates, should that server ever failed.

Today we'll shutdown the phabricator to migrate all the data on the new host and re-deploy it there (it would be nice to put it in read only mode instead, but it currently doesn't have it, it's planned for future versions). We also need to reconfigure the web server for the new setup, so there may be short periods of downtime for the wiki too.

We'll notify you when migration is complete.

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1123329 2017-01-15T18:40:40Z 2017-03-23T23:11:46Z VyOS 2.0 development digest #9: socket communication functionality, complete parser, and open tasks

Socket communication

A long-awaited (by me, anyway ;) milestone: VyConf is now capable of communicating with clients. This allows us to write a simple non-interactive client. Right now the only supported operaion is "status" (a keepalive of sorts), but the list will be growing.

I guess I should talk about the client before going into technical details of the protocol. The client will be way easier to use than what we have now. Two main problems with CLI tools from VyOS 1.x is that my_cli_bin (the command used by set/delete operations) requires a lot of environment setup, and that cli-shell-api is limited in scope. Part of the reason for this is that my_cli_bin is used in the interactive shell. Since the interactive shell of VyConf will be a standalone program rather than a bash completion hack, we are free to make the non-interactive client more idiomatic as a shell command, closer in user experience to git or s3cmd.

This is what it will look like:


SESSION=$(vycli setupSession)
vycli --session=$SESSION configure
vycli --session=$SESSION set "system host-name vyos"
vycli --session=$SESSION delete "system name-server 192.0.2.1"
vycli --session=$SESSION commit
vycli --session=$SESSION exists "service dhcp-server"
vycli --session=$SESSION commit returnValue "system host-name"
vycli --session=$SESSION --format=json show "interfaces ethernet"

As you can see, first, the top level words are subcommands, much like "git branch". Since the set of top level words is fixed anyway, this doesn't create new limitations. Second, the same client can execute both high level set/delete/commit operations and low level exists/returnValue/etc. methods. Third, the only thing it needs to operate is a session token (I'm thinking that unless it's passed in --session option, vycli should try to get it from an environment variable, but we'll see, let me know what you think about this issue). This way contributors will get an easy way to test the code even before interactive shell is complete; and when VyOS 2.0 is usable, shell scripts and people fond of working from bash rather than the domain-specific shell will have access to all system functions, without worrying about intricate environment variable setup.

The protocol

As I already said in the previous post, VyConf uses Protobuf for serialized messages. Protobuf doesn't define any framing, however, so we have to come up with something. Most popular options are delimiters and length headers. The issue with delimiters is that you have to make sure they do not appear in user input, or you risk losing a part of the message. Some programs choose to escape delimiters, other rely on unusual sequences, e.g. the backend of OPNSense uses three null bytes for it. Since Protobuf is a binary protocol, no sequence is unusual enough, so length headers look like the best option. VyConf uses 4 byte headers in network order, that are followed by a Protobuf message. This is easy enough to implement in any language, so it shouldn't be a problem when writing bindings for other languages.

The code

There is a single client library that can be used by all of the non-interactive client and the interactive shell. It will also serve as the OCaml bindings package for VyConf (Python and other languages wil need their own bindings, but with Protobuf, most of it can be autogenerated).

Parser improvements

Inactive and ephemeral nodes

The curly config parser is now complete. It supports the inactive and ephemeral properties. This is what a config with those will look like:

protocols {
  static {
    /* Inserted by a fail2ban-like script */
    #EPHEMERAL route 192.0.2.78/32 {
      blackhole;
    }
    /* DIsabled by admin */
    #INACTIVE route 203.0.113.128/25 {
      next-hop 203.0.113.1;
    }
  }
}

While I'm not sure if there are valid use cases for it, nodes can be inactive and ephemeral at the same time. Deactivating an ephemeral node that was created by scritp perhaps? Anyway, since both are a part of the config format that the "show" command will produce, we get to support both in the parser too.

Multi nodes

By multi nodes I mean nodes that may have more than one value, such as "address" in interfaces. As you remember, I suggested and implemented a new syntax for such nodes:

interfaces {
  ethernet eth0 {
    address [
      192.0.2.1/24;
      192.0.2.2/24;
    ];
  }
}

However, the parser now supports the original syntax too, that is:.

interfaces {
  ethernet eth0 {
    address 192.0.2.1/24;
    address 192.0.2.2/24;
  }
}

I didn't intend to support it originally, but it was another edge case that prompted me to add it. For config read operations to work correctly, every path in the tree must be unique. The high level Config_tree.set function maintains this invariant, but the parser gets to use lower level primitives that do not, so if a user creates a config with duplicate nodes, e.g. by careless pasting, the config tree that the parser returns will have them too, so we get to detect such situations and do something about it. Configs with duplicate tag nodes (e.g. "ethernet eth0 { ... } ethernet eth0 { ... }") are rejected as incorrect since there is no way to recover from this. Multiple non-leaf nodes with distinct children (e.g. "system { host-name vyos; } system { name-server 192.0.2.1; }") can be merged cleanly, so I've added some code to merge them by moving children of subsequent nodes under the first on and removing the extra nodes afterwards. However, since in the raw config there is no real distinction between leaf and non-leaf nodes, so in case of leaf nodes that code would simply remove all but the first. I've extended it to also move values into the first node, which equates support for the old syntax, except node comments and inactive/ephemeral properties will be inherited from the first node. Then again, this is how the parser in VyOS 1.x behaves, so nothing is lost.

While the show command in VyOS 2.0 will always use the new syntax with curly brackets, the parser will not break the principle of least astonishment for people used to the old one. Also, if we decide to write a migration utility for converting 1.x configs to 2.0, we'll be able to reuse the parser, after adding semicolons to the old config with a simple regulat expression perhaps.

Misc

Node names and unquoted values now can contain any characters that are not reserved, that is, anything but whitespace, curly braces, square brackets, and semicolons.

What's next?

Next I'm going to work on adding low level config operations (exists/returnValue/...) and set commands so that we can do some real life tests.

There's a bunch of open tasks if you want to join the development:

T254 is about preventing nodes with reserved characters in their names early in the process, at the "set" time. There's a rather nasty bug in VyOS 1.1.7 related to this: you can pass a quoted node name with spaces to set and if there is no validation rule attached to the node, as it's with "vpn l2tp remote-access authentication local-users", the node will be created. It will fail to parse correctly after you save and reload the config. We'll fix it in 1.2.0 of course, but we also need to prevent it from ever appearing in 2.0 too.

T255 is about adding the curly config renderer. While we can use the JSON serializer for testing right now, the usual format is also just easier on the eyes, and it's a relatively simple task too.

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1120870 2017-01-05T22:10:17Z 2017-01-23T13:27:38Z VyOS 2.0 development digest #8: vote for or against the new tag node syntax, and the protobuf schema

Tag node syntax

The change in tag node format I introduced in the previous post turned out quite polarizing and started quite some discussion in the comments. I created a poll in phabricator for it: https://phabricator.vyos.net/V3 , please cast your vote there.

If you missed the post, or found the explanation confusing, here's what it's all about. Right now in config files we format tag nodes (i.e. nodes that can have children without predefined names, such as interfaces and firewall rules) differently from other nodes:


/* normal node */
interfaces {
  /* tag node */
  ethernet eth0 {
    address 192.0.2.1/24
  }
  /* tag node */
  ethernet eth1 {
    address 203.0.113.1/24
  }
}

It looks nice, but complicates the parser. What I proposed and implemented in the initial parser draft is to not use any custom formatting for tag nodes:

/* normal node */
interfaces {
  /* actually a tag node, but rendering is now the same as for normal */
  ethernet {
    eth0 {
      address 192.0.2.1/24;
    }
    eth1 {
     address 203.0.113.1/24
    }
  }
}

This makes the parser noticeable simpler, but makes the syntax more verbose and adds more newlines.

If more people vote against this change than for it, I'll take time to implement it in the parser.

Note: This change only affects the config syntax, and has no effect on the command syntax. The command for the example above would still be "set interfaces ethernet eth0 address 192.0.2.1/24", in user input and in the output of "show configuration commands". Tag nodes will also be usable as edit levels regardless of the config file syntax, as in "edit interfaces tunnel; copy tun0 to tun1".

Protobuf schema

Today I wrote an initial draft of the protobuf schema that VyConf daemon will use for communication with clients (shell, CLI tool, and HTTP bridge). You can find it here: https://github.com/vyos/vyconf/blob/master/data/vyconf.proto

Right now it defines the following operations:

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1120515 2017-01-04T19:27:32Z 2017-01-23T13:27:43Z VyOS 2.0 development digest #7: Python coding guidelines for config scripts in 1.2.0, and config parser for VyConf

Python coding guidelines for 1.2.0

In some previous post I was talking about the Python wrapper for the config reading library. However, simply switching to a language that is not Perl will not automatically make that code easy to move to 2.0 when the backend is ready, neither it will automatically improve the design and architecture. It will also improve the code in general, and help keeping it readable and maintainable.

You can find the document here: http://wiki.vyos.net/wiki/Python_config_script_policy 

In short:

  • Logic for config validation, generating configs, and changing system settings/restarting services must be completely separated
  • For any configs that allow nesting (dhcpd.conf, ipsec.conf etc.) template processor must be used (as opposed to string concatenation)
  • Functions should not randomly output anything to stdout/stderr
  • Code must be unit-testable

Config parser for VyConf/VyOS 2.0

Today I pushed initial implementation of the new config lexer and parser. It already supports nodes and node comments, but doesn't support node metadata (that will be used to mark inactive and ephemeral nodes).

You can read the code (https://github.com/vyos/vyconf/blob/master/src/curly_lexer.mll and https://github.com/vyos/vyconf/blob/master/src/curly_parser.mly) and play with it by loading the .cma's into REPL. Next step is to add config renderer. Once the protobuf schema is ready we can wrap it all into a daemon and finally have something to really play with, rather than just run the unit tests.

Informally, here's what I changed in the config syntax.

Old config

interfaces {
  /* WAN interface */
  ethernet eth0 {
    address 192.0.2.1/24
    address 192.0.2.2/24
    duplex auto
  }
}

New config

interfaces {
  ethernet {
    /* WAN interface */
    eth0 {
      address [
        192.0.2.1/24;
        192.0.2.2/24;
      ];
      duplex auto;
      // This kind of comment is ignored by the parser
    }
  }
}

As you can see, the changes are:

  • Leaf nodes are now terminated by semicolons rather than newlines.
  • There is syntax for comments that are ignored by the parser
  • Multi nodes have the array of values in square brackets.
  • Tag nodes do not receive any special formatting.

I suppose the last change may be controversial, because it can lead to somewhat odd-looking constructs like:

interfaces {
  ethernet {
    eth0 {
      vif {
        21 {
          address 192.0.2.1/24
        }
      }
    }
  }
}

If you are really going to miss the old approach to tag nodes (that is "ethernet eth0 {" as opposed to "ethernet { eth0 { ...", let me know and I guess I can come up with something. The main difficulty is that, while this never occurs in configs VyOS config save produces, different tag nodes, e.g. "interfaces ethernet" and "interfaces tunnel" can be intermingled, so for parsing we have to track which ones were already created, and this will make the parser code a lot longer.

I'm pretty convinced that "address 192.0.2.1/24; address 192.0.2.2/24" is simply visual clutter and JunOS-like square bracket syntax will make it cleaner. It also solves the aforementioned problem with interleaved nodes tracking for leaf nodes.

Let me know what you think about the syntax.

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1117598 2016-12-31T09:02:03Z 2017-01-23T13:27:46Z VyOS 2.0 development digest #6: new beginner-friendly tasks, design questions, and the details of the config tree

The tasks

Both tasks from the previous post have been taken up and implemented by Phil Summers (thanks, Phil!). New tasks await.

First task was very simple: the Reference_tree module needs functions for checking facts about nodes, analogous to is_multi. For config output, and for high level set/delete/commit operations we need easy ways to know if the node is tag or leaf, or valueless, what component is responsible for it etc. It can be done mostly by analogy with is_multi function and its relatives, so it's friendly to complete beginners. But Phil Summers implemented it before I could make the post (thanks again, Phil!).

Second task is a little bit more involved but still simple enough for anyone who started learning ML not long ago. It's about loading interface definitions from a directory. In VyOS, we may have a bunch of files in /usr/share/vyos/interfaces such as firewall.xml, system.xml, ospf.xml, and so on, and we need to load them into the reference tree that is used for path validation, completion etc.

Design questions

To give you some context, I'll remind you that the vyconf shell will not be bash-based, due to having to fork and modify bash (or any other UNIX shell) to get completion from the first word to begin with, and for variety of other reasons. So, first question: do you think we should use the vyconf shell where you can enter VyOS configuration commands as login shell, or we should go for JunOS-like approach when you login to a UNIX shell and then issue a command to enter the configuration shell? You can cast your vote here: https://phabricator.vyos.net/V2 

Second question is more open-ended: we are going to printing the config as the normal VyOS config syntax, and as set commands, but what else should we support? Some considerations: since "show" will be a part of the config API, it can be used by e.g. web GUI to display the config. This means config output of XML or JSON can be a useful thing. But, which one, or perhaps both? And also we need to decide what the XML and/or JSON shouid look like, since we can go for a generic schema that keeps node names in attributes, or we can use custom tags such as <interfaces> (but then every component should provide a schema).

Now, to the "long-awaited" details of the config tree...

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1119428 2016-12-31T05:49:15Z 2017-02-01T04:11:03Z Change is coming to VyOS project

People often ask us the same questions, such as if we know about Debian 6 EOL, or when 1.2.0 will be released, or when this or that feature will be implemented. The short answer, for all of those: it depends on you. Yes, you, the VyOS users.

Here’s what it takes to run an open source project of this type. There are multiple tasks, and they all have to be done:

  • Emergency fixes and security patches

  • Routine bug fixes, cleanups, and refactoring

  • Development of new features

  • Documentation writing

  • Testing (including writing automated tests)


All those tasks need hands (ideally, connected to a brain). Emergency bug fixes and security patches needs a team of committed people who can do this job on a short notice, which is attainable in two ways, either there are people for whom it’s their primary job, or the team of committed people is large enough to have people with spare time at any given moment.

Cleanups and refactoring are also things that need a team of committed people because those are things that no one benefits from in a short run, it’s about making life easier for contributors and improving the sustainability of the project, keeping it from becoming an unmanageable mess. Development of new features needs people who are personally interested in those features and have the expertise to integrate them in a right way. It’s perfect if they also maintain their code, but if they simply hand documented and maintainable code to the maintainers team, that’s good enough.

Now, the sad truth is that VyOS Project has none of those. The commitment to using it among its users greatly exceeds the commitment to contributing to it. While we don’t know for certain how many people are using VyOS, we have at least some data. At the moment, there are 600 users of the official AMI on AWS. There were 11k+ users last month on user guide page and it’s constantly growing since the time when I took up the role of the community manager of the VyOS project. We are also aware about companies that have around 1k VyOS instances and companies that rely on VyOS in their business operations in one way or another. But still, if we talk about consumers vs. contributors, we see 99% consumers vs 1% contributors relation.


My original idea was to raise awareness of the VyOS project by introducing a new website, refreshing the forum look, activating social media channels and introducing modern collaboration tools to make participation in the project easier, open new ways how users and companies can participate and contribute. Finally bigger user base means there’s a larger pool of people and companies who can contribute to the project. We also launched commercial support with idea that if companies that using VyOS for their businesses can’t or just don’t want to participate in the project directly, the may be willing to support the project by purchasing support subscriptions.




10 months later I can admit that I was partially wrong in my thoughts. While consumer user base growing rapidly, i just can’t tell the same about contributors and this is a pity. Sure, we got a few new contributors, some of them contribute occasionally, other are more active, and some old contributors are back (Thank you guys for joining/re-joining VyOS!). We are also working with several companies that are showing interest in VyOS as a platform and contribute to the project in commercial means and via human resources, and that is great, however, it’s not enough at this scale.


At this point, I started thinking that current situation is not something that can be considered as fair and not really make sense.


This are just some of questions that came to my mind frequently:


  • Why those who not contributing literally nothing to the project, getting the same as others who spend their time and resources?

  • Why companies like ALP group using VyOS in their business and claiming publicly that they will return improvements to upstream when they are not actually returning anything? Why do some people think that they can come to IRC/Chat and demand something without contributing anything?

  • Why are those cloud providers that using VyOS for their businesses not bothering to support the project in any way?


I would like to remind you of the basic principles of the VyOS philosophy established from its start:


VyOS is a community driven project!

VyOS always will be an open source project!

VyOS will not have any commercial or any other special versions!


However, if we all want VyOS to be a great project, we all need to adhere to those principles, otherwise, nothing will happen. Community driven means that the driving force behind improvements should be those interested in them. Open source means we can’t license a proprietary component from a third party if existing open source software does not provide the feature you need. Finally, free for everyone means we all share responsibility for the success or failure of the project.


I’m happy and proud to be part of VyOS community and I really consider as my duty to help the project and the community grow. I’m doing what I can, and I expect that if you also care about the project, you will participate too.


We all can contribute to the project, no matter if you are developer or network engineer or neither of this.


There are many tasks that can be done by individuals with zero programming involved:


  • Documentation (documenting new features, improving existing wiki pages, or rewriting old documentation for Vyatta Core)

  • Support community in forums/IRC/chat (we have English and localized forums, and you can request a channel in your native language like our Japanese community did)

  • Feature requests (well described use cases from which can benefit all our community: note that a good feature request should make it easier for developers to implement it, just saying you want MPLS is not quite the same as researching existing open source implementations, trying them out, making sample configs contributors with coding skills can use for the reference, drafting CLI design and so on!)

  • Testing & Bug reports

  • Design discussions, such as those in the VyOS 2.0 development digest


If you work at the company that uses VyOS for business needs please consider  talking with CEO/CTO about:

  • Providing full/part time worker(s) to accomplish tasks listed above

  • Provide paid accounts in common clouds for development needs

  • Provide HW and license for laboratory (we need quite a lot of HW to support all of the hypervisors, same is true about licenses for interworking testing)

  • Buy commercial support & services



In January, we’d like to have a meeting with all current contributors to discuss what we can do to increase participation in the project.

Meanwhile, I would like to ask you to share this blog post to all whom it may concern.

All of the VyOS users (especially those companies that use VyOS in their business) should be aware that is time to start participate in the project if you want to keep using VyOS and rely on it in the future.


Brace yourself.

Change is coming!

]]>
Yuriy Andamasov
tag:blog.vyos.net,2013:Post/1118343 2016-12-26T19:57:31Z 2017-01-23T13:27:55Z Phabricator maintenance

We are working on the servers now, moving some things around again, and the phabricator is temporarily inaccessible. We'll let you know when it's resolved.

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1117392 2016-12-22T01:33:41Z 2017-02-01T08:54:50Z VyOS 2.0 development digest #5: doing 1.2.x and 2.0 development in parallel

There was a rather heated discussion about the 1.2.0 situation on the channel, and valid points were definitely expressed: while 2.0 is being written, 1.2.0 can't benefit from any of that work, and it's sad. We do need a working VyOS in any case, and we can't just stop doing anything about it and only work on 2.0. My original plan was to put 1.2.0 in maintenance mode once it's stabilized, but it would mean no updates at all for anyone, other than bugfixes. To make things worse, some things do need if not rewrite, but at least very deep refactoring bordering on rewrite just to make them work again, due to deep changes in the configs of e.g. StrongSWAN.

There are three main issues with reusing the old code,  as I already said: it's written in Perl, it mixes config reading and checking with logic, and it can't be tested outside VyOS. The fourth issue is that the logic for generating, writing, and applying configs is not separated in most scripts either so they don't fit the 2.0 model of more transactional commits. The question is if we can do anything about those issues to enable rewriting bits of 1.2.0 in a way that will allow reusing that code in 2.0 when the config backend and base system are ready, and what exactly should we do.

My conclusion so far is that we probably can, with some dirty hacks and extra care. Read on.

The language

I guess by now everyone agrees that Perl is a bad idea. There are few people who know it these days, and there is no justification for knowing it. The language is a minefield that lacks proper error reporting mechanism or means to convey the semantics.

If you are new to it, look at this examples:

All "error reporting" enabled, let's try to divide a string by an integer.

$ perl -e 'use strict; use warnings; print "foobar" / 42'
Argument "foobar" isn't numeric in division (/) at -e line 1.
0

A warning indeed... Didn't prevent program from producing a value though: garbage in, garbage out. And, my long time favorite: analogous issues bit me in real code a number of times!

$ perl -e 'print reverse "dog"'
dog

Even if you know that it has to do with "list context", good luck finding information about default context of this or that function in the docs. In short, if the language of VyOS 1.x wasn't Perl, a lot of bugs would be outright impossible.

Python looks like a good candidate for config scripts: it's strongly typed, the type and object system is fairly expressive, there are nice unit test libraries and template processors and other things, and it's reasonably fast. What I don't like about it and dynamically typed languages in general is that it needs a damn good test coverage because the set of errors it can detect at compile time is limited and a lot of errors make it to runtime, but there are always compromises.

But, we need bindings. VyConf will use sockets and protobuf messages for its API which makes writing bindings for pretty much any language trivial, but in 1.x.x it's more complicated. The C++/Perl library from VyOS backend is not really easy to follow, and not trivial to produce bindings for. However, we have cli-shell-api, which is already used in config scripts written in shell, and behaves as it should. It also produces fairly machine-friendly output, even though its error reporting is rudimantary (then again, error reporting of the C++ and Perl library isn't all that nice either). So, for a proof of concept, I decided to make a thin wrapper around cli-shell-api: later it can be rewritten as a real C++ binding if this approach shows its limitations. It will need some C++ library logic extraction and cleanup to replicate the behaviour (why the C++ library itself links against Perl interpreter library? Did you know it also links to specific version of the apt-pkg library that was never meant for end users and made no promise of API stability, for its version comparison function that it uses for soring names of nodes like eth0? That's another story though).

Anyway, I need to add the Python library to the vyatta-cfg package which I'll do soon, for the time being you can put the file to your VyOS (it works in 1.1.7 with python2.6) and play with it:  

Right now it exposes just a handful of functions: exists(), return_value(), return_values(), and list_nodes(). It also has is_leaf/is_tag/is_multi functions that it uses internally to produce somewhat better error reporting, though they are unnecessary in config scripts, since you already know that about nodes from templates. Those four functions are enough to write a config script for something like squid, dnsmasq, openvpn, or anything else that can reload its config on its own. It's programs that need fancy update logic that really need exists_orig or return_effective_value. Incidentally, a lot of components that need that rewrite to repair or could seriously benefit from serious overhaul are like that: for example. iptables is handled by manipulating individual rules right now even though iptables-restore is atomic, likewise openvpn is now managed by passing it the config in command line options while it's perfectly capable of reloading its config and this would make tunnel restarts a lot less disruptive, and strongswan, the holder of the least maintainable config script, is indeed capable of live reload too.

Which brings us to the next part...

The conventions

To avoid having to do two rewrites of the same code instead of just one, we need to make sure that at least substantial part of the code from VyOS 1.2.x can be reused in 2.0. For this we need to setup a set of conventions. I suggest the following, and let's discuss it.

Language version

Python 3 SHALL be used.

Rationale: well, how much longer can we all keep 2.x alive if 3.0 is just a cleaner and nicer implementation?

Coding standard

No single function shall SHOULD be longer than 100 lines.

Rationale: https://github.com/vyos/vyatta-cfg-vpn/blob/current/scripts/vpn-config.pl#L449-L1134 ;)

Logic separation and testability

This is the most important part. To be able to reuse anything, we need to separate assumptions about the environment from the core logic. To be able to test it in isolation and make sure most of the bugs are caught on developer workstations rather than test routers, we need to avoid dependendies on the global state whenever possible. Also, to fit the transactional commit model of VyOS 2.0 later, we need to keep consistency checking, generating configs, and restarting services separated.

For this, I suggest that we config scripts follow this blueprint:


def get_config():
    foo = vyos.config.return_value("foo bar")
    bar = vyos.config.return_value("baz quux")
    return {"foo": foo, "bar": bar} # Could be an object depending on size and complexity...

def verify(config):
    result do_some_checks(config)
    if checks_succees(result):
        return None
    else:
        raise ScaryException("Some error")

def generate(config):
    write_config_files(config)

def apply(config):
    restart_some_services(config)

if __name__ == '__main__':
    try:
       c = get_config()
       verify(c)
       generate(c)
       apply(c)
    except ScaryException:
        exit_gracefully()

This way the function that process the config can be tested outside of VyOS by creating the same stucture as get_config() would create by hand (or from file) and passing it as an argument. Likewise, in 2.0 we can call its verify(), update(), and apply() functions separately.

Let me know what you think.

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1117093 2016-12-21T14:09:03Z 2017-01-23T13:32:53Z VyOS 2.0 development digest #4: simple tasks for beginners, and the reasons to learn (and use) OCaml

Look, I lied again. This post is still not about the config and reference tree internals. People in the chat and elsewhere started showing some interest in learning OCaml and contributing to VyConf, and Hiroyuki Satou even already made a small pull request (it's regarding build docs rather than the code itself, but that's a good start and will help people with setting up the environment), so I decided to make a post to explain some details and address common concerns.

The easy tasks

There are a couple of tasks that can be done completely by analogy, so they are good for getting familiar with the code and making sure your build environment actually works.

The first one is about new properties of config tree nodes, "inactive" and "ephemeral", that will be used for JunOS-like activate/deactivate functionality, and for nodes that are meant to be temporary and won't make it to the saved config respectively.

The other one is about adding "hidden" and "secret" properties to the command definition schema and the reference tree, "hidden" is meant for hiding commands from completion (for feature toggles, or easter eggs ;), and "secret" is meant to hide sensitive data from unpriviliged users or for making public pastes.

Make sure you reference the task number in your commit description, as in "T225: add this and that" so that phabricator can automatically connect the commits with the task.

If you want to take them up and need any assistance, feel free to ask me in phabricator or elsewhere.

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1116186 2016-12-17T14:08:57Z 2017-01-23T13:32:59Z VyOS 2.0 development digest #3: questions for you, vyconf daemon config format, appliance directory structure, and external validators lookup

Ok, I changed my mind: before we jump into the abyss of datastructures and look how the config and reference trees work, I'll describe the changes I made in the last few days.

Also, I'd like to make it clear that if you don't respond to design questions I state in these posts, I, or whoever takes up the task, will just do what they think is right. ;)

I guess I'll start with the questions this time. First, in the comments to the first post, the person who goes by kglkgvkvsd544 suggested two features: commit-confirm by default, and an alternative solution to the partial commit where instead of loading the failsafe config, the system loads a previous revision instead, in the same way as commit-confirm does. I don't think any of those should be the only available option, but having them as configurable options may be a good idea. Let me know what you think about it.

Another very important question: we need to decide on the wire protocol that vyconf daemon will use for communication with its clients (the CLI tool, the interactive shell, and the HTTP bridge). I created a task for it, https://phabricator.vyos.net/T216, let me know what you think about it. I'm inclined towards protobuf myself.

Now to the recent changes.

vyconfd config

As I already said, VyConf is supposed to run as a daemon and keep the running config tree, the reference tree, and the user sessions in memory. Obviously, it needs to know where to get that data. It also needs to know where to write logs, where the PID file and socket file should be, and other things daemons are supposed to know. Here's what the vyconf config for VyOS may look like:

[appliance]
name = "VyOS"

data_dir = "/usr/share/vyos/"
program_dir = "/usr/libexec/vyos"
config_dir = "/etc/vyos"

# paths relative to config_dir
primary_config = "config.boot"
fallback_config = "config.failsafe"

[vyconf]
socket = "/var/run/vyconfd.sock"
pid_file = "/var/run/vyconfd.pid"
log_file = "/var/log/vyconfd.log"
That INI-like language is called TOML, it's pretty well specified and capable of some fancy stuff like arrays and hashes, apart from simple (string, string) pairs. The best part is that there are libraries for parsing it for many languages, and the one for OCaml is particularly nice and idiomatic (it uses lenses and option types to access a nested and possibly underdefined datastructure in a handy and typesafe way), like:
let pid_file = TomlLenses.(get conf (key "vyconf" |-- table |-- key "pid_file" |-- string)) in
match pid_file with
| Some f -> f
| None -> "/var/run/vyconfd.pid"
The config format and this example reflects some decisions. First, the directory structure if more FHS-friendly. What do we need /opt/vyos for, if FHS already has directories meant for exactly what we need: architecture-independent data (/usr/share), programs called by other programs and not directly by users (/usr/libexec), and config files (/etc)?
Second, all important parameters are configurable. The primary motivation for this is making VyConf usable for every appliance developer (most appliances will not be called VyOS for certain), but for ourselves and for every contributor to VyOS it's a reminder to avoid hardcoded paths anywhere: if changing it is just one line edit away, a hardcoded path is an especially bad idea.

Here's the complete directory structure I envision:
$data_dir/
  interfaces/ # interface definitions
  components/ # Component definitions
$program_dir/
  components/  # Scripts/programs that verify the appliance config, generate actual configs, and apply them
  migration/       # Migration scripts (convert appliance config if syntax changes)
  validators/       # Value validators
$config_dir/
  scripts/              # User scripts
  post-config.d/   # Post-config hooks, like vyatta-postconfig-bootup.script
  pre-commit.d/   # Pre-commit hooks
  post-commit.d/ # Post-commit hooks
  archive/             # Commit archive
This is not an exhaustive list of directories an appliance can have of course, it's just directories that have any significance for VyConf. I'm also wondering if we should introduce post-save hooks for those who want to do something beyond the built-in commit archive functionality.

External validators

As you remember, my idea is to get rid of the inflexible system of built-in "types" and make regex the only built-in constraint type, and use external validators for everything else.

External validators will be stored in the $program_dir/validators. Since many packages use the same types of values (IP addresses is a common example), and in VyOS 1.x we already have quite a lot of templates that reference the same validation scripts, making it a separate entity will simplify reusing them, since it's easy to see what validators exist, and you can also be sure that they behave like you expect (or if they don't, it's a bug).
Validator executable take two arguments, first argument is constraint string, and the second is the value to be validated (e.g. range "1-99" "42").The expected behaviour is to return 0 if the value is valid and a non-zero exit code otherwise. Validators should not produce any output, instead the user will get the message defined in the constraintError tag of the interface definition (this approach is more flexible since different things may want to use different messages, e.g. to clarify why exactly the value is not valid).

That's all for today. Feel free to comment and ask questions. The next post really will be about the config tree and the way set commands work, stay tuned.

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1114691 2016-12-12T20:45:10Z 2017-01-23T13:33:03Z VyOS 2.0 development digest #2

In the previous post we talked about the reasons for rewrite, design and implementation issues, and basic ideas. Now it's time to get to details. Today we'll mostly talk about command definitions, or rather interface definitions, since set commands is just one way to access the configuration interface.

Let's review the VyConf architecture (I included a few things in the diagram that we haven't discussed yet, ignore them for now):

At startup, VyConf will load the main config (or the fallback config, if that fails). But to know whether the config is valid, and to know what programs to call to actually configure the target applications, it needs additional data. We'll call that data "interface definitions" since it defines the configuration interface. Specifically, it defines:

  1. What config nodes (paths) are allowed (e.g. "interfaces ethernet", or "protocols ospf")
  2. What values are valid for that nodes (e.g. any IPv4 or IPv6 address for "system name-server")
  3. What script/program should be called when this or that part of the config is changed

The old way

Before we get to the new way, let's review the old way, the way it's one in the current VyOS implementation. In the current VyOS, those definitions are called "templates", no one remembers why.

This is a typical template file:
vyos@vyos# cat /opt/vyatta/share/vyatta-cfg/templates/interfaces/ethernet/node.tag/speed/node.def 
type: txt
help: Link speed
default: "auto"
syntax:expression: $VAR(@) in "auto", "10", "100", "1000", "2500", "10000"; "Speed must be auto, 10, 100, 1000, 2500, or 10000"
allowed: echo auto 10 100 1000 2500 10000

commit:expression: exec "\
	/opt/vyatta/sbin/vyatta-interfaces.pl --dev=$VAR(../@) \
	--check-speed $VAR(@) $VAR(../duplex/@)"

update: if [ ! -f /tmp/speed-duplex.$VAR(../@) ]; then
	   /opt/vyatta/sbin/vyatta-interfaces.pl --dev=$VAR(../@) \
	   	--speed-duplex $VAR(@) $VAR(../duplex/@)
	   touch /tmp/speed-duplex.$VAR(../@)
	fi

val_help: auto; Auto negotiation (default)
val_help: 10; 10 Mbit/sec
val_help: 100; 100 Mbit/sec
val_help: 1000; 1 Gbit/sec
val_help: 2500; 2.5 Gbit/sec
val_help: 10000; 10 Gbit/sec

We can spot a few issues with it already. First, the set of definitions is a huge directory tree where each directory represents a config node, e.g. "interfaces ethernet address" will be "interfaces/ethernet/node.tag/address/node.def". This makes them hard to navigate, you need to read a lot of small files to get the whole picture, and you need to do a lot of file/directory hopping to edit them. Now, try mass editing, or checking for mistakes before release...

Next, they use a custom syntax, which needs custom lexer and parser, and custom documentation (in practice, the source is your best bet, though I did write something of a syntax reference). No effortless parsing, no effortless analysis or transformation either.

Next, the value checking has its peculiarities. There is concept of type (it can be txt, u32, ipv4, ipv4net, ipv6, ipv6net, and macaddr), and there is "syntax:expression:" thing which partially duplicate each other. The types are hardcoded in the config backend and cannot be added without modifying it, even though they only define validation procedure and are not used in any other way. The "syntax:expression:" can be either "foo in bar baz quux", or "pattern $regex", or "exec $externalScript".

But, the "original sin" of those files is that they allow embedded shell scripts, as you can see. Mixing data with logic is rarely a good idea, and in this case it's especially annoying because a) you cannot test such code other than on a live system b) you have to read every single file in a package to get the complete picture, since any of them may have embedded shell.

Now to the new way.
]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1114667 2016-12-11T23:21:03Z 2016-12-30T01:23:32Z VyOS talk at the SDN meetup in Barcelona

Looks like it's a blogging day, with the VyOS 2.0 digest #2 on the way...

Tomorrow at 19:45 local time, Yuriy Andamasov (that's syncer on #vyos) gives a talk at the SDN and network programmability meetup in Barcelona.

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1114664 2016-12-11T23:15:56Z 2016-12-30T01:23:38Z Why services and not donations This was originally intended as a reply to a comment in the VyOS 2.0 digest #1 post, but that would be a rather long reply to post in comments I guess. This question is brought up once every few months perhaps, but I never got to publishing my own position on it.

Dave Smarts asked there:

>Where / how do we donate? Is there an obvious personal vs. corp. donation system?

Contributions and commitment to development are best donations

Myself I think the best thing individuals can do for the project is contribute to the design and the code. For 2.0, we are especially in need of commited people who can join the core team, so we can work together and make it happen (given the amount of design work and CS heavy lifting it needs). The foundation needs to be at least somewhat complete before "casual" contributors can add small improvement, in the initial stage every developer needs to know the full context.

While the jessie migration work is more friendly to occasional contributors who can add a fix or two when they find a bug, it also needs a lot of dedicated and unrewarding work on cleaning up the mess we inherited and removing the assumptions that are no longer true in jessie.

While we are at it, there's an area that needs contributions especially badly: documentation. If you've got a minute, come to wiki.vyos.net and document some feature. A stab page with some config examples is better than no page at all. If you don't know what to write about, you can use the old Vyatta Core docs as a reference (just don't copy anything verbatim from there! They are not under an open source license, that would be a copyright violation).

Donations create legal issues and paperwork

To be fair, I have no idea how donations must be handled, in any country. It's only any obvious if we start a non-profit "VyOS Foundation", even then starting a non-profit is quite an undertake that comes with loads of paperwork and expenses. Remember PDPC (the foundation set up by Freenode)? It had to be dissolved because donations to Freenode couldn't cover the expenses of running it.

Additionally, whether legally required or not, I believe anyone who receives donations is morally required to publish a complete financial report and disclose what they received and what they spent it on. Non-profits are legally required to do that, in any case people who donated have the right to know what happened to their donations.

Donations create a moral problem

I'll be using Wikipedia as a model project that relies on donations.

For them, donations are unquestionably morally correct because the primary purpose of donations is to stay online, and what it takes to stay online is obvious (servers, hosting, bandwidth...). MediaWiki software (for the most part) and the content of Wikipedia are community effort, and people aren't donating towards better content, they are donating towards keeping the servers online, and as long as they are online, their expectations are met. Note that anything they do with cash left after the hosting bills is often criticized, such as the salaries of the paid foundation executives, and their research projects. But, at least the funds spent on hosting are unquestinably spent the intended way.

Donations towards development, however, are a lot like the part Wikimedia Foundation is criticized for. Donations create expectations (justifiably!), but in this case the expectations are left implicit, and open to interpretations. What exactly constitutes spending money towards development? If the donation is exactly to the VyOS project (even assuming VyOS Foundation did exist), who is eligible to receive shares of it? Is programming the only appropriate activity, or it's right to hire a graphics designer to improve the website? Myself, breaking the expectations of people who donate is the last thing I want, but without knowing their expectations it's impossible to avoid.

The other issue is common for donations and crowdfunding. What's our responsibility if we fail to do something in time, or at all?

For a small project, donations from individuals are not financially viable

VyOS is a small project, whether we like it or not. Making overly optimistic estimates, let's say we have 10 000 users, and if every hundredth user donates $10 a year, we are left with $1000. Just enough to pay the bills of one particularly modest maintainer for a month.

Why services are better

  • For companies, authorizing service purchase is normally a lot easier than authorizing a donation
  • You get real value for your money, rather than a vague promise
  • Since the obligations of each party are well defined, expectations will not be broken accidentally

What to do

If you want to contribute to VyOS as an individual, join the development. If you want to contribute financially, try to persuade your company to buy support from Sentrium: at the very least is will help me (dmbaturin) pay my bills, and if we get enough customers, build a team of fulltime (or at least part time) maintainers.

As an alternative way to get guaranteed developer time, if your company is using VyOS and you have people who already contributed to it or want to, consider allocating some of their paid time for it, we will be more than grateful if you do.

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1113713 2016-12-08T01:49:38Z 2016-12-30T01:23:46Z VyOS 2.0 development digest #1

I keep talking about the future VyOS 2.0 and how we all should be doing it, but I guess my biggest mistake is not being public enough, and not being structured enough.

In the early days of VyOS, I used to post development updates, which no one would read or comment upon, so I gave up on it. Now that I think of it, I shouldn't have expected much as the size of the community was very small at the time, and there were hardly many people to read it in the first place, even though it was a critical time for the project, and input from the readers would have been very valuable.

Well, this is a critical time for the project too, and we need your input and your contributions more than ever, so I need to get to fixing my mistakes and try to make it easy for everyone to see what's going on and what we need help with.

Getting a steady stream of contributions is a very important goal. While the commercial support thing we are doing may let the maintainers focus on VyOS and ensure that things like security fixes and release builds get guaranteed attention in time, without occasional contributors who add things they personally need (while maintainers may not, I think myself I'm using maybe 30% of all VyOS features any often) the project will never realize its full potential, and may go stale.

But to make the project easy to manage and easy to contribute to, we need to solve multiple hard problems. It can be hard to get oneself to do things that promise no immediate returns, but if you looks at it the other way, we have a chance to build a system of our dreams together. As of 1.1.x and 1.2.x (the jessie branch), we'll figure it out how to maintain it until we solve those problems, but that's for another post. Right now we are talking about VyOS 2.0, which gets to be a cleanroom rewrite.

Why VyOS isn't as good as it could be, and can't be improved

I considered using "Why VyOS sucks" to catch reader's attention. It's a harsh word, and it may not be all that true, given that VyOS in its current state is way ahead of many other systems that don't even have system-wide config consistency checks, or revisions, or safe upgrades, but there are multiple problems that are so fundamental that they are impossible to fix without rewriting at least a very large part of the code.

I'll state the design problems that cannot be fixed in the current system. They affect both end users and contributors, sometimes indirectly, but very seriously.

Design problem #1: partial commits

You've seen it. You commit, there's an error somewhere, and one part of the config is applied, while the other isn't. Most of the time it's just a nuisance, you fix the issue and commit again, but if you, say, change interface address and firewall rule that is supposed to allow SSH to it, you can get locked out of your system.

The worst case, however, is when commit fails at boot. While it's good to have SSH at least, debugging it can be very frustrating, when something doesn't work, and you have no idea why, until you inspect the running config and see that something is simply missing (if you run into it in VyOS 1.x, do "load /config/config.boot" and commit, this will either work or show you why it failed). It's made worse by lack of notifications about config load failure for remote users, you can only see that error on the console.

The feature that can't be implemented due to it is what goes by "commit check" in JunOS. You can't test if your configuration will apply cleanly without actually commiting it.

It's because in the scripts, the logic for consistency checking and generating real configs (and sometimes applying them too) is mixed together. Regardless of the backend issues, every script needs to be taken apart and rewritten to separate that logic. We'll talk more about it later.

Design problem #2: read and write operations disparity

Config reads and writes are implemented in completely different ways. There is no easy programmatic API for modifying the config, and it's very hard to implement because binaries that do it rely on specific environment setup. Not impossible, but very hard to do right, and to maintain afterwards.

This blocks many things: network API and thus an easy to implement GUI, modifying the config script scripts in sane ways (we do have the script-template which does the trick, kinda, but it could be a lot better).

Design problem #3: internal representation

Now we are getting to really bad stuff. The running config is represented as a directory tree in tmpfs. If you find it hard to believe, browse /opt/vyatta/config/active, e.g. /opt/vyatta/config/active/system/time-zone/node.val

Config levels are directories, and node values are in node.val files. For every config session, a copy of the active directory is made, and mounted together with the original directory in union mount through UnionFS.

There are lots of reasons why it's bad:

  • It relies on behaviour of UnionFS, OverlayFS or another filesystem won't do. We are at mercy of unionfs-fuse developers now, and if they stop maintaining it (and I can see why they may, OverlayFS has many advantages over it), things will get interesting for us
  • It requires watching file ownership and permissions. Scripts that modify the config need to run as vyattacfg group, and if you forget to sg, you end up with a system where no one but you (or root) can make any new commits, until you fix it by hand or reboot
  • It keeps us from implementing role-based access control, since config permissions are tied to UNIX permissions, and we'd have to map it to POSIX ACLs or SELinux and re-create those access rules at boot since the running config dir is populated by loading the config
  • For large configs, it creates a fair amount of system calls and context switches, which may make system run slower than it could

Design problem #3: rollback mechanism

Due to certain details (mostly handling of default values), and the way config scripts work too, rollback cannot be done without reboot. Same issue once made Vyatta developers revert activate/deactivate feature.

It makes confirmed commit a lot less useful than it should be, especially in telecom where routers cannot be rebooted at random even in maintenance windows.

Implementation problem #1: untestable logic

We already discussed it a bit. The logic for reading the config, validating it, and generating application configs is mixed in most of the scripts. It may not look like a big deal, but for the maintainers and contributors it is. It's also amplified by the fact that there is not way to create and manipulate configs separately, the only way you can test anything is to build a complete image, boot it, and painstakingly test everything by hand, or have expect-like tool emulate testing it by hand.

You never know if your changes may possibly work until you get them to a live system. This allows syntax errors in command definitions and compilation errors in scripts to make it into builds, and it make it into a release more than one time when it wasn't immediately apparent and only appread with certain combination of options.

This can be improved a lot by testing components in isolation, but this requires that the code is written in appropriate way. If you write a calculator and start with add(), sub(), mul() etc. functions, and use them in a GUI form, you can test the logic on its own automatically, e.g. does add(2,3) equal 5, and does mul(9, 0) equal 0, does sqrt(-3) raise an exception and so on. But if you embed that logic in button event handlers, you are out of luck. That's how VyOS is for the most part, even if you mock the config subsystem so that config read functions return the test data, you need to redo the script so that every function does exactly one thing testable in isolation.

This is one of the reasons 1.2.0 is taking so long, without tests, or even ability to add them, we don't even know what's not working until we stumble upon it in manual testing.

Implementation problem #2: command definitions

This is a design problem too, but it's not so fundamental. Now we use custom syntax for command definitions (aka "templates"), which have tags such as help: or type: and embedded shell scripts. There are multiple problem with it. For example, it's not so easy to automatically generate at least a command reference from them, and you need a complete live system for that, since part of the templates is autogenerated. The other issue is that right now some components feature very extensive use of embedded shell, and some things are implemented in embedded shell scripts inside templates entirely, which makes testing even harder than it already is.

We could talk about upgrade mechanism too, but I guess I'll leave it for another post. Right now I'd like to talk about proposed solutions, and what's being done already, and what kind of work you can join.

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1111367 2016-11-28T13:29:29Z 2016-12-30T01:23:52Z Sentrium Cyber Monday promotion: get some free hands on assistance hours with every VyOS commercial support plan

Hi everyone,

This summer,  Sentrium, an IT consulting company ran by some of long time VyOS developers and users, launched commercial support for VyOS, something that we think will make VyOS more attractive for enterprise and service provider networks and give the VyOS project some funding and ensure its sustainable development and growth.

We also would like to say huge thanks to all of our existing customers for the commitment to the VyOS project and for your trust in us! You are contributing to the VyOS development and we are really happy to see such interest from companies around the world. Thank you!

Based on these few months of experience with VyOS support, we decided to adjust our support plans based on customer feedback You can view the new plans at this web page: https://sentrium.io/vyos-commercial-support/

We tried to cover common use cases which we observing:

  • Small companies and nonprofit organizations with budget constraints
  • Companies with internal expertise which just need a “formal” support contract due to the business requirements.
  • Businesses who need phone support and hands-on assistance with strict SLAs for mission critical applications.

So, at a glance, how our support from vendors:

  • All support plans cover all routers in the company, no matter if this production instance, or you just spin up another VyOS in your lab. You don’t need get to pay more when your network grows, or wait to get support contracts for new routers.

  • Initial configuration and environment review is included in each plan, and this allows us to suggest config improvements and also shorten resolution times (we can go directly to issue, bypassing topology discovery)

  • We only employ people with real networking knowledge and do not do anything to artificially make response time appear shorter than it is (e.g. by sending a form reply). We do not offer guaranteed short response time for all plans, but you can be sure the first reply you get includes some useful information about your problem. For email support, relatively long response time also allows us to do some research about your problem, try a solution in a lab, or consult a maintainer or a contributor, if we cannot offer a useful response offhand.

From Cyber Monday and until end of the year we’ll be running a promotion: everyone who buys a support subscription also gets free hand on assistance hours. The basic plan comes with one free hour, the standard plan comes with four, and the production support plan comes with eight.

We know there are still a number of people running Vyatta Core and who may want to switch to VyOS. Switching to a new system is always a concern, so if you are using Vyatta Core, you can use the hands on assistance hours for migration to VyOS. But with all improvements and, most importantly, security fixes that have been added, and will be added by 1.1.8 and then 1.2.0, upgrade is very important.

If you are running Vyatta Core 6.5 or 6.6, it can be upgraded to VyOS as if it was a new Vyatta version, with a few minor caveats. If your version is 6.4 or earlier, there are some features that may require manual config rewrite, which shouldn’t take too long unless your config is particularly large. We have successfully upgraded versions as old as 6.0 to VyOS without any problems. If, no matter how unlikely, you are still using VC 5.0, things get interesting, since it didn’t have the image upgrade yet, but even that is doable and has been done before (though if you can, we suggest that you reinstall and we can help with migrating the config).
]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1108935 2016-11-17T21:04:26Z 2016-11-17T21:05:20Z VyOS servers issue resolved

Hi everyone,

The hardware issue is resolved, and our servers are back to life.

Thanks to http://openitc.co.uk/ for hosting those machines, and for always being helpful and responsive when things go wrong!

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1108908 2016-11-17T19:07:34Z 2016-11-17T19:38:35Z Some VyOS servers are down

Hi everyone,

Due to a hardware issue our hoster is having, some our servers are down. What's down is the wiki.vyos.net/forum.vyos.net host and the packages.vyos.net hosts.

The new website, vyos.io, is up, it's hosted in a different place. If you need to download VyOS images, you can use any of the mirrors, e.g. 0.de.mirrors.vyos.net.

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1103618 2016-10-30T16:35:15Z 2016-10-31T09:30:15Z VyOS Project October Update

Hi everyone,

Last two weeks was pretty busy, and we attended multiple events, so I decided to share what happens with all of you.

On behalf of Sentrium,  I (Yuriy Andamasov, that is) with other Sentrium employees visited VMworld, OpenStack Summit, and a meeting held by an ISP consortium known as Guifi  in Barcelona this October.

Among other things, one of my goals was to find out if there’s any interest in VyOS in the enterprise and cloud markets. I’m excited to see that VyOS seems to be the best deal for many use cases. There are so many cool applications and technologies around, many of them require networking, other provide networking functions and/or orchestration functions for large deployments.

However, many people are interested in integration with management tools and other software, so we came to the conclusion that it’s a good idea to provide integration with VMware NSX and OpenStack Neutron at least, as they are quite popular. Of course, we need your help with it. If you are interested, or even already working on it, feel free to contact us, come to Phabricator or the IRC channel, or contact me personally by email (yuriy@sentrium.io) or in the RocketChat (I’m syncer there).

I’ve also had a great conversation with Netronome people regarding their SmartNICs and VyOS and how we can collaborate on open source solutions which get best from software and hardware worlds and hopefully this collaboration can bring us some truly impressive results.

As we realized and as strange it is to us, many people just never heard about VyOS and only some of them heard something about Vyatta Core, so we are trying to raise awareness of it, and that was one of the reasons I attended those events. We made some leaflets with information about the project, and I handed them out at all those events I attended. We’ll make a PDF and Scribus source files available soon so that you can print them for your own use if you are going to attend an event and promote VyOS there.

We also gave a talk about VyOS at the  Guifi meeting. Guifi members are building a big community network here in Spain, and we think that VyOS is a perfect match to this effort. Later this year we are going to give a more detailed technical talk, discuss the challenges they are having and see how VyOS can help them make their network better.

We hope this effort will help VyOS to acquire a wider community interested in open source networking. Remember that even if you are not a programmer, there are still lots of things you can help with, such as testing, feature design discussion, documentation, howtos, and just helping other community members on our forum or chat.

In other news, there will be a few more designs was added and another few will be available soon in our merchandise store

Now when this streak of community building events is over, we all can get back to working on the code. 

Daniil(dmbaturin) is working on the 1.1.8 maintenance release and messing around with the nextgen VyOS prototype, and 1.2.0 beta still needs your testing, so please, pick up it here and give it a try, but remember that it still not production ready.



]]>
Yuriy Andamasov
tag:blog.vyos.net,2013:Post/1095406 2016-10-02T19:10:05Z 2016-12-30T18:05:52Z Informal meeting of VyOS users in Barcelona

If you are in Barcelona this October, we have some news for you.

There is a plan to organize an informal meeting of VyOS users and all interested in open source routing, roughly at the same time as VMworld and the OpenStack Summit. 

The current idea is October the 17th at 19-00 for VMWorld week

and on 24th 19-00 on OpenStack Summit week, though it's not set in stone yet 

Who's in charge

Yuriy Andamasov (syncer on #vyos) is the initiator and the primary contact.
On behalf of Sentrium Yuriy and Santiago will take care about administrative questions.

Note: None of the maintainers planning to attend, so if you want to discuss specific issues with the code and contributing to it, it's likely not the best place to ask. 
Yuriy is one of VyOS infrastructure admins and the community manager, so feel free to discuss these issues with him. If you want to contribute to VyOS but don't know where to start, feel free to ask him and he can suggest some tasks to work on too.

What's planned

A walk around Barcelona places and have informal talks about VyOS, virtualization, networking and other IT and non-IT subjects. 
There is no plan to have a formal meeting with talks prepared in advance or anything else at this time, though if there is an interest in it, perhaps it's possible to arrange it.
The idea is for VyOS users and everyone interested in VyOS and open source routing in general, to come together and socialize over sightseeing/food/etc. in great city Barcelona

Want to participate?

Go to https://www.meetup.com/VyOS-Default-Route-Barcelona/events/234567678/, register, and leave your comments. If you want to sponsor laptop stickers, t-shirts, or anything else for the attendees, it will be appreciated. In exchange, we can put your company logo on the VyOS brochure that Yuriy will be distributing at VMworld and OpenStack Summit.
]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1091949 2016-09-20T19:35:09Z 2016-09-21T22:34:06Z VyOS virtual meeting notes - 14 September 2016

We hosted our first VyOS virtual meeting here in September and invited both developers and enthusiasts to attend. The meeting was held on September 14th at 18:00 UTC and all in all we had about 11 participants join. Yuriy Andamsov (syncer) brought this idea of a virtual meeting to fruition, thank you Yuriy!

Meeting summary:

VyOS 1.1.8 release
We discussed the general question of whether this should be a maintenance-only release or whether new features should be included. The community has readied a few new features which could easily be imported to this release. In general, past VyOS micro releases have included new features as long as they are safe, low risk changes. There was a lot of discussion about this topic mostly related to where developer time is best spent and whether making this a maintenance-only release would help justify more effort from the community to put towards v1.2. In the end we agreed that 1.1.8 will include backports of a few new features, but only where it's not a major headache or risk to do so.

Web GUI discussion
Mihail brought up the work he's been doing on a web GUI front-end for VyOS. His work can be found here:
https://github.com/mickvav/vyatta-webgui

https://github.com/mickvav/vyatta-accel-ppp

General consensus on a web GUI is that it's a nice to have, not a requirement at the moment for the project. We might look to integrate this at some point in the 1.2 future or beyond.

The move to Jessie (VyOS 1.2)

Here's where we are.  We have nightly builds for VyOS 1.2 based on Debian Jessie.  The original VyOS code base is challenging and there's no current automated testing system, so we need testers.  We agreed that one thing we need is visibility on what testing has been done so far.  If you have tested a 1.2 nightly build or would like to, please see this thread to view and get access to the testing matrix.

Jason Hendry mentioned a side project which some Mintel hackers had started on, using serverspec to automate tests of VyOS nightly builds.  On top of doing some manual testing of the nightly builds and contributing to the spreadsheet, he's going to look into getting the serverspec base pushed into CI.

Community Members Present

We had a lot of responses to the original phabricator thread.  Unfortunately not everyone could make it, and also a few people weren't able to join because we hit limits with maximum number of participants in Google Hangouts.  Next meeting we will try a different piece of technology.

  • Jason Hendry (jhendryUK)
  • Daniil Baturin (Dmbaturin)
  • Kim Hagen (UnicronNL)
  • Paul Fitzgerald
  • Michael Zimmerer (mtz4718)
  • Mihail Vasilev (mickvav)
  • Ewald van Geffen (Feedmytv)
  • Patrick van Staveren (trickv)
  • Yuriy Andamsov (syncer)
  • Bronislav Robenek (BillyTheCzech)
  • Amos Shapira
We took some meeting notes which are currently available only on Google Docs but will be centralized somewhere agreeable in the future.

Feedback & Next Meeting
If you would like to join the next meeting, please comment on Q55 in Phabricator to get yourself on the list.  Hope to see you there!
]]>
tag:blog.vyos.net,2013:Post/1090031 2016-09-14T13:15:55Z 2016-09-15T13:44:58Z CVE-2016-5696, development meeting, and other things

TCP vulnerability and its fix

There was a vulnerability discovered in the implementation of TCP in Linux (CVE-2016-5696) that is remotely exploitable and allows an attacker to impersonate a connected user. For a hotfix on a live system, you can add sudo sysctl net.ipv4.tcp_challenge_ack_limit=1000000000 to /config/scripts/vyatta-postconfig-bootup.script. The fix will be included in the 1.1.8 release.

Development meetup

Today at 18:00 UTC we will hold a development meeting, everyone is invited to join. It was discussed in phabricator (https://phabricator.vyos.net/Q41) originally, and most people voted for Google Hangouts, so that's what we'll use. I'm not fond of it myself, but so be it.

If you want to participate, fill the form here: https://goo.gl/forms/PBarp2bPWvAncQtJ2 so that we know your email address and can send you an invite.

It will be a semi-structured format, the agenda is the following:

  • 1.1.8 maintenance release and what backports to include in it
  • 1.2.0 and how to go about verifying that it actually works
  • VyOS 2.0 (the clean rewrite) design principles
  • Strategies for attracting more contributors

Rocket Chat

Lately we've setup Rocket Chat , an open source chat platform with some interesting features, including voice support, offline logs, and more. If it works well, we can use it for future development meetings, and possibly other things as well. If you want to give it a try, visit https://chat.vyos.io

 

1.2.0 beta2

People keep asking about 1.2.0-beta2. The truth is, if we build some image, and call it 1.2.0-beta2, nothing will change really. You can always get the latest nightly build from http://dev.packages.vyos.net/iso/current/amd64/ and start testing it. Maybe when it's stable enough for at least non-critical production environments, we will call it beta2, but until then, nightly builds is really a better way to go.

VyOS shirts

A bit to our surprise, there was some interest in VyOS shirts (https://teespring.com/stores/vyos), but they also created a bit of confusion as to what it costs and where the profit goes.

The base cost of the shirt around $15/EUR 10, so the "retail price" is some $5 higher than the base cost. Any profit from it will be used for things directly related to VyOS, such as domain renewals and the like (if the shirts somehow become popular enough to pay for anything else, we'll see what else we can do with it).

We need to come up with some convenient way to make the ledger public so that everyone can see what we received what it was used for.


]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1087183 2016-09-06T15:16:09Z 2017-01-11T11:15:40Z VyOS remote management library for Python

Someone on Facebook rightfully noted that lately there's been more work on the infrastructure than development. This is true, but that work on infrastructure was long overdue and we just had to do it some time. There is even more work on the infrastructure waiting to be done, though it's more directly related to development, like restructuring the package repos.

Anyway, it doesn't mean all development has stopped while we've been working on infrastructure. Today we released a Python library for managing VyOS routers remotely.

Before I get to the details, have a quick example of what using it is like:

import vymgmt

vyos = vymgmt.Router('192.0.2.1', 'vyos', password='vyos', port=22)

vyos.login()
vyos.configure()

vyos.set("protocols static route 203.0.113.0/25 next-hop 192.0.2.20")
vyos.delete("system options reboot-on-panic")
vyos.commit()

vyos.save()
vyos.exit()
vyos.logout()

If you want to give it a try, you can install it from PyPI ("pip install vymgmt"), it's compatible with both Python 2.7 and Python 3. You can read the API reference at http://vymgmt.readthedocs.io/en/latest/ or get the source code at https://github.com/vyos/python-vyos-mgmt .

Now to the details. This is not a true remote API, the library connects to VyOS over SSH and sends commands as if it was a user session. Surprisingly, one of the tricky parts was to find an SSH/expect library that can cope with VyOS shell environment well, and is compatible with both 2.7 and 3. All credit for this goes to our contributor who goes by Hochikong, who tried a whole bunch of them, settled with pexpect and wrote a prototype.

How the library is better than using pexpect directly, if it's a rather thin wrapper for it? First, it's definitely more convenient to just call set() or delete() or commit() than to format command strings yourself and take care of the sending and receiving lines.

Second, common error conditions are detected (through simple regex matching) and raise appropriate exceptions such as ConfigError (for set/delete failures) or CommitError for commit errors. There's also a special ConfigLocked exception (a subclass of CommitError) that is raised when commit fails due to another commit in progress, so you can recover from it by sleep() and retry. This may seem uncommon, but people who use VRRP transition scripts and the like on VyOS already reported that they ran into it.

Third, the library is aware of the state machine of VyOS sessions, and will not let you accidentally do wrong things such as trying to enter set/delete commands before entering the conf mode. By default it also doesn't let you exit configure sessions if there are uncommited or unsaved changes, though you can override it. If a timeout occursm an exception will be raised too (while pexpect returns False in this case).

Right now it only supports set, delete, and commit, of all high level methods. This should be enough for the start, but if you want something else, there are generic methods for running op and conf mode commands (run_op_mode_command() and run_conf_mode_command() respectively). We are not sure what people want most, so what we implement depends on your requests ans suggestions (and pull requests of course!). Other things that are planned but that aren't there yet are SSH public key auth and top level words other than set and delete (rename, copy etc.). We are not sure if commit-confirm is really friendly to programmatic access, but if you have any ideas how to handle it, share with us.

On an unrelated note, syncer and his graphics designer friend made a design for VyOS t-shirts. If anyone buys that stuff, the funds will be used for the project needs. The base cost is around 20 eur, but you can get them with 15% discount by using VYOSMGTLIB promo code: https://teespring.com/stores/vyos?source=blog&pr=VYOSMGTLIB

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1086457 2016-09-04T07:20:41Z 2016-09-06T18:08:08Z The new website is now live

Hi everyone,

The new website is now live. There are still some rough edges (typos, odd links, odd wording etc.), if you find anything like this, let us know. But overall it does what it's supposed to do, tells newcomers what VyOS is and provides quick links to downloads and other resources to existing users.

Here's an explanation of what happened exactly. Before that, vyos.net and vyos.org were pointing at the wiki host, and the wiki main page used to serve as our primary website. It means there are quite a few links like http://vyos.net/wiki/Something on the net, and simply pointing that domain at the host with the new website would create quite some link rot.

To avoid this, we've setup two conditional redirects, one redirects vyos.net/wiki/Something to wiki.vyos.net/wiki/Something, another one redirects everything else to vyos.io. So far it seems to work properly, but if you notice any issues with it, such as links that are not redirected correctly, let us know.

P.S. we added some merch(not much but we will add more soon) to our store, please check out https://teespring.com/stores/vyos

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1084746 2016-08-29T16:13:25Z 2016-08-29T18:02:06Z wiki.vyos.net now uses the new recaptcha (meaning: way easier to edit)

Hi everyone,

I like to joke about "wiki.vyos.net, a free encyclopedia no one bothers to edit", but there was a thing that actually made it rather inconvenient to edit for people, regardless of their willingness to edit it.

I mean the subnet captcha. If you haven't seen it, it made you calculate a broadcast address of a random subnet to register an account or edit a page. Everyone hated it, including me, and I had to add a number of active editors to a group that is exempt from it, by hand, so that it doesn't keep haunting them.

What's worse is that after a MediaWiki update it stopped working for a mysterious reason which I couldn't quickly debug, so I thought it may be a good time to reconsider it. Now there's the new reCAPTCHA instead, the one that wants you to click an "I'm not a robot" checkbox. For most users, it is trivial to solve, and it proved fairly effective at mitigating spam, so I think at least for now it may be the best solution for everyone.

Still, it's not entirely without controversy. There still may be accessibility converns and, since it's a third-party service, privacy concerns. If the captcha is giving you troubles, we still may make you exempt from it if you tell us your account and explain the issue.

If you, luckily for you, haven't seen the introduction of the subnet captcha, the story was that the wiki (a Vyatta wiki  back then, vyattawiki.net) had all its content replaced with spam overnight, and the original reCAPTCHA did nothing to protect it. We had been having spam problems for a long time by that moment, and after that event, we thought we have to come up with something that will stop spam forever, and so I wrote the subnet captcha plugin, and it worked very well. Except making edits inconvenient for legitimate users of course. Now when it's gone, I hope we'll be getting more editors.

I hope more people will come and edit some pages. We are seriously lacking in the documentation department, but if everyone does a small bit, it's not really that hard. You don't need to write big chunks at once and document complete features, a brief description and a config example is a lot better than nothing.

]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1082828 2016-08-21T08:29:04Z 2016-08-23T20:04:25Z vyos.net → wiki.vyos.net redirect You may have noticed (or will notice soon) that http://vyos.net redirects to http://wiki.vyos.net. Don't worry, this is normal.

It's the first step in preparation to roll out the new website. When it goes live, the vyos.net and vyos.io domains will point to the new website, and the wiki will live on its own wiki.vyos.(net|io) domain.

Meanwhile you can view the future website at http://vyos.io/ and give us your feedback, if you haven't already.


]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1080990 2016-08-13T21:17:13Z 2016-08-13T21:17:13Z dead.vyos.net and contributing to VyOS

Hi everyone,

This is a bit embarassing. For almost a day http://www.vyos.net (but not http://vyos.net) was sending people to the http://dead.vyos.net website, until someone on the IRC pointed it out (thanks for this!).

It happened due to an oddity in the way Apache HTTPD handles host aliases, and it's fixed now.

The dead.vyos.net website was created as a joke for giving the link to people who ask if VyOS project is dead. While the website is indeed a joke, the issue with lack of contributors is very real, and it does slow things down.

If you want to contribute to VyOS, there is a lot of work to be done, and the maintainers on the IRC and in phabricator will be happy to point you to beginner-friendly tasks and answer questions about the code and the patch submission process.


]]>
Daniil Baturin
tag:blog.vyos.net,2013:Post/1079388 2016-08-07T14:59:00Z 2016-10-02T18:58:20Z VyOS Project news

Hello, Community!

We have some great news to share! 

As some of you may already know, we are planning to run virtual meeting event for VyOS devs and users in near future.

So in case you want to participate, just fill up this form and of course join us on our dev. portal to stay in touch.

May admit that this summer is productive in all aspects:

Tremendous work towards VyOS 1.2 done and we going to present 1.2.0 beta 2 in some weeks! Thanks, to our super team!

We revived OVA distribution of VyOS some months ago and continued work in the direction of extensive VMWare Platform support; we also plan to deliver supported images for all other standard hypervisors like KVM, MS Hyper-V and VyOS on clouds markets for AWS, Azure, GCE.

VyOS is now available on SolidRun ClearFog device, credits for this great work to UnicronNL, this opens new applications for VyOS.

Ansible starting from version 2.2 able to configure VyOS, so if you using Ansible, give it a try! We on other hand working on standalone management library for Python.

We receive many requests regarding GUI, and yes we are listening to them. Last four months we dedicated a lot of time to studies regarding possible ways of delivery such  UI. Apart from technical challenges, there are other problems to take into account(like service providers don't want GUI, while SMBs and ROBOs show high demand). We are working to bring satisfaction to all users without sacrifices from any side. So basically yes, it will be GUI in near future, but there are not ETAs for now.  

We believe that affordable open source routing platform and NFV should be accessible to everyone out there

Want to participate???

Join us on social networks and spread the word about VyOS - Twitter, RedditFaceBook, Google+, Linkedin 

Participate in discussions on forum and of course join us on development portal 

VyOS is a community-driven Linux-based network operating system for routers and firewalls

Saludos,

Yuriy

]]>
Yuriy Andamasov