VyOS 2.0 development digest #9: socket communication functionality, complete parser, and open tasks

Socket communication

A long-awaited (by me, anyway ;) milestone: VyConf is now capable of communicating with clients. This allows us to write a simple non-interactive client. Right now the only supported operaion is "status" (a keepalive of sorts), but the list will be growing.

I guess I should talk about the client before going into technical details of the protocol. The client will be way easier to use than what we have now. Two main problems with CLI tools from VyOS 1.x is that my_cli_bin (the command used by set/delete operations) requires a lot of environment setup, and that cli-shell-api is limited in scope. Part of the reason for this is that my_cli_bin is used in the interactive shell. Since the interactive shell of VyConf will be a standalone program rather than a bash completion hack, we are free to make the non-interactive client more idiomatic as a shell command, closer in user experience to git or s3cmd.

This is what it will look like:

SESSION=$(vycli setupSession)
vycli --session=$SESSION configure
vycli --session=$SESSION set "system host-name vyos"
vycli --session=$SESSION delete "system name-server"
vycli --session=$SESSION commit
vycli --session=$SESSION exists "service dhcp-server"
vycli --session=$SESSION commit returnValue "system host-name"
vycli --session=$SESSION --format=json show "interfaces ethernet"

As you can see, first, the top level words are subcommands, much like "git branch". Since the set of top level words is fixed anyway, this doesn't create new limitations. Second, the same client can execute both high level set/delete/commit operations and low level exists/returnValue/etc. methods. Third, the only thing it needs to operate is a session token (I'm thinking that unless it's passed in --session option, vycli should try to get it from an environment variable, but we'll see, let me know what you think about this issue). This way contributors will get an easy way to test the code even before interactive shell is complete; and when VyOS 2.0 is usable, shell scripts and people fond of working from bash rather than the domain-specific shell will have access to all system functions, without worrying about intricate environment variable setup.

The protocol

As I already said in the previous post, VyConf uses Protobuf for serialized messages. Protobuf doesn't define any framing, however, so we have to come up with something. Most popular options are delimiters and length headers. The issue with delimiters is that you have to make sure they do not appear in user input, or you risk losing a part of the message. Some programs choose to escape delimiters, other rely on unusual sequences, e.g. the backend of OPNSense uses three null bytes for it. Since Protobuf is a binary protocol, no sequence is unusual enough, so length headers look like the best option. VyConf uses 4 byte headers in network order, that are followed by a Protobuf message. This is easy enough to implement in any language, so it shouldn't be a problem when writing bindings for other languages.

The code

There is a single client library that can be used by all of the non-interactive client and the interactive shell. It will also serve as the OCaml bindings package for VyConf (Python and other languages wil need their own bindings, but with Protobuf, most of it can be autogenerated).

Parser improvements

Inactive and ephemeral nodes

The curly config parser is now complete. It supports the inactive and ephemeral properties. This is what a config with those will look like:

protocols {
  static {
    /* Inserted by a fail2ban-like script */
    #EPHEMERAL route {
    /* DIsabled by admin */
    #INACTIVE route {

While I'm not sure if there are valid use cases for it, nodes can be inactive and ephemeral at the same time. Deactivating an ephemeral node that was created by scritp perhaps? Anyway, since both are a part of the config format that the "show" command will produce, we get to support both in the parser too.

Multi nodes

By multi nodes I mean nodes that may have more than one value, such as "address" in interfaces. As you remember, I suggested and implemented a new syntax for such nodes:

interfaces {
  ethernet eth0 {
    address [;;

However, the parser now supports the original syntax too, that is:.

interfaces {
  ethernet eth0 {

I didn't intend to support it originally, but it was another edge case that prompted me to add it. For config read operations to work correctly, every path in the tree must be unique. The high level Config_tree.set function maintains this invariant, but the parser gets to use lower level primitives that do not, so if a user creates a config with duplicate nodes, e.g. by careless pasting, the config tree that the parser returns will have them too, so we get to detect such situations and do something about it. Configs with duplicate tag nodes (e.g. "ethernet eth0 { ... } ethernet eth0 { ... }") are rejected as incorrect since there is no way to recover from this. Multiple non-leaf nodes with distinct children (e.g. "system { host-name vyos; } system { name-server; }") can be merged cleanly, so I've added some code to merge them by moving children of subsequent nodes under the first on and removing the extra nodes afterwards. However, since in the raw config there is no real distinction between leaf and non-leaf nodes, so in case of leaf nodes that code would simply remove all but the first. I've extended it to also move values into the first node, which equates support for the old syntax, except node comments and inactive/ephemeral properties will be inherited from the first node. Then again, this is how the parser in VyOS 1.x behaves, so nothing is lost.

While the show command in VyOS 2.0 will always use the new syntax with curly brackets, the parser will not break the principle of least astonishment for people used to the old one. Also, if we decide to write a migration utility for converting 1.x configs to 2.0, we'll be able to reuse the parser, after adding semicolons to the old config with a simple regulat expression perhaps.


Node names and unquoted values now can contain any characters that are not reserved, that is, anything but whitespace, curly braces, square brackets, and semicolons.

What's next?

Next I'm going to work on adding low level config operations (exists/returnValue/...) and set commands so that we can do some real life tests.

There's a bunch of open tasks if you want to join the development:

T254 is about preventing nodes with reserved characters in their names early in the process, at the "set" time. There's a rather nasty bug in VyOS 1.1.7 related to this: you can pass a quoted node name with spaces to set and if there is no validation rule attached to the node, as it's with "vpn l2tp remote-access authentication local-users", the node will be created. It will fail to parse correctly after you save and reload the config. We'll fix it in 1.2.0 of course, but we also need to prevent it from ever appearing in 2.0 too.

T255 is about adding the curly config renderer. While we can use the JSON serializer for testing right now, the usual format is also just easier on the eyes, and it's a relatively simple task too.

12 responses
Hi there, I was wondering if there was a REST interface for the vyatta. Also what are the advantages to the the sessions interface to that of a REST interface?
Just a proposal: to ease the script writing it could accept input from stdin. Your example could be more readable written in bash like this (uses bash heredoc syntax): SESSION=$(vycli setupSession) vycli --session=$SESSION --read-stdin
Mmm apparently the forum deleted most of my code, I'll repost here: http://pastebin.com/Lh3fNQZ0
Juan, since in most cases when people want to send substantial amount of data, it's all uniform set commands, I think it's better to add merge option to the client that can read files, including stdin. The config parser in vyconf is a module like any other and the parse function is easy to use programmatically and get a config tree that can be converted to set commands for sending to vyconf (not like in the current vyos backend). Using exists or return value in this fashion is futile since their output woud be left inaccessible anyway.
Sean, sorry, looks like I forgot to respond to your comment! There is no remote API of any kind for the current vyos, that's one of the goals of vyconf. The proprietary vyatta did have some API, I never quite liked the approach they took though. The session interface comes from the fact that sessions are, well, stateful. The HTTP wrapper for it therefore won't be actually RESTful either. On that topic, using URIs for set/delete etc. I also think the config paths should be in the POST data rather than the URI and if anything should be in the URI it's the top level commands (/set, /delete...), because constructing URIs from "interfaces ethernet eth0 address" is outright insanity. Or you mean why HTTP is not used internally? Well, it's performance considerations rather than philosophy, since HTTP 1.x has no persistent connections, the vyos shell would have to reconnect for every command to begin with. The other reason is that only UNIX domain sockets have the simplest "single sign-on" mechanism (PEERCRED option) on Linux and FreeBSD anyway, and SSO over a network would require a lot more complex solutions such as Kerberos, or the user would essentially have to login twice, first to the system, then to the vyos shell.
Ooops, sorry for that. Didn't notice the exists and return value (just skimmed over the text, my reading comprehension failure...). But I guess you could also implement something like that by capturing the stdout of the process and assigning it with $(command ... ) Also, I don't completely understand what you mean by "merge option"; probably something like "cat file1 - file2" where stdin (the '-') is output between file1 and file 2. If that is the case it's more than enough IMO. And last thing, in what language is written the config parser? I'd be quite happy if it would be accessed easily from python. Of course, I understand it'd lowest priority and that it's got its own share of problems.
I like that you ended up being able to support both formats for Multi Nodes. I personally like the new format better, but I understand why most people don't want to upset the apple cart.
5 visitors upvoted this post.