VyOS 2.0 development digest #3: questions for you, vyconf daemon config format, appliance directory structure, and external validators lookup

Ok, I changed my mind: before we jump into the abyss of datastructures and look how the config and reference trees work, I'll describe the changes I made in the last few days.

Also, I'd like to make it clear that if you don't respond to design questions I state in these posts, I, or whoever takes up the task, will just do what they think is right. ;)

I guess I'll start with the questions this time. First, in the comments to the first post, the person who goes by kglkgvkvsd544 suggested two features: commit-confirm by default, and an alternative solution to the partial commit where instead of loading the failsafe config, the system loads a previous revision instead, in the same way as commit-confirm does. I don't think any of those should be the only available option, but having them as configurable options may be a good idea. Let me know what you think about it.

Another very important question: we need to decide on the wire protocol that vyconf daemon will use for communication with its clients (the CLI tool, the interactive shell, and the HTTP bridge). I created a task for it, https://phabricator.vyos.net/T216, let me know what you think about it. I'm inclined towards protobuf myself.

Now to the recent changes.

vyconfd config

As I already said, VyConf is supposed to run as a daemon and keep the running config tree, the reference tree, and the user sessions in memory. Obviously, it needs to know where to get that data. It also needs to know where to write logs, where the PID file and socket file should be, and other things daemons are supposed to know. Here's what the vyconf config for VyOS may look like:

name = "VyOS"

data_dir = "/usr/share/vyos/"
program_dir = "/usr/libexec/vyos"
config_dir = "/etc/vyos"

# paths relative to config_dir
primary_config = "config.boot"
fallback_config = "config.failsafe"

socket = "/var/run/vyconfd.sock"
pid_file = "/var/run/vyconfd.pid"
log_file = "/var/log/vyconfd.log"
That INI-like language is called TOML, it's pretty well specified and capable of some fancy stuff like arrays and hashes, apart from simple (string, string) pairs. The best part is that there are libraries for parsing it for many languages, and the one for OCaml is particularly nice and idiomatic (it uses lenses and option types to access a nested and possibly underdefined datastructure in a handy and typesafe way), like:
let pid_file = TomlLenses.(get conf (key "vyconf" |-- table |-- key "pid_file" |-- string)) in
match pid_file with
| Some f -> f
| None -> "/var/run/vyconfd.pid"
The config format and this example reflects some decisions. First, the directory structure if more FHS-friendly. What do we need /opt/vyos for, if FHS already has directories meant for exactly what we need: architecture-independent data (/usr/share), programs called by other programs and not directly by users (/usr/libexec), and config files (/etc)?
Second, all important parameters are configurable. The primary motivation for this is making VyConf usable for every appliance developer (most appliances will not be called VyOS for certain), but for ourselves and for every contributor to VyOS it's a reminder to avoid hardcoded paths anywhere: if changing it is just one line edit away, a hardcoded path is an especially bad idea.

Here's the complete directory structure I envision:
  interfaces/ # interface definitions
  components/ # Component definitions
  components/  # Scripts/programs that verify the appliance config, generate actual configs, and apply them
  migration/       # Migration scripts (convert appliance config if syntax changes)
  validators/       # Value validators
  scripts/              # User scripts
  post-config.d/   # Post-config hooks, like vyatta-postconfig-bootup.script
  pre-commit.d/   # Pre-commit hooks
  post-commit.d/ # Post-commit hooks
  archive/             # Commit archive
This is not an exhaustive list of directories an appliance can have of course, it's just directories that have any significance for VyConf. I'm also wondering if we should introduce post-save hooks for those who want to do something beyond the built-in commit archive functionality.

External validators

As you remember, my idea is to get rid of the inflexible system of built-in "types" and make regex the only built-in constraint type, and use external validators for everything else.

External validators will be stored in the $program_dir/validators. Since many packages use the same types of values (IP addresses is a common example), and in VyOS 1.x we already have quite a lot of templates that reference the same validation scripts, making it a separate entity will simplify reusing them, since it's easy to see what validators exist, and you can also be sure that they behave like you expect (or if they don't, it's a bug).
Validator executable take two arguments, first argument is constraint string, and the second is the value to be validated (e.g. range "1-99" "42").The expected behaviour is to return 0 if the value is valid and a non-zero exit code otherwise. Validators should not produce any output, instead the user will get the message defined in the constraintError tag of the interface definition (this approach is more flexible since different things may want to use different messages, e.g. to clarify why exactly the value is not valid).

That's all for today. Feel free to comment and ask questions. The next post really will be about the config tree and the way set commands work, stay tuned.

14 responses
Thanks for the writeup.
Regarding configurable options for commit/commit-confirm, I think that's good. commit and commit-confirm are similar enough that changing the default shouldn't be a big deal, for whatever is decided. Protobuf is good, but have you considered flatbuffers? It's got one nice improvement over Protobuf: no marshalling/unmarshalling necessary. The data structure on the wire is the same in memory. See https://stackoverflow.com/questions/25356551/wh... Post-save hooks *could* be useful, but it smells like Feature Creep. Unless there is strong demand (or any demand) and it's not too difficult to implement later, probably best to keep it simple.
>have you considered flatbuffers? I looked into it briefly, my issue with it is that if the language is not C++, it limits the user to the memory representation of C++, which may not be bad for some applications, but can be quite limiting. Protobuf schema compilers are aware of the target language and its type system and can generetate idiomatic code. The ocaml-protoc compiler can recognize oneOf as what it really is: a sum type, and produce something like "type foo = Int_val of int | String_val of string" from "message Foo { oneof foo { Int_val int = 1; String_val string = 2}}". The Python compiler makes good use of metaclasses. I don't think in our case some performance gain would make up for the loss of expressiveness, and people are not sending thousands of commands per second anyway. >Post-save hooks *could* be useful, but it smells like Feature Creep >Unless there is strong demand (or any demand)... Well, yeah, it's pretty much of a question of demand. If we are implementing pre/post-commit hooks (and I guess we are, right now in VyOS they are used internally for commit-archive, for instance), adding post-save hooks is a matter of calling the same run_hooks function in a different place with different arguments, the only question is if anyone really wants it or not. Myself I can see possible uses for it, but I don't think I would use it myself.
Does that mean "commit-confirm" as the default state out of the box, or a configuration option so that the user can set "commit-confirm" as the baseline commit style for the system? The latter would be a great option. I'd be reluctant to make it the base state, for two reasons: - First, it deviates from a de facto industry standard. My perception is that most users are coming from a networking background, and it's important to preserve the look, feel, and behavior of other routing solutions. - Second, as VyOS finds its way out of the lab, and into production, when an operator addresses a production troubleshooting issue, it'd be unfortunate if, after applying a configuration change that resolved an issue, the change reverted back to a non-working state by default. That's not what I'd call "a sane default."
I agree, commit-confirm should never be the default (and definitely shouldn't be the only available option). There's another thing, it doesn't work well with GUI and especially with remote API, so it cannot be truly global anyway. But an option like "set system config-management commit-confirm-by-default" for those who really want it to be the default for interactive sessions may not be a bad idea.
I like trying to be more FHS friendly, but would that make upgrading versions of VyOS trickier, or is it just specifying different paths to change? I like TOML and protobuf. Seen a lot of projects lately that have been using both of those.
>but would that make upgrading versions of VyOS trickier, or is it just specifying different paths to change? That's a whole different can of worms! Right now VyOS uses a rather radical solution to the upgrade problem: it doesn't really upgrade anything!.Instead, it adds a new squashfs image with all new stuff. The unpleasant side effect is that even tiny updates require reboot, and takes up quite some space too. We don't have any consensus on how it should be done yet, one of my ideas is to use NixOS as the base distro for its Nix package manager that keeps snapshot of packages and can revert the transaction if upgrade fails. There is also an option of using Nix outside of NixOS. We all need to discuss that, and come up with a specific plan. But, the paths themselves don't change much, current VyOS uses .deb packages for its stuff anyway, which makes /opt quite redundant.
> one of my ideas is to use NixOS as the base distro for its Nix package manager that keeps snapshot of packages and can revert the transaction if upgrade fails. I agree that we should stick to FHS. NixOS is cool for its own reasons, but I think we should make sure that VyConf/VyOS 2.0 is flexible enough for various base operating systems. I apologize if that's already a design goal and I'm just "preaching to the choir."
Mario, I'm not sure if I stated it anywhere in these posts, but yes, it's definitely a goal. There is definitely an altruistic part in making it standalone so that people can use it for their own appliances, but in the end it benefits us even if no one outside VyOS contribute to it, since minimizing and isolating assumption makes code a lot easier to maintain. Unlimited and unrestricted assumptions is exactly what makes migrating the current code even to newer Debian. I routinely run the automated tests in vyconf on both Linux (my Fedora desktop mainly) and FreeBSD.
That one who has never lost the access throws me the first stone. Sometimes, you dont expect to loose access because of a config you just typed. But it happen. My point is that using the two commands "apply" and "commit", you'll never loose access for more than N minutes. When you "apply", if you are able to type the "commit" command you know everything is OK (still have to test your config) and config will not revert. If you are not able to type the "commit" command, you lost the access. Because "apply" automaticaly revert, you wont lost the access for a long time. You can then correct the little issue that made you lost access and "apply" again, then commit.
kglkgvkvsd544: My point is that commit-confirm, whatever we call it, enabled by default or, worse, made the only option, would seriously violate the principle of least surprise. I also have a mixed feeling about exclusive reliance on commit-confirm as a safety mechanism: if you don't have any out of band management, you are only protected against problems that can be reliably prevented by commit-confirm, upgrade or boot problems will still leave you helpless. "set system options commit default-confirm" (or similar) would be a good compromise, by default the system behaves as it used to and as every other system does, but if you want it different, it's just one command away.
3 visitors upvoted this post.