NeoRaider's blog

GSoC: The ECE configuration system - summary

The Google Summer of Code is almost over, so in this blog post I'll give a overview over the targets I've met (and those I haven't).

Code repositories

All code in the first two repositories has been developed by me during the GSoC. The third link shows the work I've done to integrate a ECE backend into libuci.

What is working

As described in earlier posts, my GSoC project was a configuration storage system for OpenWrt/LEDE, trying to solve various issues of the UCI config system. The principal points of this new system are

    • ubus-based config daemon maintaining a central storage database file
    • JSON-based configuration data model
    • Validation based on simplified JSON-Schema

The Wiki at https://gitlab.com/neoraider/ece/wikis/home gives a good overview of the design and the usage of ECE and describes many features in detail. The pkg-ece package feed can be used to build and install the different components of ECE on OpenWrt/LEDE easily.

If you've worked with OpenWrt/LEDE, you probably know the UCI config system. A UCI config file looks like this:

config system
        option hostname 'lede'
        option timezone 'UTC'

This format is very simple: Each file (called "package") has a number of sections (named or unnamed) of different types (this example from the "system" package has a single unnamed section of type "system"). These section contain options with single values or lists of values.

Unnamed sections are usually accessed using indices, for example a command to set the hostname would look like this:

uci set system.@system[0].hostname='betterhostname'

With the simplicity of UCI, there come various issues and missing features; these are only a few of them:

      • The fixed data model (package/section/option) makes some kinds of configuration very awkward: In the example above, the index 0 must be given for the system section, but having a second section of this kind would not make sense. In other cases, deeper configuration trees must be flattened to be stored in UCI, making the configuration harder to understand
      • All values in UCI are strings, which often causes inconsistencies (booleans are usually stored as '0'/'1', but several other pairs like 'false'/'true' and 'off'/'on' are supported as well; different users of UCI sometimes parse numbers differently)
      • UCI doesn't have built-in validation. Frontends like LuCI usually validate the entered data, but as soon as the CLI client is used, no validation is done.
      • UCI always stored the whole configuration file and not only changes from the defaults, making the storage inefficient on overlay-based filesystem setups as they are common on OpenWrt/LEDE
      • In some situations, upgrades to default values should also affect the effective values; but only if the user didn't change the values themselves. With UCI, this is not possible, as it doesn't store the information if a value was changed by a user.
      • UCI allows comments in config files, but they are lost as soon as libuci or the CLI tool is used to modify it

The configuration given above could be represented in ECE as this JSON document:

{
  "system": {
    "hostname": "lede",
    "timezone": "UCI"
  }
}

Note that this is only the external representation of the configuration; internally, it is stored in a more efficient binary format.

JSON gives us a lot of features for free: arbitrary configuration trees with proper data types. Existing standards and standard drafts like JSON Pointer and JSON Schema can be used to reference and validate configuration (the JSON Schema specification is simplified for ECE a bit though to allow more efficient validation on embedded systems).

The command for changing the hostname would look like this in ECE:

ece set /system/hostname '"betterhostname"'

The quoting is currently necessary to make the string a valid JSON document; this may change in a future version.

The whole configuration is saved in a single JSON document, but the specific format is not defined by a single schema; instead, each package can provide a schema, and the configuration tree is validated against a merged schema definition.

The schemas also provide default values for the configuration. Adding documentation for the configuration options to the schemas is planned as a future addition and might be used to support the user in configuration utilities and automatically generate web-based or other interfaces.

This gives only a small example for the usage of ECE, the abovementioned ECE Wiki contains much more information about the usage of ECE and the ideas behind it.

In addition to the daemon and a simple CLI utility, I've developed libraries for C, Lua and Shell which allow to access the configuration. While there are still some features missing (some points for future work are given in https://gitlab.com/neoraider/ece/wikis/todo ), I think most of the missing pieces can be added in the near future.

The UCI/ECE bridge

When I proposed my project for the GSoC, I didn't aim at making it a full replacement for the current UCI system, at least not in the near future. While the possibility to move some of UCI config files into the ECE config database had been my plan from the beginning, my ideas for backwards compatibility didn't go further than a one-time import from UCI to ECE, and one-way generation of UCI config files from ECE.

After talking to a few LEDE developers and package maintainers, it became clear to me and my mentors that many people are interested in replacing UCI with a better system in the not-too-far future. But for ECE to become this replacement, a real two-way binding between UCI and ECE would be necessary to allow gradual migration, so configuration utilities like LuCI (and many other utilities somehow interacting with UCI) don't need to be adjusted in a flag-day change.

An incomplete design draft for this UCI/ECE bridge has been outlined in https://gitlab.com/neoraider/ece/wikis/design/uci-bridge . The code found in the UCI ECE backend repository linked above implements a part of this bridge (it can load "static" and "named" bindings from ECE into UCI, and commit "static" bindings back to ECE) and has been implemented as an API- and ABI-compatible extension to libuci. The development of this bridge has taken a lot of time (much more time than I had originally scheduled for UCI compatibility features), as the data models of UCI and ECE are very different.

Future work

Of all points given in https://gitlab.com/neoraider/ece/wikis/todo , finalizing the database format is the most important, as any future change in the storage format will either break compatibility or involve some kind of conversion. When it is clear the format won't be changed anymore, ECE should be added to the OpenWrt community package repository to make it easily accessible to all OpenWrt and LEDE users.

After that, other points given in the TODO should be dealt with, but none of those seem too pressing to prevent actually using ECE for some software (but some of the points given in the first section of the TODO page would need to be addressed to properly support software that requires more complex configuration).

Last, but no least, I'd like to express my gratitude to my mentors and all people in the OpenWrt, LEDE and Freifunk communities who have helped me develop ECE by giving guidance and lots of useful feedback, and to Google, who allowed me to focus on this project throughout this summer.

GSoC: The ECE configuration system - midterm update

Hi,
the GSoC is nearing its midterm evaluation, so let's have a look at the current state of the "Experimental Configuration Environment" (working title)!

I started the project with defining models for configuration data, configuration diffs (that are used to modify data and store modifications) and schemas. All of the design documents I wrote can be found at: https://gitlab.com/neoraider/ece/wikis/design

I tried hard not to re-invent things that already exist in a more or less standardized way:

  • My configuration model is just JSON (without support for floating-point numbers right now, as the libubox blobmsg format ubus uses to represent JSON doesn't support them - I plan to fix this)
  • JSON Pointers (https://tools.ietf.org/html/rfc6901) are used to refer to elements of the configuration tree
  • The schema format uses a subset of JSON Schema (http://json-schema.org/)

Based on this design, I've started implementing the configuration management daemon eced. The page https://gitlab.com/neoraider/ece/wikis/implementation gives a good overview of the current features and usage of eced. Quite a lot is already working:

  • Multiple schemas can be loaded and merged
  • Default configuration is generated from the schemas
  • A ubus interface allows to query and modify the configuration
  • Config modifications can be loaded from and stored into a diff file

There's still a lot missing, so here's what I plan to do next:

  • Make something use ECE! UCI is well-integrated in OpenWrt/LEDE and can be used from C, Lua and shell scripts; all of these languages should also be supported by an ECE client library. An example schema to replace /etc/config/system does already exist and could be used to experiment with partially replacing UCI with ECE.
  • Support schema upgrades: When schemas are updated, this can partially invalidate config diffs. A way to deal with such inconsistencies must be defined.

Discussions and feedback have also led me to the decision to put a stronger focus to UCI interoperability. While I had only planned import and export of UCI configuration at first, I've realized that a two-way binding between UCI and ECE (i.e. an API-compatible libuci replacement/extension that will use ECE as backend) will be necessary for ECE to find acceptance. This will also allow to continue to use existing configuration tools like LuCI with UCI configuration converted to ECE.

I also plan to have OpenWrt/LEDE packages for ECE ready soon, so you can play around with the current implementation yourselves. Maybe I'll also set up a public host with ECE Laughing

GSoC: A new configuration system for OpenWrt/LEDE

Hi,
I'm Matthias (aka NeoRaider), and this year, I'll participate in the Google Summer of Code for the Freifunk project.

The goal of my project is to develop an alternative to the UCI configuration system, as UCI has a number of issues that make it cumbersome to use in some situations.

One of the basic issues of UCI that affects many Freifunk (or generally community mesh) firmwares is the upgrade behaviour. Mesh firmwares usually contain elaborate default configurations, which set up network interfaces and other things to allow participating in a mesh without deep knowledge of the setup.

But this setup needs to change from time to time, as the firmware is upgraded. In the Gluon firmware framework, we usually solve this by providing upgrade scripts which modify the configuration after flashing the firmware. Writing these script is often a tedious task, and the scripts easily break when the configuration differs too much from the expected one.

But the ability to change the configuration is important for many Freifunk users: They want to change the role of ethernet ports, WLAN settings, and a lot more. But UCI doesn't provide information how a setting was changed: if a script encounters an unexpected value, it can't find out if it is an old default value, or was changed by the user. This often leads to a difficult choice for the script author: either to overwrite the value unconditionally, maybe disregarding voluntary configuration changes, or not to overwrite it, rendering communities unable to change certain configuration during upgrades.

My project aims at solving this by saving the user configuration independently of the defaults provided by packages. This way, a package upgrade can change the default values, but explicit user configuration will not be affected.

Another issue is that the upgrade scripts are usually part of the packages that bring the configuration. Removing a package will leave the configuration behind, which is usually a good thing (for users which know about this and may be interested in the the old config), but for mostly-automatic upgrades, old configuration may accumulate, which can quickly become problematic on devices with very limited flash.

I plan to fix this by basing the new configuration system on schema definitions, which specify which configuration options and values are valid. The schema will probably be based on JSON, as there are already lots of existing systems for defining JSON schemas, which may be used for my project or at least serve as inspiration. This will also finally provide real datatypes for the configuration and make things more consistent (finally no more 1/true/on/yes/0/false/off/no booleans!). If we want too keep the package/section/option organization of UCI, or rather allow defining schemas for any JSON structure, is still a subject of debate.

Instead of going into detail even more in this post, I'll provide a link to my Gitlab project: https://gitlab.com/neoraider/ece

Design documents, examples for configuration and schemas, and first code will start to appear there very soon Laughing

Syndicate content