Updating Configurations

When configuration changes are introduced, a new mapping and configuration templates should be created (at least once per release where configuration changes were introduced).

When only adding new configuration options, the configuration file is still compatible with the previous version since all the options that existed before are still present. In this case, the configuration minor version number should be bumped.

When an option is removed or renamed, the configuration file is no longer compatible with the previous version. In this case, the major version number should be bumped.

Adding upgrade template files

For each configuration version, it should exist a corresponding directory under the templates. For example, for the configuration version 2.0, the directory templates/2.0 was added.

For each new version, the following files should be created:

A directory for the new version number should be created in the templates directory
The mapping.json file that specify the transformations from the previous configuration version to the new version should be added. See the format in Mapping file format
For each existing component, the corresponding configuration file jinja2 template should be added. For example, for the verifier, the verifier.j2 should be created in the directory for the new version
If special transformations that cannot be expressed through the simple operations in the mapping.json file, the adjust.py script can be added. See below the requirements for the adjust.py in Adjust script requirements

For example, when adding templates for a new version X.Y, the following directory tree should be added:

templates
└── X.Y
    ├── adjust.py (optional)
    ├── agent.j2
    ├── ca.j2
    ├── logging.j2
    ├── mapping.json
    ├── registrar.j2
    ├── tenant.j2
    ├── test.md
    └── verifier.j2

Mapping file format

For each configuration version, a mapping from the previous version to the new version is required. The mapping is provided as a JSON file, with the following fields:

version: The mapping version, in the MAJOR.MINOR format. The value assigned to this field should match the template directory name. For example, for the mapping in the templates/2.0 directory, the version field should be set as 2.0.
type: The mapping type. The supported values are update and full. See the requirements for each type below in Update mapping and Full mapping, respectively.
components: The components to be modified by the mapping. Depending on the mapping type, the contents of the components field are different. See the requirements for each type below in Update mapping and Full mapping.
subcomponents: Only present in the full mapping type, this field is required to associate sections to their parent components.
Full mapping: The type field in the mapping.json file should be set as full for the mapping to be processed as a full mapping. In this type of mapping, all the options for all the components are treated as replacements of options. If an option is omitted, it means the option is removed in the new configuration version.

Update mapping

For the update mapping, the type field in the mapping.json file should be set as update. The update mapping is used to create a new configuration version by applying changes to the existing options through operations and preserve all the other options. This mapping type is the recommended way to introduce small changes to the configuration files.

In the update mapping, the components field list the components that are modified from the previous version, and the operations applied to them. See the format of the components dictionary below in the Update mapping components format

The update mapping does not use or need the subcomponents field.

Update mapping components format

For the update mapping, the components field should be set as a dictionary which maps the sections to be modified to the operations to be applied to them. Only the sections that are modified need to be present in the dictionary, all the omitted components are preserved as they were in the previous version.

The supported operations to modify the sections are:

add: This operation adds a new option to the section. It should be assigned as a dictionary mapping the new option name to its default value.

For example, the following mapping file adds 2 new options to the [comp_a] section:

{
    "version": "3.1",
    "type": "update",
    "components": {
        "comp_a": {
            "add": {
                "new_option": "value",
                "new_option2": "value2"
            }
        }
    }
}

remove: This operation removes options that exists in the previous configuration version. An array of options to be removed should be assigned to the remove field for the section/component to be modified.

For example, the following mapping file removes 2 options from the [comp_a] section:
```
{
    "version": "3.1",
    "type": "update",
    "components": {
        "comp_a": {
            "remove": ["unused_option", "another_unused_option"]
        }
    }
}
```
replace: This operation replaces an option that exists in the previous configuration version with another in the new version. During the processing of the mapping file, the value found in the replaced option will be preserved, and assigned to the new option in the output. If the option is not found in the input configuration file, the default value is used instead.

The replace field should be set as a dictionary mapping the option to be replaced to the parameters for the new option. The dictionary should have the following fields:
- section: The section to which the new option should be added
- option: The new option name
- default: The default value to be used in case the old option is not found in the input configuration.
For example, the following mapping file replaces two options from the [comp_a] section:
```
{
    "version": "3.1",
    "type": "update",
    "components": {
        "comp_a": {
            "replace": {
                "old_option_to_replace": {
                    "section": "new_section",
                    "option": "new_option",
                    "default": "value"
                },
                "old_value": {
                    "section": "other_section",
                    "option": "other_new_option",
                    "default": "value"
                }
            }
        }
    }
}
```

Full mapping

In the full mapping, al the options of the new configuration version should be declared. If an option is omitted, it means the option is removed.

The format of the fields in the full mapping file are:

version: Should be in the MAJOR.MINOR format. The version should match the directory name
type: Should be set as full. If omitted, the mapping will be treated as as full mapping
components: Should be set as a dictionary which maps each component to a dictionary of options. See the option dictionary format below in Full mapping components format section.
subcomponents: Should be set as a dictionary mapping subcomponents to its main component. This is necessary to create a relationship between the sections of the files that are not components (e.g. The [revocations] section in the verifier.conf file should be declared in the subcomponents dictionary as "revocations": "verifier"). See the format below in Subcomponents format

Full mapping components format

For each component, the options transformations should be declared through dictionaries that map the new option name to the option it is replacing.

The upgrade script will search the options to be replaced in the old configuration, in the section provided in the section field. If the option is found, the value is preserved in the news configuration, otherwise the value provided in the default field is used instead.

As an example, follows an excerpt of the templates/2.0/mapping.json file:

"components": {
    "agent": {
        "version": {
            "section": "agent",
            "option": "version",
            "default": "2.0"
        },
        "revocation_notification_ip": {
            "section": "general",
            "option": "receive_revocation_ip",
            "default": "127.0.0.1"
        },
}

In the excerpt above, in the agent component two options are declared, version and revocation_notification_ip.

The new option revocation_notification_ip will receive the value from the receive_revocation_ip from the general section in the old configuration file. If the option is not found, the value 127.0.0.1 provided in the default field is used instead.

Subcomponents format

The configuration file for some of the components (e.g. the verifier.conf) have more than one section. The main section is named after the component (e.g. [verifier] section of the verifier.conf file). The other sections are considered subsections, or subcomponents, by the configuration upgrade script. The subsections are associated with their main section in the Subcomponents dictionary, which maps a subsection to the associated main section.

For example, the following excerpt from the templates/2.0/mapping.json file:

"subcomponents": {
    "revocations": "verifier",
    "loggers": "logging",
    "handlers": "logging",
    "formatters": "logging",
    "formatter_formatter": "logging",
    "logger_root": "logging",
    "handler_consoleHandler": "logging",
    "logger_keylime": "logging"
}

In the excerpt, the [revocations] section is declared as a subsection (or subcomponent) of the [verifier] section.

Adjust script requirements

The optional adjust.py script can perform complex operations that cannot be expressed using the mapping.json file. For example, deciding the value of an option depending on the presence of another option in the configuration file.

The adjust.py script is processed after the mapping.json is applied. For this reason, when writing the adjust.py script, the author should consider the input to be the output of the processing of the associated mapping.json file.

The only requirement for the adjust.py script is to implement the adjust() function, defined as the following:

def adjust(
    config: RawConfigParser, mapping: Dict, logger: Logger = logging.getLogger(__name__)
) -> None:

The config parameter is the result after applying the transformations defined by the mapping.json file.

The mapping parameter is the dictionary read from the mapping.json JSON file.

The optional logger parameter will receive the logger from the keylime_upgrade_config script, so that all the log messages are in a single place. It is recommended to keep the default assigned as logging.getLogger(__name__) so that the keylime_upgrade_config can set the log level accordingly.

The adjust() function should make the changes to the parser received through the config parameter directly.