1. TOMLParams

TOMLParams tests

TOML-based parameter files made better

1.1. Key features

  • Simple externalization of parameters in one or more TOML files.

  • Supports loading, saving, default values, parameter name checking and optional parameter type checking.

  • Parameters available through attribute lookup or dictionary lookup.

  • Supports hierarchical file inclusion with overriding.

  • Support for using environment variables to select parameter set (as well as API).

  • Default values can be specified in code (as Python dictionaries) or in a TOML file.

  • Full support for parameter hierarchy (using TOML Tables), still with attribute and dictionary lookup.

  • Support for writing consolidated parameters as TOML after hierarchical inclusion and resolution.

  • Can be subclassed and other attributes can be used without affecting TOML file writing.

1.2. Installation

From PyPI:

python -m pip install -U tomlparams

Source installation from Github:

python -m pip install -U git+ssh://git@github.com/gofcoe/tomlparams.git

2. Sample Usage

2.1. The Simplest Case (Defaults Only)

A TOMLParams object is initialized with a set of default values for all parameters. The defaults can be supplied as a Python dictionary from a TOML file, and all values should be TOML-serializable (so no None values).

from datetime import date, datetime
from tomlparams import TOMLParams

defaults = {
    'start_date': datetime.date(2024, 1, 1),
    'run_days': 366,
    'tolerance': 0.0001,
    'log': True,
    'locale': 'en_GB',
    'critical_event_time': datetime.datetime(2024, 7, 31, 3, 22, 22),
    'logging': {
        'format': '.csv',
        'events': ['financial', 'telecoms']
    }
}

params = TOMLParams(defaults=defaults, name='defaults')

The special value for 'defaults' for name specifies that the default values should be used and nothing should be read from a custom TOML parameters file.

Parameters are stored as attributes and can be accessed using . style attribute lookup or dictionary-style lookup with []:

>>> params.run_days
366
>>> params['start_date']
datetime.date(2024, 1, 1)
>>> params.logging.format
'.csv'
>>> params['logging']['events']
['financial', 'telecoms']

2.2. Using a TOML file to override some parameter values

If the name is set to anything other than 'defaults', that will be used as the stem name of TOML file in which to look for override parameters. This defaults to base (for base.toml). The directory in which the system looks for this TOML file can be set with standard_params_dir, and if not specified will default to ~/tomlparams (i.e. tomlparams in the user’s home directory).

So if base.toml exists in the current working directory, and contains

start_date = 2024-03-03

[logging]
format = '.json'

Then we will have the following (using the same defaults dict as before)

>>> params = TOMLParams(defaults=defaults, name='defaults')
>>> params.run_days
366
>>> params['start_date']
datetime.date(2024, 3, 3)
>>> params.logging.format
'.json'
>>> params['logging']['events']
['financial', 'telecoms']

>>> params
TOMLParams(
    start_date: datetime.date(2024, 3, 3),
    run_days: 366,
    tolerance: 0.0001,
    logging: ParamsGroup(
    	format: '.json',
    	events: ['financial', 'telecoms']
    ),
    locale: 'en_GB',
    critical_event_time: datetime.datetime(2024, 7, 31, 3, 22, 22)
)

Notice how the two values in base.toml have overridden the defaults (start_date and logging.format), but that other parameters, including logging.events, retained their values.

3. Setting defaults from a TOML File

We can set the defaults from a TOML file instead of a Python dictionary by simply giving the path to the TOML file as the value for defaults when initializing TOMLParams. If it’s an absolute path, it will be read, but if it’s a relative path, the system will look in the standard_params_dir.

This TOML file:

start_date = 2024-01-01
run_days = 366
tolerance = 0.0001
locale = "en_GB"
critical_event_time = 2024-07-31 03:22:22

[logging]
format = ".csv"
events = [
    "financial",
    "telecoms",
]

is equivalent to the dict used above.

and if stored in ./defaults.toml can be used with

>>> params = TOMLParams(
     defaults='defaults',
     standard_params_dir='.'
)

3.1. Setting a custom TOML file name and Parameter (Key) Checking

Only parameters in defaults are allowed to exist in other TOML parameter files. an exception is raised if any unexpected values are found in the TOML file.

For example, if newparams.toml is:

new_param = 'this will go badly'

and we repeated the use the same defaults, passing newparams as the name we get the following error:

>>> params = TOMLParams(
     defaults='defaults',
     name='newparam',
     standard_params_dir='.'
)
*** ERROR: The following issues were found:
 Bad key at root level - key: new_param

4. Hierarchical Inclusion

A special key, include may be used as the first line of a TOML file, and may be set to either single string name (the stem name of a TOML file to include), or a list of such names.

include = ['one', 'two']

Inclusions are processed left-to-right, before the rest of the values in the file with the include statement, but each include is only processed once, the first time it is encountered, with newer values always overriding old ones.

So if base.toml (or other named TOML file used) starts with this include line, and neither of one and two has any further inclusions, the order of parameter setting will be:

  1. values from defaults

  2. values from one.toml

  3. values from two.toml

  4. any other values

If one and two both start with

include = 'three'

then the order of processing will be:

  1. values from defaults

  2. values from three.toml (from its inclusion by one.toml)

  3. values from one.toml

  4. values from two.toml (three.toml will is not included a second time)

  5. any other values

Circular inclusion does not cause a problem, because each file is only ever included once, but is potentially confusing for the reader, so is not recommended.

Unless verbose is set to False when initializing TOMLParams, any TOML files processed are reported, in order of inclusion, listing full paths.

So running this code (available in the source repo as examples/readme/readme4.py):

from tomlparams import TOMLParams

params = TOMLParams(
     defaults='defaults2',
     name='hier',
     standard_params_dir='.'
)

print(repr(params))

we see the following output:

$ python readme4.py
Parameters set from: /home/sdf/tomlparams/examples/readme/three.toml
Parameters set from: /home/sdf/tomlparams/examples/readme/one.toml
Parameters set from: /home/sdf/tomlparams/examples/readme/two.toml
Parameters set from: /home/sdf/tomlparams/examples/readme/hier.toml
TOMLParams(
    a='hier',
    b='two',
    c='one',
    d='three',
    e='default',
    group=ParamsGroup(
        a='group hier',
        b='group two',
        c='group one',
        d='group three',
        e='group default',
        subgroup=ParamsGroup(
            a='subgroup hier',
            b='subgroup two',
            c='subgroup one',
            d='subgroup three',
            e='subgroup default'
        )
    )
)

where (in order of parameter setting): defaults2.toml (used to specify defaults) is:

# defaults2.toml

a = 'default'
b = 'default'
c = 'default'
d = 'default'
e = 'default'

[group]

a = 'group default'
b = 'group default'
c = 'group default'
d = 'group default'
e = 'group default'

[group.subgroup]

a = 'subgroup default'
b = 'subgroup default'
c = 'subgroup default'
d = 'subgroup default'
e = 'subgroup default'

These settings are overridden first by three.toml:

# three.toml

a = 'three'
b = 'three'
c = 'three'
d = 'three'

[group]

a = 'group three'
b = 'group three'
c = 'group three'
d = 'group three'

[group.subgroup]

a = 'subgroup three'
b = 'subgroup three'
c = 'subgroup three'
d = 'subgroup three'

These are next overridden by one.toml:

# one.toml

include = 'three'
a = 'one'
b = 'one'
c = 'one'


[group]

a = 'group one'
b = 'group one'
c = 'group one'

[group.subgroup]

a = 'subgroup one'
b = 'subgroup one'
c = 'subgroup one'

In turn, these are overridden by two.toml:

# two.toml

include = 'three'
a = 'two'
b = 'two'

[group]

a = 'group two'
b = 'group two'

[group.subgroup]

a = 'subgroup two'
b = 'subgroup two'

Finally, the parameters set in hier.toml over-ride all others:

# hier.toml

include = ['one', 'two']

a = 'hier'

[group]

a = 'group hier'

[group.subgroup]

a = 'subgroup hier'

4.1. Writing out the consolidated TOML file

Hierarchical file inclusion is powerful, and allows different sets of parameters to be combined easily, but it can be hard for the reader to know what the final set of parameters used is without thinking through the inclusion hierarchy. TOMLParams can write out a consolidated parameter values finally used as a single TOML file containing them all, without inclusions. The method write_consolidated_toml on a TOMLParams object will do this:

param.write_consolidated_toml('consolidated_params.toml')

5. Determination of Name of TOML Parameters File

When a TOMLParams object is instantiated, the second argument—name—determines which set of parameters is used. Many options for setting this are supported. In determining which parameters file to use, there are two relevant directories:

  • the standard parameters directory can be set by passing in a path as standard_params_dir. If this is not specified, explicitly, ~/{params_name} will be used if the params_name argument has been set on instantiating TOMLParams, or ~/tomlparams otherwise.

  • the user parameters directory can be set by passing in a path as user_params_dir. If this is not specified, explicitly, ~/user{params_name} will be used if the params_name argument has been set on instantiating TOMLParams, or ~/usertomlparams otherwise.

  1. If name is set to foo, TOMLParams will search for foo.toml in both the standard parameters directory and the user parameters directory. If it is found in either it will be used. If it is not found at all, an exception will be raised. If it is found in both places, an exception will also be raised.

    (This behaviour is designed to allow code to supply standard parameters, possibly in a read-only directory and for users also to have their own parameters in another directory, while not allowing either to take priority over the other, so that which parameters are used is unambiguous.)

  2. If name is set to either of the special values default or defaults, the default values will be used for all parameters, whether those were supplied as a Python dictionary or in a TOML file (specified by the defaults argument).

  3. If name is not supplied, or is passed as None, TOMLParams will look the value in an environment variable. That variable defaults to TOMLPARAMS, but can be set by setting the argument env_var when instantiating a TOMLParams object. If this environment variable is set to foo, foo.toml will be searched for in both parameters directories exactly as if it had been provided as the value of name.

    THe environment variable is particularly convenient when several programs need to share a set of parameters. Most commonly, the environment variable will be specified on the command line before the Python command, e.g.

    TOMLPARAMS=bar python simulate.py

    if the Python program using TOMLParams is simulate.py. Alternatively, a value may be export a shell start-up file, or in a script, using export TOMLPARAMS=bar.

  4. If name is not supplied, and the relevant environment variable is not set, the value of the argument base_params_dir will be used; this defaults to base.

5.1. Idiomatic command-line use

If you want to use base.toml as the standard parameters, but allow people to list the parameters file (stem) on the command line for a program simulate.py, with ~/simparams as the standard parameters directory, and defaults being read from defaults.toml in that directory, the usual procedure would be to use code like this:

import sys
from tomlparams import TOMLParams

class Simulate:
    def __init__(params, ...)


if __name__ == '__main__:
    name = sys.argv[1] if len(sys.argv) > 1 else 'base'
    params = TOMLParams('defaults.toml', name, params_name='simparams')
    sim = Simulate(params, ...)

6. Environment Variables

As noted above, there are two environment variables that can affect the behaviour of TOMLParams.

  • If the name argument is not passed toTOMLParams (or is passed as None) the environment variable TOMLPARAMS will be consulted (by default). The name of the environment variable to consult can be changed by passing a different value for env_var.

  • Any value for the variable check_types can be overridden by setting the environment variable TOMLPARAMSCHECKING to warn, error or off. A variable other than TOMLPARAMSCHECKING can be specified using the argument type_check_env_var when initializing TOMLParams.

7. Type Checking

Type checking is performed against the types of any values present in the passed-in defaults. Three levels of action are configurable:

  • WARN (default) — a warning is sent to stderr but processing continues)

  • ERROR — processing stops. All type-checking and key-checking errors are collected and reported before exit

  • OFF — no type checking. (Useful when using polymorphic parameters.)

Type checking of any collections (e.g. list, set and tuple) present in defaults is also performed and the selected action taken if a type appears in a TOML array that is not present in the corresponding collection in the defaults.

Action on a type checking mismatch can be configured in two ways:

  • Via the environment variable specified by the type_check_env_var setting (defaults to TOMLPARAMSCHECKING) with allowed levels warn, error, and off.

  • Via the check_types setting.

The environment variable takes precedence over the setting where set.

8. Command line tool tomlparams

As part of installing the TOMLParams library, a command line tool called tomlparams is made available. This provides a few functions, the most important of which is the ability to copy usage examples to your current working directory.

Running the command by itself prints usage information:

$ tomlparams
TOMLParams

USAGE:
    tomlparams help      --- show this message
    tomlparams version   --- report version number
    tomlparams examples  --- copy the examples to ./tomlparams_examples

Documentation: https://tomlparams.readthedocs.io/
Source code:   https://github.com/smartdatafoundry.com/tomlparams
Website:       https://tomlparams.com

Installation:

    python -m pip install -U tomlparams

The examples can be copied as follows:

$ pwd
/home/sdf/tomlparams

$ tomlparams examples
Examples copied to /home/sdf/tomlparams/tomlparams_examples.

The readme subdirectory of the resulting tomlparams_examples contains all the examples from this README.md file.

9. Documentation and API Help

The documentation for tomlparams is available at tomlparams.readthedocs.io. More detailed help on all the arguments available when initializing TOMLParams is available from the documentation, and also from the help system within Python using:

>>> import tomlparams
>>> help(tomlparams.TOMLParams)