# TOMLParams

[![TOMLParams tests](https://github.com/smartdatafoundry/tomlparams/actions/workflows/push_main.yml/badge.svg)](https://github.com/smartdatafoundry/tomlparams/actions/workflows/push_main.yml)

TOML-based parameter files made better

## Key features

 * Simple externalization of parameters in one or more TOML files.
 * Supports loading, saving, default values, parameter name checking
   and optional parameter type checking.
 * Parameters available through attribute lookup or dictionary lookup.
 * Supports hierarchical file inclusion with overriding.
 * Support for using environment variables to select parameter set (as well as API).
 * Default values can be specified in code (as Python dictionaries) or in a TOML file.
 * Full support for parameter hierarchy (using TOML Tables),
   still with attribute and dictionary lookup.
 * Support for writing consolidated parameters as TOML after hierarchical inclusion
   and resolution.
 * Can be subclassed and other attributes can be used without affecting TOML file writing.

## Installation

From PyPI:
```
python -m pip install -U tomlparams
```

Source installation from Github:
```
python -m pip install -U git+ssh://git@github.com/gofcoe/tomlparams.git
```

# Sample Usage

## The Simplest Case (Defaults Only)

A TOMLParams object is initialized with a set of default values for all parameters.
The defaults can be supplied as a Python dictionary from a TOML file, and all
values should be TOML-serializable (so no `None` values).

```python
from datetime import date, datetime
from tomlparams import TOMLParams

defaults = {
    'start_date': datetime.date(2024, 1, 1),
    'run_days': 366,
    'tolerance': 0.0001,
    'log': True,
    'locale': 'en_GB',
    'critical_event_time': datetime.datetime(2024, 7, 31, 3, 22, 22),
    'logging': {
        'format': '.csv',
        'events': ['financial', 'telecoms']
    }
}

params = TOMLParams(defaults=defaults, name='defaults')
```

The special value for `'defaults'` for name specifies that the default values
should be used and nothing should be read from a custom TOML parameters file.

Parameters are stored as attributes and can be accessed using `.` style attribute
lookup or dictionary-style lookup with `[]`:

```python
>>> params.run_days
366
>>> params['start_date']
datetime.date(2024, 1, 1)
>>> params.logging.format
'.csv'
>>> params['logging']['events']
['financial', 'telecoms']
```

## Using a TOML file to override some parameter values

If the `name` is set to anything other than `'defaults'`, that will be
used as the stem name of TOML file in which to look for override parameters.
This defaults to `base` (for `base.toml`). The directory in which the system
looks for this TOML file can be set with `standard_params_dir`, and if not specified
will default to `~/tomlparams` (i.e. `tomlparams` in the user's home directory).

So if `base.toml` exists in the current working directory, and contains

```toml
start_date = 2024-03-03

[logging]
format = '.json'
```

Then we will have the following (using the same `defaults` dict as before)
```python
>>> params = TOMLParams(defaults=defaults, name='defaults')
>>> params.run_days
366
>>> params['start_date']
datetime.date(2024, 3, 3)
>>> params.logging.format
'.json'
>>> params['logging']['events']
['financial', 'telecoms']

>>> params
TOMLParams(
    start_date: datetime.date(2024, 3, 3),
    run_days: 366,
    tolerance: 0.0001,
    logging: ParamsGroup(
    	format: '.json',
    	events: ['financial', 'telecoms']
    ),
    locale: 'en_GB',
    critical_event_time: datetime.datetime(2024, 7, 31, 3, 22, 22)
)
```

Notice how the two values in `base.toml` have overridden the defaults
(`start_date` and `logging.format`), but that other parameters,
including `logging.events`, retained their values.

# Setting defaults from a TOML File

We can set the defaults from a TOML file instead of a Python dictionary by
simply giving the path to the TOML file as the value for `defaults` when initializing
`TOMLParams`. If it's an absolute path, it will be read, but if it's a relative path,
the system will look in the `standard_params_dir`.

This TOML file:
```toml
start_date = 2024-01-01
run_days = 366
tolerance = 0.0001
locale = "en_GB"
critical_event_time = 2024-07-31 03:22:22

[logging]
format = ".csv"
events = [
    "financial",
    "telecoms",
]
```
is equivalent to the dict used above.

and if stored in `./defaults.toml` can be used with
```python
>>> params = TOMLParams(
     defaults='defaults',
     standard_params_dir='.'
)

```

## Setting a custom TOML file name and Parameter (Key) Checking

Only parameters in `defaults` are allowed to exist in other TOML parameter files.
an exception is raised if any unexpected values are found in the TOML file.

For example, if `newparams.toml` is:

```toml
new_param = 'this will go badly'
```

and we repeated the use the same defaults, passing `newparams` as the `name`
we get the following error:
```
>>> params = TOMLParams(
     defaults='defaults',
     name='newparam',
     standard_params_dir='.'
)
*** ERROR: The following issues were found:
 Bad key at root level - key: new_param
```

# Hierarchical Inclusion

A special key, `include` may be used as the first line of a TOML file,
and may be set to either single string name
(the stem name of a TOML file to include),
or a list of such names.

```toml
include = ['one', 'two']
```

Inclusions are processed left-to-right, before the rest of the values in the
file with the `include` statement, but each include is only processed once,
the first time it is encountered, with newer values always overriding old ones.

So if `base.toml` (or other named TOML file used)
starts with this `include` line, and neither of `one` and `two`
has any further inclusions, the order of parameter setting will be:

  1. values from defaults
  2. values from `one.toml`
  3. values from `two.toml`
  4. any other values

If `one` and `two` both start with

```toml
include = 'three'
```

then the order of processing will be:

  1. values from defaults
  2. values from `three.toml` (from its inclusion by `one.toml`)
  3. values from `one.toml`
  4. values from `two.toml` (`three.toml` will is *not* included a second time)
  5. any other values

Circular inclusion does not cause a problem, because each file is only
ever included once, but is potentially confusing for the reader, so is
not recommended.

Unless `verbose` is set to `False` when initializing `TOMLParams`, any
TOML files processed are reported, in order of inclusion, listing full
paths.


So running this code (available in the source repo
as `examples/readme/readme4.py`):
```python
from tomlparams import TOMLParams

params = TOMLParams(
     defaults='defaults2',
     name='hier',
     standard_params_dir='.'
)

print(repr(params))
```
we see the following output:
```none
$ python readme4.py
Parameters set from: /home/sdf/tomlparams/examples/readme/three.toml
Parameters set from: /home/sdf/tomlparams/examples/readme/one.toml
Parameters set from: /home/sdf/tomlparams/examples/readme/two.toml
Parameters set from: /home/sdf/tomlparams/examples/readme/hier.toml
TOMLParams(
    a='hier',
    b='two',
    c='one',
    d='three',
    e='default',
    group=ParamsGroup(
        a='group hier',
        b='group two',
        c='group one',
        d='group three',
        e='group default',
        subgroup=ParamsGroup(
            a='subgroup hier',
            b='subgroup two',
            c='subgroup one',
            d='subgroup three',
            e='subgroup default'
        )
    )
)
```
where (in order of parameter setting):
`defaults2.toml` (used to specify defaults) is:
```toml
# defaults2.toml

a = 'default'
b = 'default'
c = 'default'
d = 'default'
e = 'default'

[group]

a = 'group default'
b = 'group default'
c = 'group default'
d = 'group default'
e = 'group default'

[group.subgroup]

a = 'subgroup default'
b = 'subgroup default'
c = 'subgroup default'
d = 'subgroup default'
e = 'subgroup default'
```
These settings are overridden first by `three.toml`:
```toml
# three.toml

a = 'three'
b = 'three'
c = 'three'
d = 'three'

[group]

a = 'group three'
b = 'group three'
c = 'group three'
d = 'group three'

[group.subgroup]

a = 'subgroup three'
b = 'subgroup three'
c = 'subgroup three'
d = 'subgroup three'
```

These are next overridden by `one.toml`:

```toml
# one.toml

include = 'three'
a = 'one'
b = 'one'
c = 'one'


[group]

a = 'group one'
b = 'group one'
c = 'group one'

[group.subgroup]

a = 'subgroup one'
b = 'subgroup one'
c = 'subgroup one'
```

In turn, these are overridden by `two.toml`:

```toml
# two.toml

include = 'three'
a = 'two'
b = 'two'

[group]

a = 'group two'
b = 'group two'

[group.subgroup]

a = 'subgroup two'
b = 'subgroup two'
```

Finally, the parameters set in `hier.toml` over-ride all others:

```toml
# hier.toml

include = ['one', 'two']

a = 'hier'

[group]

a = 'group hier'

[group.subgroup]

a = 'subgroup hier'
```

## Writing out the consolidated TOML file

Hierarchical file inclusion is powerful, and allows different sets of
parameters to be combined easily, but it can be hard for the reader to
know what the final set of parameters used is without thinking through
the inclusion hierarchy.  TOMLParams can write out a consolidated
parameter values finally used as a single TOML file containing them
all, without inclusions. The method `write_consolidated_toml` on
a `TOMLParams` object will do this:

```python
param.write_consolidated_toml('consolidated_params.toml')
```

# Determination of Name of TOML Parameters File

When a `TOMLParams` object is instantiated, the second
argument---`name`---determines which set of parameters is used.
Many options for setting this are supported. In determining which
parameters file to use, there are two relevant directories:

 * the _standard parameters directory_ can be set by passing in
   a path as `standard_params_dir`. If this is not specified,
   explicitly, `~/{params_name}` will be used if the `params_name`
   argument has been set on instantiating TOMLParams,
   or `~/tomlparams` otherwise.
 * the _user parameters directory_ can be set by passing in a
   path as `user_params_dir`. If this is not specified,
   explicitly, `~/user{params_name}` will be used if the `params_name`
   argument has been set on instantiating TOMLParams,
   or `~/usertomlparams` otherwise.

1. If `name` is set to `foo`, TOMLParams will search for `foo.toml`
   in both the standard parameters directory and the user parameters
   directory. If it is found in either it will be used. If it is not
   found at all, an exception will be raised. If it is found in both
   places, an exception will also be raised.

   (This behaviour is designed to allow code to supply standard
   parameters, possibly in a read-only directory and for users
   also to have their own parameters in another directory, while
   not allowing either to take priority over the other, so that
   which parameters are used is unambiguous.)

2. If `name` is set to either of the special values `default` or
   `defaults`, the default values will be used for all parameters,
   whether those were supplied as a Python dictionary or in a
   TOML file (specified by the `defaults` argument).

3. If `name` is not supplied, or is passed as `None`, TOMLParams
   will look the value in an environment variable. That variable
   defaults to `TOMLPARAMS`, but can be set by setting the argument
   `env_var` when instantiating a `TOMLParams` object.
   If this environment variable is set to `foo`, `foo.toml` will
   be searched for in both parameters directories exactly as if it
   had been provided as the value of `name`.

   THe environment variable is particularly convenient when several
   programs need to share a set of parameters. Most commonly, the
   environment variable will be specified on the command line before
   the Python command, e.g.

       TOMLPARAMS=bar python simulate.py

   if the Python program using `TOMLParams` is `simulate.py`.
   Alternatively, a value may be  export a shell start-up file,
   or in a script, using `export TOMLPARAMS=bar`.

4. If `name` is not supplied, and the relevant environment variable
   is not set, the value of the argument `base_params_dir` will be
   used; this defaults to `base`.


## Idiomatic command-line use

If you want to use `base.toml` as the standard parameters, but allow
people to list the parameters file (stem) on the command line for a program
`simulate.py`, with `~/simparams` as the standard parameters directory,
and defaults being read from `defaults.toml` in that directory,
the usual procedure would be to use code like this:

```python
import sys
from tomlparams import TOMLParams

class Simulate:
    def __init__(params, ...)


if __name__ == '__main__:
    name = sys.argv[1] if len(sys.argv) > 1 else 'base'
    params = TOMLParams('defaults.toml', name, params_name='simparams')
    sim = Simulate(params, ...)

```


# Environment Variables

As noted above, there are two environment variables that can affect
the behaviour of TOMLParams.

 * If the `name` argument is not passed to`TOMLParams` (or is passed as `None`)
   the environment variable `TOMLPARAMS` will be consulted (by default).
   The name of the environment variable to consult can be changed by
   passing a different value for `env_var`.

 * Any value for the variable `check_types` can be overridden by setting
   the environment variable `TOMLPARAMSCHECKING` to `warn`, `error` or `off`.
   A variable other than `TOMLPARAMSCHECKING` can be specified using
   the argument `type_check_env_var` when initializing `TOMLParams`.


# Type Checking

Type checking is performed against the types of any values present in
the passed-in `defaults`. Three levels of action are configurable:

* `WARN` (default) — a warning is sent to `stderr` but processing continues)
* `ERROR` — processing stops. All type-checking and key-checking errors are collected
   and reported before exit
* `OFF` — no type checking. (Useful when using polymorphic parameters.)

Type checking of any collections (e.g. `list`, `set` and `tuple`)
present in `defaults` is also performed and the selected action taken
if a type appears in a TOML array that is not present in the
corresponding collection in the defaults.

Action on a type checking mismatch can be configured in two ways:

* Via the environment variable specified by the
  `type_check_env_var` setting (defaults to `TOMLPARAMSCHECKING`)
  with allowed levels `warn`, `error`, and `off`.
* Via the `check_types` setting.

The environment variable takes precedence over the setting where set.

# Command line tool tomlparams

As part of installing the TOMLParams library, a command line tool
called tomlparams is made available. This provides a few functions,
the most important of which is the ability to copy usage examples
to your current working directory.

Running the command by itself prints usage information:

```bash
$ tomlparams
TOMLParams

USAGE:
    tomlparams help      --- show this message
    tomlparams version   --- report version number
    tomlparams examples  --- copy the examples to ./tomlparams_examples

Documentation: https://tomlparams.readthedocs.io/
Source code:   https://github.com/smartdatafoundry.com/tomlparams
Website:       https://tomlparams.com

Installation:

    python -m pip install -U tomlparams

```

The examples can be copied as follows:

```
$ pwd
/home/sdf/tomlparams

$ tomlparams examples
Examples copied to /home/sdf/tomlparams/tomlparams_examples.
```

The `readme` subdirectory of the resulting `tomlparams_examples`
contains all the examples from this `README.md` file.

# Documentation and API Help

The documentation for `tomlparams` is available at
[tomlparams.readthedocs.io](https://tomlparams.readthedocs.io).
More detailed help on all the arguments available when initializing
`TOMLParams` is available from the documentation,
and also from the help system within Python using:

```python
>>> import tomlparams
>>> help(tomlparams.TOMLParams)
```