Base-Configuration-Modifications

The base-configuration-modifications nomenclature is at the core of customizing instrumentation in obsinfo.

Classes using base-configure-modification:

Stage (in obsinfo/instrumentation/stage.py)
Datalogger (in obsinfo/instrumentation/instrument_component.py)
Sensor (in obsinfo/instrumentation/instrument_component.py)
Preamplifier (in obsinfo/instrumentation/instrument_component.py)
Instrumentation (in obsinfo/instrumentation/instrumentation.py)
Location (in obsinfo/helper_classes/location.py)
Timing changes (not sure it’s enabled yet!)

YAML structure:

{element}:
    base:
        {element}_property1
        {element}_property2
        {element}_property3
        ...
        configuration_default: <str>
        configuration_definitions:
            {CONFIG_NAME1}:
                (configuration_description): <str>
                {element}_propertyN
                ...
            {CONFIG_NAME2}:
                (configuration_description): <str>
                {element}_propertyM
                ...
            ...
    configuration: <str>
    modifications:
        {element}_propertyY
        ...
        {element_specific_modifier1}
        ...
    *channel_modifications:
        <CH-IDENTIFIER>:
            *base*: <file reference>
            *configuration: <str>
            modifications:
                {element}_property
                ...
        <CH-IDENTIFIER>:
            ...
        ...
    **stage_modifications:
        <STAGE-NUMBER-CODE>:
            *base*: <file reference>
            *configuration: <str>
            {element}_property
            ...
        <STAGE-NUMBER-CODE>:
            ...
        ...

‘*’: channel_modifications only exist in instrumentation elements
‘**’: stage_modifications only exist in instrument_component and instrumentation:channel_modifications elements`

ORDER OF PRIORITY

stage_modifications > channel_modifications > modifications > configuration > base

Multi-level priorities

instrumentation elements contain instrument_component elements, which contain stage elemetns. Each of these can have configurations and modifications. The order of priority is

I THINK THE ORDER SHOULD BE:

instrumentation_level_declaration > instrument_component_level_declaration > stage_level_declaration

The highest-level configuration is chosen, then all of the modifications are evaluated, from highest to lowest level.

This means that a modification introduced at a lower level will override a higher-level configuration. We do this so that the high-level user gets out what they put in, but a consequence is that unseen lower-level modifications can override what the user expected from his-her configuration.

WE RECOMMEND AGAINST USING MODIFICATIONS AT THE LOWER LEVELS, UNLESS IT IS ABSOLUTELY SOMETHING THAT SHOULD IMPLEMENTED FOR THE GIVEN ELEMENT.

Specification in schemas

Every element that uses the base-config nomenclature has the following element declarations in it’s JSON schema file:

Name	properties	required
`{element}`	`base` `configuration` `modifications` `notes`	`base`
`base`	{properties} `configuration_default` `configurations`	specified properties
`modifications`	`base` `configuration` {properties}	none
`configurations_map`	map of configuration names (=> `configuration_definition`)	NA
`configuration_definition`	{properties} `configuration_description`	none

There is also a base_properties element that lists all of the properties in a base element. Originally this was used with allOf to avoid repetition, but allOf validation errors are impossible to read so we now explicitly state properties in each element. In each of the other elements, I separate the base_properties from the element-specific properties by a blank line, for clarity. The base_properties element is now just a reference.

Implementation in the code

When a class has a base-configuration-modification nomenclature, calls to ObsMetaData.get_super() are replaced by calls to ObsMetaData.base_configured_element(). The latter evaluates the base: configuration: modification: structure and replaces values as appropriate before handing off to ObsMetaData.get_super()

base_configured_element() should, in order:

Check if the configuration has been updated
return the given configuration
Apply local (base-config) changes to the configuration
Apply higher-level (channel-mods?) changes to the configuration

Here is an explanation of the philosophy, the codes involved and the potential bugs.

Most of the modifications are handled by the ObsMetaData class defined in obsmetadata.py. I’ll start by outlining what is done in obsmetadata.py before going on to specific implementations in the element classes.

`obsmetadata.py`

`get_super()`

Essentially a “super” dict.get(), adding the possibility to override the returned value by one in modifs_list dicts. With safe_update() and get_configured_modified_base(), I don’t think I need it anymore

`get_configured_modified_base()`

def get_configured_modified_base(self, higher_modifs={}):
“”” Return a fully configured and modified base_dict

Values in higher-modifs outrank those in self. Modifications outrank configurations. Uses safe_update() to only change specified elements.

Args:

self (ObsMetadata): base-configuration-modification
dictionary. Must have “base”, can have “configuration” and “modification” AND NOTHING ELSE.

higher_modifs (dict or ObsMetadata): modifications
dictionary. Can have “base”, “configuration” and/or “modification” AND NOTHING ELSE

Returns:

base_dict (:class:`ObsMetadata): fully configured and modified
attribute dictionary

Raises:

ValueError: if self or higher_modifs contain keys other than “base”,
“configuration” and/or “modification”

“””

`safe_update()`

Simplifies combining base elements and their modifications.

def safe_update(self, update_dict, allow_overwrite=True):
    """
    Update that only changes explicitly specfied fields

    Drills recursively through dicts inside the dict, only changing fields
    which are specified in update_dict

    Args:
        update_dict (dict or :class:`ObsMetadata`): dictionary containing
            fields to update
        allow_overwrite (bool): allow a field that was originally a dict
            to be overwritten by a field that is not a dict
    ...

Files/classes involved

If possible, only involve the classes that directly have the base-channel-modification structure:

locations.py: Locations class

instrumentation.py: Instrumentation class

instrument_component.py: InstrumentComponent superclass and Datalogger, Preamplifier and Sensor subclasses

stage.py: Stage class

processing.py?: Processing class? or Timing class? (not yet done)

For Locations and Timing the implementation should be fairly easy because at one level. We write here the philosophy/implementation for the ``Instrumentation`` -> channel -> Instrument -> ``Instrument_Component`` -> Stages -> ``Stage`` -> Filter chain:

`Instrumentation` class

input attributes dict is split into base_dict, modifications, channel_modifications and the shortcut serial_number
The shortcut is inserted into modifications
modifications is split into ic_modifs (keys = datalogger, sensor and preamplifier) and modifications (the rest).
if modifications['base'] exists, replace ``base_dict`.
if modifications['configuration'] exists, set base_dict["configuration"]
Safe_update base_dict with given configuration
Safe_update result with modifications
Create equipment attribute.
Create channels attribute in a loop for each channel:
1. Get channel_specific attributes from the updated base_dict.
2. Extract channel_modifications corresponding to the given channel
3. Split the selected channel_modifications` into InstrumentComponent-related and other
1. Safe_update the channel_specific attributes with the non-ic channel-specific modifications
e. Safe_update ic_modifs with ic-related channel modifications. g. Pass attributes and ic_modifs down to Channel()

`Channel` class

Combine attributes and channel_default into new_attributes_dict

2. Create several attributes 2. Create instrument attribute (Instrument class), passing down

new_attributes_dict and ic_modifications

`Instrument` class

Loop through ic_types: datalogger, sensor, preamplifier
1. Pass attributes_dict[ic_type] and ic_modifications[ic_type] to InstrumentComponent.construct(attributes_dict, modifs, ic_type)
Combine the response stages from the 3 ic_types
Calculate overall sensitivity

`InstrumentComponent` class

base-configuration-modification module

Split attributes_dict into
1. creates ic_base_dict, ic_modifs, and ic_response_modifs from attributes_dict[ic_type]
2. creates higher_modifs from modifs[ic_type], then higher_base, higher_config and serial_number (shortcut) from higher_modifs
Creates instrument attribute as an Instrument, passing down new_attributes_dict, ic_modifications and channel_modifications

Handling `channel_modifications` and `stage_modifications`

channel_modifications and stage_modifications are handled in the Channel and Stage classes, respectively. These classes update the modifications dictionary with the qualifying dictionaries.

Here is a plot of how dictionaries are passed through the classes, followed by extracts of the actual codes:

subnetwork/subnetwork.py:

def __init__(self, attributes_dict=None, station_only=False):
# … self.stations = Stations(attributes_dict.get(“stations”, None),

station_only, self.stations_operators)

`subnetwork/station.py`:

Passes channel_modifications down to Instrumentation

def __init__(self, code, attributes_dict, station_only=False,
             stations_operators=None):
# ...
    instr_dict = attributes_dict.get('instrumentation', None)
    channel_modifs = attributes_dict.get('channel_modifications', {})
    if instr_dict:
        self.instrumentation = Instrumentation(
            instr_dict, self.locations, start_date, end_date,
            channel_modifs, self.serial_number)

`instrumentation/instrumentation.py`:

Passes channel_modifications down to Channel

def __init__(self, attributes_dict_or_list, locations,
             start_date, end_date, channel_modifs={},
             serial_number=None):
    # ...
    self.channels = [Channel(label, attributes, locations,
                             start_date, end_date,
                             self.equipment.obspy_equipment,
                             channel_default, channel_modifs)
                     for label, attributes in das_channels.items()]

where das_channels comes from instr_dict['channels'] and channel_default comes from das_channels['default']

`channel.py`:

Selects the channel modifications to pass down to Instrument

Initializing a Channel class calls

def __init__(self, label, attributes, locations,
             start_date, end_date, equipment, channel_default={},
             channel_modifs={}):
    # ...
    selected_channel_modifs = self.get_selected_channel_modifs(
        self.channel_id_code, channel_modifs)
    self.instrument = Instrument(self.das_channel, selected_channel_modifs)
    # ...

and Channel.get_selected_channel_modifs() is:

Modify this to take ``modifications`` as well?

`instrument.py`:

def __init__(self, attributes, channel_modifs={}):
    # ...
    for ic_name in ('datalogger', 'sensor', 'preamplifier'):
        key = ic_name + '_configuration'
        config_selector = attributes.get_configured_element(key, channel_modifs)
        ic_obj = InstrumentComponent.dynamic_class_constructor(
            component, attributes_dict, channel_modifs, config_selector)
        setattr(self, ic_name, ic_obj)  # equivalent to self.ic_name = ic_obj
    # ...

which I changed to

for ic_type in ('datalogger', 'sensor', 'preamplifier'):
    ic_config_key = ic_type + '_configuration'
    if ic_type in channel_modifs:
        ic_modifs = channel_modifs[ic_type]
        # Pop out keywords
        config = ic_modifs.pop('configuration', None)
        sn = ic_modifs.pop('serial_number', None)
        base = ic_modifs.pop('base', None)
        # replace ic by channel_modifs[ic_type][]'base'] if it exists
        if base is not None:
            logger.info('Replacing {ic_type}')
            attributes[ic_type] = base
        if sn is not None:
            if 'equipment' in ic_modifs:
                if 'serial_number' in ic_modifs['equipment']:
                    logger.warning('equipment:serial_number and serial_number specified, equipment:serial_number overrides')
                else:
                    ic_modifs['equipment']['serial_number'] = sn
            else:
                ic_modifs['equipment'] = {'serial_number': sn}
        if config is not None:
            # For now, just replace v0.110 "*_configuration" keyword
            if ic_config_key in attributes:
                msg = 'attributes[{}]={} replaced by {{"{}": {{"configuration": {}}}}}'.format(
                    ic_config_key, attributes[ic_config_key], ic_type, config)
                warnings.warn(msg)
                logger.warning(msg)
            attributes[ic_config_key] = config
    config_selector = attributes.get_configured_element(ic_config_key,
                                                        channel_modifs)
    ic_obj = InstrumentComponent.dynamic_class_constructor(
        ic_type, attributes, channel_modifs, config_selector)
    setattr(self, ic_type, ic_obj)  # equivalent to self.ic_type = ic_obj

in order to handle configurations, serial_numbers and the base` element: For each of the instrument_component s found in the dictionary, it

defines a config_key (“datalogger_configuration”, for example)

checks if the instrument_component is named in the channel_modifs dict, if so: - pops out keywords (‘configuration’, ‘serial_number’ and ‘base’)

(shouldn’t need to pop any more, now that modifications are separated)

if there is a ‘base’ keyword, replace attributes[instrument_component][‘base’] by this one

if there is a ‘serial_number’ keyword, sets channel_modif[instrument_component][‘equipment’][‘serial_number’]

if there is a ‘configuration’ keyword, set attributes[config_key] to the given value

uses ObsMetadata.get_configured_element() to choose the configuration between channel_modifs[config_key] and attributes[config_key]

creates the instrument component using InstrumentComponent.dynamic_class_constructor(ic_type, attributes, channel_modifs, config_selector)

where channel_modifs is the selected_channel_modifs in Channel.__init()__

But I think it can be simplified now

The handling of the configuration names looks confused to me.

config_selector won’t need ic_name + 'configuration' in v0.111, as the congfiguration will use the same keyword (configuration) for each InstrumentComponent field.

`instrument_component.py`:

InstrumentComponent.dynamic_class_constructor(ic_type, attributes, channel_modifs, config_selector) selects the appropriate component from the attributes dict and passes it on to the specific component’s dynamic_class_constructor method:

We use this static method rather than __init__()`` in order to directly create and pass back one of the subclasses (Sensor, Datalogger or Preamplifier)

@staticmethod
def dynamic_class_constructor(component_type, attributes_dict,
                              channel_modif={}, config_selector=''):
    # ...
    selected_config = InstrumentComponent.retrieve_configuration(
        component_type, attributes_dict[component_type], config_selector)

    if component_type == 'datalogger':
        theclass = Datalogger
    elif component_type == 'sensor':
        theclass = Sensor
    elif component_type == 'preamplifier':
        theclass = Preamplifier
    # ...
    obj = theclass.dynamic_class_constructor(
        ObsMetadata(attributes_dict[component_type]),
        channel_modif.get(component_type, {}),
        selected_config)
    return obj

Here is the meat of the dynamic_class_constructor for each compoent (Datalogger, Sensor, or Preamplifier) class

def dynamic_class_constructor(cls, attributes_dict, channel_modif={},
                              selected_config={}):

    # ...
    stages_list = attributes_dict.get_configured_element(
        'stages', {}, selected_config, None)
    config_description = attributes_dict.get_configured_element(
        'configuration_description', channel_modif, selected_config, '')

    stages = Stages(stages_list,
                    channel_modif.get('stage_modifications', {}),
                    selected_config.get('stage_modifications', {}),
                    None)

    obj = cls(Equipment(ObsMetadata(attributes_dict.get('equipment', None)),
                        channel_modif.get('equipment', {}),
                        selected_config.get('equipment', {})),
              stages,
              config_description)

    return obj

`stages.py`:

Passes stage_modifications (now in channel_modif and selected_config down to Stage

def __init__(self, attribute_list, channel_modif={}, selected_config={},
             correction=None, ext_config_name=None):
    # ...
        self.stages = []
        for s, i in zip(attribute_list, range(0, len(attribute_list))):
            # Assign correction value
            if correction is None:
                correction = None
            elif i == len(attribute_list)-1:
                correction = correction
            else:
                correction = 0
            self.stages.append(Stage(ObsMetadata(s),
                                     channel_modif,
                                     selected_config,
                                     correction,
                                     i+1,
                                     ext_config_name))
            # ...

`stage.py`:

Handles the stage_modifications, passing on any values that match the stage_sequence_number:

def __init__(self, attributes_dict, channel_modif_list={},
             selected_config={}, correction=None,
             sequence_number=-1, ext_config_name=None):
    stage_modif = self.get_stage_modifications(
        channel_modif_list, str(sequence_number - 1))
    self.configuration = od.base_get_configuration_name(ext_config_name)
    kwargs = {'channel_modification': stage_modif,
              'selected_configuration': selected_config,
              'ext_config_name': ext_config_name}
    name = od.base_configured_element('name', default='', **kwargs)

    # ...

Base-Configuration-Modifications

Classes using base-configure-modification:

YAML structure:

ORDER OF PRIORITY

Multi-level priorities

Specification in schemas

Implementation in the code

obsmetadata.py

get_super()

get_configured_modified_base()

safe_update()

Files/classes involved

Instrumentation class

Channel class

Instrument class

InstrumentComponent class

Handling channel_modifications and stage_modifications

subnetwork/station.py:

instrumentation/instrumentation.py:

channel.py:

instrument.py:

instrument_component.py:

stages.py:

stage.py: