Base-Configuration-Modifications

The base-configuration-modifications nomenclature is at the core of customizing instrumentation in obsinfo.

Classes using base-configure-modification:

• Stage (in obsinfo/instrumentation/stage.py)

• Datalogger (in obsinfo/instrumentation/instrument_component.py)

• Sensor (in obsinfo/instrumentation/instrument_component.py)

• Preamplifier (in obsinfo/instrumentation/instrument_component.py)

• Instrumentation (in obsinfo/instrumentation/instrumentation.py)

• Location (in obsinfo/helper_classes/location.py)

• Timing changes (not sure it’s enabled yet!)

YAML structure:

{element}:
    base:
        {element}_property1
        {element}_property2
        {element}_property3
        ...
        configuration_default: <str>
        configuration_definitions:
            {CONFIG_NAME1}:
                (configuration_description): <str>
                {element}_propertyN
                ...
            {CONFIG_NAME2}:
                (configuration_description): <str>
                {element}_propertyM
                ...
            ...
    configuration: <str>
    modifications:
        {element}_propertyY
        ...
        {element_specific_modifier1}
        ...
    *channel_modifications:
        <CH-IDENTIFIER>:
            *base*: <file reference>
            *configuration: <str>
            modifications:
                {element}_property
                ...
        <CH-IDENTIFIER>:
            ...
        ...
    **stage_modifications:
        <STAGE-NUMBER-CODE>:
            *base*: <file reference>
            *configuration: <str>
            {element}_property
            ...
        <STAGE-NUMBER-CODE>:
            ...
        ...

‘*’

channel_modifications only exist in instrumentation elements

‘**’

stage_modifications only exist in instrument_component and instrumentation:channel_modifications elements`

ORDER OF PRIORITY

stage_modifications > channel_modifications > modifications > configuration > base

Multi-level priorities

instrumentation elements contain instrument_component elements, which contain stage elemetns. Each of these can have configurations and modifications. The order of priority is

I THINK THE ORDER SHOULD BE:

instrumentation_level_declaration > instrument_component_level_declaration > stage_level_declaration

The highest-level configuration is chosen, then all of the modifications are evaluated, from highest to lowest level.

This means that a modification introduced at a lower level will override a higher-level configuration. We do this so that the high-level user gets out what they put in, but a consequence is that unseen lower-level modifications can override what the user expected from his-her configuration.

WE RECOMMEND AGAINST USING MODIFICATIONS AT THE LOWER LEVELS, UNLESS IT IS ABSOLUTELY SOMETHING THAT SHOULD IMPLEMENTED FOR THE GIVEN ELEMENT.

Specification in schemas

Every element that uses the base-config nomenclature has the following element declarations in it’s JSON schema file:

Name	properties	required
{element}	base configuration modifications notes	base
base	{properties} configuration_default configurations	specified properties
modifications	base configuration {properties}	none
configurations_map	map of configuration names (=> configuration_definition)	NA
configuration_definition	{properties} configuration_description	none

There is also a base_properties element that lists all of the properties in a base element. Originally this was used with allOf to avoid repetition, but allOf validation errors are impossible to read so we now explicitly state properties in each element. In each of the other elements, I separate the base_properties from the element-specific properties by a blank line, for clarity. The base_properties element is now just a reference.

Implementation in the code

When a class has a base-configuration-modification nomenclature, calls to ObsMetadata.get_super() are replaced by calls to ObsMetadata.base_configured_element(). The latter evaluates the base: configuration: modification: structure and replaces values as appropriate before handing off to ObsMetadata.get_super()

base_configured_element() should, in order:

1. Check if the configuration has been updated

2. return the given configuration

3. Apply local (base-config) changes to the configuration

4. Apply higher-level (channel-mods?) changes to the configuration

Here is an explanation of the philosophy, the codes involved and the potential bugs.

Handling base-configuration-modification

Most of the modifications are handled by the ObsMetadata class defined in obsmetadata.py. I’ll start by outlining what is done in obsmetadata.py before going on to specific implementations in the element classes.

obsmetadata.py

get_super()

Essentially a “super” dict.get(), adding the possibility to override the returned value by one in modifs_list dicts. With safe_update() and get_configured_modified_base(), I don’t think I need it anymore

get_configured_modified_base()

def get_configured_modified_base(self, higher_modifs={}):
    """
    Return a fully configured and modified base_dict

    Values in higher-modifs outrank those in self.  Modifications outrank
    configurations.  Uses safe_update() to only change specified elements.

    Args:
        self (:class:`ObsMetadata`): base-configuration-modification
            dictionary.  Must have "base", can have "configuration" and
            "modification" AND NOTHING ELSE.
        higher_modifs (dict or :class:`ObsMetadata`): modifications
            dictionary.  Can have "base", "configuration" and/or
            "modification" AND NOTHING ELSE
    Returns:
        base_dict (:class:`ObsMetadata): fully configured and modified
            attribute dictionary
    Raises:
        ValueError: if self or higher_modifs contain keys other than "base",
            "configuration" and/or "modification"
    """

safe_update()

Simplifies combining base elements and their modifications.

def safe_update(self, update_dict, allow_overwrite=True):
    """
    Update that only changes explicitly specfied fields

    Drills recursively through dicts inside the dict, only changing fields
    which are specified in update_dict

    Args:
        update_dict (dict or :class:`ObsMetadata`): dictionary containing
            fields to update
        allow_overwrite (bool): allow a field that was originally a dict
            to be overwritten by a field that is not a dict
    ...

Files/classes involved

If possible, only involve the classes that directly have the base-channel-modification structure:

• locations.py: Locations class

• instrumentation.py: Instrumentation class

• instrument_component.py: InstrumentComponent superclass and Datalogger, Preamplifier and Sensor subclasses

• stage.py: Stage class

• processing.py?: Processing class? or Timing class? (not yet done)

For Locations and Timing the implementation should be fairly easy because at one level. We write here the philosophy/implementation for the ``Instrumentation`` -> channel -> Instrument -> ``Instrument_Component`` -> Stages -> ``Stage`` -> Filter chain:

Instrumentation class

1. input attributes dict is split into base_dict, modifications, channel_modifications and the shortcut serial_number

2. The shortcut is inserted into modifications

3. modifications is split into ic_modifs (keys = datalogger, sensor and preamplifier) and modifications (the rest).

4. if modifications['base'] exists, replace base_dict.

5. if modifications['configuration'] exists, set base_dict["configuration"]

6. Safe_update base_dict with given configuration

7. Safe_update result with modifications

8. Create equipment attribute.

9. Create channels attribute in a loop for each channel:

   1. Get channel_specific attributes from the updated base_dict.

   2. Extract channel_modifications corresponding to the given channel

   3. Split the selected channel_modifications into InstrumentComponent-related and other

   4. Safe_update the channel_specific attributes with the non-ic channel-specific modifications.

   5. Safe_update ic_modifs with ic-related channel modifications.

   6. Pass attributes and ic_modifs down to Channel()

Channel class

1. Combine attributes and channel_default into new_attributes_dict

2. Create several attributes

3. Create instrument attribute (Instrument class), passing down new_attributes_dict and ic_modifications

Instrument class

1. Loop through ic_types: datalogger, sensor, preamplifier

   1. Pass attributes_dict[ic_type] and ic_modifications[ic_type] to InstrumentComponent.construct(attributes_dict, modifs, ic_type)

2. Combine the response stages from the 3 ic_types

3. Calculate overall sensitivity

InstrumentComponent class

base-configuration-modification module

1. Split attributes_dict into

   1. creates ic_base_dict, ic_modifs, and ic_response_modifs from attributes_dict[ic_type]

   2. creates higher_modifs from modifs[ic_type], then higher_base, higher_config and serial_number (shortcut) from higher_modifs

2. Creates instrument attribute as an Instrument, passing down new_attributes_dict, ic_modifications and channel_modifications

Handling channel_modifications and stage_modifications

channel_modifications and stage_modifications are handled in the Channel and Stage classes, respectively. These classes update the modifications dictionary with the qualifying dictionaries.

Here is a plot of how dictionaries are passed through the classes, followed by extracts of the actual codes:

subnetwork/subnetwork.py:

def __init__(self, attributes_dict=None, station_only=False):
    # ...
    self.stations = Stations(attributes_dict.get("stations", None),
                             station_only, self.stations_operators)

subnetwork/station.py:

Passes channel_modifications down to Instrumentation

def __init__(self, code, attributes_dict, station_only=False,
             stations_operators=None):
# ...
    instr_dict = attributes_dict.get('instrumentation', None)
    channel_modifs = attributes_dict.get('channel_modifications', {})
    if instr_dict:
        self.instrumentation = Instrumentation(
            instr_dict, self.locations, start_date, end_date,
            channel_modifs, self.serial_number)

instrumentation/instrumentation.py:

Passes channel_modifications down to Channel

def __init__(self, attributes_dict_or_list, locations,
             start_date, end_date, channel_modifs={},
             serial_number=None):
    # ...
    self.channels = [Channel(label, attributes, locations,
                             start_date, end_date,
                             self.equipment.obspy_equipment,
                             channel_default, channel_modifs)
                     for label, attributes in das_channels.items()]

where das_channels comes from instr_dict['channels'] and channel_default comes from das_channels['default']

channel.py:

Selects the channel modifications to pass down to Instrument

Initializing a Channel class calls

def __init__(self, label, attributes, locations,
             start_date, end_date, equipment, channel_default={},
             channel_modifs={}):
    # ...
    selected_channel_modifs = self.get_selected_channel_modifs(
        self.channel_id_code, channel_modifs)
    self.instrument = Instrument(self.das_channel, selected_channel_modifs)
    # ...

and Channel.get_selected_channel_modifs() is:

def get_selected_channel_modifs(self, id_code, channel_modifs):
    """Select a channel_modification by id_code and channel label."""
    # Get general default
    default_channel_mod = channel_modifs.get("*", {})
    if not default_channel_mod:
        default_channel_mod = channel_modifs.get("*-*", {})
    # Get defaults by location and orientation
    default_channel_loc = channel_modifs.get(
        self.orientation_code + "-*", {})
    default_channel_orient = channel_modifs.get(
        "*-" + self.location_code, {})

    # Get modifications for this particular channel
    chmod = channel_modifs.get(id_code, {})
    if not chmod:
        # If id code not found, try with just the orientation part
        chmod = channel_modifs.get(id_code[0:1], {})

    # Gather all modifications in a single channel_modifs
    # Priority order: particular mods > orientation-specific
    #                 > location-specific > general default
    for k, v in default_channel_loc.items():
        if k not in chmod:
            chmod[k] = v
    for k, v in default_channel_orient.items():
        if k not in chmod:
            chmod[k] = v
    for k, v in default_channel_mod.items():
        if k not in chmod:
            chmod[k] = v
    return chmod

Modify this to take ``modifications`` as well?

instrument.py:

def __init__(self, attributes, channel_modifs={}):
    # ...
    for ic_name in ('datalogger', 'sensor', 'preamplifier'):
        key = ic_name + '_configuration'
        config_selector = attributes.get_configured_element(key, channel_modifs)
        ic_obj = InstrumentComponent.dynamic_class_constructor(
            component, attributes_dict, channel_modifs, config_selector)
        setattr(self, ic_name, ic_obj)  # equivalent to self.ic_name = ic_obj
    # ...

which I changed to

for ic_type in ('datalogger', 'sensor', 'preamplifier'):
    ic_config_key = ic_type + '_configuration'
    if ic_type in channel_modifs:
        ic_modifs = channel_modifs[ic_type]
        # Pop out keywords
        config = ic_modifs.pop('configuration', None)
        sn = ic_modifs.pop('serial_number', None)
        base = ic_modifs.pop('base', None)
        # replace ic by channel_modifs[ic_type][]'base'] if it exists
        if base is not None:
            logger.info('Replacing {ic_type}')
            attributes[ic_type] = base
        if sn is not None:
            if 'equipment' in ic_modifs:
                if 'serial_number' in ic_modifs['equipment']:
                    logger.warning('equipment:serial_number and serial_number specified, equipment:serial_number overrides')
                else:
                    ic_modifs['equipment']['serial_number'] = sn
            else:
                ic_modifs['equipment'] = {'serial_number': sn}
        if config is not None:
            # For now, just replace v0.110 "*_configuration" keyword
            if ic_config_key in attributes:
                msg = 'attributes[{}]={} replaced by {{"{}": {{"configuration": {}}}}}'.format(
                    ic_config_key, attributes[ic_config_key], ic_type, config)
                warnings.warn(msg)
                logger.warning(msg)
            attributes[ic_config_key] = config
    config_selector = attributes.get_configured_element(ic_config_key,
                                                        channel_modifs)
    ic_obj = InstrumentComponent.dynamic_class_constructor(
        ic_type, attributes, channel_modifs, config_selector)
    setattr(self, ic_type, ic_obj)  # equivalent to self.ic_type = ic_obj

in order to handle configurations, serial_numbers and the base` element: For each of the instrument_component s found in the dictionary, it

• defines a config_key (“datalogger_configuration”, for example)

• checks if the instrument_component is named in the channel_modifs dict, if so:

  • pops out keywords (‘configuration’, ‘serial_number’ and ‘base’) (shouldn’t need to pop any more, now that modifications are separated)

  • if there is a ‘base’ keyword, replace attributes[instrument_component]['base'] by this one

  • if there is a serial_number` keyword, sets channel_modif[instrument_component]['equipment']['serial_number']

  • if there is a configuration` keyword, set attributes[config_key] to the given value

• uses ObsMetadata.get_configured_element() to choose the configuration between channel_modifs[config_key] and attributes[config_key]

• creates the instrument component using InstrumentComponent.dynamic_class_constructor(ic_type, attributes, channel_modifs, config_selector)

where channel_modifs is the selected_channel_modifs in Channel.__init()__

But I think it can be simplified now

The handling of the configuration names looks confused to me.

config_selector won’t need ic_name + 'configuration' in v0.111, as the congfiguration will use the same keyword (configuration) for each InstrumentComponent field.

instrument_component.py:

InstrumentComponent.dynamic_class_constructor(ic_type, attributes, channel_modifs, config_selector) selects the appropriate component from the attributes dict and passes it on to the specific component’s dynamic_class_constructor method:

We use this static method rather than __init__()`` in order to directly create and pass back one of the subclasses (Sensor, Datalogger or Preamplifier)

@staticmethod
def dynamic_class_constructor(component_type, attributes_dict,
                              channel_modif={}, config_selector=''):
    # ...
    selected_config = InstrumentComponent.retrieve_configuration(
        component_type, attributes_dict[component_type], config_selector)

    if component_type == 'datalogger':
        theclass = Datalogger
    elif component_type == 'sensor':
        theclass = Sensor
    elif component_type == 'preamplifier':
        theclass = Preamplifier
    # ...
    obj = theclass.dynamic_class_constructor(
        ObsMetadata(attributes_dict[component_type]),
        channel_modif.get(component_type, {}),
        selected_config)
    return obj

Here is the meat of the dynamic_class_constructor for each component (Datalogger, Sensor, or Preamplifier) class

def dynamic_class_constructor(cls, attributes_dict, channel_modif={},
                              selected_config={}):

    # ...
    stages_list = attributes_dict.get_configured_element(
        'stages', {}, selected_config, None)
    config_description = attributes_dict.get_configured_element(
        'configuration_description', channel_modif, selected_config, '')

    stages = Stages(stages_list,
                    channel_modif.get('stage_modifications', {}),
                    selected_config.get('stage_modifications', {}),
                    None)

    obj = cls(Equipment(ObsMetadata(attributes_dict.get('equipment', None)),
                        channel_modif.get('equipment', {}),
                        selected_config.get('equipment', {})),
              stages,
              config_description)

    return obj

stages.py:

Passes stage_modifications (now in channel_modif and selected_config down to Stage

def __init__(self, attribute_list, channel_modif={}, selected_config={},
             correction=None, ext_config_name=None):
    # ...
        self.stages = []
        for s, i in zip(attribute_list, range(0, len(attribute_list))):
            # Assign correction value
            if correction is None:
                correction = None
            elif i == len(attribute_list)-1:
                correction = correction
            else:
                correction = 0
            self.stages.append(Stage(ObsMetadata(s),
                                     channel_modif,
                                     selected_config,
                                     correction,
                                     i+1,
                                     ext_config_name))
            # ...

stage.py:

Handles the stage_modifications, passing on any values that match the stage_sequence_number:

def __init__(self, attributes_dict, channel_modif_list={},
             selected_config={}, correction=None,
             sequence_number=-1, ext_config_name=None):
    stage_modif = self.get_stage_modifications(
        channel_modif_list, str(sequence_number - 1))
    self.configuration = od.base_get_configuration_name(ext_config_name)
    kwargs = {'channel_modification': stage_modif,
              'selected_configuration': selected_config,
              'ext_config_name': ext_config_name}
    name = od.base_configured_element('name', default='', **kwargs)

    # ...