Step

class jwst.stpipe.Step(name=None, parent=None, config_file=None, _validate_kwds=True, **kws)

Bases: object

Create a Step instance.

Parameters:

• name (str, optional) – The name of the Step instance. Used in logging messages and in cache filenames. If not provided, one will be generated based on the class name.
• parent (Step instance, optional) – The parent step of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.
• config_file (str path, optional) – The path to the config file that this step was initialized with. Use to determine relative path names.
• **kws (dict) – Additional parameters to set. These will be set as member variables on the new Step instance.

Attributes Summary

input_dir
make_output_path	Return function that creates the output path
prefetch_references
reference_file_types
spec

Methods Summary

__call__(*args)	Run handles the generic setup and teardown that happens with the running of each step.
call(*args, **kwargs)	Creates and runs a new instance of the class.
closeout([to_close, to_del])	Close out step processing
default_output_file([input_file])	Create a default filename based on the input name
default_suffix()	Return a default suffix based on the step
from_cmdline(args)	Create a step from a configuration file.
from_config_file(config_file[, parent, name])	Create a step from a configuration file.
from_config_section(config[, parent, name, …])	Create a step from a configuration file fragment.
get_ref_override(reference_file_type)	Determine and return any override for reference_file_type.
get_reference_file(input_file, …)	Get a reference file from CRDS.
load_as_level2_asn(obj)	Load object as an association
load_as_level3_asn(obj)	Load object as an association
load_spec_file([preserve_comments])
make_input_path(file_path)	Create an input path for a given file path
merge_config(config, config_file)
open_model(obj)	Open a datamodel
print_configspec([stream])
process(*args)	This is where real work happens.
reference_uri_to_cache_path(reference_uri)	Convert an abstract CRDS reference URI to an absolute file path in the CRDS cache.
resolve_file_name(file_name)	Resolve a file name expressed relative to this Step’s configuration file.
run(*args)	Run handles the generic setup and teardown that happens with the running of each step.
save_model(model[, suffix, idx, …])	Saves the given model using the step/pipeline’s naming scheme
search_attr(attribute[, default, parent_first])	Return first non-None attribute in step heirarchy
set_primary_input(obj[, exclusive])	Sets the name of the master input file and input directory.

Attributes Documentation

input_dir

make_output_path

Return function that creates the output path

prefetch_references = True

reference_file_types = []

spec = "\n pre_hooks = string_list(default=list())\n post_hooks = string_list(default=list())\n output_file = output_file(default=None) # File to save output to.\n output_dir = string(default=None) # Directory path for output files\n output_ext = string(default='.fits') # Default type of output\n output_use_model = boolean(default=False) # When saving use `DataModel.meta.filename`\n output_use_index = boolean(default=True) # Append index.\n save_results = boolean(default=False) # Force save results\n skip = boolean(default=False) # Skip this step\n suffix = string(default=None) # Default suffix for output files\n search_output_file = boolean(default=True) # Use outputfile define in parent step\n input_dir = string(default=None) # Input directory\n "

Methods Documentation

__call__(*args)

Run handles the generic setup and teardown that happens with the running of each step. The real work that is unique to each step type is done in the process method.

classmethod call(*args, **kwargs)

Creates and runs a new instance of the class.

To set configuration parameters, pass a config_file path or keyword arguments. Keyword arguments override those in the specified config_file.

Any positional *args will be passed along to the step’s process method.

Note: this method creates a new instance of Step with the given config_file if supplied, plus any extra *args and **kwargs. If you create an instance of a Step, set parameters, and then use this call() method, it will ignore previously-set parameters, as it creates a new instance of the class with only the config_file, *args and **kwargs passed to the call() method.

If not used with a config_file or specific *args and **kwargs, it would be better to use the run method, which does not create a new instance but simply runs the existing instance of the Step class.

closeout(to_close=None, to_del=None)

Close out step processing

Parameters:

• to_close ([object(, ..)]) – List of objects with a close method to execute The objects will also be deleted
• to_del ([object(, ..)]) – List of objects to simply delete

Notes

Other operations, such as forced garbage collection will also be done.

default_output_file(input_file=None)

Create a default filename based on the input name

default_suffix()

Return a default suffix based on the step

static from_cmdline(args)

Create a step from a configuration file.

Parameters:args (list of str) – Commandline arguments Returns:step – If the config file has a class parameter, the return value will be as instance of that class.

Any parameters found in the config file will be set as member variables on the returned Step instance.

Return type:Step instance

classmethod from_config_file(config_file, parent=None, name=None)

Create a step from a configuration file.

Parameters:

• config_file (path or readable file-like object) – The config file to load parameters from
• parent (Step instance, optional) – The parent step of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.
• name (str, optional) – If provided, use that name for the returned instance. If not provided, the following are tried (in order): - The name parameter in the config file - The filename of the config file - The name of returned class

Returns:

step – If the config file has a class parameter, the return value will be as instance of that class. The class parameter in the config file must specify a subclass of cls. If the configuration file has no class parameter, then an instance of cls is returned.

Any parameters found in the config file will be set as member variables on the returned Step instance.

Return type:

Step instance

classmethod from_config_section(config, parent=None, name=None, config_file=None)

Create a step from a configuration file fragment.

Parameters:

• config (configobj.Section instance) – The config file fragment containing parameters for this step only.
• parent (Step instance, optional) – The parent step of this step. Used to determine a fully-qualified name for this step, and to determine the mode in which to run this step.
• name (str, optional) – If provided, use that name for the returned instance. If not provided, try the following (in order): - The name parameter in the config file fragment - The name of returned class
• config_file (str, optional) – The path to the config file that created this step, if any. This is used to resolve relative file name parameters in the config file.

Returns:

step – Any parameters found in the config file fragment will be set as member variables on the returned Step instance.

Return type:

instance of cls

get_ref_override(reference_file_type)

Determine and return any override for reference_file_type.

Returns: Return type:override_filepath or None.

get_reference_file(input_file, reference_file_type)

Get a reference file from CRDS.

If the configuration file or commandline parameters override the reference file, it will be automatically used when calling this function.

Parameters:

• input_file (jwst.datamodels.ModelBase instance) – A model of the input file. Metadata on this input file will be used by the CRDS “bestref” algorithm to obtain a reference file.
• reference_file_type (string) – The type of reference file to retrieve. For example, to retrieve a flat field reference file, this would be ‘flat’.

Returns:

reference_file

Return type:

path of reference file, a string

load_as_level2_asn(obj)

Load object as an association

Loads the specified object into a Level2 association. If necessary, prepend Step.input_dir to all members.

Parameters:obj (object) – Object to load as a Level2 association Returns:association – Association Return type:jwst.associations.lib.rules_level2_base.DMSLevel2bBase

load_as_level3_asn(obj)

Load object as an association

Loads the specified object into a Level3 association. If necessary, prepend Step.input_dir to all members.

Parameters:obj (object) – Object to load as a Level3 association Returns:association – Association Return type:jwst.associations.lib.rules_level3_base.DMS_Level3_Base

classmethod load_spec_file(preserve_comments=False)

make_input_path(file_path)

Create an input path for a given file path

If file_path has no directory path, use self.input_dir as the directory path.

Parameters:file_path (str or obj) – The supplied file path to check and modify. If anything other than str, the object is simply passed back. Returns:full_path – File path using input_dir if the input had no directory path. Return type:str or obj

classmethod merge_config(config, config_file)

open_model(obj)

Open a datamodel

Primarily a wrapper around DataModel.open to handle Step peculiarities

Parameters:obj (object) – The object to open Returns:datamodel – Object opened as a datamodel Return type:DataModel

classmethod print_configspec(stream=<_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>)

process(*args)

This is where real work happens. Every Step subclass has to override this method. The default behaviour is to raise a NotImplementedError exception.

reference_uri_to_cache_path(reference_uri)

Convert an abstract CRDS reference URI to an absolute file path in the CRDS cache. Reference URI’s are typically output to dataset headers to record the reference files used.

e.g. ‘crds://jwst_miri_flat_0177.fits’ –>

‘/grp/crds/cache/references/jwst/jwst_miri_flat_0177.fits’

The CRDS cache is typically located relative to env var CRDS_PATH with default value /grp/crds/cache. See also https://jwst-crds.stsci.edu

resolve_file_name(file_name)

Resolve a file name expressed relative to this Step’s configuration file.

run(*args)

Run handles the generic setup and teardown that happens with the running of each step. The real work that is unique to each step type is done in the process method.

save_model(model, suffix=None, idx=None, output_file=None, force=False, format=None, **components)

Saves the given model using the step/pipeline’s naming scheme

Parameters:

• model (jwst.datamodels.Model instance) – The model to save.
• suffix (str) – The suffix to add to the filename.
• idx (object) – Index identifier.
• output_file (str) – Use this file name instead of what the Step default would be.
• force (bool) – Regardless of whether save_results is False and no output_file is specified, try saving.
• format (str) – The format of the file name. This is a format string that defines where suffix and the other components go in the file name.
• components (dict) – Other components to add to the file name.

Returns:

output_paths – List of output file paths the model(s) were saved in.

Return type:

[str[, ..]]

search_attr(attribute, default=None, parent_first=False)

Return first non-None attribute in step heirarchy

Parameters:

• attribute (str) – The attribute to retrieve
• default (obj) – If attribute is not found, the value to use
• parent_first (bool) – If True, allow parent definition to override step version

Returns:

value – Attribute value or default if not found

Return type:

obj

set_primary_input(obj, exclusive=True)

Sets the name of the master input file and input directory. Used to generate output file names.

Parameters:

• obj (str or DataModel) – The object to base the name on. If a datamodel, use Datamodel.meta.filename.
• exclusive (bool) – If True, only set if an input name is not already used by a parent Step. Otherwise, always set.