runner.py

get_hook(ref)

Gets a hook based on its reference. The reference should be formatted {package}.{subpackage}.{...}:{name}. Used to retrieve a hook based on a reference provided to the CLI.

Parameters:

Name	Type	Description	Default
ref	str	the hook reference	required

Returns:

Type	Description
NodeHook	the hook

Raises:

Type	Description
TypeError	if the retrieved object is not a hook.

get_module(module_ref)

Function that loads a module based on a reference string

Parameters:

Name	Type	Description	Default
module_ref	str	reference to the module, e.g. "my.module"	required

Returns:

Type	Description
ModuleType	the module object

Raises:

Type	Description
ModuleNotFoundError	if the module is not found

get_node(ref, _get_obj=get_obj)

Gets a node based on its reference. The reference should be formatted {package}.{subpackage}.{...}:{name}. Used to retrieve a node based on a reference provided to the CLI.

Parameters:

Name	Type	Description	Default
ref	str	the node reference	required
_get_obj	Callable[[str], object]	callable to get an object based on its reference	get_obj

Returns:

Type	Description
Callable	the node

Raises: TypeError: if the retrieved object is not a node.

get_obj(ref, _get_module=get_module)

Gets an object based on its reference. The reference should be formatted {package}.{subpackage}.{...}:{name}. Used to retrieve nodes and pipelines based on references provided to the CLI.

Examples:

>>> from ordeq import Node
>>> node = get_obj("path.to.module:export_to_mssql")  # doctest: +SKIP
>>> isinstance(node, Node)  # doctest: +SKIP
True

>>> from ordeq.framework.pipeline import is_pipeline
>>> pipeline = get_obj("path.to.module:everything")  # doctest: +SKIP
>>> is_pipeline(pipeline)  # doctest: +SKIP
True

>>> from ordeq.framework import Hook
>>> log_hook = get_obj("path.to.module:LogHook")  # doctest: +SKIP
>>> isinstance(log_hook, Hook)  # doctest: +SKIP
True

Parameters:

Name	Type	Description	Default
ref	str	reference to the object	required
_get_module	Callable[[str], ModuleType]	method to retrieve the module of the object	get_module

Returns:

Type	Description
object	the object

Raises:

Type	Description
ModuleNotFoundError	if the module cannot be found
AttributeError	if the object is not found in the module.

get_pipeline(ref, _get_obj=get_obj)

Gets a pipeline based on its reference. The reference should be formatted {package}.{subpackage}.{...}:{name}. Used to retrieve a pipeline based on a reference provided to the CLI.

Parameters:

Name	Type	Description	Default
ref	str	the pipeline reference	required
_get_obj	Callable[[str], object]	callable to get an object based on its reference	get_obj

Returns:

Type	Description
Pipeline	the pipeline

Raises:

Type	Description
TypeError	if the retrieved object is not a pipeline.

run_node_refs(node_refs, hook_refs, *, save='all')

Runs nodes by their references. References should be formatted {package}.{subpackage}.{...}:{object-name}. This method should not be called directly by users.

Parameters:

Name	Type	Description	Default
node_refs	tuple[str, ...]	node references	required
hook_refs	tuple[str]	hook references	required
save	SaveMode	save mode to run with, defaults to "all"	'all'

Returns:

Type	Description
DataStoreType	RunReceipt

run_pipeline_ref(pipeline_ref, hook_refs, *, save='all')

Runs a pipeline by its references. References should be formatted {package}.{subpackage}.{...}:{object-name}. This method should not be called directly by users.

Parameters:

Name	Type	Description	Default
pipeline_ref	str	pipeline reference	required
hook_refs	tuple[str]	hook references	required
save	SaveMode	save mode to run with, defaults to "all"	'all'

Returns:

Type	Description
DataStoreType	RunReceipt