Pipeline Node Factories
shtk.PipelineNodeFactory subclasses are templates used to define the properties of shtk.PipelineNode instances. shtk.PipelineNode instances are nodes within a directed acyclic graph (DAG) that represent a process pipeline.
shtk.PipelineNodeFactory instances are useful because they enable (1) running process pipelines multiple times (2) displaying the commands that will be run as part of the pipeline prior to executing the command and (3) composing partial pipelines into more complex process pipelines.
PipelineNodeFactory instances are usually constructed by a shtk.Shell.command(), or by using the pipe operator to connect multiple PipelineProcessFactory instances together into a single process pipeline.
shtk.PipelineNodeFactory
- class shtk.PipelineNodeFactory(stdin_factory=None, stdout_factory=None, stderr_factory=None)
Bases:
ABCAbstract base class defining a template for building PipelineNode’s
- Parameters
stdin_factory (None or StreamFactory) – Template for stdin Stream’s (Default value: None)
stdout_factory (None or StreamFactory) – Template for stdout Stream’s (Default value: None)
stderr_factory (None or StreamFactory) – Template for stderr Stream’s (Default value: None)
- stdin_factory
Template for stdin Stream’s (Default value: None)
- Type
None or StreamFactory
- stdout_factory
Template for stdout Stream’s (Default value: None)
- Type
None or StreamFactory
- stderr_factory
Template for stderr Stream’s (Default value: None)
- Type
None or StreamFactory
- children
templates for children
- Type
list of PipelineNodeFactory
- stdin(arg, mode='r')
Sets the stdin stream factory (in-place)
- Parameters
arg (str, pathlib.Path, StreamFactory, or None) –
If arg is an str or pathlib.Path, it is treated as a filename and stdin will be read from that file.
If arg is a StreamFactory it is used directly to create streams for stdin.
If None, stdin reads from os.devnull
mode – The mode in which to open the file, if opened. Only relevant if arg is a str or pathlib.Path. Must be one of (‘r’, ‘rb’). (Default value = ‘r’)
- Returns
Altered self
- Return type
- stdout(arg, mode='w')
Sets the stdout stream factory (in-place)
- Parameters
arg (str, pathlib.Path, StreamFactory, or None) –
If arg is an str or pathlib.Path, it is treated as a filename and stdout will write to that file.
If arg is a StreamFactory it is used directly to create streams for stdout.
If None, stdout writes to os.devnull
mode – The mode in which to open the file, if opened. Only relevant if arg is a str or pathlib.Path. Must be one of (‘w’, ‘wb’, ‘a’, ‘ab’). (Default value = ‘w’)
- Returns
Altered self
- Return type
- stderr(arg, mode='w')
Sets the stderr stream factory (in-place)
- Parameters
arg (str, pathlib.Path, StreamFactory, or None) –
If arg is an str or pathlib.Path, it is treated as a filename and stderr will write to that file.
If arg is a StreamFactory it is used directly to create streams for stderr.
If None, stderr writes to os.devnull
mode – The mode in which to open the file, if opened. Only relevant if arg is a str or pathlib.Path. Must be one of (‘w’, ‘wb’, ‘a’, ‘ab’). (Default value = ‘w’)
- Returns
Altered self
- Return type
- async build(job, stdin_stream=None, stdout_stream=None, stderr_stream=None)
Creates and executes PipelineNode’s and self-defined StreamFactories
If self.std{in,out,err}_factory is defined, it is pased to the child as the preferred stream. Otherwise the std{in,out,err}_stream parameters are used.
- Parameters
job (Job) – job from which to pull environment variables, current working directory, etc.
stdin_stream (Stream) – Stream instance to pass to PipelineNode as stdin
stdout_stream (Stream) – Stream instance to pass to PipelineNode as stdout
stderr_stream (Stream) – Stream instance to pass to PipelineNode as stderr
- Returns
The constructed PipelineNode instance
- Return type
- abstract async build_inner(job, stdin_stream, stdout_stream, stderr_stream)
Abstract method used for instantiating PipelineNodes. This method is wrapped by build() which handles stream management prior to passing them to build_inner().
- Parameters
job (Job) – The job from which to pull the current working directory and environment variables for subprocesses.
stdin_stream (Stream) – The Stream instance to be used as the PipelineNode’s stdin_stream.
stdout_stream (Stream) – The Stream instance to be used as the PipelineNode’s stdout_stream.
stderr_stream (Stream) – The Stream instance to be used as the PipelineNode’s stderr_stream.
- Returns
An instantiated PipelineNode.
- Return type
shtk.PipelineChannelFactory
- class shtk.PipelineChannelFactory(left, right, **kwargs)
Bases:
PipelineNodeFactoryPipelineChannelFactory is a template for creating PipelineChannel.
PipelineChannelFactory creates PipelineChannel instances representing a chain of subprocesses with each feeding stdout to the next subprocess’s stdin.
- Parameters
left (PipelineNodeFactory) – A PipelineNodeFactory that will create a series of processes that should write to stdout.
right (PipelineNodeFactory) – A PipelineNodeFactory that will create a series of processes that should read from stdin.
- left
A PipelineNodeFactory that will create a series of processes that should write to stdout.
- Type
- right
A PipelineNodeFactory that will create a series of processes that should read from stdin.
- Type
- children
[left, right]
- Type
list of PipelineNodeFactory
- pipe_stream
The PipeStreamFactory that will be used to redirect left’s stdout to right’s stdin.
- Type
- async build_inner(job, stdin_stream, stdout_stream, stderr_stream)
Instantiates a PipelineChannel
- async build(job, stdin_stream=None, stdout_stream=None, stderr_stream=None)
Creates and executes PipelineNode’s and self-defined StreamFactories
If self.std{in,out,err}_factory is defined, it is pased to the child as the preferred stream. Otherwise the std{in,out,err}_stream parameters are used.
- Parameters
job (Job) – job from which to pull environment variables, current working directory, etc.
stdin_stream (Stream) – Stream instance to pass to PipelineNode as stdin
stdout_stream (Stream) – Stream instance to pass to PipelineNode as stdout
stderr_stream (Stream) – Stream instance to pass to PipelineNode as stderr
- Returns
The constructed PipelineNode instance
- Return type
- stderr(arg, mode='w')
Sets the stderr stream factory (in-place)
- Parameters
arg (str, pathlib.Path, StreamFactory, or None) –
If arg is an str or pathlib.Path, it is treated as a filename and stderr will write to that file.
If arg is a StreamFactory it is used directly to create streams for stderr.
If None, stderr writes to os.devnull
mode – The mode in which to open the file, if opened. Only relevant if arg is a str or pathlib.Path. Must be one of (‘w’, ‘wb’, ‘a’, ‘ab’). (Default value = ‘w’)
- Returns
Altered self
- Return type
- stdin(arg, mode='r')
Sets the stdin stream factory (in-place)
- Parameters
arg (str, pathlib.Path, StreamFactory, or None) –
If arg is an str or pathlib.Path, it is treated as a filename and stdin will be read from that file.
If arg is a StreamFactory it is used directly to create streams for stdin.
If None, stdin reads from os.devnull
mode – The mode in which to open the file, if opened. Only relevant if arg is a str or pathlib.Path. Must be one of (‘r’, ‘rb’). (Default value = ‘r’)
- Returns
Altered self
- Return type
- stdout(arg, mode='w')
Sets the stdout stream factory (in-place)
- Parameters
arg (str, pathlib.Path, StreamFactory, or None) –
If arg is an str or pathlib.Path, it is treated as a filename and stdout will write to that file.
If arg is a StreamFactory it is used directly to create streams for stdout.
If None, stdout writes to os.devnull
mode – The mode in which to open the file, if opened. Only relevant if arg is a str or pathlib.Path. Must be one of (‘w’, ‘wb’, ‘a’, ‘ab’). (Default value = ‘w’)
- Returns
Altered self
- Return type
shtk.PipelineProcessFactory
- class shtk.PipelineProcessFactory(*args, env=None, cwd=None, close_fds=True)
Bases:
PipelineNodeFactoryTemplate for a PipelineProcess which runs a command as a subprocess
- Parameters
*args (list of str or pathlib.Path) – The command to run and its arguments for the instantiated PipelineProcess instances.
environment (dict) – The environment variables to use for the instantiated PipelineProcess instances (Default value = {}).
cwd (str or pathlib.Path) – The current working directory for the instantiated PipelineProcess instances (Default value = None).
close_fds (bool) – If true, close_fds will be passed to the equivalent of subprocess.Popen().
- async build(job, stdin_stream=None, stdout_stream=None, stderr_stream=None)
Creates and executes PipelineNode’s and self-defined StreamFactories
If self.std{in,out,err}_factory is defined, it is pased to the child as the preferred stream. Otherwise the std{in,out,err}_stream parameters are used.
- Parameters
job (Job) – job from which to pull environment variables, current working directory, etc.
stdin_stream (Stream) – Stream instance to pass to PipelineNode as stdin
stdout_stream (Stream) – Stream instance to pass to PipelineNode as stdout
stderr_stream (Stream) – Stream instance to pass to PipelineNode as stderr
- Returns
The constructed PipelineNode instance
- Return type
- async build_inner(job, stdin_stream, stdout_stream, stderr_stream)
Instantiates a PipelineProcess
- stderr(arg, mode='w')
Sets the stderr stream factory (in-place)
- Parameters
arg (str, pathlib.Path, StreamFactory, or None) –
If arg is an str or pathlib.Path, it is treated as a filename and stderr will write to that file.
If arg is a StreamFactory it is used directly to create streams for stderr.
If None, stderr writes to os.devnull
mode – The mode in which to open the file, if opened. Only relevant if arg is a str or pathlib.Path. Must be one of (‘w’, ‘wb’, ‘a’, ‘ab’). (Default value = ‘w’)
- Returns
Altered self
- Return type
- stdin(arg, mode='r')
Sets the stdin stream factory (in-place)
- Parameters
arg (str, pathlib.Path, StreamFactory, or None) –
If arg is an str or pathlib.Path, it is treated as a filename and stdin will be read from that file.
If arg is a StreamFactory it is used directly to create streams for stdin.
If None, stdin reads from os.devnull
mode – The mode in which to open the file, if opened. Only relevant if arg is a str or pathlib.Path. Must be one of (‘r’, ‘rb’). (Default value = ‘r’)
- Returns
Altered self
- Return type
- stdout(arg, mode='w')
Sets the stdout stream factory (in-place)
- Parameters
arg (str, pathlib.Path, StreamFactory, or None) –
If arg is an str or pathlib.Path, it is treated as a filename and stdout will write to that file.
If arg is a StreamFactory it is used directly to create streams for stdout.
If None, stdout writes to os.devnull
mode – The mode in which to open the file, if opened. Only relevant if arg is a str or pathlib.Path. Must be one of (‘w’, ‘wb’, ‘a’, ‘ab’). (Default value = ‘w’)
- Returns
Altered self
- Return type