luigi.contrib.ssh module¶
Light-weight remote execution library and utilities.
There are some examples in the unittest but I added another that is more luigi-specific in the examples directory (examples/ssh_remote_execution.py)
RemoteContext
is meant to provide functionality similar to that of the
standard library subprocess module, but where the commands executed are run on
a remote machine instead, without the user having to think about prefixing
everything with “ssh” and credentials etc.
Using this mini library (which is just a convenience wrapper for subprocess),
RemoteTarget
is created to let you stream data from a remotely stored file using
the luigi FileSystemTarget
semantics.
As a bonus, RemoteContext
also provides a really cool feature that let’s you
set up ssh tunnels super easily using a python context manager (there is an example
in the integration part of unittests).
This can be super convenient when you want secure communication using a non-secure protocol or circumvent firewalls (as long as they are open for ssh traffic).
- exception luigi.contrib.ssh.RemoteCalledProcessError(returncode, command, host, output=None)[source]¶
Bases:
CalledProcessError
- class luigi.contrib.ssh.RemoteContext(host, **kwargs)[source]¶
Bases:
object
- check_output(cmd)[source]¶
Execute a shell command remotely and return the output.
Simplified version of Popen when you only want the output as a string and detect any errors.
- tunnel(local_port, remote_port=None, remote_host='localhost')[source]¶
Open a tunnel between localhost:local_port and remote_host:remote_port via the host specified by this context.
Remember to close() the returned “tunnel” object in order to clean up after yourself when you are done with the tunnel.
- class luigi.contrib.ssh.RemoteFileSystem(host, **kwargs)[source]¶
Bases:
FileSystem
- listdir(path)[source]¶
Return a list of files rooted in path.
This returns an iterable of the files rooted at
path
. This is intended to be a recursive listing.- Parameters:
path (str) – a path within the FileSystem to list.
Note: This method is optional, not all FileSystem subclasses implements it.
- mkdir(path, parents=True, raise_if_exists=False)[source]¶
Create directory at location
path
Creates the directory at
path
and implicitly create parent directories if they do not already exist.- Parameters:
path (str) – a path within the FileSystem to create as a directory.
parents (bool) – Create parent directories when necessary. When parents=False and the parent directory doesn’t exist, raise luigi.target.MissingParentDirectory
raise_if_exists (bool) – raise luigi.target.FileAlreadyExists if the folder already exists.
- class luigi.contrib.ssh.AtomicRemoteFileWriter(fs, path)[source]¶
Bases:
OutputPipeProcessWrapper
- property tmp_path¶
- property fs¶
- class luigi.contrib.ssh.RemoteTarget(path, host, format=None, **kwargs)[source]¶
Bases:
FileSystemTarget
Target used for reading from remote files.
The target is implemented using ssh commands streaming data over the network.
Initializes a FileSystemTarget instance.
- Parameters:
path – the path associated with this FileSystemTarget.
- property fs¶
The
FileSystem
associated with this FileSystemTarget.
- open(mode='r')[source]¶
Open the FileSystem target.
This method returns a file-like object which can either be read from or written to depending on the specified mode.
- Parameters:
mode (str) – the mode r opens the FileSystemTarget in read-only mode, whereas w will open the FileSystemTarget in write mode. Subclasses can implement additional options. Using b is not supported; initialize with format=Nop instead.