luigi.contrib.ssh module

Light-weight remote execution library and utilities.

There are some examples in the unittest but I added another that is more luigi-specific in the examples directory (examples/ssh_remote_execution.py)

RemoteContext is meant to provide functionality similar to that of the standard library subprocess module, but where the commands executed are run on a remote machine instead, without the user having to think about prefixing everything with “ssh” and credentials etc.

Using this mini library (which is just a convenience wrapper for subprocess), RemoteTarget is created to let you stream data from a remotely stored file using the luigi FileSystemTarget semantics.

As a bonus, RemoteContext also provides a really cool feature that let’s you set up ssh tunnels super easily using a python context manager (there is an example in the integration part of unittests).

This can be super convenient when you want secure communication using a non-secure protocol or circumvent firewalls (as long as they are open for ssh traffic).

exception luigi.contrib.ssh.RemoteCalledProcessError(returncode, command, host, output=None)[source]

Bases: subprocess.CalledProcessError

class luigi.contrib.ssh.RemoteContext(host, **kwargs)[source]

Bases: object

Popen(cmd, **kwargs)[source]

Remote Popen.

check_output(cmd)[source]

Execute a shell command remotely and return the output.

Simplified version of Popen when you only want the output as a string and detect any errors.

tunnel(*args, **kwds)[source]

Open a tunnel between localhost:local_port and remote_host:remote_port via the host specified by this context.

Remember to close() the returned “tunnel” object in order to clean up after yourself when you are done with the tunnel.

class luigi.contrib.ssh.RemoteFileSystem(host, **kwargs)[source]

Bases: luigi.target.FileSystem

exists(path)[source]

Return True if file or directory at path exist, False otherwise.

listdir(path)[source]
isdir(path)[source]

Return True if directory at path exist, False otherwise.

remove(path, recursive=True)[source]

Remove file or directory at location path.

mkdir(path, parents=True, raise_if_exists=False)[source]
put(local_path, path)[source]
get(path, local_path)[source]
class luigi.contrib.ssh.AtomicRemoteFileWriter(fs, path)[source]

Bases: luigi.format.OutputPipeProcessWrapper

close()[source]
tmp_path
fs
class luigi.contrib.ssh.RemoteTarget(path, host, format=None, **kwargs)[source]

Bases: luigi.target.FileSystemTarget

Target used for reading from remote files.

The target is implemented using ssh commands streaming data over the network.

fs
open(mode='r')[source]
put(local_path)[source]
get(local_path)[source]