luigi.target module

The abstract Target class. It is a central concept of Luigi and represents the state of the workflow.

class luigi.target.Target

Bases: object

A Target is a resource generated by a Task.

For example, a Target might correspond to a file in HDFS or data in a database. The Target interface defines one method that must be overridden: exists(), which signifies if the Target has been created or not.

Typically, a Task will define one or more Targets as output, and the Task is considered complete if and only if each of its output Targets exist.

abstract exists()

Returns True if the Target exists and False otherwise.

exception luigi.target.FileSystemException

Bases: Exception

Base class for generic file system exceptions.

exception luigi.target.FileAlreadyExists

Bases: FileSystemException

Raised when a file system operation can’t be performed because a directory exists but is required to not exist.

exception luigi.target.MissingParentDirectory

Bases: FileSystemException

Raised when a parent directory doesn’t exist. (Imagine mkdir without -p)

exception luigi.target.NotADirectory

Bases: FileSystemException

Raised when a file system operation can’t be performed because an expected directory is actually a file.

class luigi.target.FileSystem

Bases: object

FileSystem abstraction used in conjunction with FileSystemTarget.

Typically, a FileSystem is associated with instances of a FileSystemTarget. The instances of the FileSystemTarget will delegate methods such as FileSystemTarget.exists() and FileSystemTarget.remove() to the FileSystem.

Methods of FileSystem raise FileSystemException if there is a problem completing the operation.

abstract exists(path)

Return True if file or directory at path exist, False otherwise

Parameters:

path (str) – a path within the FileSystem to check for existence.

abstract remove(path, recursive=True, skip_trash=True)

Remove file or directory at location path

Parameters:

• path (str) – a path within the FileSystem to remove.

• recursive (bool) – if the path is a directory, recursively remove the directory and all of its descendants. Defaults to True.

mkdir(path, parents=True, raise_if_exists=False)

Create directory at location path

Creates the directory at path and implicitly create parent directories if they do not already exist.

Parameters:

• path (str) – a path within the FileSystem to create as a directory.

• parents (bool) – Create parent directories when necessary. When parents=False and the parent directory doesn’t exist, raise luigi.target.MissingParentDirectory

• raise_if_exists (bool) – raise luigi.target.FileAlreadyExists if the folder already exists.

isdir(path)

Return True if the location at path is a directory. If not, return False.

Parameters:

path (str) – a path within the FileSystem to check as a directory.

Note: This method is optional, not all FileSystem subclasses implements it.

listdir(path)

Return a list of files rooted in path.

This returns an iterable of the files rooted at path. This is intended to be a recursive listing.

Parameters:

path (str) – a path within the FileSystem to list.

Note: This method is optional, not all FileSystem subclasses implements it.

move(path, dest)

Move a file, as one would expect.

rename_dont_move(path, dest)

Potentially rename path to dest, but don’t move it into the dest folder (if it is a folder). This relates to Atomic Writes Problem.

This method has a reasonable but not bullet proof default implementation. It will just do move() if the file doesn’t exists() already.

rename(*args, **kwargs)

Alias for move()

copy(path, dest)

Copy a file or a directory with contents. Currently, LocalFileSystem and MockFileSystem support only single file copying but S3Client copies either a file or a directory as required.

class luigi.target.FileSystemTarget(path)

Bases: Target

Base class for FileSystem Targets like LocalTarget and HdfsTarget.

A FileSystemTarget has an associated FileSystem to which certain operations can be delegated. By default, exists() and remove() are delegated to the FileSystem, which is determined by the fs property.

Methods of FileSystemTarget raise FileSystemException if there is a problem completing the operation.

Usage:

target = FileSystemTarget('~/some_file.txt')
target = FileSystemTarget(pathlib.Path('~') / 'some_file.txt')
target.exists()  # False

Initializes a FileSystemTarget instance.

Parameters:

path – the path associated with this FileSystemTarget.

abstract property fs

The FileSystem associated with this FileSystemTarget.

abstract open(mode)

Open the FileSystem target.

This method returns a file-like object which can either be read from or written to depending on the specified mode.

Parameters:

mode (str) – the mode r opens the FileSystemTarget in read-only mode, whereas w will open the FileSystemTarget in write mode. Subclasses can implement additional options. Using b is not supported; initialize with format=Nop instead.

exists()

Returns True if the path for this FileSystemTarget exists; False otherwise.

This method is implemented by using fs.

remove()

Remove the resource at the path specified by this FileSystemTarget.

This method is implemented by using fs.

temporary_path()

A context manager that enables a reasonably short, general and magic-less way to solve the Atomic Writes Problem.

• On entering, it will create the parent directories so the temporary_path is writeable right away. This step uses FileSystem.mkdir().

• On exiting, it will move the temporary file if there was no exception thrown. This step uses FileSystem.rename_dont_move()

The file system operations will be carried out by calling them on fs.

The typical use case looks like this:

class MyTask(luigi.Task):
    def output(self):
        return MyFileSystemTarget(...)

    def run(self):
        with self.output().temporary_path() as self.temp_output_path:
            run_some_external_command(output_path=self.temp_output_path)

class luigi.target.AtomicLocalFile(path)

Bases: BufferedWriter

Abstract class to create a Target that creates a temporary file in the local filesystem before moving it to its final destination.

This class is just for the writing part of the Target. See luigi.local_target.LocalTarget for example

close()

Flush and close the IO object.

This method has no effect if the file is already closed.

generate_tmp_path(path)

move_to_final_destination()

property tmp_path