luigi.target module¶
The abstract Target
class.
It is a central concept of Luigi and represents the state of the workflow.
- class luigi.target.Target[source]¶
Bases:
object
A Target is a resource generated by a
Task
.For example, a Target might correspond to a file in HDFS or data in a database. The Target interface defines one method that must be overridden:
exists()
, which signifies if the Target has been created or not.Typically, a
Task
will define one or more Targets as output, and the Task is considered complete if and only if each of its output Targets exist.
- exception luigi.target.FileSystemException[source]¶
Bases:
Exception
Base class for generic file system exceptions.
- exception luigi.target.FileAlreadyExists[source]¶
Bases:
FileSystemException
Raised when a file system operation can’t be performed because a directory exists but is required to not exist.
- exception luigi.target.MissingParentDirectory[source]¶
Bases:
FileSystemException
Raised when a parent directory doesn’t exist. (Imagine mkdir without -p)
- exception luigi.target.NotADirectory[source]¶
Bases:
FileSystemException
Raised when a file system operation can’t be performed because an expected directory is actually a file.
- class luigi.target.FileSystem[source]¶
Bases:
object
FileSystem abstraction used in conjunction with
FileSystemTarget
.Typically, a FileSystem is associated with instances of a
FileSystemTarget
. The instances of theFileSystemTarget
will delegate methods such asFileSystemTarget.exists()
andFileSystemTarget.remove()
to the FileSystem.Methods of FileSystem raise
FileSystemException
if there is a problem completing the operation.- abstract exists(path)[source]¶
Return
True
if file or directory atpath
exist,False
otherwise- Parameters:
path (str) – a path within the FileSystem to check for existence.
- abstract remove(path, recursive=True, skip_trash=True)[source]¶
Remove file or directory at location
path
- Parameters:
path (str) – a path within the FileSystem to remove.
recursive (bool) – if the path is a directory, recursively remove the directory and all of its descendants. Defaults to
True
.
- mkdir(path, parents=True, raise_if_exists=False)[source]¶
Create directory at location
path
Creates the directory at
path
and implicitly create parent directories if they do not already exist.- Parameters:
path (str) – a path within the FileSystem to create as a directory.
parents (bool) – Create parent directories when necessary. When parents=False and the parent directory doesn’t exist, raise luigi.target.MissingParentDirectory
raise_if_exists (bool) – raise luigi.target.FileAlreadyExists if the folder already exists.
- isdir(path)[source]¶
Return
True
if the location atpath
is a directory. If not, returnFalse
.- Parameters:
path (str) – a path within the FileSystem to check as a directory.
Note: This method is optional, not all FileSystem subclasses implements it.
- listdir(path)[source]¶
Return a list of files rooted in path.
This returns an iterable of the files rooted at
path
. This is intended to be a recursive listing.- Parameters:
path (str) – a path within the FileSystem to list.
Note: This method is optional, not all FileSystem subclasses implements it.
- rename_dont_move(path, dest)[source]¶
Potentially rename
path
todest
, but don’t move it into thedest
folder (if it is a folder). This relates to Atomic Writes Problem.This method has a reasonable but not bullet proof default implementation. It will just do
move()
if the file doesn’texists()
already.
- class luigi.target.FileSystemTarget(path)[source]¶
Bases:
Target
Base class for FileSystem Targets like
LocalTarget
andHdfsTarget
.A FileSystemTarget has an associated
FileSystem
to which certain operations can be delegated. By default,exists()
andremove()
are delegated to theFileSystem
, which is determined by thefs
property.Methods of FileSystemTarget raise
FileSystemException
if there is a problem completing the operation.- Usage:
target = FileSystemTarget('~/some_file.txt') target = FileSystemTarget(pathlib.Path('~') / 'some_file.txt') target.exists() # False
Initializes a FileSystemTarget instance.
- Parameters:
path – the path associated with this FileSystemTarget.
- abstract property fs¶
The
FileSystem
associated with this FileSystemTarget.
- abstract open(mode)[source]¶
Open the FileSystem target.
This method returns a file-like object which can either be read from or written to depending on the specified mode.
- Parameters:
mode (str) – the mode r opens the FileSystemTarget in read-only mode, whereas w will open the FileSystemTarget in write mode. Subclasses can implement additional options. Using b is not supported; initialize with format=Nop instead.
- exists()[source]¶
Returns
True
if the path for this FileSystemTarget exists;False
otherwise.This method is implemented by using
fs
.
- remove()[source]¶
Remove the resource at the path specified by this FileSystemTarget.
This method is implemented by using
fs
.
- temporary_path()[source]¶
A context manager that enables a reasonably short, general and magic-less way to solve the Atomic Writes Problem.
On entering, it will create the parent directories so the temporary_path is writeable right away. This step uses
FileSystem.mkdir()
.On exiting, it will move the temporary file if there was no exception thrown. This step uses
FileSystem.rename_dont_move()
The file system operations will be carried out by calling them on
fs
.The typical use case looks like this:
class MyTask(luigi.Task): def output(self): return MyFileSystemTarget(...) def run(self): with self.output().temporary_path() as self.temp_output_path: run_some_external_command(output_path=self.temp_output_path)
- class luigi.target.AtomicLocalFile(path)[source]¶
Bases:
BufferedWriter
Abstract class to create a Target that creates a temporary file in the local filesystem before moving it to its final destination.
This class is just for the writing part of the Target. See
luigi.local_target.LocalTarget
for example- close()[source]¶
Flush and close the IO object.
This method has no effect if the file is already closed.
- property tmp_path¶