Provides access to HDFS using the HdfsTarget, a subclass of Target.

class, format=None, is_tmp=False, fs=None)[source]



Open the FileSystem target.

This method returns a file-like object which can either be read from or written to depending on the specified mode.

Parameters:mode (str) – the mode r opens the FileSystemTarget in read-only mode, whereas w will open the FileSystemTarget in write mode. Subclasses can implement additional options. Using b is not supported; initialize with format=Nop instead.

Remove the resource at the path specified by this FileSystemTarget.

This method is implemented by using fs.

rename(path, raise_if_exists=False)[source]

Does not change self.path.

Unlike move_dir(), rename() might cause nested directories. See spotify/luigi#522

move(path, raise_if_exists=False)[source]

Alias for rename()


Move using rename_dont_move

New since after luigi v2.1: Does not change self.path

One could argue that the implementation should use the mkdir+raise_if_exists approach, but we at Spotify have had more trouble with that over just using plain mv. See spotify/luigi#557


Copy to destination directory.


Currently only works with hadoopcli

class, format=None, client=None, flag='_SUCCESS')[source]


Defines a target directory with a flag-file (defaults to _SUCCESS) used to signify job success.

This checks for two things:

  • the path exists (just like the HdfsTarget)
  • the _SUCCESS file exists within the directory.

Because Hadoop outputs into a directory and not a single file, the path is assumed to be a directory.

Initializes a HdfsFlagTarget.

  • path (str) – the directory where the files are stored.
  • client
  • flag (str) –

Returns True if the path for this FileSystemTarget exists; False otherwise.

This method is implemented by using fs.