Provides access to HDFS using the HdfsTarget, a subclass of Target.

class, format=None, is_tmp=False, fs=None)[source]



Open the FileSystem target.

This method returns a file-like object which can either be read from or written to depending on the specified mode.

Parameters:mode (str) – the mode r opens the FileSystemTarget in read-only mode, whereas w will open the FileSystemTarget in write mode. Subclasses can implement additional options.

Remove the resource at the path specified by this FileSystemTarget.

This method is implemented by using fs.

rename(path, raise_if_exists=False)[source]

Does not change self.path.

Unlike move_dir(), rename() might cause nested directories. See spotify/luigi#522

move(path, raise_if_exists=False)[source]

Alias for rename()


Move using rename_dont_move

New since after luigi v2.1: Does not change self.path

One could argue that the implementation should use the mkdir+raise_if_exists approach, but we at Spotify have had more trouble with that over just using plain mv. See spotify/luigi#557


Copy to destination directory.


Currently only works with hadoopcli

class, format=None, client=None, flag='_SUCCESS')[source]


Defines a target directory with a flag-file (defaults to _SUCCESS) used to signify job success.

This checks for two things:

  • the path exists (just like the HdfsTarget)
  • the _SUCCESS file exists within the directory.

Because Hadoop outputs into a directory and not a single file, the path is assumed to be a directory.

Initializes a HdfsFlagTarget.

  • path (str) – the directory where the files are stored.
  • client
  • flag (str) –

Returns True if the path for this FileSystemTarget exists; False otherwise.

This method is implemented by using fs.