luigi.contrib.hdfs.target module

Provides access to HDFS using the HdfsTarget, a subclass of Target.

class luigi.contrib.hdfs.target.HdfsTarget(path=None, format=None, is_tmp=False, fs=None)[source]

Bases: luigi.target.FileSystemTarget

fs
glob_exists(expected_files)[source]
open(mode='r')[source]
remove(skip_trash=False)[source]
rename(path, raise_if_exists=False)[source]

Does not change self.path.

Unlike move_dir(), rename() might cause nested directories. See spotify/luigi#522

move(path, raise_if_exists=False)[source]

Alias for rename()

move_dir(path)[source]

Move using rename_dont_move

New since after luigi v2.1: Does not change self.path

One could argue that the implementation should use the mkdir+raise_if_exists approach, but we at Spotify have had more trouble with that over just using plain mv. See spotify/luigi#557

copy(dst_dir)[source]

Copy to destination directory.

is_writable()[source]

Currently only works with hadoopcli