luigi.contrib.hdfs.hadoopcli_clients module

The implementations of the hdfs clients. The hadoop cli client and the snakebite client.

luigi.contrib.hdfs.hadoopcli_clients.create_hadoopcli_client()

Given that we want one of the hadoop cli clients (unlike snakebite), this one will return the right one.

class luigi.contrib.hdfs.hadoopcli_clients.HdfsClient

Bases: luigi.contrib.hdfs.abstract_client.HdfsFileSystem

This client uses Apache 2.x syntax for file system commands, which also matched CDH4.

recursive_listdir_cmd = ['-ls', '-R']

static call_check(command)

exists(path)

Use hadoop fs -stat to check file existence.

move(path, dest)

Move a file, as one would expect.

remove(path, recursive=True, skip_trash=False)

Remove file or directory at location path

Parameters:

• path (str) – a path within the FileSystem to remove.
• recursive (bool) – if the path is a directory, recursively remove the directory and all of its descendants. Defaults to True.

chmod(path, permissions, recursive=False)

chown(path, owner, group, recursive=False)

count(path)

Count contents in a directory

copy(path, destination)

Copy a file or a directory with contents. Currently, LocalFileSystem and MockFileSystem support only single file copying but S3Client copies either a file or a directory as required.

put(local_path, destination)

get(path, local_destination)

getmerge(path, local_destination, new_line=False)

mkdir(path, parents=True, raise_if_exists=False)

Create directory at location path

Creates the directory at path and implicitly create parent directories if they do not already exist.

Parameters:

• path (str) – a path within the FileSystem to create as a directory.
• parents (bool) – Create parent directories when necessary. When parents=False and the parent directory doesn’t exist, raise luigi.target.MissingParentDirectory
• raise_if_exists (bool) – raise luigi.target.FileAlreadyExists if the folder already exists.

listdir(path, ignore_directories=False, ignore_files=False, include_size=False, include_type=False, include_time=False, recursive=False)

Return a list of files rooted in path.

This returns an iterable of the files rooted at path. This is intended to be a recursive listing.

Parameters:path (str) – a path within the FileSystem to list.

Note: This method is optional, not all FileSystem subclasses implements it.

touchz(path)

class luigi.contrib.hdfs.hadoopcli_clients.HdfsClientCdh3

Bases: luigi.contrib.hdfs.hadoopcli_clients.HdfsClient

This client uses CDH3 syntax for file system commands.

mkdir(path, parents=True, raise_if_exists=False)

No explicit -p switch, this version of Hadoop always creates parent directories.

remove(path, recursive=True, skip_trash=False)

Remove file or directory at location path

Parameters:

• path (str) – a path within the FileSystem to remove.
• recursive (bool) – if the path is a directory, recursively remove the directory and all of its descendants. Defaults to True.

class luigi.contrib.hdfs.hadoopcli_clients.HdfsClientApache1

Bases: luigi.contrib.hdfs.hadoopcli_clients.HdfsClientCdh3

This client uses Apache 1.x syntax for file system commands, which are similar to CDH3 except for the file existence check.

recursive_listdir_cmd = ['-lsr']

exists(path)

Use hadoop fs -stat to check file existence.