luigi.contrib.hdfs package

Module contents

Provides access to HDFS using the HdfsTarget, a subclass of Target. You can configure what client by setting the “client” config under the “hdfs” section in the configuration, or using the --hdfs-client command line option. “hadoopcli” is the slowest, but should work out of the box. “snakebite” is the fastest, but requires Snakebite to be installed.

Since the hdfs functionality is quite big in luigi, it’s split into smaller files under luigi/contrib/hdfs/*.py. But for the sake of convenience and API stability, everything is reexported under luigi.contrib.hdfs.