luigi.contrib.hdfs package¶
Submodules¶
- luigi.contrib.hdfs.abstract_client module
- luigi.contrib.hdfs.clients module
- luigi.contrib.hdfs.config module
- luigi.contrib.hdfs.error module
- luigi.contrib.hdfs.format module
- luigi.contrib.hdfs.hadoopcli_clients module
- luigi.contrib.hdfs.snakebite_client module
- luigi.contrib.hdfs.target module
- luigi.contrib.hdfs.webhdfs_client module
Module contents¶
Provides access to HDFS using the HdfsTarget
, a subclass of Target
.
You can configure what client by setting the “client” config under the “hdfs” section in the configuration, or using the --hdfs-client
command line option.
“hadoopcli” is the slowest, but should work out of the box. “snakebite” is the fastest, but requires Snakebite to be installed.
Since the hdfs functionality is quite big in luigi, it’s split into smaller
files under luigi/contrib/hdfs/*.py
. But for the sake of convenience and
API stability, everything is reexported under luigi.contrib.hdfs
.