luigi.contrib.hdfs package

Submodules

• luigi.contrib.hdfs.abstract_client module
  • HdfsFileSystem
    • HdfsFileSystem.rename()
    • HdfsFileSystem.rename_dont_move()
    • HdfsFileSystem.remove()
    • HdfsFileSystem.chmod()
    • HdfsFileSystem.chown()
    • HdfsFileSystem.count()
    • HdfsFileSystem.copy()
    • HdfsFileSystem.put()
    • HdfsFileSystem.get()
    • HdfsFileSystem.mkdir()
    • HdfsFileSystem.listdir()
    • HdfsFileSystem.touchz()
• luigi.contrib.hdfs.clients module
  • get_autoconfig_client()
  • exists()
  • rename()
  • remove()
  • mkdir()
  • listdir()
• luigi.contrib.hdfs.config module
  • hdfs
    • hdfs.client_version
    • hdfs.namenode_host
    • hdfs.namenode_port
    • hdfs.client
    • hdfs.tmp_dir
  • hadoopcli
    • hadoopcli.command
    • hadoopcli.version
  • load_hadoop_cmd()
  • get_configured_hadoop_version()
  • get_configured_hdfs_client()
  • tmppath()
• luigi.contrib.hdfs.error module
  • HDFSCliError
• luigi.contrib.hdfs.format module
  • HdfsAtomicWriteError
  • HdfsReadPipe
  • HdfsAtomicWritePipe
    • HdfsAtomicWritePipe.abort()
    • HdfsAtomicWritePipe.close()
  • HdfsAtomicWriteDirPipe
    • HdfsAtomicWriteDirPipe.abort()
    • HdfsAtomicWriteDirPipe.close()
  • PlainFormat
    • PlainFormat.input
    • PlainFormat.output
    • PlainFormat.hdfs_writer()
    • PlainFormat.hdfs_reader()
    • PlainFormat.pipe_reader()
    • PlainFormat.pipe_writer()
  • PlainDirFormat
    • PlainDirFormat.input
    • PlainDirFormat.output
    • PlainDirFormat.hdfs_writer()
    • PlainDirFormat.hdfs_reader()
    • PlainDirFormat.pipe_reader()
    • PlainDirFormat.pipe_writer()
  • CompatibleHdfsFormat
    • CompatibleHdfsFormat.output
    • CompatibleHdfsFormat.pipe_writer()
    • CompatibleHdfsFormat.pipe_reader()
    • CompatibleHdfsFormat.hdfs_writer()
    • CompatibleHdfsFormat.hdfs_reader()
• luigi.contrib.hdfs.hadoopcli_clients module
  • create_hadoopcli_client()
  • HdfsClient
    • HdfsClient.recursive_listdir_cmd
    • HdfsClient.call_check()
    • HdfsClient.exists()
    • HdfsClient.move()
    • HdfsClient.remove()
    • HdfsClient.chmod()
    • HdfsClient.chown()
    • HdfsClient.count()
    • HdfsClient.copy()
    • HdfsClient.put()
    • HdfsClient.get()
    • HdfsClient.getmerge()
    • HdfsClient.mkdir()
    • HdfsClient.listdir()
    • HdfsClient.touchz()
  • HdfsClientCdh3
    • HdfsClientCdh3.mkdir()
    • HdfsClientCdh3.remove()
  • HdfsClientApache1
    • HdfsClientApache1.recursive_listdir_cmd
    • HdfsClientApache1.exists()
• luigi.contrib.hdfs.target module
  • HdfsTarget
    • HdfsTarget.fs
    • HdfsTarget.glob_exists()
    • HdfsTarget.open()
    • HdfsTarget.remove()
    • HdfsTarget.rename()
    • HdfsTarget.move()
    • HdfsTarget.move_dir()
    • HdfsTarget.copy()
    • HdfsTarget.is_writable()
  • HdfsFlagTarget
    • HdfsFlagTarget.exists()
• luigi.contrib.hdfs.webhdfs_client module
  • webhdfs
    • webhdfs.port
    • webhdfs.user
    • webhdfs.client_type
  • WebHdfsClient
    • WebHdfsClient.url
    • WebHdfsClient.client
    • WebHdfsClient.walk()
    • WebHdfsClient.exists()
    • WebHdfsClient.upload()
    • WebHdfsClient.download()
    • WebHdfsClient.remove()
    • WebHdfsClient.read()
    • WebHdfsClient.move()
    • WebHdfsClient.mkdir()
    • WebHdfsClient.chmod()
    • WebHdfsClient.chown()
    • WebHdfsClient.count()
    • WebHdfsClient.copy()
    • WebHdfsClient.put()
    • WebHdfsClient.get()
    • WebHdfsClient.listdir()
    • WebHdfsClient.touchz()

Module contents

Provides access to HDFS using the HdfsTarget, a subclass of Target. You can configure what client by setting the “client” config under the “hdfs” section in the configuration, or using the --hdfs-client command line option. “hadoopcli” is the slowest, but should work out of the box.

Since the hdfs functionality is quite big in luigi, it’s split into smaller files under luigi/contrib/hdfs/*.py. But for the sake of convenience and API stability, everything is reexported under luigi.contrib.hdfs.