luigi.contrib.webhdfs module

Provides a WebHdfsTarget using the Python hdfs

This module is DEPRECATED and does not play well with rest of luigi’s hdfs contrib module. You can consider migrating to luigi.contrib.hdfs.webhdfs_client.WebHdfsClient

class luigi.contrib.webhdfs.WebHdfsTarget(path, client=None, format=None)[source]

Bases: luigi.target.FileSystemTarget

fs = None
open(mode='r')[source]

Open the FileSystem target.

This method returns a file-like object which can either be read from or written to depending on the specified mode.

Parameters:mode (str) – the mode r opens the FileSystemTarget in read-only mode, whereas w will open the FileSystemTarget in write mode. Subclasses can implement additional options.
class luigi.contrib.webhdfs.ReadableWebHdfsFile(path, client)[source]

Bases: object

read()[source]
readlines(char='\n')[source]
close()[source]
class luigi.contrib.webhdfs.AtomicWebHdfsFile(path, client)[source]

Bases: luigi.target.AtomicLocalFile

An Hdfs file that writes to a temp file and put to WebHdfs on close.

move_to_final_destination()[source]