luigi.contrib.ftp module¶
This library is a wrapper of ftplib or pysftp. It is convenient to move data from/to (S)FTP servers.
There is an example on how to use it (example/ftp_experiment_outputs.py)
You can also find unittest for each class.
Be aware that normal ftp does not provide secure communication.
-
class
luigi.contrib.ftp.
RemoteFileSystem
(host, username=None, password=None, port=None, tls=False, timeout=60, sftp=False, pysftp_conn_kwargs=None)[source]¶ Bases:
luigi.target.FileSystem
-
exists
(path, mtime=None)[source]¶ Return True if file or directory at path exist, False otherwise.
Additional check on modified time when mtime is passed in.
Return False if the file’s modified time is older mtime.
-
-
class
luigi.contrib.ftp.
AtomicFtpFile
(fs, path)[source]¶ Bases:
luigi.target.AtomicLocalFile
Simple class that writes to a temp file and upload to ftp on close().
Also cleans up the temp file if close is not invoked.
Initializes an AtomicFtpfile instance. :param fs: :param path: :type path: str
-
fs
¶
-
-
class
luigi.contrib.ftp.
RemoteTarget
(path, host, format=None, username=None, password=None, port=None, mtime=None, tls=False, timeout=60, sftp=False, pysftp_conn_kwargs=None)[source]¶ Bases:
luigi.target.FileSystemTarget
Target used for reading from remote files.
The target is implemented using intermediate files on the local system. On Python2, these files may not be cleaned up.
-
fs
¶
-
open
(mode)[source]¶ Open the FileSystem target.
This method returns a file-like object which can either be read from or written to depending on the specified mode.
Parameters: mode (str) – the mode r opens the FileSystemTarget in read-only mode, whereas w will open the FileSystemTarget in write mode. Subclasses can implement additional options.
-