luigi.contrib.dropbox module¶
- class luigi.contrib.dropbox.DropboxClient(token, user_agent='Luigi')[source]¶
Bases:
FileSystem
Dropbox client for authentication, designed to be used by the
DropboxTarget
class.- Parameters:
token (str) – Dropbox Oauth2 Token. See
DropboxTarget
for more information about generating a token
- exists(path)[source]¶
Return
True
if file or directory atpath
exist,False
otherwise- Parameters:
path (str) – a path within the FileSystem to check for existence.
- remove(path, recursive=True, skip_trash=True)[source]¶
Remove file or directory at location
path
- Parameters:
path (str) – a path within the FileSystem to remove.
recursive (bool) – if the path is a directory, recursively remove the directory and all of its descendants. Defaults to
True
.
- mkdir(path, parents=True, raise_if_exists=False)[source]¶
Create directory at location
path
Creates the directory at
path
and implicitly create parent directories if they do not already exist.- Parameters:
path (str) – a path within the FileSystem to create as a directory.
parents (bool) – Create parent directories when necessary. When parents=False and the parent directory doesn’t exist, raise luigi.target.MissingParentDirectory
raise_if_exists (bool) – raise luigi.target.FileAlreadyExists if the folder already exists.
- isdir(path)[source]¶
Return
True
if the location atpath
is a directory. If not, returnFalse
.- Parameters:
path (str) – a path within the FileSystem to check as a directory.
Note: This method is optional, not all FileSystem subclasses implements it.
- listdir(path, **kwargs)[source]¶
Return a list of files rooted in path.
This returns an iterable of the files rooted at
path
. This is intended to be a recursive listing.- Parameters:
path (str) – a path within the FileSystem to list.
Note: This method is optional, not all FileSystem subclasses implements it.
- class luigi.contrib.dropbox.ReadableDropboxFile(path, client)[source]¶
Bases:
object
Represents a file inside the Dropbox cloud which will be read
- Parameters:
path (str) – Dropbpx path of the file to be read (always starting with /)
client (DropboxClient) – a DropboxClient object (initialized with a valid token)
- class luigi.contrib.dropbox.AtomicWritableDropboxFile(path, client)[source]¶
Bases:
AtomicLocalFile
Represents a file that will be created inside the Dropbox cloud
- Parameters:
path (str) – Destination path inside Dropbox
client (DropboxClient) – a DropboxClient object (initialized with a valid token, for the desired account)
- class luigi.contrib.dropbox.DropboxTarget(path, token, format=None, user_agent='Luigi')[source]¶
Bases:
FileSystemTarget
A Dropbox filesystem target.
Create an Dropbox Target for storing data in a dropbox.com account
About the path parameter
The path must start with ‘/’ and should not end with ‘/’ (even if it is a directory). The path must not contain adjacent slashes (‘/files//img.jpg’ is an invalid path)
If the app has ‘App folder’ access, then / will refer to this app folder (which mean that there is no need to prepend the name of the app to the path) Otherwise, if the app has ‘full access’, then / will refer to the root of the Dropbox folder
About the token parameter:
The Dropbox target requires a valid OAuth2 token as a parameter (which means that a Dropbox API app must be created. This app can have ‘App folder’ access or ‘Full Dropbox’, as desired).
Information about generating the token can be read here:
https://dropbox-sdk-python.readthedocs.io/en/latest/api/oauth.html#dropbox.oauth.DropboxOAuth2Flow
https://blogs.dropbox.com/developers/2014/05/generate-an-access-token-for-your-own-account/
- Parameters:
path (str) – Remote path in Dropbox (starting with ‘/’).
token (str) – a valid OAuth2 Dropbox token.
format (luigi.Format) – the luigi format to use (e.g. luigi.format.Nop)
- property fs¶
The
FileSystem
associated with this FileSystemTarget.
- temporary_path()[source]¶
A context manager that enables a reasonably short, general and magic-less way to solve the Atomic Writes Problem.
On entering, it will create the parent directories so the temporary_path is writeable right away. This step uses
FileSystem.mkdir()
.On exiting, it will move the temporary file if there was no exception thrown. This step uses
FileSystem.rename_dont_move()
The file system operations will be carried out by calling them on
fs
.The typical use case looks like this:
class MyTask(luigi.Task): def output(self): return MyFileSystemTarget(...) def run(self): with self.output().temporary_path() as self.temp_output_path: run_some_external_command(output_path=self.temp_output_path)
- open(mode)[source]¶
Open the FileSystem target.
This method returns a file-like object which can either be read from or written to depending on the specified mode.
- Parameters:
mode (str) – the mode r opens the FileSystemTarget in read-only mode, whereas w will open the FileSystemTarget in write mode. Subclasses can implement additional options. Using b is not supported; initialize with format=Nop instead.