luigi.contrib.opener module

OpenerTarget support, allows easier testing and configuration by abstracting out the LocalTarget, S3Target, and MockTarget types.

Example:

from luigi.contrib.opener import OpenerTarget

OpenerTarget('/local/path.txt')
OpenerTarget('s3://zefr/remote/path.txt')

exception luigi.contrib.opener.OpenerError

Bases: luigi.target.FileSystemException

The base exception thrown by openers

exception luigi.contrib.opener.NoOpenerError

Bases: luigi.contrib.opener.OpenerError

Thrown when there is no opener for the given protocol

exception luigi.contrib.opener.InvalidQuery

Bases: luigi.contrib.opener.OpenerError

Thrown when an opener is passed unexpected arguments

class luigi.contrib.opener.OpenerRegistry(openers=None)

Bases: object

An opener registry that stores a number of opener objects used to parse Target URIs

Parameters:openers (list) – A list of objects inherited from the Opener class.

get_opener(name)

Retrieve an opener for the given protocol

Parameters:name (string) – name of the opener to open Raises:NoOpenerError – if no opener has been registered of that name

add(opener)

Adds an opener to the registry

Parameters:opener (Opener inherited object) – Opener object

open(target_uri, **kwargs)

Open target uri.

Parameters:target_uri (string) – Uri to open Returns:Target object

class luigi.contrib.opener.Opener

Bases: object

Base class for Opener objects.

allowed_kwargs = {}

filter_kwargs = True

classmethod conform_query(query)

Converts the query string from a target uri, uses cls.allowed_kwargs, and cls.filter_kwargs to drive logic.

Parameters:query (urllib.parse.unsplit(uri)query) – Unparsed query string Returns:Dictionary of parsed values, everything in cls.allowed_kwargs with values set to True will be parsed as json strings.

classmethod get_target(scheme, path, fragment, username, password, hostname, port, query, **kwargs)

Override this method to use values from the parsed uri to initialize the expected target.

class luigi.contrib.opener.MockOpener

Bases: luigi.contrib.opener.Opener

Mock target opener, works like LocalTarget but files are all in memory.

example: * mock://foo/bar.txt

names = ['mock']

allowed_kwargs = {'format': False, 'is_tmp': True, 'mirror_on_stderr': True}

classmethod get_target(scheme, path, fragment, username, password, hostname, port, query, **kwargs)

Override this method to use values from the parsed uri to initialize the expected target.

class luigi.contrib.opener.LocalOpener

Bases: luigi.contrib.opener.Opener

Local filesystem opener, works with any valid system path. This is the default opener and will be used if you don’t indicate which opener.

examples: * file://relative/foo/bar/baz.txt (opens a relative file) * file:///home/user (opens a directory from a absolute path) * foo/bar.baz (file:// is the default opener)

names = ['file']

allowed_kwargs = {'format': False, 'is_tmp': True}

classmethod get_target(scheme, path, fragment, username, password, hostname, port, query, **kwargs)

Override this method to use values from the parsed uri to initialize the expected target.

class luigi.contrib.opener.S3Opener

Bases: luigi.contrib.opener.Opener

Opens a target stored on Amazon S3 storage

examples: * s3://bucket/foo/bar.txt * s3://bucket/foo/bar.txt?aws_access_key_id=xxx&aws_secret_access_key=yyy

names = ['s3', 's3n']

allowed_kwargs = {'client': True, 'format': False}

filter_kwargs = False

classmethod get_target(scheme, path, fragment, username, password, hostname, port, query, **kwargs)

Override this method to use values from the parsed uri to initialize the expected target.