luigi.contrib.opener module

OpenerTarget support, allows easier testing and configuration by abstracting out the LocalTarget, S3Target, and MockTarget types.

Example:

from luigi.contrib.opener import OpenerTarget

OpenerTarget('/local/path.txt')
OpenerTarget('s3://zefr/remote/path.txt')
exception luigi.contrib.opener.OpenerError[source]

Bases: FileSystemException

The base exception thrown by openers

exception luigi.contrib.opener.NoOpenerError[source]

Bases: OpenerError

Thrown when there is no opener for the given protocol

exception luigi.contrib.opener.InvalidQuery[source]

Bases: OpenerError

Thrown when an opener is passed unexpected arguments

class luigi.contrib.opener.OpenerRegistry(openers=None)[source]

Bases: object

An opener registry that stores a number of opener objects used to parse Target URIs

Parameters:

openers (list) – A list of objects inherited from the Opener class.

get_opener(name)[source]

Retrieve an opener for the given protocol

Parameters:

name (string) – name of the opener to open

Raises:

NoOpenerError – if no opener has been registered of that name

add(opener)[source]

Adds an opener to the registry

Parameters:

opener (Opener inherited object) – Opener object

open(target_uri, **kwargs)[source]

Open target uri.

Parameters:

target_uri (string) – Uri to open

Returns:

Target object

class luigi.contrib.opener.Opener[source]

Bases: object

Base class for Opener objects.

allowed_kwargs = {}
filter_kwargs = True
classmethod conform_query(query)[source]

Converts the query string from a target uri, uses cls.allowed_kwargs, and cls.filter_kwargs to drive logic.

Parameters:

query (urllib.parse.unsplit(uri).query) – Unparsed query string

Returns:

Dictionary of parsed values, everything in cls.allowed_kwargs with values set to True will be parsed as json strings.

classmethod get_target(scheme, path, fragment, username, password, hostname, port, query, **kwargs)[source]

Override this method to use values from the parsed uri to initialize the expected target.

class luigi.contrib.opener.MockOpener[source]

Bases: Opener

Mock target opener, works like LocalTarget but files are all in memory.

example: * mock://foo/bar.txt

names = ['mock']
allowed_kwargs = {'format': False, 'is_tmp': True, 'mirror_on_stderr': True}
classmethod get_target(scheme, path, fragment, username, password, hostname, port, query, **kwargs)[source]

Override this method to use values from the parsed uri to initialize the expected target.

class luigi.contrib.opener.LocalOpener[source]

Bases: Opener

Local filesystem opener, works with any valid system path. This is the default opener and will be used if you don’t indicate which opener.

examples: * file://relative/foo/bar/baz.txt (opens a relative file) * file:///home/user (opens a directory from a absolute path) * foo/bar.baz (file:// is the default opener)

names = ['file']
allowed_kwargs = {'format': False, 'is_tmp': True}
classmethod get_target(scheme, path, fragment, username, password, hostname, port, query, **kwargs)[source]

Override this method to use values from the parsed uri to initialize the expected target.

class luigi.contrib.opener.S3Opener[source]

Bases: Opener

Opens a target stored on Amazon S3 storage

examples: * s3://bucket/foo/bar.txt * s3://bucket/foo/bar.txt?aws_access_key_id=xxx&aws_secret_access_key=yyy

names = ['s3', 's3n']
allowed_kwargs = {'client': True, 'format': False}
filter_kwargs = False
classmethod get_target(scheme, path, fragment, username, password, hostname, port, query, **kwargs)[source]

Override this method to use values from the parsed uri to initialize the expected target.

luigi.contrib.opener.OpenerTarget(target_uri, **kwargs)

Open target uri.

Parameters:

target_uri (string) – Uri to open

Returns:

Target object