luigi.contrib.opener

OpenerTarget support, allows easier testing and configuration by abstracting out the LocalTarget, S3Target, and MockTarget types.

Example:

from luigi.contrib.opener import OpenerTarget

OpenerTarget('/local/path.txt')
OpenerTarget('s3://zefr/remote/path.txt')

Functions

OpenerTarget(target_uri, **kwargs)

Open target uri.

Classes

LocalOpener()	Local filesystem opener, works with any valid system path.
MockOpener()	Mock target opener, works like LocalTarget but files are all in memory.
Opener()	Base class for Opener objects.
OpenerRegistry([openers])	An opener registry that stores a number of opener objects used to parse Target URIs
S3Opener()	Opens a target stored on Amazon S3 storage

Exceptions

InvalidQuery	Thrown when an opener is passed unexpected arguments
NoOpenerError	Thrown when there is no opener for the given protocol
OpenerError	The base exception thrown by openers

exception luigi.contrib.opener.OpenerError

The base exception thrown by openers

exception luigi.contrib.opener.NoOpenerError

Thrown when there is no opener for the given protocol

exception luigi.contrib.opener.InvalidQuery

Thrown when an opener is passed unexpected arguments

class luigi.contrib.opener.OpenerRegistry(openers=None)

An opener registry that stores a number of opener objects used to parse Target URIs

Parameters:

openers (list) – A list of objects inherited from the Opener class.

get_opener(name)

Retrieve an opener for the given protocol

Parameters:

name (string) – name of the opener to open

Raises:

NoOpenerError – if no opener has been registered of that name

add(opener)

Adds an opener to the registry

Parameters:

opener (Opener inherited object) – Opener object

open(target_uri, **kwargs)

Open target uri.

Parameters:

target_uri (string) – Uri to open

Returns:

Target object

class luigi.contrib.opener.Opener

Base class for Opener objects.

allowed_kwargs: dict[str, bool] = {}

filter_kwargs = True

classmethod conform_query(query)

Converts the query string from a target uri, uses cls.allowed_kwargs, and cls.filter_kwargs to drive logic.

Parameters:

query (urllib.parse.unsplit(uri).query) – Unparsed query string

Returns:

Dictionary of parsed values, everything in cls.allowed_kwargs with values set to True will be parsed as json strings.

classmethod get_target(scheme, path, fragment, username, password, hostname, port, query, **kwargs)

Override this method to use values from the parsed uri to initialize the expected target.

class luigi.contrib.opener.MockOpener

Mock target opener, works like LocalTarget but files are all in memory.

example: * mock://foo/bar.txt

names = ['mock']

allowed_kwargs: dict[str, bool] = {'format': False, 'is_tmp': True, 'mirror_on_stderr': True}

classmethod get_target(scheme, path, fragment, username, password, hostname, port, query, **kwargs)

Override this method to use values from the parsed uri to initialize the expected target.

class luigi.contrib.opener.LocalOpener

Local filesystem opener, works with any valid system path. This is the default opener and will be used if you don’t indicate which opener.

examples: * file://relative/foo/bar/baz.txt (opens a relative file) * file:///home/user (opens a directory from a absolute path) * foo/bar.baz (file:// is the default opener)

names = ['file']

allowed_kwargs: dict[str, bool] = {'format': False, 'is_tmp': True}

classmethod get_target(scheme, path, fragment, username, password, hostname, port, query, **kwargs)

Override this method to use values from the parsed uri to initialize the expected target.

class luigi.contrib.opener.S3Opener

Opens a target stored on Amazon S3 storage

examples: * s3://bucket/foo/bar.txt * s3://bucket/foo/bar.txt?aws_access_key_id=xxx&aws_secret_access_key=yyy

names = ['s3', 's3n']

allowed_kwargs: dict[str, bool] = {'client': True, 'format': False}

filter_kwargs = False

classmethod get_target(scheme, path, fragment, username, password, hostname, port, query, **kwargs)

Override this method to use values from the parsed uri to initialize the expected target.

luigi.contrib.opener.OpenerTarget(target_uri, **kwargs)

Open target uri.

Parameters:

target_uri (string) – Uri to open

Returns:

Target object