luigi.contrib.opener module¶
OpenerTarget support, allows easier testing and configuration by abstracting out the LocalTarget, S3Target, and MockTarget types.
Example:
from luigi.contrib.opener import OpenerTarget
OpenerTarget('/local/path.txt')
OpenerTarget('s3://zefr/remote/path.txt')
- exception luigi.contrib.opener.OpenerError[source]¶
Bases:
FileSystemException
The base exception thrown by openers
- exception luigi.contrib.opener.NoOpenerError[source]¶
Bases:
OpenerError
Thrown when there is no opener for the given protocol
- exception luigi.contrib.opener.InvalidQuery[source]¶
Bases:
OpenerError
Thrown when an opener is passed unexpected arguments
- class luigi.contrib.opener.OpenerRegistry(openers=None)[source]¶
Bases:
object
An opener registry that stores a number of opener objects used to parse Target URIs
- Parameters:
openers (list) – A list of objects inherited from the Opener class.
- get_opener(name)[source]¶
Retrieve an opener for the given protocol
- Parameters:
name (string) – name of the opener to open
- Raises:
NoOpenerError – if no opener has been registered of that name
- class luigi.contrib.opener.Opener[source]¶
Bases:
object
Base class for Opener objects.
- allowed_kwargs = {}¶
- filter_kwargs = True¶
- classmethod conform_query(query)[source]¶
Converts the query string from a target uri, uses cls.allowed_kwargs, and cls.filter_kwargs to drive logic.
- Parameters:
query (urllib.parse.unsplit(uri).query) – Unparsed query string
- Returns:
Dictionary of parsed values, everything in cls.allowed_kwargs with values set to True will be parsed as json strings.
- class luigi.contrib.opener.MockOpener[source]¶
Bases:
Opener
Mock target opener, works like LocalTarget but files are all in memory.
example: * mock://foo/bar.txt
- names = ['mock']¶
- allowed_kwargs = {'format': False, 'is_tmp': True, 'mirror_on_stderr': True}¶
- class luigi.contrib.opener.LocalOpener[source]¶
Bases:
Opener
Local filesystem opener, works with any valid system path. This is the default opener and will be used if you don’t indicate which opener.
examples: * file://relative/foo/bar/baz.txt (opens a relative file) * file:///home/user (opens a directory from a absolute path) * foo/bar.baz (file:// is the default opener)
- names = ['file']¶
- allowed_kwargs = {'format': False, 'is_tmp': True}¶
- class luigi.contrib.opener.S3Opener[source]¶
Bases:
Opener
Opens a target stored on Amazon S3 storage
examples: * s3://bucket/foo/bar.txt * s3://bucket/foo/bar.txt?aws_access_key_id=xxx&aws_secret_access_key=yyy
- names = ['s3', 's3n']¶
- allowed_kwargs = {'client': True, 'format': False}¶
- filter_kwargs = False¶
- luigi.contrib.opener.OpenerTarget(target_uri, **kwargs)¶
Open target uri.
- Parameters:
target_uri (string) – Uri to open
- Returns:
Target object