luigi.contrib.presto

Classes

`PrestoClient`(connection[, sleep_time])	Helper class wrapping pyhive.presto.Connection for executing presto queries and tracking progress
`PrestoTarget`(client, catalog, database, table)	Target for presto-accessible tables
`PrestoTask`(args, *kwargs)	Task for executing presto queries During its executions tracking url and percentage progress are set
`WithPrestoClient`(name, bases, attrs)	A metaclass for injecting PrestoClient as a _client field into a new instance of class T Presto connection options are taken from T-instance fields Fields should have the same names as in pyhive.presto.Cursor
`presto`(args, *kwargs)

class luigi.contrib.presto.presto(*args, **kwargs)[source]

host

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

Any value provided on the command line:

To the root task (eg. --param xyz)

Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).

With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See Parameters from config Ingestion

Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

port: Parameter whose value is an int.

user

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

Any value provided on the command line:

To the root task (eg. --param xyz)

Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).

With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See Parameters from config Ingestion

Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

catalog

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

Any value provided on the command line:

To the root task (eg. --param xyz)

Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).

With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See Parameters from config Ingestion

Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

password

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

Any value provided on the command line:

To the root task (eg. --param xyz)

Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).

With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See Parameters from config Ingestion

Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

protocol

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

Any value provided on the command line:

To the root task (eg. --param xyz)

Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).

With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See Parameters from config Ingestion

Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

poll_interval: Parameter whose value is a float.

class luigi.contrib.presto.PrestoClient(connection, sleep_time=1)[source]

Helper class wrapping pyhive.presto.Connection for executing presto queries and tracking progress

property percentage_progress

Returns:: percentage of query overall progress

property info_uri

Returns:: query UI link

execute(query, parameters=None, mode=None)[source]

Parameters:

query – query to run
parameters – parameters should be injected in the query
mode – “fetch” - yields rows, “watch” - yields log entries

Returns:

class luigi.contrib.presto.WithPrestoClient(name, bases, attrs)[source]: A metaclass for injecting PrestoClient as a _client field into a new instance of class T Presto connection options are taken from T-instance fields Fields should have the same names as in pyhive.presto.Cursor

class luigi.contrib.presto.PrestoTarget(client, catalog, database, table, partition=None)[source]

Target for presto-accessible tables

count()[source]

exists()[source]

Returns:: True if given table exists and there are any rows in a given partition False if no rows in the partition exists or table is absent

class luigi.contrib.presto.PrestoTask(*args, **kwargs)[source]

Task for executing presto queries During its executions tracking url and percentage progress are set

property host: Host of the RDBMS. Implementation should support hostname:port to encode port.

property port: Override to specify port separately from host.

property user

property username

property schema

property password

property catalog

property poll_interval

property source

property partition

property protocol

property session_props

property requests_session

property requests_kwargs

query = None

run()[source]

The task run method, to be overridden in a subclass.

See Task.run

output()[source]: Override with an RDBMS Target (e.g. PostgresTarget or RedshiftTarget) to record execution in a marker table