luigi package

Subpackages

Module contents

Package containing core luigi functionality.

class luigi.Task(*args, **kwargs)[source]

Bases: object

This is the base class of all Luigi Tasks, the base unit of work in Luigi.

A Luigi Task describes a unit or work.

The key methods of a Task, which must be implemented in a subclass are:

  • run() - the computation done by this task.
  • requires() - the list of Tasks that this Task depends on.
  • output() - the output Target that this Task creates.

Each Parameter of the Task should be declared as members:

class MyTask(luigi.Task):
    count = luigi.IntParameter()
    second_param = luigi.Parameter()

In addition to any declared properties and methods, there are a few non-declared properties, which are created by the Register metaclass:

priority = 0

Priority of the task: the scheduler should favor available tasks with higher priority values first. See Task priority

disabled = False
resources = {}

Resources used by the task. Should be formatted like {“scp”: 1} to indicate that the task requires 1 unit of the scp resource.

worker_timeout = None

Number of seconds after which to time out the run function. No timeout if set to 0. Defaults to 0 or worker-timeout value in config file Only works when using multiple workers.

max_batch_size = inf

Maximum number of tasks to run together as a batch. Infinite by default

batchable

True if this instance can be run as part of a batch. By default, True if it has any batched parameters

retry_count

Override this positive integer to have different retry_count at task level Check [scheduler]

disable_hard_timeout

Override this positive integer to have different disable_hard_timeout at task level. Check [scheduler]

disable_window_seconds

Override this positive integer to have different disable_window_seconds at task level. Check [scheduler]

owner_email

Override this to send out additional error emails to task owner, in addition to the one defined in the global configuration. This should return a string or a list of strings. e.g. ‘test@exmaple.com’ or [‘test1@example.com’, ‘test2@example.com’]

use_cmdline_section

Property used by core config such as –workers etc. These will be exposed without the class as prefix.

classmethod event_handler(event)[source]

Decorator for adding event handlers.

trigger_event(event, *args, **kwargs)[source]

Trigger that calls all of the specified events associated with this class.

accepts_messages

For configuring which scheduler messages can be received. When falsy, this tasks does not accept any message. When True, all messages are accepted.

task_module

Returns what Python module to import to get access to this class.

task_namespace = '__not_user_specified'

This value can be overriden to set the namespace that will be used. (See Namespaces, families and ids) If it’s not specified and you try to read this value anyway, it will return garbage. Please use get_task_namespace() to read the namespace.

Note that setting this value with @property will not work, because this is a class level value.

classmethod get_task_namespace()[source]

The task family for the given class.

Note: You normally don’t want to override this.

task_family = 'Task'
classmethod get_task_family()[source]

The task family for the given class.

If task_namespace is not set, then it’s simply the name of the class. Otherwise, <task_namespace>. is prefixed to the class name.

Note: You normally don’t want to override this.

classmethod get_params()[source]

Returns all of the Parameters for this Task.

classmethod batch_param_names()[source]
classmethod get_param_names(include_significant=False)[source]
classmethod get_param_values(params, args, kwargs)[source]

Get the values of the parameters from the args and kwargs.

Parameters:
  • params – list of (param_name, Parameter).
  • args – positional arguments
  • kwargs – keyword arguments.
Returns:

list of (name, value) tuples, one for each parameter.
param_args
initialized()[source]

Returns True if the Task is initialized and False otherwise.

classmethod from_str_params(params_str)[source]

Creates an instance from a str->str hash.

Parameters:params_str – dict of param name -> value as string.
to_str_params(only_significant=False, only_public=False)[source]

Convert all parameters to a str->str hash.

clone(cls=None, **kwargs)[source]

Creates a new instance from an existing instance where some of the args have changed.

There’s at least two scenarios where this is useful (see test/clone_test.py):

  • remove a lot of boiler plate when you have recursive dependencies and lots of args
  • there’s task inheritance and some logic is on the base class
Parameters:
  • cls –
  • kwargs –
Returns:
complete()[source]

If the task has any outputs, return True if all outputs exist. Otherwise, return False.

However, you may freely override this method with custom logic.

classmethod bulk_complete(parameter_tuples)[source]

Returns those of parameter_tuples for which this Task is complete.

Override (with an efficient implementation) for efficient scheduling with range tools. Keep the logic consistent with that of complete().

output()[source]

The output that this Task produces.

The output of the Task determines if the Task needs to be run–the task is considered finished iff the outputs all exist. Subclasses should override this method to return a single Target or a list of Target instances.

Implementation note
If running multiple workers, the output must be a resource that is accessible by all workers, such as a DFS or database. Otherwise, workers might compute the same output since they don’t see the work done by other workers.

See Task.output

requires()[source]

The Tasks that this Task depends on.

A Task will only run if all of the Tasks that it requires are completed. If your Task does not require any other Tasks, then you don’t need to override this method. Otherwise, a Subclasses can override this method to return a single Task, a list of Task instances, or a dict whose values are Task instances.

See Task.requires

process_resources()[source]

Override in “template” tasks which provide common resource functionality but allow subclasses to specify additional resources while preserving the name for consistent end-user experience.

input()[source]

Returns the outputs of the Tasks returned by requires()

See Task.input

Returns:a list of Target objects which are specified as outputs of all required Tasks.
deps()[source]

Internal method used by the scheduler.

Returns the flattened list of requires.

run()[source]

The task run method, to be overridden in a subclass.

See Task.run

on_failure(exception)[source]

Override for custom error handling.

This method gets called if an exception is raised in run(). The returned value of this method is json encoded and sent to the scheduler as the expl argument. Its string representation will be used as the body of the error email sent out if any.

Default behavior is to return a string representation of the stack trace.

on_success()[source]

Override for doing custom completion handling for a larger class of tasks

This method gets called when run() completes without raising any exceptions.

The returned value is json encoded and sent to the scheduler as the expl argument.

Default behavior is to send an None value

no_unpicklable_properties(**kwds)[source]

Remove unpicklable properties before dump task and resume them after.

This method could be called in subtask’s dump method, to ensure unpicklable properties won’t break dump.

This method is a context-manager which can be called as below:

class luigi.Config(*args, **kwargs)[source]

Bases: luigi.task.Task

Class for configuration. See Configuration classes.

class luigi.ExternalTask(*args, **kwargs)[source]

Bases: luigi.task.Task

Subclass for references to external dependencies.

An ExternalTask’s does not have a run implementation, which signifies to the framework that this Task’s output() is generated outside of Luigi.

run = None
class luigi.WrapperTask(*args, **kwargs)[source]

Bases: luigi.task.Task

Use for tasks that only wrap other tasks and that by definition are done if all their requirements exist.

complete()[source]

If the task has any outputs, return True if all outputs exist. Otherwise, return False.

However, you may freely override this method with custom logic.

luigi.namespace(namespace=None, scope='')[source]

Call to set namespace of tasks declared after the call.

It is often desired to call this function with the keyword argument scope=__name__.

The scope keyword makes it so that this call is only effective for task classes with a matching [*] __module__. The default value for scope is the empty string, which means all classes. Multiple calls with the same scope simply replace each other.

The namespace of a Task can also be changed by specifying the property task_namespace.

class Task2(luigi.Task):
    task_namespace = 'namespace2'

This explicit setting takes priority over whatever is set in the namespace() method, and it’s also inherited through normal python inheritence.

There’s no equivalent way to set the task_family.

New since Luigi 2.6.0: scope keyword argument.

[*]When there are multiple levels of matching module scopes like a.b vs a.b.c, the more specific one (a.b.c) wins.

See also

The new and better scaling auto_namespace()

luigi.auto_namespace(scope='')[source]

Same as namespace(), but instead of a constant namespace, it will be set to the __module__ of the task class. This is desirable for these reasons:

  • Two tasks with the same name will not have conflicting task families
  • It’s more pythonic, as modules are Python’s recommended way to do namespacing.
  • It’s traceable. When you see the full name of a task, you can immediately identify where it is defined.

We recommend calling this function from your package’s outermost __init__.py file. The file contents could look like this:

import luigi

luigi.auto_namespace(scope=__name__)

To reset an auto_namespace() call, you can use namespace(scope='my_scope'). But this will not be needed (and is also discouraged) if you use the scope kwarg.

New since Luigi 2.6.0.

class luigi.Target[source]

Bases: object

A Target is a resource generated by a Task.

For example, a Target might correspond to a file in HDFS or data in a database. The Target interface defines one method that must be overridden: exists(), which signifies if the Target has been created or not.

Typically, a Task will define one or more Targets as output, and the Task is considered complete if and only if each of its output Targets exist.

exists()[source]

Returns True if the Target exists and False otherwise.

class luigi.LocalTarget(path=None, format=None, is_tmp=False)[source]

Bases: luigi.target.FileSystemTarget

fs = <luigi.local_target.LocalFileSystem object>
makedirs()[source]

Create all parent folders if they do not exist.

open(mode='r')[source]

Open the FileSystem target.

This method returns a file-like object which can either be read from or written to depending on the specified mode.

Parameters:mode (str) – the mode r opens the FileSystemTarget in read-only mode, whereas w will open the FileSystemTarget in write mode. Subclasses can implement additional options.
move(new_path, raise_if_exists=False)[source]
move_dir(new_path)[source]
remove()[source]

Remove the resource at the path specified by this FileSystemTarget.

This method is implemented by using fs.

copy(new_path, raise_if_exists=False)[source]
fn
class luigi.RemoteScheduler(url='http://localhost:8082/', connect_timeout=None)[source]

Bases: object

Scheduler proxy object. Talks to a RemoteSchedulerResponder.

add_scheduler_message_response(*args, **kwargs)
add_task(*args, **kwargs)
  • add task identified by task_id if it doesn’t exist
  • if deps is not None, update dependency list
  • update status of task
  • add additional workers/stakeholders
  • update priority when needed
add_task_batcher(*args, **kwargs)
add_worker(*args, **kwargs)
announce_scheduling_failure(*args, **kwargs)
count_pending(*args, **kwargs)
decrease_running_task_resources(*args, **kwargs)
dep_graph(*args, **kwargs)
disable_worker(*args, **kwargs)
fetch_error(*args, **kwargs)
forgive_failures(*args, **kwargs)
get_running_task_resources(*args, **kwargs)
get_scheduler_message_response(*args, **kwargs)
get_task_progress_percentage(*args, **kwargs)
get_task_status_message(*args, **kwargs)
get_work(*args, **kwargs)
graph(*args, **kwargs)
inverse_dep_graph(*args, **kwargs)
is_pause_enabled(*args, **kwargs)
is_paused(*args, **kwargs)
pause(*args, **kwargs)
ping(*args, **kwargs)
prune(*args, **kwargs)
re_enable_task(*args, **kwargs)
resource_list(*args, **kwargs)

Resources usage info and their consumers (tasks).

send_scheduler_message(*args, **kwargs)
set_task_progress_percentage(*args, **kwargs)
set_task_status_message(*args, **kwargs)
set_worker_processes(*args, **kwargs)
task_list(*args, **kwargs)

Query for a subset of tasks by status.

Query for a subset of tasks by task_id.

Parameters:task_str –
Returns:
unpause(*args, **kwargs)
update_resource(*args, **kwargs)
update_resources(*args, **kwargs)
worker_list(*args, **kwargs)
exception luigi.RPCError(message, sub_exception=None)[source]

Bases: exceptions.Exception

class luigi.Parameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]

Bases: object

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

  • Any value provided on the command line:
    • To the root task (eg. --param xyz)
    • Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).
  • With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See Parameters from config Ingestion
  • Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

Parameters:
  • default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
  • significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
  • description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
  • config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
  • positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
  • always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
  • batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
  • visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC
has_task_value(task_name, param_name)[source]
task_value(task_name, param_name)[source]
parse(x)[source]

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:x (str) – the value to parse.
Returns:the parsed value.
serialize(x)[source]

Opposite of parse().

Converts the value x to a string.

Parameters:x – the value to serialize.
normalize(x)[source]

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

next_in_enumeration(_value)[source]

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:value – The value
Returns:The next value, like “value + 1”. Or None if there’s no enumerable ordering.
class luigi.DateParameter(interval=1, start=None, **kwargs)[source]

Bases: luigi.parameter._DateParameterBase

Parameter whose value is a date.

A DateParameter is a Date string formatted YYYY-MM-DD. For example, 2013-07-10 specifies July 10, 2013.

DateParameters are 90% of the time used to be interpolated into file system paths or the like. Here is a gentle reminder of how to interpolate date parameters into strings:

class MyTask(luigi.Task):
    date = luigi.DateParameter()

    def run(self):
        templated_path = "/my/path/to/my/dataset/{date:%Y/%m/%d}/"
        instantiated_path = templated_path.format(date=self.date)
        # print(instantiated_path) --> /my/path/to/my/dataset/2016/06/09/
        # ... use instantiated_path ...

To set this parameter to default to the current day. You can write code like this:

import datetime

class MyTask(luigi.Task):
    date = luigi.DateParameter(default=datetime.date.today())
date_format = '%Y-%m-%d'
next_in_enumeration(value)[source]

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:value – The value
Returns:The next value, like “value + 1”. Or None if there’s no enumerable ordering.
normalize(value)[source]

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.MonthParameter(interval=1, start=None, **kwargs)[source]

Bases: luigi.parameter.DateParameter

Parameter whose value is a date, specified to the month (day of date is “rounded” to first of the month).

A MonthParameter is a Date string formatted YYYY-MM. For example, 2013-07 specifies July of 2013.

date_format = '%Y-%m'
next_in_enumeration(value)[source]

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:value – The value
Returns:The next value, like “value + 1”. Or None if there’s no enumerable ordering.
normalize(value)[source]

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.YearParameter(interval=1, start=None, **kwargs)[source]

Bases: luigi.parameter.DateParameter

Parameter whose value is a date, specified to the year (day and month of date is “rounded” to first day of the year).

A YearParameter is a Date string formatted YYYY.

date_format = '%Y'
next_in_enumeration(value)[source]

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:value – The value
Returns:The next value, like “value + 1”. Or None if there’s no enumerable ordering.
normalize(value)[source]

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.DateHourParameter(interval=1, start=None, **kwargs)[source]

Bases: luigi.parameter._DatetimeParameterBase

Parameter whose value is a datetime specified to the hour.

A DateHourParameter is a ISO 8601 formatted date and time specified to the hour. For example, 2013-07-10T19 specifies July 10, 2013 at 19:00.

date_format = '%Y-%m-%dT%H'
class luigi.DateMinuteParameter(interval=1, start=None, **kwargs)[source]

Bases: luigi.parameter._DatetimeParameterBase

Parameter whose value is a datetime specified to the minute.

A DateMinuteParameter is a ISO 8601 formatted date and time specified to the minute. For example, 2013-07-10T1907 specifies July 10, 2013 at 19:07.

The interval parameter can be used to clamp this parameter to every N minutes, instead of every minute.

date_format = '%Y-%m-%dT%H%M'
deprecated_date_format = '%Y-%m-%dT%HH%M'
parse(s)[source]

Parses a string to a datetime.

class luigi.DateSecondParameter(interval=1, start=None, **kwargs)[source]

Bases: luigi.parameter._DatetimeParameterBase

Parameter whose value is a datetime specified to the second.

A DateSecondParameter is a ISO 8601 formatted date and time specified to the second. For example, 2013-07-10T190738 specifies July 10, 2013 at 19:07:38.

The interval parameter can be used to clamp this parameter to every N seconds, instead of every second.

date_format = '%Y-%m-%dT%H%M%S'
class luigi.DateIntervalParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]

Bases: luigi.parameter.Parameter

A Parameter whose value is a DateInterval.

Date Intervals are specified using the ISO 8601 date notation for dates (eg. “2015-11-04”), months (eg. “2015-05”), years (eg. “2015”), or weeks (eg. “2015-W35”). In addition, it also supports arbitrary date intervals provided as two dates separated with a dash (eg. “2015-11-04-2015-12-04”).

Parameters:
  • default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
  • significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
  • description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
  • config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
  • positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
  • always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
  • batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
  • visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC
parse(s)[source]

Parses a DateInterval from the input.

see luigi.date_interval
for details on the parsing of DateIntervals.
class luigi.TimeDeltaParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]

Bases: luigi.parameter.Parameter

Class that maps to timedelta using strings in any of the following forms:

  • n {w[eek[s]]|d[ay[s]]|h[our[s]]|m[inute[s]|s[second[s]]} (e.g. “1 week 2 days” or “1 h”)
    Note: multiple arguments must be supplied in longest to shortest unit order
  • ISO 8601 duration PnDTnHnMnS (each field optional, years and months not supported)
  • ISO 8601 duration PnW

See https://en.wikipedia.org/wiki/ISO_8601#Durations

Parameters:
  • default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
  • significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
  • description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
  • config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
  • positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
  • always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
  • batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
  • visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC
parse(input)[source]

Parses a time delta from the input.

See TimeDeltaParameter for details on supported formats.

serialize(x)[source]

Converts datetime.timedelta to a string

Parameters:x – the value to serialize.
class luigi.IntParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]

Bases: luigi.parameter.Parameter

Parameter whose value is an int.

Parameters:
  • default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
  • significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
  • description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
  • config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
  • positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
  • always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
  • batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
  • visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC
parse(s)[source]

Parses an int from the string using int().

next_in_enumeration(value)[source]

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:value – The value
Returns:The next value, like “value + 1”. Or None if there’s no enumerable ordering.
class luigi.FloatParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]

Bases: luigi.parameter.Parameter

Parameter whose value is a float.

Parameters:
  • default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
  • significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
  • description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
  • config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
  • positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
  • always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
  • batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
  • visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC
parse(s)[source]

Parses a float from the string using float().

class luigi.BoolParameter(*args, **kwargs)[source]

Bases: luigi.parameter.Parameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

IMPLICIT_PARSING = 'implicit'
EXPLICIT_PARSING = 'explicit'
parsing = 'implicit'
parse(val)[source]

Parses a bool from the string, matching ‘true’ or ‘false’ ignoring case.

normalize(value)[source]

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.TaskParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]

Bases: luigi.parameter.Parameter

A parameter that takes another luigi task class.

When used programatically, the parameter should be specified directly with the luigi.task.Task (sub) class. Like MyMetaTask(my_task_param=my_tasks.MyTask). On the command line, you specify the luigi.task.Task.get_task_family(). Like

$ luigi --module my_tasks MyMetaTask --my_task_param my_namespace.MyTask

Where my_namespace.MyTask is defined in the my_tasks python module.

When the luigi.task.Task class is instantiated to an object. The value will always be a task class (and not a string).

Parameters:
  • default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
  • significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
  • description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
  • config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
  • positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
  • always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
  • batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
  • visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC
parse(input)[source]

Parse a task_famly using the Register

serialize(cls)[source]

Converts the luigi.task.Task (sub) class to its family name.

class luigi.ListParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]

Bases: luigi.parameter.Parameter

Parameter whose value is a list.

In the task definition, use

class MyTask(luigi.Task):
  grades = luigi.ListParameter()

    def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

At the command line, use

$ luigi --module my_tasks MyTask --grades <JSON string>

Simple example with two grades:

$ luigi --module my_tasks MyTask --grades '[100,70]'
Parameters:
  • default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
  • significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
  • description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
  • config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
  • positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
  • always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
  • batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
  • visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC
normalize(x)[source]

Ensure that struct is recursively converted to a tuple so it can be hashed.

Parameters:x (str) – the value to parse.
Returns:the normalized (hashable/immutable) value.
parse(x)[source]

Parse an individual value from the input.

Parameters:x (str) – the value to parse.
Returns:the parsed value.
serialize(x)[source]

Opposite of parse().

Converts the value x to a string.

Parameters:x – the value to serialize.
class luigi.TupleParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]

Bases: luigi.parameter.ListParameter

Parameter whose value is a tuple or tuple of tuples.

In the task definition, use

class MyTask(luigi.Task):
  book_locations = luigi.TupleParameter()

    def run(self):
        for location in self.book_locations:
            print("Go to page %d, line %d" % (location[0], location[1]))

At the command line, use

$ luigi --module my_tasks MyTask --book_locations <JSON string>

Simple example with two grades:

$ luigi --module my_tasks MyTask --book_locations '((12,3),(4,15),(52,1))'
Parameters:
  • default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
  • significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
  • description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
  • config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
  • positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
  • always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
  • batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
  • visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC
parse(x)[source]

Parse an individual value from the input.

Parameters:x (str) – the value to parse.
Returns:the parsed value.
class luigi.EnumParameter(*args, **kwargs)[source]

Bases: luigi.parameter.Parameter

A parameter whose value is an Enum.

In the task definition, use

class Model(enum.Enum):
  Honda = 1
  Volvo = 2

class MyTask(luigi.Task):
  my_param = luigi.EnumParameter(enum=Model)

At the command line, use,

$ luigi --module my_tasks MyTask --my-param Honda
parse(s)[source]

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:x (str) – the value to parse.
Returns:the parsed value.
serialize(e)[source]

Opposite of parse().

Converts the value x to a string.

Parameters:x – the value to serialize.
class luigi.DictParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]

Bases: luigi.parameter.Parameter

Parameter whose value is a dict.

In the task definition, use

class MyTask(luigi.Task):
  tags = luigi.DictParameter()

    def run(self):
        logging.info("Find server with role: %s", self.tags['role'])
        server = aws.ec2.find_my_resource(self.tags)

At the command line, use

$ luigi --module my_tasks MyTask --tags <JSON string>

Simple example with two tags:

$ luigi --module my_tasks MyTask --tags '{"role": "web", "env": "staging"}'

It can be used to define dynamic parameters, when you do not know the exact list of your parameters (e.g. list of tags, that are dynamically constructed outside Luigi), or you have a complex parameter containing logically related values (like a database connection config).

Parameters:
  • default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
  • significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
  • description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
  • config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
  • positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
  • always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
  • batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
  • visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC
normalize(value)[source]

Ensure that dictionary parameter is converted to a _FrozenOrderedDict so it can be hashed.

parse(s)[source]

Parses an immutable and ordered dict from a JSON string using standard JSON library.

We need to use an immutable dictionary, to create a hashable parameter and also preserve the internal structure of parsing. The traversal order of standard dict is undefined, which can result various string representations of this parameter, and therefore a different task id for the task containing this parameter. This is because task id contains the hash of parameters’ JSON representation.

Parameters:s – String to be parse
serialize(x)[source]

Opposite of parse().

Converts the value x to a string.

Parameters:x – the value to serialize.
luigi.run(*args, **kwargs)[source]
luigi.build(tasks, worker_scheduler_factory=None, **env_params)[source]

Run internally, bypassing the cmdline parsing.

Useful if you have some luigi code that you want to run internally. Example:

luigi.build([MyTask1(), MyTask2()], local_scheduler=True)

One notable difference is that build defaults to not using the identical process lock. Otherwise, build would only be callable once from each process.

Parameters:
  • tasks –
  • worker_scheduler_factory –
  • env_params –
Returns:

True if there were no scheduling errors, even if tasks may fail.
class luigi.Event[source]

Bases: object

DEPENDENCY_DISCOVERED = 'event.core.dependency.discovered'
DEPENDENCY_MISSING = 'event.core.dependency.missing'
DEPENDENCY_PRESENT = 'event.core.dependency.present'
BROKEN_TASK = 'event.core.task.broken'
START = 'event.core.start'
PROGRESS = 'event.core.progress'

This event can be fired by the task itself while running. The purpose is for the task to report progress, metadata or any generic info so that event handler listening for this can keep track of the progress of running task.

FAILURE = 'event.core.failure'
SUCCESS = 'event.core.success'
PROCESSING_TIME = 'event.core.processing_time'
TIMEOUT = 'event.core.timeout'
PROCESS_FAILURE = 'event.core.process_failure'
class luigi.NumericalParameter(left_op=<built-in function le>, right_op=<built-in function lt>, *args, **kwargs)[source]

Bases: luigi.parameter.Parameter

Parameter whose value is a number of the specified type, e.g. int or float and in the range specified.

In the task definition, use

class MyTask(luigi.Task):
    my_param_1 = luigi.NumericalParameter(
        var_type=int, min_value=-3, max_value=7) # -3 <= my_param_1 < 7
    my_param_2 = luigi.NumericalParameter(
        var_type=int, min_value=-3, max_value=7, left_op=operator.lt, right_op=operator.le) # -3 < my_param_2 <= 7

At the command line, use

$ luigi --module my_tasks MyTask --my-param-1 -3 --my-param-2 -2
Parameters:
  • var_type (function) – The type of the input variable, e.g. int or float.
  • min_value – The minimum value permissible in the accepted values range. May be inclusive or exclusive based on left_op parameter. This should be the same type as var_type.
  • max_value – The maximum value permissible in the accepted values range. May be inclusive or exclusive based on right_op parameter. This should be the same type as var_type.
  • left_op (function) – The comparison operator for the left-most comparison in the expression min_value left_op value right_op value. This operator should generally be either operator.lt or operator.le. Default: operator.le.
  • right_op (function) – The comparison operator for the right-most comparison in the expression min_value left_op value right_op value. This operator should generally be either operator.lt or operator.le. Default: operator.lt.
parse(s)[source]

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:x (str) – the value to parse.
Returns:the parsed value.
class luigi.ChoiceParameter(var_type=<type 'str'>, *args, **kwargs)[source]

Bases: luigi.parameter.Parameter

A parameter which takes two values:
  1. an instance of Iterable and
  2. the class of the variables to convert to.

In the task definition, use

class MyTask(luigi.Task):
    my_param = luigi.ChoiceParameter(choices=[0.1, 0.2, 0.3], var_type=float)

At the command line, use

$ luigi --module my_tasks MyTask --my-param 0.1

Consider using EnumParameter for a typed, structured alternative. This class can perform the same role when all choices are the same type and transparency of parameter value on the command line is desired.

Parameters:
  • var_type (function) – The type of the input variable, e.g. str, int, float, etc. Default: str
  • choices – An iterable, all of whose elements are of var_type to restrict parameter choices to.
parse(s)[source]

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:x (str) – the value to parse.
Returns:the parsed value.
normalize(var)[source]

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.OptionalParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]

Bases: luigi.parameter.Parameter

A Parameter that treats empty string as None

Parameters:
  • default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
  • significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
  • description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
  • config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
  • positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
  • always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
  • batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
  • visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC
serialize(x)[source]

Opposite of parse().

Converts the value x to a string.

Parameters:x – the value to serialize.
parse(x)[source]

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:x (str) – the value to parse.
Returns:the parsed value.