luigi

Package containing core luigi functionality.

class luigi.Task(*args, **kwargs)

This is the base class of all Luigi Tasks, the base unit of work in Luigi.

A Luigi Task describes a unit or work.

The key methods of a Task, which must be implemented in a subclass are:

• run() - the computation done by this task.

• requires() - the list of Tasks that this Task depends on.

• output() - the output Target that this Task creates.

Each Parameter of the Task should be declared as members:

class MyTask(luigi.Task):
    count = luigi.IntParameter()
    second_param = luigi.Parameter()

In addition to any declared properties and methods, there are a few non-declared properties, which are created by the Register metaclass:

priority = 0

Priority of the task: the scheduler should favor available tasks with higher priority values first. See Task priority

disabled = False

resources: Dict[str, Any] = {}

Resources used by the task. Should be formatted like {“scp”: 1} to indicate that the task requires 1 unit of the scp resource.

worker_timeout: int | None = None

Number of seconds after which to time out the run function. No timeout if set to 0. Defaults to 0 or worker-timeout value in config

max_batch_size = inf

Maximum number of tasks to run together as a batch. Infinite by default

property batchable

True if this instance can be run as part of a batch. By default, True if it has any batched parameters

property retry_count

Override this positive integer to have different retry_count at task level Check [scheduler]

property disable_hard_timeout

Override this positive integer to have different disable_hard_timeout at task level. Check [scheduler]

property disable_window

Override this positive integer to have different disable_window at task level. Check [scheduler]

property disable_window_seconds

property owner_email

Override this to send out additional error emails to task owner, in addition to the one defined in the global configuration. This should return a string or a list of strings. e.g. ‘test@exmaple.com’ or [‘test1@example.com’, ‘test2@example.com’]

property use_cmdline_section

Property used by core config such as –workers etc. These will be exposed without the class as prefix.

classmethod event_handler(event)

Decorator for adding event handlers.

classmethod remove_event_handler(event, callback)

Function to remove the event handler registered previously by the cls.event_handler decorator.

trigger_event(event, *args, **kwargs)

Trigger that calls all of the specified events associated with this class.

property accepts_messages

For configuring which scheduler messages can be received. When falsy, this tasks does not accept any message. When True, all messages are accepted.

property task_module

Returns what Python module to import to get access to this class.

task_namespace = '__not_user_specified'

This value can be overridden to set the namespace that will be used. (See Namespaces, families and ids) If it’s not specified and you try to read this value anyway, it will return garbage. Please use get_task_namespace() to read the namespace.

Note that setting this value with @property will not work, because this is a class level value.

classmethod get_task_namespace()

The task family for the given class.

Note: You normally don’t want to override this.

task_family = 'Task'

classmethod get_task_family()

The task family for the given class.

If task_namespace is not set, then it’s simply the name of the class. Otherwise, <task_namespace>. is prefixed to the class name.

Note: You normally don’t want to override this.

classmethod get_params()

Returns all of the Parameters for this Task.

classmethod batch_param_names()

classmethod get_param_names(include_significant=False)

classmethod get_param_values(params, args, kwargs)

Get the values of the parameters from the args and kwargs.

Parameters:

• params – list of (param_name, Parameter).

• args – positional arguments

• kwargs – keyword arguments.

Returns:

list of (name, value) tuples, one for each parameter.

property param_args

initialized()

Returns True if the Task is initialized and False otherwise.

classmethod from_str_params(params_str)

Creates an instance from a str->str hash.

Parameters:

params_str – dict of param name -> value as string.

to_str_params(only_significant=False, only_public=False)

Convert all parameters to a str->str hash.

clone(cls=None, **kwargs)

Creates a new instance from an existing instance where some of the args have changed.

There’s at least two scenarios where this is useful (see test/clone_test.py):

• remove a lot of boiler plate when you have recursive dependencies and lots of args

• there’s task inheritance and some logic is on the base class

Parameters:

• cls

• kwargs

Returns:

complete()

If the task has any outputs, return True if all outputs exist. Otherwise, return False.

However, you may freely override this method with custom logic.

classmethod bulk_complete(parameter_tuples)

Returns those of parameter_tuples for which this Task is complete.

Override (with an efficient implementation) for efficient scheduling with range tools. Keep the logic consistent with that of complete().

output()

The output that this Task produces.

The output of the Task determines if the Task needs to be run–the task is considered finished iff the outputs all exist. Subclasses should override this method to return a single Target or a list of Target instances.

Implementation note

If running multiple workers, the output must be a resource that is accessible by all workers, such as a DFS or database. Otherwise, workers might compute the same output since they don’t see the work done by other workers.

See Task.output

requires()

The Tasks that this Task depends on.

A Task will only run if all of the Tasks that it requires are completed. If your Task does not require any other Tasks, then you don’t need to override this method. Otherwise, a subclass can override this method to return a single Task, a list of Task instances, or a dict whose values are Task instances.

See Task.requires

process_resources()

Override in “template” tasks which provide common resource functionality but allow subclasses to specify additional resources while preserving the name for consistent end-user experience.

input()

Returns the outputs of the Tasks returned by requires()

See Task.input

Returns:

a list of Target objects which are specified as outputs of all required Tasks.

deps()

Internal method used by the scheduler.

Returns the flattened list of requires.

run()

The task run method, to be overridden in a subclass.

See Task.run

on_failure(exception)

Override for custom error handling.

This method gets called if an exception is raised in run(). The returned value of this method is json encoded and sent to the scheduler as the expl argument. Its string representation will be used as the body of the error email sent out if any.

Default behavior is to return a string representation of the stack trace.

on_success()

Override for doing custom completion handling for a larger class of tasks

This method gets called when run() completes without raising any exceptions.

The returned value is json encoded and sent to the scheduler as the expl argument.

Default behavior is to send an None value

no_unpicklable_properties()

Remove unpicklable properties before dump task and resume them after.

This method could be called in subtask’s dump method, to ensure unpicklable properties won’t break dump.

This method is a context-manager which can be called as below:

class luigi.Config(*args, **kwargs)

Class for configuration. See Configuration classes.

class luigi.ExternalTask(*args, **kwargs)

Subclass for references to external dependencies.

An ExternalTask’s does not have a run implementation, which signifies to the framework that this Task’s output() is generated outside of Luigi.

run = None

class luigi.WrapperTask(*args, **kwargs)

Use for tasks that only wrap other tasks and that by definition are done if all their requirements exist.

complete()

If the task has any outputs, return True if all outputs exist. Otherwise, return False.

However, you may freely override this method with custom logic.

luigi.namespace(namespace=None, scope='')

Call to set namespace of tasks declared after the call.

It is often desired to call this function with the keyword argument scope=__name__.

The scope keyword makes it so that this call is only effective for task classes with a matching [*] __module__. The default value for scope is the empty string, which means all classes. Multiple calls with the same scope simply replace each other.

The namespace of a Task can also be changed by specifying the property task_namespace.

class Task2(luigi.Task):
    task_namespace = 'namespace2'

This explicit setting takes priority over whatever is set in the namespace() method, and it’s also inherited through normal python inheritence.

There’s no equivalent way to set the task_family.

New since Luigi 2.6.0: scope keyword argument.

[*]

When there are multiple levels of matching module scopes like a.b vs a.b.c, the more specific one (a.b.c) wins.

See also

The new and better scaling auto_namespace()

luigi.auto_namespace(scope='')

Same as namespace(), but instead of a constant namespace, it will be set to the __module__ of the task class. This is desirable for these reasons:

• Two tasks with the same name will not have conflicting task families

• It’s more pythonic, as modules are Python’s recommended way to do namespacing.

• It’s traceable. When you see the full name of a task, you can immediately identify where it is defined.

We recommend calling this function from your package’s outermost __init__.py file. The file contents could look like this:

import luigi

luigi.auto_namespace(scope=__name__)

To reset an auto_namespace() call, you can use namespace(scope='my_scope'). But this will not be needed (and is also discouraged) if you use the scope kwarg.

New since Luigi 2.6.0.

class luigi.DynamicRequirements(requirements, custom_complete=None)

Wraps dynamic requirements yielded in tasks’s run methods to control how completeness checks of (e.g.) large chunks of tasks are performed. Besides the wrapped requirements, instances of this class can be passed an optional function custom_complete that might implement an optimized check for completeness. If set, the function will be called with a single argument, complete_fn, which should be used to perform the per-task check. Example:

class SomeTaskWithDynamicRequirements(luigi.Task):
    ...

    def run(self):
        large_chunk_of_tasks = [OtherTask(i=i) for i in range(10000)]

        def custom_complete(complete_fn):
            # example: assume OtherTask always write into the same directory, so just check
            #          if the first task is complete, and compare basenames for the rest
            if not complete_fn(large_chunk_of_tasks[0]):
                return False
            paths = [task.output().path for task in large_chunk_of_tasks]
            basenames = os.listdir(os.path.dirname(paths[0]))  # a single fs call
            return all(os.path.basename(path) in basenames for path in paths)

        yield DynamicRequirements(large_chunk_of_tasks, custom_complete)

requirements

The original, wrapped requirements.

custom_complete

The optional, custom function performing the completeness check of the wrapped requirements.

property flat_requirements

property paths

complete(complete_fn=None)

class luigi.Target

A Target is a resource generated by a Task.

For example, a Target might correspond to a file in HDFS or data in a database. The Target interface defines one method that must be overridden: exists(), which signifies if the Target has been created or not.

Typically, a Task will define one or more Targets as output, and the Task is considered complete if and only if each of its output Targets exist.

abstractmethod exists()

Returns True if the Target exists and False otherwise.

class luigi.LocalTarget(path=None, format=None, is_tmp=False)

Initializes a FileSystemTarget instance.

Parameters:

path – the path associated with this FileSystemTarget.

fs = <luigi.local_target.LocalFileSystem object>

makedirs()

Create all parent folders if they do not exist.

open(mode='r')

Open the FileSystem target.

This method returns a file-like object which can either be read from or written to depending on the specified mode.

Parameters:

mode (str) – the mode r opens the FileSystemTarget in read-only mode, whereas w will open the FileSystemTarget in write mode. Subclasses can implement additional options. Using b is not supported; initialize with format=Nop instead.

move(new_path, raise_if_exists=False)

move_dir(new_path)

remove()

Remove the resource at the path specified by this FileSystemTarget.

This method is implemented by using fs.

copy(new_path, raise_if_exists=False)

property fn

class luigi.RemoteScheduler(url='http://localhost:8082/', connect_timeout=None)

Scheduler proxy object. Talks to a RemoteSchedulerResponder.

close()

add_scheduler_message_response(task_id, message_id, response)

add_task(task_id=None, status='PENDING', runnable=True, deps=None, new_deps=None, expl=None, resources=None, priority=0, family='', module=None, params=None, param_visibilities=None, accepts_messages=False, assistant=False, tracking_url=None, worker=None, batchable=None, batch_id=None, retry_policy_dict=None, owners=None, **kwargs)

• add task identified by task_id if it doesn’t exist

• if deps is not None, update dependency list

• update status of task

• add additional workers/stakeholders

• update priority when needed

add_task_batcher(worker, task_family, batched_args, max_batch_size=inf)

add_worker(worker, info, **kwargs)

announce_scheduling_failure(task_name, family, params, expl, owners, **kwargs)

count_pending(worker)

decrease_running_task_resources(task_id, decrease_resources)

dep_graph(task_id, include_done=True, **kwargs)

disable_worker(worker)

fetch_error(task_id, **kwargs)

forgive_failures(task_id=None)

get_running_task_resources(task_id)

get_scheduler_message_response(task_id, message_id)

get_task_progress_percentage(task_id)

get_task_status_message(task_id)

get_work(host=None, assistant=False, current_tasks=None, worker=None, **kwargs)

graph(**kwargs)

has_task_history()

inverse_dep_graph(task_id, include_done=True, **kwargs)

is_pause_enabled()

is_paused()

mark_as_done(task_id=None)

pause()

ping(**kwargs)

prune()

re_enable_task(task_id)

report_task_statistics(task_id, statistics)

resource_list()

Resources usage info and their consumers (tasks).

send_scheduler_message(worker, task, content)

set_task_progress_percentage(task_id, progress_percentage)

set_task_status_message(task_id, status_message)

set_worker_processes(worker, n)

task_list(status='', upstream_status='', limit=True, search=None, max_shown_tasks=None, **kwargs)

Query for a subset of tasks by status.

task_search(task_str, **kwargs)

Query for a subset of tasks by task_id.

Parameters:

task_str

Returns:

unpause()

update_metrics_task_started(task)

update_resource(resource, amount)

update_resources(**resources)

worker_list(include_running=True, **kwargs)

exception luigi.RPCError(message, sub_exception=None)

class luigi.Parameter(default: ~luigi.parameter.T | ~luigi.parameter._NoValueType = <no_value>, is_global: bool = False, significant: bool = True, description: str | None = None, config_path: ~luigi.parameter.ConfigPath | None = None, positional: bool = True, always_in_help: bool = False, batch_method: ~typing.Callable[[~typing.Iterable[~typing.Any]], ~typing.Any] | None = None, visibility: ~luigi.parameter.ParameterVisibility = ParameterVisibility.PUBLIC)

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

• Any value provided on the command line:

  • To the root task (eg. --param xyz)

  • Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).

• With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See Parameters from config Ingestion

• Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

has_task_value(task_name, param_name)

task_value(task_name, param_name)

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

normalize(x)

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

next_in_enumeration(value)

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:

value – The value

Returns:

The next value, like “value + 1”. Or None if there’s no enumerable ordering.

class luigi.DateParameter(default: ~datetime.date | ~luigi.parameter._NoValueType = <no_value>, interval: int = 1, start: ~datetime.date | None = None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

Parameter whose value is a date.

A DateParameter is a Date string formatted YYYY-MM-DD. For example, 2013-07-10 specifies July 10, 2013.

DateParameters are 90% of the time used to be interpolated into file system paths or the like. Here is a gentle reminder of how to interpolate date parameters into strings:

class MyTask(luigi.Task):
    date = luigi.DateParameter()

    def run(self):
        templated_path = "/my/path/to/my/dataset/{date:%Y/%m/%d}/"
        instantiated_path = templated_path.format(date=self.date)
        # print(instantiated_path) --> /my/path/to/my/dataset/2016/06/09/
        # ... use instantiated_path ...

To set this parameter to default to the current day. You can write code like this:

import datetime

class MyTask(luigi.Task):
    date = luigi.DateParameter(default=datetime.date.today())

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

date_format = '%Y-%m-%d'

next_in_enumeration(value)

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:

value – The value

Returns:

The next value, like “value + 1”. Or None if there’s no enumerable ordering.

normalize(x)

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.MonthParameter(default: ~datetime.date | ~luigi.parameter._NoValueType = <no_value>, interval: int = 1, start: ~datetime.date | None = None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

Parameter whose value is a date, specified to the month (day of date is “rounded” to first of the month).

A MonthParameter is a Date string formatted YYYY-MM. For example, 2013-07 specifies July of 2013. Task objects constructed from code accept date (ignoring the day value) or Month.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

date_format = '%Y-%m'

next_in_enumeration(value)

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:

value – The value

Returns:

The next value, like “value + 1”. Or None if there’s no enumerable ordering.

normalize(x)

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.YearParameter(default: ~datetime.date | ~luigi.parameter._NoValueType = <no_value>, interval: int = 1, start: ~datetime.date | None = None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

Parameter whose value is a date, specified to the year (day and month of date is “rounded” to first day of the year).

A YearParameter is a Date string formatted YYYY. Task objects constructed from code accept date (ignoring the month and day values) or Year.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

date_format = '%Y'

next_in_enumeration(value)

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:

value – The value

Returns:

The next value, like “value + 1”. Or None if there’s no enumerable ordering.

normalize(x)

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.DateHourParameter(default: ~datetime.datetime | ~luigi.parameter._NoValueType = <no_value>, interval: int = 1, start: ~datetime.datetime | None = None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

Parameter whose value is a datetime specified to the hour.

A DateHourParameter is a ISO 8601 formatted date and time specified to the hour. For example, 2013-07-10T19 specifies July 10, 2013 at 19:00.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

date_format = '%Y-%m-%dT%H'

class luigi.DateMinuteParameter(default: ~datetime.datetime | ~luigi.parameter._NoValueType = <no_value>, interval: int = 1, start: ~datetime.datetime | None = None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

Parameter whose value is a datetime specified to the minute.

A DateMinuteParameter is a ISO 8601 formatted date and time specified to the minute. For example, 2013-07-10T1907 specifies July 10, 2013 at 19:07.

The interval parameter can be used to clamp this parameter to every N minutes, instead of every minute.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

date_format = '%Y-%m-%dT%H%M'

deprecated_date_format = '%Y-%m-%dT%HH%M'

parse(x)

Parses a string to a datetime.

class luigi.DateSecondParameter(default: ~datetime.datetime | ~luigi.parameter._NoValueType = <no_value>, interval: int = 1, start: ~datetime.datetime | None = None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

Parameter whose value is a datetime specified to the second.

A DateSecondParameter is a ISO 8601 formatted date and time specified to the second. For example, 2013-07-10T190738 specifies July 10, 2013 at 19:07:38.

The interval parameter can be used to clamp this parameter to every N seconds, instead of every second.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

date_format = '%Y-%m-%dT%H%M%S'

class luigi.DateIntervalParameter(default: ~luigi.parameter.T | ~luigi.parameter._NoValueType = <no_value>, is_global: bool = False, significant: bool = True, description: str | None = None, config_path: ~luigi.parameter.ConfigPath | None = None, positional: bool = True, always_in_help: bool = False, batch_method: ~typing.Callable[[~typing.Iterable[~typing.Any]], ~typing.Any] | None = None, visibility: ~luigi.parameter.ParameterVisibility = ParameterVisibility.PUBLIC)

A Parameter whose value is a DateInterval.

Date Intervals are specified using the ISO 8601 date notation for dates (eg. “2015-11-04”), months (eg. “2015-05”), years (eg. “2015”), or weeks (eg. “2015-W35”). In addition, it also supports arbitrary date intervals provided as two dates separated with a dash (eg. “2015-11-04-2015-12-04”).

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(x)

Parses a DateInterval from the input.

see luigi.date_interval

for details on the parsing of DateIntervals.

class luigi.TimeDeltaParameter(default: ~luigi.parameter.T | ~luigi.parameter._NoValueType = <no_value>, is_global: bool = False, significant: bool = True, description: str | None = None, config_path: ~luigi.parameter.ConfigPath | None = None, positional: bool = True, always_in_help: bool = False, batch_method: ~typing.Callable[[~typing.Iterable[~typing.Any]], ~typing.Any] | None = None, visibility: ~luigi.parameter.ParameterVisibility = ParameterVisibility.PUBLIC)

Class that maps to timedelta using strings in any of the following forms:

• A bare number is interpreted as duration in seconds.

• n {w[eek[s]]|d[ay[s]]|h[our[s]]|m[inute[s]|s[second[s]]} (e.g. “1 week 2 days” or “1 h”)

  Note: multiple arguments must be supplied in longest to shortest unit order

• ISO 8601 duration PnDTnHnMnS (each field optional, years and months not supported)

• ISO 8601 duration PnW

See https://en.wikipedia.org/wiki/ISO_8601#Durations

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(x)

Parses a time delta from the input.

See TimeDeltaParameter for details on supported formats.

serialize(x)

Converts datetime.timedelta to a string

Parameters:

x – the value to serialize.

class luigi.StrParameter(default: ~luigi.parameter.T | ~luigi.parameter._NoValueType = <no_value>, is_global: bool = False, significant: bool = True, description: str | None = None, config_path: ~luigi.parameter.ConfigPath | None = None, positional: bool = True, always_in_help: bool = False, batch_method: ~typing.Callable[[~typing.Iterable[~typing.Any]], ~typing.Any] | None = None, visibility: ~luigi.parameter.ParameterVisibility = ParameterVisibility.PUBLIC)

Parameter whose value is a str.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

class luigi.IntParameter(default: ~luigi.parameter.T | ~luigi.parameter._NoValueType = <no_value>, is_global: bool = False, significant: bool = True, description: str | None = None, config_path: ~luigi.parameter.ConfigPath | None = None, positional: bool = True, always_in_help: bool = False, batch_method: ~typing.Callable[[~typing.Iterable[~typing.Any]], ~typing.Any] | None = None, visibility: ~luigi.parameter.ParameterVisibility = ParameterVisibility.PUBLIC)

Parameter whose value is an int.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(x)

Parses an int from the string using int().

next_in_enumeration(value)

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:

value – The value

Returns:

The next value, like “value + 1”. Or None if there’s no enumerable ordering.

class luigi.FloatParameter(default: ~luigi.parameter.T | ~luigi.parameter._NoValueType = <no_value>, is_global: bool = False, significant: bool = True, description: str | None = None, config_path: ~luigi.parameter.ConfigPath | None = None, positional: bool = True, always_in_help: bool = False, batch_method: ~typing.Callable[[~typing.Iterable[~typing.Any]], ~typing.Any] | None = None, visibility: ~luigi.parameter.ParameterVisibility = ParameterVisibility.PUBLIC)

Parameter whose value is a float.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(x)

Parses a float from the string using float().

class luigi.BoolParameter(default: bool | ~luigi.parameter._NoValueType = <no_value>, parsing: str | None = None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

IMPLICIT_PARSING = 'implicit'

EXPLICIT_PARSING = 'explicit'

parsing = 'implicit'

parse(x)

Parses a bool from the string, matching ‘true’ or ‘false’ ignoring case.

normalize(x)

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.PathParameter(default: ~pathlib.Path | ~luigi.parameter._NoValueType = <no_value>, *, absolute: bool = False, exists: bool = False, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

Parameter whose value is a path.

In the task definition, use

class MyTask(luigi.Task):
    existing_file_path = luigi.PathParameter(exists=True)
    new_file_path = luigi.PathParameter()

    def run(self):
        # Get data from existing file
        with self.existing_file_path.open("r", encoding="utf-8") as f:
            data = f.read()

        # Output message in new file
        self.new_file_path.parent.mkdir(parents=True, exist_ok=True)
        with self.new_file_path.open("w", encoding="utf-8") as f:
            f.write("hello from a PathParameter => ")
            f.write(data)

At the command line, use

$ luigi --module my_tasks MyTask --existing-file-path <path> --new-file-path <path>

Parameters:

• absolute (bool) – If set to True, the given path is converted to an absolute path.

• exists (bool) – If set to True, a ValueError is raised if the path does not exist.

normalize(x)

Normalize the given value to a pathlib.Path object.

class luigi.TaskParameter(default: ~luigi.parameter.T | ~luigi.parameter._NoValueType = <no_value>, is_global: bool = False, significant: bool = True, description: str | None = None, config_path: ~luigi.parameter.ConfigPath | None = None, positional: bool = True, always_in_help: bool = False, batch_method: ~typing.Callable[[~typing.Iterable[~typing.Any]], ~typing.Any] | None = None, visibility: ~luigi.parameter.ParameterVisibility = ParameterVisibility.PUBLIC)

A parameter that takes another luigi task class.

When used programatically, the parameter should be specified directly with the luigi.task.Task (sub) class. Like MyMetaTask(my_task_param=my_tasks.MyTask). On the command line, you specify the luigi.task.Task.get_task_family(). Like

$ luigi --module my_tasks MyMetaTask --my_task_param my_namespace.MyTask

Where my_namespace.MyTask is defined in the my_tasks python module.

When the luigi.task.Task class is instantiated to an object. The value will always be a task class (and not a string).

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(x)

Parse a task_famly using the Register

serialize(x)

Converts the luigi.task.Task (sub) class to its family name.

class luigi.ListParameter(default: ~luigi.parameter.ListT | ~luigi.parameter._NoValueType = <no_value>, *, schema=None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

Parameter whose value is a list.

In the task definition, use

class MyTask(luigi.Task):
  grades = luigi.ListParameter()

    def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

At the command line, use

$ luigi --module my_tasks MyTask --grades <JSON string>

Simple example with two grades:

$ luigi --module my_tasks MyTask --grades '[100,70]'

It is possible to provide a JSON schema that should be validated by the given value:

class MyTask(luigi.Task):
  grades = luigi.ListParameter(
    schema={
      "type": "array",
      "items": {
        "type": "number",
        "minimum": 0,
        "maximum": 10
      },
      "minItems": 1
    }
  )

  def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

Using this schema, the following command will work:

$ luigi --module my_tasks MyTask --numbers '[1, 8.7, 6]'

while these commands will fail because the parameter is not valid:

$ luigi --module my_tasks MyTask --numbers '[]'  # must have at least 1 element
$ luigi --module my_tasks MyTask --numbers '[-999, 999]'  # elements must be in [0, 10]

Finally, the provided schema can be a custom validator:

custom_validator = jsonschema.Draft4Validator(
  schema={
    "type": "array",
    "items": {
      "type": "number",
      "minimum": 0,
      "maximum": 10
    },
    "minItems": 1
  }
)

class MyTask(luigi.Task):
  grades = luigi.ListParameter(schema=custom_validator)

  def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

normalize(x)

Ensure that struct is recursively converted to a tuple so it can be hashed.

Parameters:

x (str) – the value to parse.

Returns:

the normalized (hashable/immutable) value.

parse(x)

Parse an individual value from the input.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

class luigi.TupleParameter(default: ~luigi.parameter.ListT | ~luigi.parameter._NoValueType = <no_value>, *, schema=None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

Parameter whose value is a tuple or tuple of tuples.

In the task definition, use

class MyTask(luigi.Task):
  book_locations = luigi.TupleParameter()

    def run(self):
        for location in self.book_locations:
            print("Go to page %d, line %d" % (location[0], location[1]))

At the command line, use

$ luigi --module my_tasks MyTask --book_locations <JSON string>

Simple example with two grades:

$ luigi --module my_tasks MyTask --book_locations '((12,3),(4,15),(52,1))'

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(x)

Parse an individual value from the input.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

class luigi.EnumParameter(default: ~luigi.parameter.EnumParameterType | ~luigi.parameter._NoValueType = <no_value>, *, enum: ~typing.Type[~luigi.parameter.EnumParameterType] | None = None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

A parameter whose value is an Enum.

In the task definition, use

class Model(enum.Enum):
  Honda = 1
  Volvo = 2

class MyTask(luigi.Task):
  my_param = luigi.EnumParameter(enum=Model)

At the command line, use,

$ luigi --module my_tasks MyTask --my-param Honda

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

class luigi.DictParameter(default: ~luigi.parameter.DictT | ~luigi.parameter._NoValueType = <no_value>, *, schema=None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

Parameter whose value is a dict.

In the task definition, use

class MyTask(luigi.Task):
  tags = luigi.DictParameter()

    def run(self):
        logging.info("Find server with role: %s", self.tags['role'])
        server = aws.ec2.find_my_resource(self.tags)

At the command line, use

$ luigi --module my_tasks MyTask --tags <JSON string>

Simple example with two tags:

$ luigi --module my_tasks MyTask --tags '{"role": "web", "env": "staging"}'

It can be used to define dynamic parameters, when you do not know the exact list of your parameters (e.g. list of tags, that are dynamically constructed outside Luigi), or you have a complex parameter containing logically related values (like a database connection config).

It is possible to provide a JSON schema that should be validated by the given value:

class MyTask(luigi.Task):
  tags = luigi.DictParameter(
    schema={
      "type": "object",
      "patternProperties": {
        ".*": {"type": "string", "enum": ["web", "staging"]},
      }
    }
  )

  def run(self):
    logging.info("Find server with role: %s", self.tags['role'])
    server = aws.ec2.find_my_resource(self.tags)

Using this schema, the following command will work:

$ luigi --module my_tasks MyTask --tags '{"role": "web", "env": "staging"}'

while this command will fail because the parameter is not valid:

$ luigi --module my_tasks MyTask --tags '{"role": "UNKNOWN_VALUE", "env": "staging"}'

Finally, the provided schema can be a custom validator:

custom_validator = jsonschema.Draft4Validator(
  schema={
    "type": "object",
    "patternProperties": {
      ".*": {"type": "string", "enum": ["web", "staging"]},
    }
  }
)

class MyTask(luigi.Task):
  tags = luigi.DictParameter(schema=custom_validator)

  def run(self):
    logging.info("Find server with role: %s", self.tags['role'])
    server = aws.ec2.find_my_resource(self.tags)

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

normalize(x)

Ensure that dictionary parameter is converted to a FrozenOrderedDict so it can be hashed.

parse(x)

Parses an immutable and ordered dict from a JSON string using standard JSON library.

We need to use an immutable dictionary, to create a hashable parameter and also preserve the internal structure of parsing. The traversal order of standard dict is undefined, which can result various string representations of this parameter, and therefore a different task id for the task containing this parameter. This is because task id contains the hash of parameters’ JSON representation.

Parameters:

s – String to be parse

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

class luigi.EnumListParameter(default: ~typing.Tuple[~luigi.parameter.EnumParameterType, ...] | ~luigi.parameter._NoValueType = <no_value>, *, enum: ~typing.Type[~luigi.parameter.EnumParameterType] | None = None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

A parameter whose value is a comma-separated list of Enum. Values should come from the same enum.

Values are taken to be a list, i.e. order is preserved, duplicates may occur, and empty list is possible.

In the task definition, use

class Model(enum.Enum):
  Honda = 1
  Volvo = 2

class MyTask(luigi.Task):
  my_param = luigi.EnumListParameter(enum=Model)

At the command line, use,

$ luigi --module my_tasks MyTask --my-param Honda,Volvo

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

luigi.run(*args, **kwargs)

Please dont use. Instead use luigi binary.

Run from cmdline using argparse.

Parameters:

use_dynamic_argparse – Deprecated and ignored

luigi.build(tasks, worker_scheduler_factory=None, detailed_summary=False, **env_params)

Run internally, bypassing the cmdline parsing.

Useful if you have some luigi code that you want to run internally. Example:

luigi.build([MyTask1(), MyTask2()], local_scheduler=True)

One notable difference is that build defaults to not using the identical process lock. Otherwise, build would only be callable once from each process.

Parameters:

• tasks

• worker_scheduler_factory

• env_params

Returns:

True if there were no scheduling errors, even if tasks may fail.

class luigi.Event

DEPENDENCY_DISCOVERED = 'event.core.dependency.discovered'

DEPENDENCY_MISSING = 'event.core.dependency.missing'

DEPENDENCY_PRESENT = 'event.core.dependency.present'

BROKEN_TASK = 'event.core.task.broken'

START = 'event.core.start'

PROGRESS = 'event.core.progress'

This event can be fired by the task itself while running. The purpose is for the task to report progress, metadata or any generic info so that event handler listening for this can keep track of the progress of running task.

FAILURE = 'event.core.failure'

SUCCESS = 'event.core.success'

PROCESSING_TIME = 'event.core.processing_time'

TIMEOUT = 'event.core.timeout'

PROCESS_FAILURE = 'event.core.process_failure'

class luigi.NumericalParameter(default: ~luigi.parameter.NumericalType | ~luigi.parameter._NoValueType = <no_value>, *, var_type: ~typing.Type[~luigi.parameter.NumericalType] | None = None, min_value: ~luigi.parameter.NumericalType | None = None, max_value: ~luigi.parameter.NumericalType | None = None, left_op=<built-in function le>, right_op=<built-in function lt>, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

Parameter whose value is a number of the specified type, e.g. int or float and in the range specified.

In the task definition, use

class MyTask(luigi.Task):
    my_param_1 = luigi.NumericalParameter(
        var_type=int, min_value=-3, max_value=7) # -3 <= my_param_1 < 7
    my_param_2 = luigi.NumericalParameter(
        var_type=int, min_value=-3, max_value=7, left_op=operator.lt, right_op=operator.le) # -3 < my_param_2 <= 7

At the command line, use

$ luigi --module my_tasks MyTask --my-param-1 -3 --my-param-2 -2

Parameters:

• var_type (function) – The type of the input variable, e.g. int or float.

• min_value – The minimum value permissible in the accepted values range. May be inclusive or exclusive based on left_op parameter. This should be the same type as var_type.

• max_value – The maximum value permissible in the accepted values range. May be inclusive or exclusive based on right_op parameter. This should be the same type as var_type.

• left_op (function) – The comparison operator for the left-most comparison in the expression min_value left_op value right_op value. This operator should generally be either operator.lt or operator.le. Default: operator.le.

• right_op (function) – The comparison operator for the right-most comparison in the expression min_value left_op value right_op value. This operator should generally be either operator.lt or operator.le. Default: operator.lt.

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

class luigi.ChoiceParameter(default: ~luigi.parameter.ChoiceType | ~luigi.parameter._NoValueType = <no_value>, *, choices: ~typing.Sequence[~luigi.parameter.ChoiceType] | None = None, var_type: ~typing.Type[~luigi.parameter.ChoiceType] = <class 'str'>, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

A parameter which takes two values:

1. an instance of Iterable and

2. the class of the variables to convert to.

In the task definition, use

class MyTask(luigi.Task):
    my_param = luigi.ChoiceParameter(choices=[0.1, 0.2, 0.3], var_type=float)

At the command line, use

$ luigi --module my_tasks MyTask --my-param 0.1

Consider using EnumParameter for a typed, structured alternative. This class can perform the same role when all choices are the same type and transparency of parameter value on the command line is desired.

Parameters:

• var_type (function) – The type of the input variable, e.g. str, int, float, etc. Default: str

• choices – An iterable, all of whose elements are of var_type to restrict parameter choices to.

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

normalize(x)

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.ChoiceListParameter(default: ~typing.Tuple[~luigi.parameter.ChoiceType, ...] | ~luigi.parameter._NoValueType = <no_value>, var_type: ~typing.Type[~luigi.parameter.ChoiceType] = <class 'str'>, choices: ~typing.Sequence[~luigi.parameter.ChoiceType] | None = None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

A parameter which takes two values:

1. an instance of Iterable and

2. the class of the variables to convert to.

Values are taken to be a list, i.e. order is preserved, duplicates may occur, and empty list is possible.

In the task definition, use

class MyTask(luigi.Task):
    my_param = luigi.ChoiceListParameter(choices=['foo', 'bar', 'baz'], var_type=str)

At the command line, use

$ luigi --module my_tasks MyTask --my-param foo,bar

Consider using EnumListParameter for a typed, structured alternative. This class can perform the same role when all choices are the same type and transparency of parameter value on the command line is desired.

Parameters:

• var_type (function) – The type of the input variable, e.g. str, int, float, etc. Default: str

• choices – An iterable, all of whose elements are of var_type to restrict parameter choices to.

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:

x (str) – the value to parse.

Returns:

the parsed value.

normalize(x)

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:

x – the value to serialize.

class luigi.OptionalParameter(default: _OptT | None | _NoValueType = None, **kwargs: Unpack[_ParameterKwargs])

Class to parse optional parameters.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

expected_type

alias of str

class luigi.OptionalStrParameter(default: _OptT | None | _NoValueType = None, **kwargs: Unpack[_ParameterKwargs])

Class to parse optional str parameters.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

expected_type

alias of str

class luigi.OptionalIntParameter(default: _OptT | None | _NoValueType = None, **kwargs: Unpack[_ParameterKwargs])

Class to parse optional int parameters.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

expected_type

alias of int

class luigi.OptionalFloatParameter(default: _OptT | None | _NoValueType = None, **kwargs: Unpack[_ParameterKwargs])

Class to parse optional float parameters.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

expected_type

alias of float

class luigi.OptionalBoolParameter(default: _OptT | None | _NoValueType = None, **kwargs: Unpack[_ParameterKwargs])

Class to parse optional bool parameters.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

expected_type

alias of bool

class luigi.OptionalPathParameter(default: _OptT | None | _NoValueType = None, **kwargs: Unpack[_ParameterKwargs])

Class to parse optional path parameters.

Parameters:

• absolute (bool) – If set to True, the given path is converted to an absolute path.

• exists (bool) – If set to True, a ValueError is raised if the path does not exist.

expected_type: type = (<class 'str'>, <class 'pathlib._local.Path'>)

class luigi.OptionalDictParameter(default: _OptT | None | _NoValueType = None, **kwargs: Unpack[_ParameterKwargs])

Class to parse optional dict parameters.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

expected_type

alias of FrozenOrderedDict

class luigi.OptionalListParameter(default: _OptT | None | _NoValueType = None, **kwargs: Unpack[_ParameterKwargs])

Class to parse optional list parameters.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

expected_type

alias of tuple

class luigi.OptionalTupleParameter(default: _OptT | None | _NoValueType = None, **kwargs: Unpack[_ParameterKwargs])

Class to parse optional tuple parameters.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.

• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.

• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.

• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.

• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.

• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.

• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

• visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

expected_type

alias of tuple

class luigi.OptionalChoiceParameter(default: ~luigi.parameter.ChoiceType | None | ~luigi.parameter._NoValueType = <no_value>, var_type: ~typing.Type[~luigi.parameter.ChoiceType] = <class 'str'>, choices: ~typing.Sequence[~luigi.parameter.ChoiceType] | None = None, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

Class to parse optional choice parameters.

Parameters:

• var_type (function) – The type of the input variable, e.g. str, int, float, etc. Default: str

• choices – An iterable, all of whose elements are of var_type to restrict parameter choices to.

class luigi.OptionalNumericalParameter(default: ~luigi.parameter.NumericalType | None | ~luigi.parameter._NoValueType = <no_value>, **kwargs: ~typing.Unpack[~luigi.parameter._ParameterKwargs])

Class to parse optional numerical parameters.

Parameters:

• var_type (function) – The type of the input variable, e.g. int or float.

• min_value – The minimum value permissible in the accepted values range. May be inclusive or exclusive based on left_op parameter. This should be the same type as var_type.

• max_value – The maximum value permissible in the accepted values range. May be inclusive or exclusive based on right_op parameter. This should be the same type as var_type.

• left_op (function) – The comparison operator for the left-most comparison in the expression min_value left_op value right_op value. This operator should generally be either operator.lt or operator.le. Default: operator.le.

• right_op (function) – The comparison operator for the right-most comparison in the expression min_value left_op value right_op value. This operator should generally be either operator.lt or operator.le. Default: operator.lt.

class luigi.LuigiStatusCode(*values)

All possible status codes for the attribute status in LuigiRunResult when the argument detailed_summary=True in luigi.run() / luigi.build. Here are the codes and what they mean:

Status Code Name	Meaning
SUCCESS	There were no failed tasks or missing dependencies
SUCCESS_WITH_RETRY	There were failed tasks but they all succeeded in a retry
FAILED	There were failed tasks
FAILED_AND_SCHEDULING_FAILED	There were failed tasks and tasks whose scheduling failed
SCHEDULING_FAILED	There were tasks whose scheduling failed
NOT_RUN	There were tasks that were not granted run permission by the scheduler
MISSING_EXT	There were missing external dependencies

SUCCESS = (':)', 'there were no failed tasks or missing dependencies')

SUCCESS_WITH_RETRY = (':)', 'there were failed tasks but they all succeeded in a retry')

FAILED = (':(', 'there were failed tasks')

FAILED_AND_SCHEDULING_FAILED = (':(', 'there were failed tasks and tasks whose scheduling failed')

SCHEDULING_FAILED = (':(', 'there were tasks whose scheduling failed')

NOT_RUN = (':|', 'there were tasks that were not granted run permission by the scheduler')

MISSING_EXT = (':|', 'there were missing external dependencies')

Modules

batch_notifier	Library for sending batch notifications from the Luigi scheduler.
cmdline
cmdline_parser	This module contains luigi internal parsing logic.
configuration
contrib	Package containing optional and-on functionality.
date_interval	luigi.date_interval provides convenient classes for date algebra.
db_task_history	Provides a database backend to the central scheduler.
event	Definitions needed for events.
execution_summary	This module provide the function summary() that is used for printing an execution summary at the end of luigi invocations.
format
freezing	Internal-only module with immutable data structures.
interface	This module contains the bindings for command line integration and dynamic loading of tasks
local_target	LocalTarget provides a concrete implementation of a Target class that uses files on the local file system
lock	Locking functionality when launching things from the command line.
metrics
mock	This module provides a class MockTarget, an implementation of Target.
mypy	Plugin that provides support for luigi.Task
notifications	Supports sending emails when tasks fail.
parameter	Parameters are one of the core concepts of Luigi.
process	Contains some helper functions to run luigid in daemon mode
retcodes	Module containing the logic for exit codes for the luigi binary.
rpc	Implementation of the REST interface between the workers and the server.
safe_extractor	This module provides a class SafeExtractor that offers a secure way to extract tar files while mitigating path traversal vulnerabilities, which can occur when files inside the archive are crafted to escape the intended extraction directory.
scheduler	The system for scheduling tasks and executing them in order.
server	Simple REST server that takes commands in a JSON payload Interface to the Scheduler class.
setup_logging	This module contains helper classes for configuring logging for luigid and workers via command line arguments and options from config files.
target	The abstract Target class.
task	The abstract Task class.
task_history	Abstract class for task history.
task_register	Define the centralized register of all Task classes.
task_status	Possible values for a Task's status in the Scheduler
tools	Sort of a standard library for doing stuff with Tasks at a somewhat abstract level.
util	Using inherits and requires to ease parameter pain
worker	The worker communicates with the scheduler and does two things: