luigi.parameter module¶

Parameters are one of the core concepts of Luigi. All Parameters sit on Task classes. See Parameter for more info on how to define parameters.

class luigi.parameter.ParameterVisibility(value)[source]¶

Bases: IntEnum

Possible values for the parameter visibility option. Public is the default. See Parameters for more info.

PUBLIC = 0¶

HIDDEN = 1¶

PRIVATE = 2¶

classmethod has_value(value)[source]¶

serialize()[source]¶

exception luigi.parameter.ParameterException[source]¶

Bases: Exception

Base exception.

exception luigi.parameter.MissingParameterException[source]¶

Bases: ParameterException

Exception signifying that there was a missing Parameter.

exception luigi.parameter.UnknownParameterException[source]¶

Bases: ParameterException

Exception signifying that an unknown Parameter was supplied.

exception luigi.parameter.DuplicateParameterException[source]¶

Bases: ParameterException

Exception signifying that a Parameter was specified multiple times.

exception luigi.parameter.OptionalParameterTypeWarning[source]¶

Bases: UserWarning

Warning class for OptionalParameterMixin with wrong type.

exception luigi.parameter.UnconsumedParameterWarning[source]¶

Bases: UserWarning

Warning class for parameters that are not consumed by the task.

class luigi.parameter.Parameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=ParameterVisibility.PUBLIC)[source]¶

Bases: object

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

Any value provided on the command line:

To the root task (eg. --param xyz)

Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).

With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See Parameters from config Ingestion

Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

has_task_value(task_name, param_name)[source]¶

task_value(task_name, param_name)[source]¶

parse(x)[source]¶

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:: x (str) – the value to parse.
Returns:: the parsed value.

serialize(x)[source]¶

Opposite of parse().

Converts the value x to a string.

Parameters:: x – the value to serialize.

normalize(x)[source]¶

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

next_in_enumeration(value)[source]¶

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:: value – The value
Returns:: The next value, like “value + 1”. Or None if there’s no enumerable ordering.

class luigi.parameter.OptionalParameterMixin[source]¶

Bases: object

Mixin to make a parameter class optional and treat empty string as None.

expected_type¶: alias of None

serialize(x)[source]¶: Parse the given value if the value is not None else return an empty string.

parse(x)[source]¶: Parse the given value if it is a string (empty strings are parsed to None).

normalize(x)[source]¶: Normalize the given value if it is not None.

next_in_enumeration(value)[source]¶

class luigi.parameter.OptionalParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=ParameterVisibility.PUBLIC)[source]¶

Bases: OptionalParameterMixin, Parameter

Class to parse optional parameters.

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

expected_type¶: alias of str

class luigi.parameter.OptionalStrParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=ParameterVisibility.PUBLIC)[source]¶

Bases: OptionalParameterMixin, Parameter

Class to parse optional str parameters.

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

expected_type¶: alias of str

class luigi.parameter.DateParameter(interval=1, start=None, **kwargs)[source]¶

Bases: _DateParameterBase

Parameter whose value is a date.

A DateParameter is a Date string formatted YYYY-MM-DD. For example, 2013-07-10 specifies July 10, 2013.

DateParameters are 90% of the time used to be interpolated into file system paths or the like. Here is a gentle reminder of how to interpolate date parameters into strings:

class MyTask(luigi.Task):
    date = luigi.DateParameter()

    def run(self):
        templated_path = "/my/path/to/my/dataset/{date:%Y/%m/%d}/"
        instantiated_path = templated_path.format(date=self.date)
        # print(instantiated_path) --> /my/path/to/my/dataset/2016/06/09/
        # ... use instantiated_path ...

To set this parameter to default to the current day. You can write code like this:

import datetime

class MyTask(luigi.Task):
    date = luigi.DateParameter(default=datetime.date.today())

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

date_format = '%Y-%m-%d'¶

next_in_enumeration(value)[source]¶

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:: value – The value
Returns:: The next value, like “value + 1”. Or None if there’s no enumerable ordering.

normalize(value)[source]¶

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.parameter.MonthParameter(interval=1, start=None, **kwargs)[source]¶

Bases: DateParameter

Parameter whose value is a date, specified to the month (day of date is “rounded” to first of the month).

A MonthParameter is a Date string formatted YYYY-MM. For example, 2013-07 specifies July of 2013. Task objects constructed from code accept date (ignoring the day value) or Month.

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

date_format = '%Y-%m'¶

next_in_enumeration(value)[source]¶

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:: value – The value
Returns:: The next value, like “value + 1”. Or None if there’s no enumerable ordering.

normalize(value)[source]¶

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.parameter.YearParameter(interval=1, start=None, **kwargs)[source]¶

Bases: DateParameter

Parameter whose value is a date, specified to the year (day and month of date is “rounded” to first day of the year).

A YearParameter is a Date string formatted YYYY. Task objects constructed from code accept date (ignoring the month and day values) or Year.

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

date_format = '%Y'¶

next_in_enumeration(value)[source]¶

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:: value – The value
Returns:: The next value, like “value + 1”. Or None if there’s no enumerable ordering.

normalize(value)[source]¶

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.parameter.DateHourParameter(interval=1, start=None, **kwargs)[source]¶

Bases: _DatetimeParameterBase

Parameter whose value is a datetime specified to the hour.

A DateHourParameter is a ISO 8601 formatted date and time specified to the hour. For example, 2013-07-10T19 specifies July 10, 2013 at 19:00.

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

date_format = '%Y-%m-%dT%H'¶

class luigi.parameter.DateMinuteParameter(interval=1, start=None, **kwargs)[source]¶

Bases: _DatetimeParameterBase

Parameter whose value is a datetime specified to the minute.

A DateMinuteParameter is a ISO 8601 formatted date and time specified to the minute. For example, 2013-07-10T1907 specifies July 10, 2013 at 19:07.

The interval parameter can be used to clamp this parameter to every N minutes, instead of every minute.

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

date_format = '%Y-%m-%dT%H%M'¶

deprecated_date_format = '%Y-%m-%dT%HH%M'¶

parse(s)[source]¶: Parses a string to a datetime.

class luigi.parameter.DateSecondParameter(interval=1, start=None, **kwargs)[source]¶

Bases: _DatetimeParameterBase

Parameter whose value is a datetime specified to the second.

A DateSecondParameter is a ISO 8601 formatted date and time specified to the second. For example, 2013-07-10T190738 specifies July 10, 2013 at 19:07:38.

The interval parameter can be used to clamp this parameter to every N seconds, instead of every second.

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

date_format = '%Y-%m-%dT%H%M%S'¶

class luigi.parameter.IntParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=ParameterVisibility.PUBLIC)[source]¶

Bases: Parameter

Parameter whose value is an int.

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(s)[source]¶: Parses an int from the string using int().

next_in_enumeration(value)[source]¶

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:: value – The value
Returns:: The next value, like “value + 1”. Or None if there’s no enumerable ordering.

class luigi.parameter.OptionalIntParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=ParameterVisibility.PUBLIC)[source]¶

Bases: OptionalParameterMixin, IntParameter

Class to parse optional int parameters.

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

expected_type¶: alias of int

class luigi.parameter.FloatParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=ParameterVisibility.PUBLIC)[source]¶

Bases: Parameter

Parameter whose value is a float.

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(s)[source]¶: Parses a float from the string using float().

class luigi.parameter.OptionalFloatParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=ParameterVisibility.PUBLIC)[source]¶

Bases: OptionalParameterMixin, FloatParameter

Class to parse optional float parameters.

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

expected_type¶: alias of float

class luigi.parameter.BoolParameter(*args, **kwargs)[source]¶

Bases: Parameter

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

IMPLICIT_PARSING = 'implicit'¶

EXPLICIT_PARSING = 'explicit'¶

parsing = 'implicit'¶

parse(val)[source]¶: Parses a bool from the string, matching ‘true’ or ‘false’ ignoring case.

normalize(value)[source]¶

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.parameter.OptionalBoolParameter(*args, **kwargs)[source]¶

Bases: OptionalParameterMixin, BoolParameter

Class to parse optional bool parameters.

expected_type¶: alias of bool

class luigi.parameter.DateIntervalParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=ParameterVisibility.PUBLIC)[source]¶

Bases: Parameter

A Parameter whose value is a DateInterval.

Date Intervals are specified using the ISO 8601 date notation for dates (eg. “2015-11-04”), months (eg. “2015-05”), years (eg. “2015”), or weeks (eg. “2015-W35”). In addition, it also supports arbitrary date intervals provided as two dates separated with a dash (eg. “2015-11-04-2015-12-04”).

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(s)[source]¶

Parses a DateInterval from the input.

see luigi.date_interval: for details on the parsing of DateIntervals.

class luigi.parameter.TimeDeltaParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=ParameterVisibility.PUBLIC)[source]¶

Bases: Parameter

Class that maps to timedelta using strings in any of the following forms:

A bare number is interpreted as duration in seconds.

n {w[eek[s]]|d[ay[s]]|h[our[s]]|m[inute[s]|s[second[s]]} (e.g. “1 week 2 days” or “1 h”)
Note: multiple arguments must be supplied in longest to shortest unit order

ISO 8601 duration PnDTnHnMnS (each field optional, years and months not supported)

ISO 8601 duration PnW

See https://en.wikipedia.org/wiki/ISO_8601#Durations

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(input)[source]¶

Parses a time delta from the input.

See TimeDeltaParameter for details on supported formats.

serialize(x)[source]¶

Converts datetime.timedelta to a string

Parameters:: x – the value to serialize.

class luigi.parameter.TaskParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=ParameterVisibility.PUBLIC)[source]¶

Bases: Parameter

A parameter that takes another luigi task class.

When used programatically, the parameter should be specified directly with the luigi.task.Task (sub) class. Like MyMetaTask(my_task_param=my_tasks.MyTask). On the command line, you specify the luigi.task.Task.get_task_family(). Like

$ luigi --module my_tasks MyMetaTask --my_task_param my_namespace.MyTask

Where my_namespace.MyTask is defined in the my_tasks python module.

When the luigi.task.Task class is instantiated to an object. The value will always be a task class (and not a string).

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(input)[source]¶: Parse a task_famly using the Register

serialize(cls)[source]¶: Converts the luigi.task.Task (sub) class to its family name.

class luigi.parameter.EnumParameter(*args, **kwargs)[source]¶

Bases: Parameter

A parameter whose value is an Enum.

In the task definition, use

class Model(enum.Enum):
  Honda = 1
  Volvo = 2

class MyTask(luigi.Task):
  my_param = luigi.EnumParameter(enum=Model)

At the command line, use,

$ luigi --module my_tasks MyTask --my-param Honda

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(s)[source]¶

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:: x (str) – the value to parse.
Returns:: the parsed value.

serialize(e)[source]¶

Opposite of parse().

Converts the value x to a string.

Parameters:: x – the value to serialize.

class luigi.parameter.EnumListParameter(*args, **kwargs)[source]¶

Bases: Parameter

A parameter whose value is a comma-separated list of Enum. Values should come from the same enum.

Values are taken to be a list, i.e. order is preserved, duplicates may occur, and empty list is possible.

In the task definition, use

class Model(enum.Enum):
  Honda = 1
  Volvo = 2

class MyTask(luigi.Task):
  my_param = luigi.EnumListParameter(enum=Model)

At the command line, use,

$ luigi --module my_tasks MyTask --my-param Honda,Volvo

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(s)[source]¶

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:: x (str) – the value to parse.
Returns:: the parsed value.

serialize(enum_values)[source]¶

Opposite of parse().

Converts the value x to a string.

Parameters:: x – the value to serialize.

class luigi.parameter.DictParameter(*args, schema=None, **kwargs)[source]¶

Bases: Parameter

Parameter whose value is a dict.

In the task definition, use

class MyTask(luigi.Task):
  tags = luigi.DictParameter()

    def run(self):
        logging.info("Find server with role: %s", self.tags['role'])
        server = aws.ec2.find_my_resource(self.tags)

At the command line, use

$ luigi --module my_tasks MyTask --tags <JSON string>

Simple example with two tags:

$ luigi --module my_tasks MyTask --tags '{"role": "web", "env": "staging"}'

It can be used to define dynamic parameters, when you do not know the exact list of your parameters (e.g. list of tags, that are dynamically constructed outside Luigi), or you have a complex parameter containing logically related values (like a database connection config).

It is possible to provide a JSON schema that should be validated by the given value:

class MyTask(luigi.Task):
  tags = luigi.DictParameter(
    schema={
      "type": "object",
      "patternProperties": {
        ".*": {"type": "string", "enum": ["web", "staging"]},
      }
    }
  )

  def run(self):
    logging.info("Find server with role: %s", self.tags['role'])
    server = aws.ec2.find_my_resource(self.tags)

Using this schema, the following command will work:

$ luigi --module my_tasks MyTask --tags '{"role": "web", "env": "staging"}'

while this command will fail because the parameter is not valid:

$ luigi --module my_tasks MyTask --tags '{"role": "UNKNOWN_VALUE", "env": "staging"}'

Finally, the provided schema can be a custom validator:

custom_validator = jsonschema.Draft4Validator(
  schema={
    "type": "object",
    "patternProperties": {
      ".*": {"type": "string", "enum": ["web", "staging"]},
    }
  }
)

class MyTask(luigi.Task):
  tags = luigi.DictParameter(schema=custom_validator)

  def run(self):
    logging.info("Find server with role: %s", self.tags['role'])
    server = aws.ec2.find_my_resource(self.tags)

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

normalize(value)[source]¶: Ensure that dictionary parameter is converted to a FrozenOrderedDict so it can be hashed.

parse(source)[source]¶

Parses an immutable and ordered dict from a JSON string using standard JSON library.

We need to use an immutable dictionary, to create a hashable parameter and also preserve the internal structure of parsing. The traversal order of standard dict is undefined, which can result various string representations of this parameter, and therefore a different task id for the task containing this parameter. This is because task id contains the hash of parameters’ JSON representation.

Parameters:: s – String to be parse

serialize(x)[source]¶

Opposite of parse().

Converts the value x to a string.

Parameters:: x – the value to serialize.

class luigi.parameter.OptionalDictParameter(*args, schema=None, **kwargs)[source]¶

Bases: OptionalParameterMixin, DictParameter

Class to parse optional dict parameters.

expected_type¶: alias of FrozenOrderedDict

class luigi.parameter.ListParameter(*args, schema=None, **kwargs)[source]¶

Bases: Parameter

Parameter whose value is a list.

In the task definition, use

class MyTask(luigi.Task):
  grades = luigi.ListParameter()

    def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

At the command line, use

$ luigi --module my_tasks MyTask --grades <JSON string>

Simple example with two grades:

$ luigi --module my_tasks MyTask --grades '[100,70]'

It is possible to provide a JSON schema that should be validated by the given value:

class MyTask(luigi.Task):
  grades = luigi.ListParameter(
    schema={
      "type": "array",
      "items": {
        "type": "number",
        "minimum": 0,
        "maximum": 10
      },
      "minItems": 1
    }
  )

  def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

Using this schema, the following command will work:

$ luigi --module my_tasks MyTask --numbers '[1, 8.7, 6]'

while these commands will fail because the parameter is not valid:

$ luigi --module my_tasks MyTask --numbers '[]'  # must have at least 1 element
$ luigi --module my_tasks MyTask --numbers '[-999, 999]'  # elements must be in [0, 10]

Finally, the provided schema can be a custom validator:

custom_validator = jsonschema.Draft4Validator(
  schema={
    "type": "array",
    "items": {
      "type": "number",
      "minimum": 0,
      "maximum": 10
    },
    "minItems": 1
  }
)

class MyTask(luigi.Task):
  grades = luigi.ListParameter(schema=custom_validator)

  def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

normalize(x)[source]¶

Ensure that struct is recursively converted to a tuple so it can be hashed.

Parameters:: x (str) – the value to parse.
Returns:: the normalized (hashable/immutable) value.

parse(x)[source]¶

Parse an individual value from the input.

Parameters:: x (str) – the value to parse.
Returns:: the parsed value.

serialize(x)[source]¶

Opposite of parse().

Converts the value x to a string.

Parameters:: x – the value to serialize.

class luigi.parameter.OptionalListParameter(*args, schema=None, **kwargs)[source]¶

Bases: OptionalParameterMixin, ListParameter

Class to parse optional list parameters.

expected_type¶: alias of tuple

class luigi.parameter.TupleParameter(*args, schema=None, **kwargs)[source]¶

Bases: ListParameter

Parameter whose value is a tuple or tuple of tuples.

In the task definition, use

class MyTask(luigi.Task):
  book_locations = luigi.TupleParameter()

    def run(self):
        for location in self.book_locations:
            print("Go to page %d, line %d" % (location[0], location[1]))

At the command line, use

$ luigi --module my_tasks MyTask --book_locations <JSON string>

Simple example with two grades:

$ luigi --module my_tasks MyTask --book_locations '((12,3),(4,15),(52,1))'

Parameters:

default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
visibility – A Parameter whose value is a ParameterVisibility. Default value is ParameterVisibility.PUBLIC

parse(x)[source]¶

Parse an individual value from the input.

Parameters:: x (str) – the value to parse.
Returns:: the parsed value.

class luigi.parameter.OptionalTupleParameter(*args, schema=None, **kwargs)[source]¶

Bases: OptionalParameterMixin, TupleParameter

Class to parse optional tuple parameters.

expected_type¶: alias of tuple

class luigi.parameter.NumericalParameter(left_op=<built-in function le>, right_op=<built-in function lt>, *args, **kwargs)[source]¶

Bases: Parameter

Parameter whose value is a number of the specified type, e.g. int or float and in the range specified.

In the task definition, use

class MyTask(luigi.Task):
    my_param_1 = luigi.NumericalParameter(
        var_type=int, min_value=-3, max_value=7) # -3 <= my_param_1 < 7
    my_param_2 = luigi.NumericalParameter(
        var_type=int, min_value=-3, max_value=7, left_op=operator.lt, right_op=operator.le) # -3 < my_param_2 <= 7

At the command line, use

$ luigi --module my_tasks MyTask --my-param-1 -3 --my-param-2 -2

Parameters:

var_type (function) – The type of the input variable, e.g. int or float.
min_value – The minimum value permissible in the accepted values range. May be inclusive or exclusive based on left_op parameter. This should be the same type as var_type.
max_value – The maximum value permissible in the accepted values range. May be inclusive or exclusive based on right_op parameter. This should be the same type as var_type.
left_op (function) – The comparison operator for the left-most comparison in the expression min_value left_op value right_op value. This operator should generally be either operator.lt or operator.le. Default: operator.le.
right_op (function) – The comparison operator for the right-most comparison in the expression min_value left_op value right_op value. This operator should generally be either operator.lt or operator.le. Default: operator.lt.

parse(s)[source]¶

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:: x (str) – the value to parse.
Returns:: the parsed value.

class luigi.parameter.OptionalNumericalParameter(*args, **kwargs)[source]¶

Bases: OptionalParameterMixin, NumericalParameter

Class to parse optional numerical parameters.

class luigi.parameter.ChoiceParameter(var_type=<class 'str'>, *args, **kwargs)[source]¶

Bases: Parameter

A parameter which takes two values:

an instance of Iterable and
the class of the variables to convert to.

In the task definition, use

class MyTask(luigi.Task):
    my_param = luigi.ChoiceParameter(choices=[0.1, 0.2, 0.3], var_type=float)

At the command line, use

$ luigi --module my_tasks MyTask --my-param 0.1

Consider using EnumParameter for a typed, structured alternative. This class can perform the same role when all choices are the same type and transparency of parameter value on the command line is desired.

Parameters:

var_type (function) – The type of the input variable, e.g. str, int, float, etc. Default: str
choices – An iterable, all of whose elements are of var_type to restrict parameter choices to.

parse(s)[source]¶

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:: x (str) – the value to parse.
Returns:: the parsed value.

normalize(var)[source]¶

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

class luigi.parameter.ChoiceListParameter(*args, **kwargs)[source]¶

Bases: ChoiceParameter

A parameter which takes two values:

an instance of Iterable and
the class of the variables to convert to.

Values are taken to be a list, i.e. order is preserved, duplicates may occur, and empty list is possible.

In the task definition, use

class MyTask(luigi.Task):
    my_param = luigi.ChoiceListParameter(choices=['foo', 'bar', 'baz'], var_type=str)

At the command line, use

$ luigi --module my_tasks MyTask --my-param foo,bar

Consider using EnumListParameter for a typed, structured alternative. This class can perform the same role when all choices are the same type and transparency of parameter value on the command line is desired.

Parameters:

var_type (function) – The type of the input variable, e.g. str, int, float, etc. Default: str
choices – An iterable, all of whose elements are of var_type to restrict parameter choices to.

parse(s)[source]¶

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:: x (str) – the value to parse.
Returns:: the parsed value.

normalize(var)[source]¶

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

serialize(values)[source]¶

Opposite of parse().

Converts the value x to a string.

Parameters:: x – the value to serialize.

class luigi.parameter.OptionalChoiceParameter(*args, **kwargs)[source]¶

Bases: OptionalParameterMixin, ChoiceParameter

Class to parse optional choice parameters.

class luigi.parameter.PathParameter(*args, absolute=False, exists=False, **kwargs)[source]¶

Bases: Parameter

Parameter whose value is a path.

In the task definition, use

class MyTask(luigi.Task):
    existing_file_path = luigi.PathParameter(exists=True)
    new_file_path = luigi.PathParameter()

    def run(self):
        # Get data from existing file
        with self.existing_file_path.open("r", encoding="utf-8") as f:
            data = f.read()

        # Output message in new file
        self.new_file_path.parent.mkdir(parents=True, exist_ok=True)
        with self.new_file_path.open("w", encoding="utf-8") as f:
            f.write("hello from a PathParameter => ")
            f.write(data)

At the command line, use

$ luigi --module my_tasks MyTask --existing-file-path <path> --new-file-path <path>

Parameters:

absolute (bool) – If set to True, the given path is converted to an absolute path.
exists (bool) – If set to True, a ValueError is raised if the path does not exist.

normalize(x)[source]¶: Normalize the given value to a pathlib.Path object.

class luigi.parameter.OptionalPathParameter(*args, absolute=False, exists=False, **kwargs)[source]¶

Bases: OptionalParameter, PathParameter

Class to parse optional path parameters.

Parameters:

absolute (bool) – If set to True, the given path is converted to an absolute path.
exists (bool) – If set to True, a ValueError is raised if the path does not exist.

expected_type = (<class 'str'>, <class 'pathlib.Path'>)¶

luigi.parameter module¶

Luigi

Navigation

Related Topics