luigi.parameter module

Parameters are one of the core concepts of Luigi. All Parameters sit on Task classes. See Parameter for more info on how to define parameters.

exception luigi.parameter.ParameterException

Bases: exceptions.Exception

Base exception.

exception luigi.parameter.MissingParameterException

Bases: luigi.parameter.ParameterException

Exception signifying that there was a missing Parameter.

exception luigi.parameter.UnknownParameterException

Bases: luigi.parameter.ParameterException

Exception signifying that an unknown Parameter was supplied.

exception luigi.parameter.DuplicateParameterException

Bases: luigi.parameter.ParameterException

Exception signifying that a Parameter was specified multiple times.

class luigi.parameter.Parameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None)

Bases: object

Parameter whose value is a str, and a base class for other parameter types.

Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:

class MyTask(luigi.Task):
    foo = luigi.Parameter()

class RequiringTask(luigi.Task):
    def requires(self):
        return MyTask(foo="hello")

    def run(self):
        print(self.requires().foo)  # prints "hello"

This makes it possible to instantiate multiple tasks, eg MyTask(foo='bar') and MyTask(foo='baz'). The task will then have the foo attribute set appropriately.

When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate a = TaskA(x=44) then a.x == 44. When the value is not provided, the value will be resolved in this order of falling priority:

• Any value provided on the command line:
  • To the root task (eg. --param xyz)
  • Then to the class, using the qualified task name syntax (eg. --TaskA-param xyz).
• With [TASK_NAME]>PARAM_NAME: <serialized value> syntax. See Parameters from config Ingestion
• Any default value set using the default flag.

Parameter objects may be reused, but you must then set the positional=False flag.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

has_task_value(task_name, param_name)

task_value(task_name, param_name)

parse(x)

Parse an individual value from the input.

The default implementation is the identity function, but subclasses should override this method for specialized parsing.

Parameters:x (str) – the value to parse. Returns:the parsed value.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:x – the value to serialize.

normalize(x)

Given a parsed parameter value, normalizes it.

The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.

This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as x // 2 * 2.

next_in_enumeration(_value)

If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the luigi.execution_summary module for pretty printing purposes. Enabling it to pretty print tasks like MyTask(num=1), MyTask(num=2), MyTask(num=3) to MyTask(num=1..3).

Parameters:value – The value Returns:The next value, like “value + 1”. Or None if there’s no enumerable ordering.

class luigi.parameter.DateParameter(interval=1, start=None, **kwargs)

Bases: luigi.parameter._DateParameterBase

Parameter whose value is a date.

A DateParameter is a Date string formatted YYYY-MM-DD. For example, 2013-07-10 specifies July 10, 2013.

DateParameters are 90% of the time used to be interpolated into file system paths or the like. Here is a gentle reminder of how to interpolate date parameters into strings:

class MyTask(luigi.Task):
    date = luigi.DateParameter()

    def run(self):
        templated_path = "/my/path/to/my/dataset/{date:%Y/%m/%d}/"
        instantiated_path = templated_path.format(date=self.date)
        # print(instantiated_path) --> /my/path/to/my/dataset/2016/06/09/
        # ... use instantiated_path ...

To set this parameter to default to the current day. You can write code like this:

import datetime

class MyTask(luigi.Task):
    date = luigi.DateParameter(default=datetime.date.today())

date_format = '%Y-%m-%d'

next_in_enumeration(value)

normalize(value)

class luigi.parameter.MonthParameter(interval=1, start=None, **kwargs)

Bases: luigi.parameter.DateParameter

Parameter whose value is a date, specified to the month (day of date is “rounded” to first of the month).

A MonthParameter is a Date string formatted YYYY-MM. For example, 2013-07 specifies July of 2013.

date_format = '%Y-%m'

next_in_enumeration(value)

normalize(value)

class luigi.parameter.YearParameter(interval=1, start=None, **kwargs)

Bases: luigi.parameter.DateParameter

Parameter whose value is a date, specified to the year (day and month of date is “rounded” to first day of the year).

A YearParameter is a Date string formatted YYYY.

date_format = '%Y'

next_in_enumeration(value)

normalize(value)

class luigi.parameter.DateHourParameter(interval=1, start=None, **kwargs)

Bases: luigi.parameter._DatetimeParameterBase

Parameter whose value is a datetime specified to the hour.

A DateHourParameter is a ISO 8601 formatted date and time specified to the hour. For example, 2013-07-10T19 specifies July 10, 2013 at 19:00.

date_format = '%Y-%m-%dT%H'

class luigi.parameter.DateMinuteParameter(interval=1, start=None, **kwargs)

Bases: luigi.parameter._DatetimeParameterBase

Parameter whose value is a datetime specified to the minute.

A DateMinuteParameter is a ISO 8601 formatted date and time specified to the minute. For example, 2013-07-10T1907 specifies July 10, 2013 at 19:07.

The interval parameter can be used to clamp this parameter to every N minutes, instead of every minute.

date_format = '%Y-%m-%dT%H%M'

deprecated_date_format = '%Y-%m-%dT%HH%M'

parse(s)

class luigi.parameter.DateSecondParameter(interval=1, start=None, **kwargs)

Bases: luigi.parameter._DatetimeParameterBase

Parameter whose value is a datetime specified to the second.

A DateSecondParameter is a ISO 8601 formatted date and time specified to the second. For example, 2013-07-10T190738 specifies July 10, 2013 at 19:07:38.

The interval parameter can be used to clamp this parameter to every N seconds, instead of every second.

date_format = '%Y-%m-%dT%H%M%S'

class luigi.parameter.IntParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None)

Bases: luigi.parameter.Parameter

Parameter whose value is an int.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

parse(s)

Parses an int from the string using int().

next_in_enumeration(value)

class luigi.parameter.FloatParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None)

Bases: luigi.parameter.Parameter

Parameter whose value is a float.

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

parse(s)

Parses a float from the string using float().

class luigi.parameter.BoolParameter(*args, **kwargs)

Bases: luigi.parameter.Parameter

A Parameter whose value is a bool. This parameter have an implicit default value of False.

parse(s)

Parses a bool from the string, matching ‘true’ or ‘false’ ignoring case.

normalize(value)

class luigi.parameter.DateIntervalParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None)

Bases: luigi.parameter.Parameter

A Parameter whose value is a DateInterval.

Date Intervals are specified using the ISO 8601 date notation for dates (eg. “2015-11-04”), months (eg. “2015-05”), years (eg. “2015”), or weeks (eg. “2015-W35”). In addition, it also supports arbitrary date intervals provided as two dates separated with a dash (eg. “2015-11-04-2015-12-04”).

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

parse(s)

Parses a DateInterval from the input.

see luigi.date_interval

for details on the parsing of DateIntervals.

class luigi.parameter.TimeDeltaParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None)

Bases: luigi.parameter.Parameter

Class that maps to timedelta using strings in any of the following forms:

• n {w[eek[s]]|d[ay[s]]|h[our[s]]|m[inute[s]|s[second[s]]} (e.g. “1 week 2 days” or “1 h”)

  Note: multiple arguments must be supplied in longest to shortest unit order

• ISO 8601 duration PnDTnHnMnS (each field optional, years and months not supported)
• ISO 8601 duration PnW

See https://en.wikipedia.org/wiki/ISO_8601#Durations

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

parse(input)

Parses a time delta from the input.

See TimeDeltaParameter for details on supported formats.

serialize(x)

Converts datetime.timedelta to a string

Parameters:x – the value to serialize.

class luigi.parameter.TaskParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None)

Bases: luigi.parameter.Parameter

A parameter that takes another luigi task class.

When used programatically, the parameter should be specified directly with the luigi.task.Task (sub) class. Like MyMetaTask(my_task_param=my_tasks.MyTask). On the command line, you specify the luigi.task.Task.get_task_family(). Like

$ luigi --module my_tasks MyMetaTask --my_task_param my_namespace.MyTask

Where my_namespace.MyTask is defined in the my_tasks python module.

When the luigi.task.Task class is instantiated to an object. The value will always be a task class (and not a string).

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

parse(input)

Parse a task_famly using the Register

serialize(cls)

Converts the luigi.task.Task (sub) class to its family name.

class luigi.parameter.EnumParameter(*args, **kwargs)

Bases: luigi.parameter.Parameter

A parameter whose value is an Enum.

In the task definition, use

class Model(enum.Enum):
  Honda = 1
  Volvo = 2

class MyTask(luigi.Task):
  my_param = luigi.EnumParameter(enum=Model)

At the command line, use,

$ luigi --module my_tasks MyTask --my-param Honda

parse(s)

serialize(e)

class luigi.parameter.DictParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None)

Bases: luigi.parameter.Parameter

Parameter whose value is a dict.

In the task definition, use

class MyTask(luigi.Task):
  tags = luigi.DictParameter()

    def run(self):
        logging.info("Find server with role: %s", self.tags['role'])
        server = aws.ec2.find_my_resource(self.tags)

At the command line, use

$ luigi --module my_tasks MyTask --tags <JSON string>

Simple example with two tags:

$ luigi --module my_tasks MyTask --tags '{"role": "web", "env": "staging"}'

It can be used to define dynamic parameters, when you do not know the exact list of your parameters (e.g. list of tags, that are dynamically constructed outside Luigi), or you have a complex parameter containing logically related values (like a database connection config).

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

normalize(value)

Ensure that dictionary parameter is converted to a _FrozenOrderedDict so it can be hashed.

parse(s)

Parses an immutable and ordered dict from a JSON string using standard JSON library.

We need to use an immutable dictionary, to create a hashable parameter and also preserve the internal structure of parsing. The traversal order of standard dict is undefined, which can result various string representations of this parameter, and therefore a different task id for the task containing this parameter. This is because task id contains the hash of parameters’ JSON representation.

Parameters:s – String to be parse

serialize(x)

class luigi.parameter.ListParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None)

Bases: luigi.parameter.Parameter

Parameter whose value is a list.

In the task definition, use

class MyTask(luigi.Task):
  grades = luigi.ListParameter()

    def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

At the command line, use

$ luigi --module my_tasks MyTask --grades <JSON string>

Simple example with two grades:

$ luigi --module my_tasks MyTask --grades '[100,70]'

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

normalize(x)

Ensure that list parameter is converted to a tuple so it can be hashed.

Parameters:x (str) – the value to parse. Returns:the normalized (hashable/immutable) value.

parse(x)

Parse an individual value from the input.

Parameters:x (str) – the value to parse. Returns:the parsed value.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:x – the value to serialize.

class luigi.parameter.TupleParameter(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None)

Bases: luigi.parameter.Parameter

Parameter whose value is a tuple or tuple of tuples.

In the task definition, use

class MyTask(luigi.Task):
  book_locations = luigi.TupleParameter()

    def run(self):
        for location in self.book_locations:
            print("Go to page %d, line %d" % (location[0], location[1]))

At the command line, use

$ luigi --module my_tasks MyTask --book_locations <JSON string>

Simple example with two grades:

$ luigi --module my_tasks MyTask --book_locations '((12,3),(4,15),(52,1))'

Parameters:

• default – the default value for this parameter. This should match the type of the Parameter, i.e. datetime.date for DateParameter or int for IntParameter. By default, no default is stored and the value must be specified at runtime.
• significant (bool) – specify False if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default: True.
• description (str) – A human-readable string describing the purpose of this Parameter. For command-line invocations, this will be used as the help string shown to users. Default: None.
• config_path (dict) – a dictionary with entries section and name specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default: None.
• positional (bool) – If true, you can set the argument as a positional argument. It’s true by default but we recommend positional=False for abstract base classes and similar cases.
• always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
• batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run

parse(x)

Parse an individual value from the input.

Parameters:x (str) – the value to parse. Returns:the parsed value.

serialize(x)

Opposite of parse().

Converts the value x to a string.

Parameters:x – the value to serialize.

class luigi.parameter.NumericalParameter(left_op=<built-in function le>, right_op=<built-in function lt>, *args, **kwargs)

Bases: luigi.parameter.Parameter

Parameter whose value is a number of the specified type, e.g. int or float and in the range specified.

In the task definition, use

class MyTask(luigi.Task):
    my_param_1 = luigi.NumericalParameter(
        var_type=int, min_value=-3, max_value=7) # -3 <= my_param_1 < 7
    my_param_2 = luigi.NumericalParameter(
        var_type=int, min_value=-3, max_value=7, left_op=operator.lt, right_op=operator.le) # -3 < my_param_2 <= 7

At the command line, use

$ luigi --module my_tasks MyTask --my-param-1 -3 --my-param-2 -2

Parameters:

• var_type (function) – The type of the input variable, e.g. int or float.
• min_value – The minimum value permissible in the accepted values range. May be inclusive or exclusive based on left_op parameter. This should be the same type as var_type.
• max_value – The maximum value permissible in the accepted values range. May be inclusive or exclusive based on right_op parameter. This should be the same type as var_type.
• left_op (function) – The comparison operator for the left-most comparison in the expression min_value left_op value right_op value. This operator should generally be either operator.lt or operator.le. Default: operator.le.
• right_op (function) – The comparison operator for the right-most comparison in the expression min_value left_op value right_op value. This operator should generally be either operator.lt or operator.le. Default: operator.lt.

parse(s)

class luigi.parameter.ChoiceParameter(var_type=<type 'str'>, *args, **kwargs)

Bases: luigi.parameter.Parameter

A parameter which takes two values:

1. an instance of Iterable and
2. the class of the variables to convert to.

In the task definition, use

class MyTask(luigi.Task):
    my_param = luigi.ChoiceParameter(choices=[0.1, 0.2, 0.3], var_type=float)

At the command line, use

$ luigi --module my_tasks MyTask --my-param 0.1

Consider using EnumParameter for a typed, structured alternative. This class can perform the same role when all choices are the same type and transparency of parameter value on the command line is desired.

Parameters:

• var_type (function) – The type of the input variable, e.g. str, int, float, etc. Default: str
• choices – An iterable, all of whose elements are of var_type to restrict parameter choices to.

parse(s)