luigi package¶
Subpackages¶
- luigi.configuration package
- luigi.contrib package
- Subpackages
- luigi.contrib.hdfs package
- Submodules
- luigi.contrib.hdfs.abstract_client module
- luigi.contrib.hdfs.clients module
- luigi.contrib.hdfs.config module
- luigi.contrib.hdfs.error module
- luigi.contrib.hdfs.format module
- luigi.contrib.hdfs.hadoopcli_clients module
- luigi.contrib.hdfs.snakebite_client module
- luigi.contrib.hdfs.target module
- luigi.contrib.hdfs.webhdfs_client module
- Module contents
- Submodules
- luigi.contrib.hdfs package
- Submodules
- luigi.contrib.azureblob module
- luigi.contrib.batch module
- luigi.contrib.beam_dataflow module
- luigi.contrib.bigquery module
- luigi.contrib.bigquery_avro module
- luigi.contrib.datadog_metric module
- luigi.contrib.dataproc module
- luigi.contrib.docker_runner module
- luigi.contrib.dropbox module
- luigi.contrib.ecs module
- luigi.contrib.esindex module
- luigi.contrib.external_daily_snapshot module
- luigi.contrib.external_program module
- luigi.contrib.ftp module
- luigi.contrib.gcp module
- luigi.contrib.gcs module
- luigi.contrib.hadoop module
- luigi.contrib.hadoop_jar module
- luigi.contrib.hive module
- luigi.contrib.kubernetes module
- luigi.contrib.lsf module
- luigi.contrib.lsf_runner module
- luigi.contrib.mongodb module
- luigi.contrib.mrrunner module
- luigi.contrib.mssqldb module
- luigi.contrib.mysqldb module
- luigi.contrib.opener module
- luigi.contrib.pai module
- luigi.contrib.pig module
- luigi.contrib.postgres module
- luigi.contrib.presto module
- luigi.contrib.prometheus_metric module
- luigi.contrib.pyspark_runner module
- luigi.contrib.rdbms module
- luigi.contrib.redis_store module
- luigi.contrib.redshift module
- luigi.contrib.s3 module
- luigi.contrib.salesforce module
- luigi.contrib.scalding module
- luigi.contrib.sge module
- luigi.contrib.sge_runner module
- luigi.contrib.simulate module
- luigi.contrib.spark module
- luigi.contrib.sparkey module
- luigi.contrib.sqla module
- luigi.contrib.ssh module
- luigi.contrib.target module
- luigi.contrib.webhdfs module
- Module contents
- Subpackages
- luigi.tools package
Submodules¶
- luigi.batch_notifier module
- luigi.cmdline module
- luigi.cmdline_parser module
- luigi.date_interval module
- luigi.db_task_history module
- luigi.event module
- luigi.execution_summary module
- luigi.format module
- luigi.freezing module
- luigi.interface module
- luigi.local_target module
- luigi.lock module
- luigi.metrics module
- luigi.mock module
- luigi.notifications module
- luigi.parameter module
- luigi.process module
- luigi.retcodes module
- luigi.rpc module
- luigi.scheduler module
- luigi.server module
- luigi.setup_logging module
- luigi.six module
- luigi.target module
- luigi.task module
- luigi.task_history module
- luigi.task_register module
- luigi.task_status module
- luigi.util module
- luigi.worker module
Module contents¶
Package containing core luigi functionality.
-
class
luigi.
Task
(*args, **kwargs)[source]¶ Bases:
object
This is the base class of all Luigi Tasks, the base unit of work in Luigi.
A Luigi Task describes a unit or work.
The key methods of a Task, which must be implemented in a subclass are:
run()
- the computation done by this task.requires()
- the list of Tasks that this Task depends on.output()
- the outputTarget
that this Task creates.
Each
Parameter
of the Task should be declared as members:class MyTask(luigi.Task): count = luigi.IntParameter() second_param = luigi.Parameter()
In addition to any declared properties and methods, there are a few non-declared properties, which are created by the
Register
metaclass:-
priority
= 0¶ Priority of the task: the scheduler should favor available tasks with higher priority values first. See Task priority
-
disabled
= False¶
-
resources
= {}¶ Resources used by the task. Should be formatted like {“scp”: 1} to indicate that the task requires 1 unit of the scp resource.
-
worker_timeout
= None¶ Number of seconds after which to time out the run function. No timeout if set to 0. Defaults to 0 or worker-timeout value in config
-
max_batch_size
= inf¶ Maximum number of tasks to run together as a batch. Infinite by default
-
batchable
¶ True if this instance can be run as part of a batch. By default, True if it has any batched parameters
-
retry_count
¶ Override this positive integer to have different
retry_count
at task level Check [scheduler]
-
disable_hard_timeout
¶ Override this positive integer to have different
disable_hard_timeout
at task level. Check [scheduler]
-
disable_window_seconds
¶ Override this positive integer to have different
disable_window_seconds
at task level. Check [scheduler]
-
owner_email
¶ Override this to send out additional error emails to task owner, in addition to the one defined in the global configuration. This should return a string or a list of strings. e.g. ‘test@exmaple.com’ or [‘test1@example.com’, ‘test2@example.com’]
-
use_cmdline_section
¶ Property used by core config such as –workers etc. These will be exposed without the class as prefix.
-
trigger_event
(event, *args, **kwargs)[source]¶ Trigger that calls all of the specified events associated with this class.
-
accepts_messages
¶ For configuring which scheduler messages can be received. When falsy, this tasks does not accept any message. When True, all messages are accepted.
-
task_module
¶ Returns what Python module to import to get access to this class.
-
task_namespace
= '__not_user_specified'¶ This value can be overriden to set the namespace that will be used. (See Namespaces, families and ids) If it’s not specified and you try to read this value anyway, it will return garbage. Please use
get_task_namespace()
to read the namespace.Note that setting this value with
@property
will not work, because this is a class level value.
-
classmethod
get_task_namespace
()[source]¶ The task family for the given class.
Note: You normally don’t want to override this.
-
task_family
= 'Task'¶
-
classmethod
get_task_family
()[source]¶ The task family for the given class.
If
task_namespace
is not set, then it’s simply the name of the class. Otherwise,<task_namespace>.
is prefixed to the class name.Note: You normally don’t want to override this.
-
classmethod
get_param_values
(params, args, kwargs)[source]¶ Get the values of the parameters from the args and kwargs.
Parameters: - params – list of (param_name, Parameter).
- args – positional arguments
- kwargs – keyword arguments.
Returns: list of (name, value) tuples, one for each parameter.
-
param_args
¶
-
classmethod
from_str_params
(params_str)[source]¶ Creates an instance from a str->str hash.
Parameters: params_str – dict of param name -> value as string.
-
to_str_params
(only_significant=False, only_public=False)[source]¶ Convert all parameters to a str->str hash.
-
clone
(cls=None, **kwargs)[source]¶ Creates a new instance from an existing instance where some of the args have changed.
There’s at least two scenarios where this is useful (see test/clone_test.py):
- remove a lot of boiler plate when you have recursive dependencies and lots of args
- there’s task inheritance and some logic is on the base class
Parameters: - cls –
- kwargs –
Returns:
-
complete
()[source]¶ If the task has any outputs, return
True
if all outputs exist. Otherwise, returnFalse
.However, you may freely override this method with custom logic.
-
classmethod
bulk_complete
(parameter_tuples)[source]¶ Returns those of parameter_tuples for which this Task is complete.
Override (with an efficient implementation) for efficient scheduling with range tools. Keep the logic consistent with that of complete().
-
output
()[source]¶ The output that this Task produces.
The output of the Task determines if the Task needs to be run–the task is considered finished iff the outputs all exist. Subclasses should override this method to return a single
Target
or a list ofTarget
instances.- Implementation note
- If running multiple workers, the output must be a resource that is accessible by all workers, such as a DFS or database. Otherwise, workers might compute the same output since they don’t see the work done by other workers.
See Task.output
-
requires
()[source]¶ The Tasks that this Task depends on.
A Task will only run if all of the Tasks that it requires are completed. If your Task does not require any other Tasks, then you don’t need to override this method. Otherwise, a subclass can override this method to return a single Task, a list of Task instances, or a dict whose values are Task instances.
See Task.requires
-
process_resources
()[source]¶ Override in “template” tasks which provide common resource functionality but allow subclasses to specify additional resources while preserving the name for consistent end-user experience.
-
input
()[source]¶ Returns the outputs of the Tasks returned by
requires()
See Task.input
Returns: a list of Target
objects which are specified as outputs of all required Tasks.
-
on_failure
(exception)[source]¶ Override for custom error handling.
This method gets called if an exception is raised in
run()
. The returned value of this method is json encoded and sent to the scheduler as the expl argument. Its string representation will be used as the body of the error email sent out if any.Default behavior is to return a string representation of the stack trace.
-
class
luigi.
Config
(*args, **kwargs)[source]¶ Bases:
luigi.task.Task
Class for configuration. See Configuration classes.
-
class
luigi.
ExternalTask
(*args, **kwargs)[source]¶ Bases:
luigi.task.Task
Subclass for references to external dependencies.
An ExternalTask’s does not have a run implementation, which signifies to the framework that this Task’s
output()
is generated outside of Luigi.-
run
= None¶
-
-
class
luigi.
WrapperTask
(*args, **kwargs)[source]¶ Bases:
luigi.task.Task
Use for tasks that only wrap other tasks and that by definition are done if all their requirements exist.
-
luigi.
namespace
(namespace=None, scope='')[source]¶ Call to set namespace of tasks declared after the call.
It is often desired to call this function with the keyword argument
scope=__name__
.The
scope
keyword makes it so that this call is only effective for task classes with a matching [*]__module__
. The default value forscope
is the empty string, which means all classes. Multiple calls with the same scope simply replace each other.The namespace of a
Task
can also be changed by specifying the propertytask_namespace
.class Task2(luigi.Task): task_namespace = 'namespace2'
This explicit setting takes priority over whatever is set in the
namespace()
method, and it’s also inherited through normal python inheritence.There’s no equivalent way to set the
task_family
.New since Luigi 2.6.0:
scope
keyword argument.[*] When there are multiple levels of matching module scopes like a.b
vsa.b.c
, the more specific one (a.b.c
) wins.See also
The new and better scaling
auto_namespace()
-
luigi.
auto_namespace
(scope='')[source]¶ Same as
namespace()
, but instead of a constant namespace, it will be set to the__module__
of the task class. This is desirable for these reasons:- Two tasks with the same name will not have conflicting task families
- It’s more pythonic, as modules are Python’s recommended way to do namespacing.
- It’s traceable. When you see the full name of a task, you can immediately identify where it is defined.
We recommend calling this function from your package’s outermost
__init__.py
file. The file contents could look like this:import luigi luigi.auto_namespace(scope=__name__)
To reset an
auto_namespace()
call, you can usenamespace(scope='my_scope')
. But this will not be needed (and is also discouraged) if you use thescope
kwarg.New since Luigi 2.6.0.
-
class
luigi.
Target
[source]¶ Bases:
object
A Target is a resource generated by a
Task
.For example, a Target might correspond to a file in HDFS or data in a database. The Target interface defines one method that must be overridden:
exists()
, which signifies if the Target has been created or not.Typically, a
Task
will define one or more Targets as output, and the Task is considered complete if and only if each of its output Targets exist.
-
class
luigi.
LocalTarget
(path=None, format=None, is_tmp=False)[source]¶ Bases:
luigi.target.FileSystemTarget
-
fs
= <luigi.local_target.LocalFileSystem object>¶
-
open
(mode='r')[source]¶ Open the FileSystem target.
This method returns a file-like object which can either be read from or written to depending on the specified mode.
Parameters: mode (str) – the mode r opens the FileSystemTarget in read-only mode, whereas w will open the FileSystemTarget in write mode. Subclasses can implement additional options. Using b is not supported; initialize with format=Nop instead.
-
remove
()[source]¶ Remove the resource at the path specified by this FileSystemTarget.
This method is implemented by using
fs
.
-
fn
¶
-
-
class
luigi.
RemoteScheduler
(url='http://localhost:8082/', connect_timeout=None)[source]¶ Bases:
object
Scheduler proxy object. Talks to a RemoteSchedulerResponder.
-
add_scheduler_message_response
(*args, **kwargs)¶
-
add_task
(*args, **kwargs)¶ - add task identified by task_id if it doesn’t exist
- if deps is not None, update dependency list
- update status of task
- add additional workers/stakeholders
- update priority when needed
-
add_task_batcher
(*args, **kwargs)¶
-
add_worker
(*args, **kwargs)¶
-
announce_scheduling_failure
(*args, **kwargs)¶
-
count_pending
(*args, **kwargs)¶
-
decrease_running_task_resources
(*args, **kwargs)¶
-
dep_graph
(*args, **kwargs)¶
-
disable_worker
(*args, **kwargs)¶
-
fetch_error
(*args, **kwargs)¶
-
forgive_failures
(*args, **kwargs)¶
-
get_running_task_resources
(*args, **kwargs)¶
-
get_scheduler_message_response
(*args, **kwargs)¶
-
get_task_progress_percentage
(*args, **kwargs)¶
-
get_task_status_message
(*args, **kwargs)¶
-
get_work
(*args, **kwargs)¶
-
graph
(*args, **kwargs)¶
-
has_task_history
(*args, **kwargs)¶
-
inverse_dep_graph
(*args, **kwargs)¶
-
is_pause_enabled
(*args, **kwargs)¶
-
is_paused
(*args, **kwargs)¶
-
mark_as_done
(*args, **kwargs)¶
-
pause
(*args, **kwargs)¶
-
ping
(*args, **kwargs)¶
-
prune
(*args, **kwargs)¶
-
re_enable_task
(*args, **kwargs)¶
-
resource_list
(*args, **kwargs)¶ Resources usage info and their consumers (tasks).
-
send_scheduler_message
(*args, **kwargs)¶
-
set_task_progress_percentage
(*args, **kwargs)¶
-
set_task_status_message
(*args, **kwargs)¶
-
set_worker_processes
(*args, **kwargs)¶
-
task_list
(*args, **kwargs)¶ Query for a subset of tasks by status.
-
task_search
(*args, **kwargs)¶ Query for a subset of tasks by task_id.
Parameters: task_str – Returns:
-
unpause
(*args, **kwargs)¶
-
update_metrics_task_started
(*args, **kwargs)¶
-
update_resource
(*args, **kwargs)¶
-
update_resources
(*args, **kwargs)¶
-
worker_list
(*args, **kwargs)¶
-
-
class
luigi.
Parameter
(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]¶ Bases:
object
Parameter whose value is a
str
, and a base class for other parameter types.Parameters are objects set on the Task class level to make it possible to parameterize tasks. For instance:
class MyTask(luigi.Task): foo = luigi.Parameter() class RequiringTask(luigi.Task): def requires(self): return MyTask(foo="hello") def run(self): print(self.requires().foo) # prints "hello"
This makes it possible to instantiate multiple tasks, eg
MyTask(foo='bar')
andMyTask(foo='baz')
. The task will then have thefoo
attribute set appropriately.When a task is instantiated, it will first use any argument as the value of the parameter, eg. if you instantiate
a = TaskA(x=44)
thena.x == 44
. When the value is not provided, the value will be resolved in this order of falling priority:- Any value provided on the command line:
- To the root task (eg.
--param xyz
) - Then to the class, using the qualified task name syntax (eg.
--TaskA-param xyz
).
- To the root task (eg.
- With
[TASK_NAME]>PARAM_NAME: <serialized value>
syntax. See Parameters from config Ingestion - Any default value set using the
default
flag.
Parameter objects may be reused, but you must then set the
positional=False
flag.Parameters: - default – the default value for this parameter. This should match the type of the
Parameter, i.e.
datetime.date
forDateParameter
orint
forIntParameter
. By default, no default is stored and the value must be specified at runtime. - significant (bool) – specify
False
if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default:True
. - description (str) – A human-readable string describing the purpose of this Parameter.
For command-line invocations, this will be used as the help string
shown to users. Default:
None
. - config_path (dict) – a dictionary with entries
section
andname
specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default:None
. - positional (bool) – If true, you can set the argument as a
positional argument. It’s true by default but we recommend
positional=False
for abstract base classes and similar cases. - always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
- batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
- visibility – A Parameter whose value is a
ParameterVisibility
. Default value is ParameterVisibility.PUBLIC
-
parse
(x)[source]¶ Parse an individual value from the input.
The default implementation is the identity function, but subclasses should override this method for specialized parsing.
Parameters: x (str) – the value to parse. Returns: the parsed value.
-
serialize
(x)[source]¶ Opposite of
parse()
.Converts the value
x
to a string.Parameters: x – the value to serialize.
-
normalize
(x)[source]¶ Given a parsed parameter value, normalizes it.
The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.
This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as
x // 2 * 2
.
-
next_in_enumeration
(_value)[source]¶ If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the
luigi.execution_summary
module for pretty printing purposes. Enabling it to pretty print tasks likeMyTask(num=1), MyTask(num=2), MyTask(num=3)
toMyTask(num=1..3)
.Parameters: value – The value Returns: The next value, like “value + 1”. Or None
if there’s no enumerable ordering.
- Any value provided on the command line:
-
class
luigi.
DateParameter
(interval=1, start=None, **kwargs)[source]¶ Bases:
luigi.parameter._DateParameterBase
Parameter whose value is a
date
.A DateParameter is a Date string formatted
YYYY-MM-DD
. For example,2013-07-10
specifies July 10, 2013.DateParameters are 90% of the time used to be interpolated into file system paths or the like. Here is a gentle reminder of how to interpolate date parameters into strings:
class MyTask(luigi.Task): date = luigi.DateParameter() def run(self): templated_path = "/my/path/to/my/dataset/{date:%Y/%m/%d}/" instantiated_path = templated_path.format(date=self.date) # print(instantiated_path) --> /my/path/to/my/dataset/2016/06/09/ # ... use instantiated_path ...
To set this parameter to default to the current day. You can write code like this:
import datetime class MyTask(luigi.Task): date = luigi.DateParameter(default=datetime.date.today())
-
date_format
= '%Y-%m-%d'¶
-
next_in_enumeration
(value)[source]¶ If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the
luigi.execution_summary
module for pretty printing purposes. Enabling it to pretty print tasks likeMyTask(num=1), MyTask(num=2), MyTask(num=3)
toMyTask(num=1..3)
.Parameters: value – The value Returns: The next value, like “value + 1”. Or None
if there’s no enumerable ordering.
-
normalize
(value)[source]¶ Given a parsed parameter value, normalizes it.
The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.
This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as
x // 2 * 2
.
-
-
class
luigi.
MonthParameter
(interval=1, start=None, **kwargs)[source]¶ Bases:
luigi.parameter.DateParameter
Parameter whose value is a
date
, specified to the month (day ofdate
is “rounded” to first of the month).A MonthParameter is a Date string formatted
YYYY-MM
. For example,2013-07
specifies July of 2013. Task objects constructed from code acceptdate
(ignoring the day value) orMonth
.-
date_format
= '%Y-%m'¶
-
next_in_enumeration
(value)[source]¶ If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the
luigi.execution_summary
module for pretty printing purposes. Enabling it to pretty print tasks likeMyTask(num=1), MyTask(num=2), MyTask(num=3)
toMyTask(num=1..3)
.Parameters: value – The value Returns: The next value, like “value + 1”. Or None
if there’s no enumerable ordering.
-
normalize
(value)[source]¶ Given a parsed parameter value, normalizes it.
The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.
This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as
x // 2 * 2
.
-
-
class
luigi.
YearParameter
(interval=1, start=None, **kwargs)[source]¶ Bases:
luigi.parameter.DateParameter
Parameter whose value is a
date
, specified to the year (day and month ofdate
is “rounded” to first day of the year).A YearParameter is a Date string formatted
YYYY
. Task objects constructed from code acceptdate
(ignoring the month and day values) orYear
.-
date_format
= '%Y'¶
-
next_in_enumeration
(value)[source]¶ If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the
luigi.execution_summary
module for pretty printing purposes. Enabling it to pretty print tasks likeMyTask(num=1), MyTask(num=2), MyTask(num=3)
toMyTask(num=1..3)
.Parameters: value – The value Returns: The next value, like “value + 1”. Or None
if there’s no enumerable ordering.
-
normalize
(value)[source]¶ Given a parsed parameter value, normalizes it.
The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.
This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as
x // 2 * 2
.
-
-
class
luigi.
DateHourParameter
(interval=1, start=None, **kwargs)[source]¶ Bases:
luigi.parameter._DatetimeParameterBase
Parameter whose value is a
datetime
specified to the hour.A DateHourParameter is a ISO 8601 formatted date and time specified to the hour. For example,
2013-07-10T19
specifies July 10, 2013 at 19:00.-
date_format
= '%Y-%m-%dT%H'¶
-
-
class
luigi.
DateMinuteParameter
(interval=1, start=None, **kwargs)[source]¶ Bases:
luigi.parameter._DatetimeParameterBase
Parameter whose value is a
datetime
specified to the minute.A DateMinuteParameter is a ISO 8601 formatted date and time specified to the minute. For example,
2013-07-10T1907
specifies July 10, 2013 at 19:07.The interval parameter can be used to clamp this parameter to every N minutes, instead of every minute.
-
date_format
= '%Y-%m-%dT%H%M'¶
-
deprecated_date_format
= '%Y-%m-%dT%HH%M'¶
-
-
class
luigi.
DateSecondParameter
(interval=1, start=None, **kwargs)[source]¶ Bases:
luigi.parameter._DatetimeParameterBase
Parameter whose value is a
datetime
specified to the second.A DateSecondParameter is a ISO 8601 formatted date and time specified to the second. For example,
2013-07-10T190738
specifies July 10, 2013 at 19:07:38.The interval parameter can be used to clamp this parameter to every N seconds, instead of every second.
-
date_format
= '%Y-%m-%dT%H%M%S'¶
-
-
class
luigi.
DateIntervalParameter
(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]¶ Bases:
luigi.parameter.Parameter
A Parameter whose value is a
DateInterval
.Date Intervals are specified using the ISO 8601 date notation for dates (eg. “2015-11-04”), months (eg. “2015-05”), years (eg. “2015”), or weeks (eg. “2015-W35”). In addition, it also supports arbitrary date intervals provided as two dates separated with a dash (eg. “2015-11-04-2015-12-04”).
Parameters: - default – the default value for this parameter. This should match the type of the
Parameter, i.e.
datetime.date
forDateParameter
orint
forIntParameter
. By default, no default is stored and the value must be specified at runtime. - significant (bool) – specify
False
if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default:True
. - description (str) – A human-readable string describing the purpose of this Parameter.
For command-line invocations, this will be used as the help string
shown to users. Default:
None
. - config_path (dict) – a dictionary with entries
section
andname
specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default:None
. - positional (bool) – If true, you can set the argument as a
positional argument. It’s true by default but we recommend
positional=False
for abstract base classes and similar cases. - always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
- batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
- visibility – A Parameter whose value is a
ParameterVisibility
. Default value is ParameterVisibility.PUBLIC
-
parse
(s)[source]¶ Parses a
DateInterval
from the input.- see
luigi.date_interval
- for details on the parsing of DateIntervals.
- see
- default – the default value for this parameter. This should match the type of the
Parameter, i.e.
-
class
luigi.
TimeDeltaParameter
(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]¶ Bases:
luigi.parameter.Parameter
Class that maps to timedelta using strings in any of the following forms:
n {w[eek[s]]|d[ay[s]]|h[our[s]]|m[inute[s]|s[second[s]]}
(e.g. “1 week 2 days” or “1 h”)- Note: multiple arguments must be supplied in longest to shortest unit order
- ISO 8601 duration
PnDTnHnMnS
(each field optional, years and months not supported) - ISO 8601 duration
PnW
See https://en.wikipedia.org/wiki/ISO_8601#Durations
Parameters: - default – the default value for this parameter. This should match the type of the
Parameter, i.e.
datetime.date
forDateParameter
orint
forIntParameter
. By default, no default is stored and the value must be specified at runtime. - significant (bool) – specify
False
if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default:True
. - description (str) – A human-readable string describing the purpose of this Parameter.
For command-line invocations, this will be used as the help string
shown to users. Default:
None
. - config_path (dict) – a dictionary with entries
section
andname
specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default:None
. - positional (bool) – If true, you can set the argument as a
positional argument. It’s true by default but we recommend
positional=False
for abstract base classes and similar cases. - always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
- batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
- visibility – A Parameter whose value is a
ParameterVisibility
. Default value is ParameterVisibility.PUBLIC
-
parse
(input)[source]¶ Parses a time delta from the input.
See
TimeDeltaParameter
for details on supported formats.
-
class
luigi.
IntParameter
(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]¶ Bases:
luigi.parameter.Parameter
Parameter whose value is an
int
.Parameters: - default – the default value for this parameter. This should match the type of the
Parameter, i.e.
datetime.date
forDateParameter
orint
forIntParameter
. By default, no default is stored and the value must be specified at runtime. - significant (bool) – specify
False
if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default:True
. - description (str) – A human-readable string describing the purpose of this Parameter.
For command-line invocations, this will be used as the help string
shown to users. Default:
None
. - config_path (dict) – a dictionary with entries
section
andname
specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default:None
. - positional (bool) – If true, you can set the argument as a
positional argument. It’s true by default but we recommend
positional=False
for abstract base classes and similar cases. - always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
- batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
- visibility – A Parameter whose value is a
ParameterVisibility
. Default value is ParameterVisibility.PUBLIC
-
next_in_enumeration
(value)[source]¶ If your Parameter type has an enumerable ordering of values. You can choose to override this method. This method is used by the
luigi.execution_summary
module for pretty printing purposes. Enabling it to pretty print tasks likeMyTask(num=1), MyTask(num=2), MyTask(num=3)
toMyTask(num=1..3)
.Parameters: value – The value Returns: The next value, like “value + 1”. Or None
if there’s no enumerable ordering.
- default – the default value for this parameter. This should match the type of the
Parameter, i.e.
-
class
luigi.
FloatParameter
(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]¶ Bases:
luigi.parameter.Parameter
Parameter whose value is a
float
.Parameters: - default – the default value for this parameter. This should match the type of the
Parameter, i.e.
datetime.date
forDateParameter
orint
forIntParameter
. By default, no default is stored and the value must be specified at runtime. - significant (bool) – specify
False
if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default:True
. - description (str) – A human-readable string describing the purpose of this Parameter.
For command-line invocations, this will be used as the help string
shown to users. Default:
None
. - config_path (dict) – a dictionary with entries
section
andname
specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default:None
. - positional (bool) – If true, you can set the argument as a
positional argument. It’s true by default but we recommend
positional=False
for abstract base classes and similar cases. - always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
- batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
- visibility – A Parameter whose value is a
ParameterVisibility
. Default value is ParameterVisibility.PUBLIC
- default – the default value for this parameter. This should match the type of the
Parameter, i.e.
-
class
luigi.
BoolParameter
(*args, **kwargs)[source]¶ Bases:
luigi.parameter.Parameter
A Parameter whose value is a
bool
. This parameter has an implicit default value ofFalse
. For the command line interface this means that the value isFalse
unless you add"--the-bool-parameter"
to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"
), e.g. when you configure the default value to beTrue
. This is called explicit parsing. When omitting the parameter value, it is still consideredTrue
but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.You can toggle between the two parsing modes on a per-parameter base via
class MyTask(luigi.Task): implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING) explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)
or globally by
luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING
for all bool parameters instantiated after this line.
-
IMPLICIT_PARSING
= 'implicit'¶
-
EXPLICIT_PARSING
= 'explicit'¶
-
parsing
= 'implicit'¶
-
normalize
(value)[source]¶ Given a parsed parameter value, normalizes it.
The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.
This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as
x // 2 * 2
.
-
-
class
luigi.
TaskParameter
(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]¶ Bases:
luigi.parameter.Parameter
A parameter that takes another luigi task class.
When used programatically, the parameter should be specified directly with the
luigi.task.Task
(sub) class. LikeMyMetaTask(my_task_param=my_tasks.MyTask)
. On the command line, you specify theluigi.task.Task.get_task_family()
. Like$ luigi --module my_tasks MyMetaTask --my_task_param my_namespace.MyTask
Where
my_namespace.MyTask
is defined in themy_tasks
python module.When the
luigi.task.Task
class is instantiated to an object. The value will always be a task class (and not a string).Parameters: - default – the default value for this parameter. This should match the type of the
Parameter, i.e.
datetime.date
forDateParameter
orint
forIntParameter
. By default, no default is stored and the value must be specified at runtime. - significant (bool) – specify
False
if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default:True
. - description (str) – A human-readable string describing the purpose of this Parameter.
For command-line invocations, this will be used as the help string
shown to users. Default:
None
. - config_path (dict) – a dictionary with entries
section
andname
specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default:None
. - positional (bool) – If true, you can set the argument as a
positional argument. It’s true by default but we recommend
positional=False
for abstract base classes and similar cases. - always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
- batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
- visibility – A Parameter whose value is a
ParameterVisibility
. Default value is ParameterVisibility.PUBLIC
-
serialize
(cls)[source]¶ Converts the
luigi.task.Task
(sub) class to its family name.
- default – the default value for this parameter. This should match the type of the
Parameter, i.e.
-
class
luigi.
ListParameter
(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]¶ Bases:
luigi.parameter.Parameter
Parameter whose value is a
list
.In the task definition, use
class MyTask(luigi.Task): grades = luigi.ListParameter() def run(self): sum = 0 for element in self.grades: sum += element avg = sum / len(self.grades)
At the command line, use
$ luigi --module my_tasks MyTask --grades <JSON string>
Simple example with two grades:
$ luigi --module my_tasks MyTask --grades '[100,70]'
Parameters: - default – the default value for this parameter. This should match the type of the
Parameter, i.e.
datetime.date
forDateParameter
orint
forIntParameter
. By default, no default is stored and the value must be specified at runtime. - significant (bool) – specify
False
if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default:True
. - description (str) – A human-readable string describing the purpose of this Parameter.
For command-line invocations, this will be used as the help string
shown to users. Default:
None
. - config_path (dict) – a dictionary with entries
section
andname
specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default:None
. - positional (bool) – If true, you can set the argument as a
positional argument. It’s true by default but we recommend
positional=False
for abstract base classes and similar cases. - always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
- batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
- visibility – A Parameter whose value is a
ParameterVisibility
. Default value is ParameterVisibility.PUBLIC
-
normalize
(x)[source]¶ Ensure that struct is recursively converted to a tuple so it can be hashed.
Parameters: x (str) – the value to parse. Returns: the normalized (hashable/immutable) value.
- default – the default value for this parameter. This should match the type of the
Parameter, i.e.
-
class
luigi.
TupleParameter
(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]¶ Bases:
luigi.parameter.ListParameter
Parameter whose value is a
tuple
ortuple
of tuples.In the task definition, use
class MyTask(luigi.Task): book_locations = luigi.TupleParameter() def run(self): for location in self.book_locations: print("Go to page %d, line %d" % (location[0], location[1]))
At the command line, use
$ luigi --module my_tasks MyTask --book_locations <JSON string>
Simple example with two grades:
$ luigi --module my_tasks MyTask --book_locations '((12,3),(4,15),(52,1))'
Parameters: - default – the default value for this parameter. This should match the type of the
Parameter, i.e.
datetime.date
forDateParameter
orint
forIntParameter
. By default, no default is stored and the value must be specified at runtime. - significant (bool) – specify
False
if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default:True
. - description (str) – A human-readable string describing the purpose of this Parameter.
For command-line invocations, this will be used as the help string
shown to users. Default:
None
. - config_path (dict) – a dictionary with entries
section
andname
specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default:None
. - positional (bool) – If true, you can set the argument as a
positional argument. It’s true by default but we recommend
positional=False
for abstract base classes and similar cases. - always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
- batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
- visibility – A Parameter whose value is a
ParameterVisibility
. Default value is ParameterVisibility.PUBLIC
- default – the default value for this parameter. This should match the type of the
Parameter, i.e.
-
class
luigi.
EnumParameter
(*args, **kwargs)[source]¶ Bases:
luigi.parameter.Parameter
A parameter whose value is an
Enum
.In the task definition, use
class Model(enum.Enum): Honda = 1 Volvo = 2 class MyTask(luigi.Task): my_param = luigi.EnumParameter(enum=Model)
At the command line, use,
$ luigi --module my_tasks MyTask --my-param Honda
-
class
luigi.
DictParameter
(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]¶ Bases:
luigi.parameter.Parameter
Parameter whose value is a
dict
.In the task definition, use
class MyTask(luigi.Task): tags = luigi.DictParameter() def run(self): logging.info("Find server with role: %s", self.tags['role']) server = aws.ec2.find_my_resource(self.tags)
At the command line, use
$ luigi --module my_tasks MyTask --tags <JSON string>
Simple example with two tags:
$ luigi --module my_tasks MyTask --tags '{"role": "web", "env": "staging"}'
It can be used to define dynamic parameters, when you do not know the exact list of your parameters (e.g. list of tags, that are dynamically constructed outside Luigi), or you have a complex parameter containing logically related values (like a database connection config).
Parameters: - default – the default value for this parameter. This should match the type of the
Parameter, i.e.
datetime.date
forDateParameter
orint
forIntParameter
. By default, no default is stored and the value must be specified at runtime. - significant (bool) – specify
False
if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default:True
. - description (str) – A human-readable string describing the purpose of this Parameter.
For command-line invocations, this will be used as the help string
shown to users. Default:
None
. - config_path (dict) – a dictionary with entries
section
andname
specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default:None
. - positional (bool) – If true, you can set the argument as a
positional argument. It’s true by default but we recommend
positional=False
for abstract base classes and similar cases. - always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
- batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
- visibility – A Parameter whose value is a
ParameterVisibility
. Default value is ParameterVisibility.PUBLIC
-
normalize
(value)[source]¶ Ensure that dictionary parameter is converted to a FrozenOrderedDict so it can be hashed.
-
parse
(source)[source]¶ Parses an immutable and ordered
dict
from a JSON string using standard JSON library.We need to use an immutable dictionary, to create a hashable parameter and also preserve the internal structure of parsing. The traversal order of standard
dict
is undefined, which can result various string representations of this parameter, and therefore a different task id for the task containing this parameter. This is because task id contains the hash of parameters’ JSON representation.Parameters: s – String to be parse
- default – the default value for this parameter. This should match the type of the
Parameter, i.e.
-
luigi.
run
(*args, **kwargs)[source]¶ Please dont use. Instead use luigi binary.
Run from cmdline using argparse.
Parameters: use_dynamic_argparse – Deprecated and ignored
-
luigi.
build
(tasks, worker_scheduler_factory=None, detailed_summary=False, **env_params)[source]¶ Run internally, bypassing the cmdline parsing.
Useful if you have some luigi code that you want to run internally. Example:
luigi.build([MyTask1(), MyTask2()], local_scheduler=True)
One notable difference is that build defaults to not using the identical process lock. Otherwise, build would only be callable once from each process.
Parameters: - tasks –
- worker_scheduler_factory –
- env_params –
Returns: True if there were no scheduling errors, even if tasks may fail.
-
class
luigi.
Event
[source]¶ Bases:
object
-
DEPENDENCY_DISCOVERED
= 'event.core.dependency.discovered'¶
-
DEPENDENCY_MISSING
= 'event.core.dependency.missing'¶
-
DEPENDENCY_PRESENT
= 'event.core.dependency.present'¶
-
BROKEN_TASK
= 'event.core.task.broken'¶
-
START
= 'event.core.start'¶
-
PROGRESS
= 'event.core.progress'¶ This event can be fired by the task itself while running. The purpose is for the task to report progress, metadata or any generic info so that event handler listening for this can keep track of the progress of running task.
-
FAILURE
= 'event.core.failure'¶
-
SUCCESS
= 'event.core.success'¶
-
PROCESSING_TIME
= 'event.core.processing_time'¶
-
TIMEOUT
= 'event.core.timeout'¶
-
PROCESS_FAILURE
= 'event.core.process_failure'¶
-
-
class
luigi.
NumericalParameter
(left_op=<built-in function le>, right_op=<built-in function lt>, *args, **kwargs)[source]¶ Bases:
luigi.parameter.Parameter
Parameter whose value is a number of the specified type, e.g.
int
orfloat
and in the range specified.In the task definition, use
class MyTask(luigi.Task): my_param_1 = luigi.NumericalParameter( var_type=int, min_value=-3, max_value=7) # -3 <= my_param_1 < 7 my_param_2 = luigi.NumericalParameter( var_type=int, min_value=-3, max_value=7, left_op=operator.lt, right_op=operator.le) # -3 < my_param_2 <= 7
At the command line, use
$ luigi --module my_tasks MyTask --my-param-1 -3 --my-param-2 -2
Parameters: - var_type (function) – The type of the input variable, e.g. int or float.
- min_value – The minimum value permissible in the accepted values range. May be inclusive or exclusive based on left_op parameter. This should be the same type as var_type.
- max_value – The maximum value permissible in the accepted values range. May be inclusive or exclusive based on right_op parameter. This should be the same type as var_type.
- left_op (function) – The comparison operator for the left-most comparison in
the expression
min_value left_op value right_op value
. This operator should generally be eitheroperator.lt
oroperator.le
. Default:operator.le
. - right_op (function) – The comparison operator for the right-most comparison in
the expression
min_value left_op value right_op value
. This operator should generally be eitheroperator.lt
oroperator.le
. Default:operator.lt
.
-
class
luigi.
ChoiceParameter
(var_type=<type 'str'>, *args, **kwargs)[source]¶ Bases:
luigi.parameter.Parameter
- A parameter which takes two values:
- an instance of
Iterable
and - the class of the variables to convert to.
- an instance of
In the task definition, use
class MyTask(luigi.Task): my_param = luigi.ChoiceParameter(choices=[0.1, 0.2, 0.3], var_type=float)
At the command line, use
$ luigi --module my_tasks MyTask --my-param 0.1
Consider using
EnumParameter
for a typed, structured alternative. This class can perform the same role when all choices are the same type and transparency of parameter value on the command line is desired.Parameters: - var_type (function) – The type of the input variable, e.g. str, int, float, etc. Default: str
- choices – An iterable, all of whose elements are of var_type to restrict parameter choices to.
-
parse
(s)[source]¶ Parse an individual value from the input.
The default implementation is the identity function, but subclasses should override this method for specialized parsing.
Parameters: x (str) – the value to parse. Returns: the parsed value.
-
normalize
(var)[source]¶ Given a parsed parameter value, normalizes it.
The value can either be the result of parse(), the default value or arguments passed into the task’s constructor by instantiation.
This is very implementation defined, but can be used to validate/clamp valid values. For example, if you wanted to only accept even integers, and “correct” odd values to the nearest integer, you can implement normalize as
x // 2 * 2
.
-
class
luigi.
OptionalParameter
(default=<object object>, is_global=False, significant=True, description=None, config_path=None, positional=True, always_in_help=False, batch_method=None, visibility=<ParameterVisibility.PUBLIC: 0>)[source]¶ Bases:
luigi.parameter.Parameter
A Parameter that treats empty string as None
Parameters: - default – the default value for this parameter. This should match the type of the
Parameter, i.e.
datetime.date
forDateParameter
orint
forIntParameter
. By default, no default is stored and the value must be specified at runtime. - significant (bool) – specify
False
if the parameter should not be treated as part of the unique identifier for a Task. An insignificant Parameter might also be used to specify a password or other sensitive information that should not be made public via the scheduler. Default:True
. - description (str) – A human-readable string describing the purpose of this Parameter.
For command-line invocations, this will be used as the help string
shown to users. Default:
None
. - config_path (dict) – a dictionary with entries
section
andname
specifying a config file entry from which to read the default value for this parameter. DEPRECATED. Default:None
. - positional (bool) – If true, you can set the argument as a
positional argument. It’s true by default but we recommend
positional=False
for abstract base classes and similar cases. - always_in_help (bool) – For the –help option in the command line parsing. Set true to always show in –help.
- batch_method (function(iterable[A])->A) – Method to combine an iterable of parsed parameter values into a single value. Used when receiving batched parameter lists from the scheduler. See Batching multiple parameter values into a single run
- visibility – A Parameter whose value is a
ParameterVisibility
. Default value is ParameterVisibility.PUBLIC
- default – the default value for this parameter. This should match the type of the
Parameter, i.e.
-
class
luigi.
LuigiStatusCode
[source]¶ Bases:
enum.Enum
All possible status codes for the attribute
status
inLuigiRunResult
when the argumentdetailed_summary=True
in luigi.run() / luigi.build. Here are the codes and what they mean:Status Code Name Meaning SUCCESS There were no failed tasks or missing dependencies SUCCESS_WITH_RETRY There were failed tasks but they all succeeded in a retry FAILED There were failed tasks FAILED_AND_SCHEDULING_FAILED There were failed tasks and tasks whose scheduling failed SCHEDULING_FAILED There were tasks whose scheduling failed NOT_RUN There were tasks that were not granted run permission by the scheduler MISSING_EXT There were missing external dependencies -
SUCCESS
= (':)', 'there were no failed tasks or missing dependencies')¶
-
SUCCESS_WITH_RETRY
= (':)', 'there were failed tasks but they all succeeded in a retry')¶
-
FAILED
= (':(', 'there were failed tasks')¶
-
FAILED_AND_SCHEDULING_FAILED
= (':(', 'there were failed tasks and tasks whose scheduling failed')¶
-
SCHEDULING_FAILED
= (':(', 'there were tasks whose scheduling failed')¶
-
NOT_RUN
= (':|', 'there were tasks that were not granted run permission by the scheduler')¶
-
MISSING_EXT
= (':|', 'there were missing external dependencies')¶
-