Configuration
=============

All configuration can be done by adding configuration files.

Supported config parsers:

* ``cfg`` (default), based on Python's standard ConfigParser_. Values may refer to environment variables using ``${ENVVAR}`` syntax.
* ``toml``

.. _ConfigParser: https://docs.python.org/3/library/configparser.html

You can choose right parser via ``LUIGI_CONFIG_PARSER`` environment variable. For example, ``LUIGI_CONFIG_PARSER=toml``.

Default (cfg) parser are looked for in:

* ``/etc/luigi/client.cfg`` (deprecated)
* ``/etc/luigi/luigi.cfg``
* ``client.cfg`` (deprecated)
* ``luigi.cfg``
* ``LUIGI_CONFIG_PATH`` environment variable

`TOML <https://github.com/toml-lang/toml>`_ parser are looked for in:

* ``/etc/luigi/luigi.toml``
* ``luigi.toml``
* ``LUIGI_CONFIG_PATH`` environment variable

Both config lists increase in priority (from low to high). The order only
matters in case of key conflicts (see docs for ConfigParser.read_).
These files are meant for both the client and ``luigid``.
If you decide to specify your own configuration you should make sure
that both the client and ``luigid`` load it properly.

.. _ConfigParser.read: https://docs.python.org/3.6/library/configparser.html#configparser.ConfigParser.read

The config file is broken into sections, each controlling a different part of the config.

Example cfg config:

.. code:: ini

    [hadoop]
    version=cdh4
    streaming_jar=/usr/lib/hadoop-xyz/hadoop-streaming-xyz-123.jar

    [core]
    scheduler_host=luigi-host.mycompany.foo

Example toml config:

.. code:: python

    [hadoop]
    version = "cdh4"
    streaming_jar = "/usr/lib/hadoop-xyz/hadoop-streaming-xyz-123.jar"

    [core]
    scheduler_host = "luigi-host.mycompany.foo"

Also see `examples/config.toml
<https://github.com/spotify/luigi/blob/master/examples/config.toml>`_
for more complex example.

.. _ParamConfigIngestion:

Parameters from config Ingestion
--------------------------------

All parameters can be overridden from configuration files. For instance if you
have a Task definition:

.. code:: python

    class DailyReport(luigi.contrib.hadoop.JobTask):
        date = luigi.DateParameter(default=datetime.date.today())
        # ...

Then you can override the default value for ``DailyReport().date`` by providing
it in the configuration:

.. code:: ini

    [DailyReport]
    date=2012-01-01

.. _ConfigClasses:

Configuration classes
*********************

Using the :ref:`ParamConfigIngestion` method, we derive the
conventional way to do global configuration. Imagine this configuration.

.. code:: ini

    [mysection]
    option=hello
    intoption=123


We can create a :py:class:`~luigi.Config` class:

.. code:: python

    import luigi

    # Config classes should be camel cased
    class mysection(luigi.Config):
        option = luigi.Parameter(default='world')
        intoption = luigi.IntParameter(default=555)

    mysection().option
    mysection().intoption


Configurable options
--------------------

Luigi comes with a lot of configurable options. Below, we describe each
section and the parameters available within it.


[core]
------

These parameters control core Luigi behavior, such as error e-mails and
interactions between the worker and scheduler.

autoload_range
  .. versionadded:: 2.8.11

  If false, prevents range tasks from autoloading. They can still be loaded
  using ``--module luigi.tools.range``. Defaults to true. Setting this to true
  explicitly disables the deprecation warning.

default_scheduler_host
  Hostname of the machine running the scheduler. Defaults to localhost.

default_scheduler_port
  Port of the remote scheduler api process. Defaults to 8082.

default_scheduler_url
  Full path to remote scheduler. Defaults to ``http://localhost:8082/``.
  For TLS support use the URL scheme: ``https``,
  example: ``https://luigi.example.com:443/``
  (Note: you will have to terminate TLS using an HTTP proxy)
  You can also use this to connect to a local Unix socket using the
  non-standard URI scheme: ``http+unix``
  example: ``http+unix://%2Fvar%2Frun%2Fluigid%2Fluigid.sock/``

hdfs_tmp_dir
  Base directory in which to store temporary files on hdfs. Defaults to
  tempfile.gettempdir()

history_filename
  If set, specifies a filename for Luigi to write stuff (currently just
  job id) to in mapreduce job's output directory. Useful in a
  configuration where no history is stored in the output directory by
  Hadoop.

log_level
  The default log level to use when no logging_conf_file is set. Must be
  a valid name of a `Python log level
  <https://docs.python.org/2/library/logging.html#logging-levels>`_.
  Default is ``DEBUG``.

logging_conf_file
  Location of the logging configuration file.

no_configure_logging
  If true, logging is not configured. Defaults to false.

parallel_scheduling
  If true, the scheduler will compute complete functions of tasks in
  parallel using multiprocessing. This can significantly speed up
  scheduling, but requires that all tasks can be pickled.
  Defaults to false.

parallel_scheduling_processes
  The number of processes to use for parallel scheduling. If not specified
  the default number of processes will be the total number of CPUs available.

rpc_connect_timeout
  Number of seconds to wait before timing out when making an API call.
  Defaults to 10.0

rpc_retry_attempts
  The maximum number of retries to connect the central scheduler before giving up.
  Defaults to 3

rpc_retry_wait
  Number of seconds to wait before the next attempt will be started to
  connect to the central scheduler between two retry attempts.
  Defaults to 30


[cors]
------

.. versionadded:: 2.8.0

These parameters control ``/api/<method>`` ``CORS`` behaviour (see: `W3C Cross-Origin Resource Sharing
<http://www.w3.org/TR/cors/>`_).

enabled
  Enables CORS support.
  Defaults to false.

allowed_origins
  A list of allowed origins. Used only if ``allow_any_origin`` is false.
  Configure in JSON array format, e.g. ["foo", "bar"].
  Defaults to empty.

allow_any_origin
  Accepts requests from any origin.
  Defaults to false.

allow_null_origin
  Allows the request to set ``null`` value of the ``Origin`` header.
  Defaults to false.

max_age
  Content of ``Access-Control-Max-Age``.
  Defaults to 86400 (24 hours).

allowed_methods
  Content of ``Access-Control-Allow-Methods``.
  Defaults to ``GET, OPTIONS``.

allowed_headers
  Content of ``Access-Control-Allow-Headers``.
  Defaults to ``Accept, Content-Type, Origin``.

exposed_headers
  Content of ``Access-Control-Expose-Headers``.
  Defaults to empty string (will NOT be sent as a response header).

allow_credentials
  Indicates that the actual request can include user credentials.
  Defaults to false.

.. _worker-config:

[worker]
--------

These parameters control Luigi worker behavior.

count_uniques
  If true, workers will only count unique pending jobs when deciding
  whether to stay alive. So if a worker can't get a job to run and other
  workers are waiting on all of its pending jobs, the worker will die.
  ``worker_keep_alive`` must be ``true`` for this to have any effect. Defaults
  to false.

keep_alive
  If true, workers will stay alive when they run out of jobs to run, as
  long as they have some pending job waiting to be run. Defaults to
  false.

ping_interval
  Number of seconds to wait between pinging scheduler to let it know
  that the worker is still alive. Defaults to 1.0.

task_limit
  .. versionadded:: 1.0.25

  Maximum number of tasks to schedule per invocation. Upon exceeding it,
  the worker will issue a warning and proceed with the workflow obtained
  thus far. Prevents incidents due to spamming of the scheduler, usually
  accidental. Default: no limit.

timeout
  .. versionadded:: 1.0.20

  Number of seconds after which to kill a task which has been running
  for too long. This provides a default value for all tasks, which can
  be overridden by setting the ``worker_timeout`` property in any task.
  Default value is 0, meaning no timeout.

wait_interval
  Number of seconds for the worker to wait before asking the scheduler
  for another job after the scheduler has said that it does not have any
  available jobs.

wait_jitter
  Duration of jitter to add to the worker wait interval such that the multiple
  workers do not ask the scheduler for another job at the same time, in seconds.
  Default: 5.0

max_keep_alive_idle_duration
  .. versionadded:: 2.8.4

  Maximum duration in seconds to keep worker alive while in idle state.
  Default: 0 (Indefinitely)

max_reschedules
  The maximum number of times that a job can be automatically
  rescheduled by a worker before it will stop trying. Workers will
  reschedule a job if it is found to not be done when attempting to run
  a dependent job. This defaults to 1.

retry_external_tasks
  If true, incomplete external tasks (i.e. tasks where the ``run()`` method is
  NotImplemented) will be retested for completion while Luigi is running.
  This means that if external dependencies are satisfied after a workflow has
  started, any tasks dependent on that resource will be eligible for running.
  Note: Every time the task remains incomplete, it will count as FAILED, so
  normal retry logic applies (see: ``retry_count`` and ``retry_delay``).
  This setting works best with ``worker_keep_alive: true``.
  If false, external tasks will only be evaluated when Luigi is first invoked.
  In this case, Luigi will not check whether external dependencies are
  satisfied  while a workflow is in progress, so dependent tasks will remain
  PENDING until the workflow is reinvoked.
  Defaults to false for backwards compatibility.

no_install_shutdown_handler
  By default, workers will stop requesting new work and finish running
  pending tasks after receiving a ``SIGUSR1`` signal. This provides a hook
  for gracefully shutting down workers that are in the process of running
  (potentially expensive) tasks. If set to true, Luigi will NOT install
  this shutdown hook on workers. Note this hook does not work on Windows
  operating systems, or when jobs are launched outside the main execution
  thread.
  Defaults to false.

send_failure_email
  Controls whether the worker will send e-mails on task and scheduling
  failures. If set to false, workers will only send e-mails on
  framework errors during scheduling and all other e-mail must be
  handled by the scheduler.
  Defaults to true.

check_unfulfilled_deps
  If true, the worker checks for completeness of dependencies before running a
  task. In case unfulfilled dependencies are detected, an exception is raised
  and the task will not run. This mechanism is useful to detect situations
  where tasks do not create their outputs properly, or when targets were
  removed after the dependency tree was built. It is recommended to disable
  this feature only when the completeness checks are known to be bottlenecks,
  e.g. when the ``exists()`` calls of the dependencies' outputs are
  resource-intensive.
  Defaults to true.

force_multiprocessing
  By default, luigi uses multiprocessing when *more than one* worker process is
  requested. When set to true, multiprocessing is used independent of the
  number of workers.
  Defaults to false.

check_complete_on_run
  By default, luigi tasks are marked as 'done' when they finish running without
  raising an error. When set to true, tasks will also verify that their outputs
  exist when they finish running, and will fail immediately if the outputs are
  missing.
  Defaults to false.

cache_task_completion
  By default, luigi task processes might check the completion status multiple
  times per task which is a safe way to avoid potential inconsistencies. For
  tasks with many dynamic dependencies, yielded in multiple stages, this might
  become expensive, e.g. in case the per-task completion check entails remote
  resources. When set to true, completion checks are cached so that tasks
  declared as complete once are not checked again.
  Defaults to false.


[elasticsearch]
---------------

These parameters control use of elasticsearch

marker_index
  Defaults to "update_log".

marker_doc_type
  Defaults to "entry".


[email]
-------

General parameters

force_send
  If true, e-mails are sent in all run configurations (even if stdout is
  connected to a tty device).  Defaults to False.

format
  Type of e-mail to send. Valid values are "plain", "html" and "none".
  When set to html, tracebacks are wrapped in <pre> tags to get fixed-
  width font. When set to none, no e-mails will be sent.

  Default value is plain.

method
  Valid values are "smtp", "sendgrid", "ses" and "sns". SES and SNS are
  services of Amazon web services. SendGrid is an email delivery service.
  The default value is "smtp".

  In order to send messages through Amazon SNS or SES set up your AWS
  config files or run Luigi on an EC2 instance with proper instance
  profile.

  In order to use sendgrid, fill in your sendgrid API key in the
  `[sendgrid]`_ section.

  In order to use smtp, fill in the appropriate fields in the `[smtp]`_
  section.

prefix
  Optional prefix to add to the subject line of all e-mails. For
  example, setting this to "[LUIGI]" would change the subject line of an
  e-mail from "Luigi: Framework error" to "[LUIGI] Luigi: Framework
  error"

receiver
  Recipient of all error e-mails. If this is not set, no error e-mails
  are sent when Luigi crashes unless the crashed job has owners set. If
  Luigi is run from the command line, no e-mails will be sent unless
  output is redirected to a file.

  Set it to SNS Topic ARN if you want to receive notifications through
  Amazon SNS. Make sure to set method to sns in this case too.

sender
  User name in from field of error e-mails.
  Default value: luigi-client@<server_name>

traceback_max_length
  Maximum length for traceback included in error email. Default is 5000.


[batch_notifier]
----------------

Parameters controlling the contents of batch notifications sent from the
scheduler

email_interval_minutes
  Number of minutes between e-mail sends. Making this larger results in
  fewer, bigger e-mails.
  Defaults to 60.

batch_mode
  Controls how tasks are grouped together in the e-mail. Suppose we have
  the following sequence of failures:

  1. TaskA(a=1, b=1)
  2. TaskA(a=1, b=1)
  3. TaskA(a=2, b=1)
  4. TaskA(a=1, b=2)
  5. TaskB(a=1, b=1)

  For any setting of batch_mode, the batch e-mail will record 5 failures
  and mention them in the subject. The difference is in how they will
  be displayed in the body. Here are example bodies with error_messages
  set to 0.

  "all" only groups together failures for the exact same task:

  - TaskA(a=1, b=1) (2 failures)
  - TaskA(a=1, b=2) (1 failure)
  - TaskA(a=2, b=1) (1 failure)
  - TaskB(a=1, b=1) (1 failure)

  "family" groups together failures for tasks of the same family:

  - TaskA (4 failures)
  - TaskB (1 failure)

  "unbatched_params" groups together tasks that look the same after
  removing batched parameters. So if TaskA has a batch_method set for
  parameter a, we get the following:

  - TaskA(b=1) (3 failures)
  - TaskA(b=2) (1 failure)
  - TaskB(a=1, b=2) (1 failure)

  Defaults to "unbatched_params", which is identical to "all" if you are
  not using batched parameters.

error_lines
  Number of lines to include from each error message in the batch
  e-mail. This can be used to keep e-mails shorter while preserving the
  more useful information usually found near the bottom of stack traces.
  This can be set to 0 to include all lines. If you don't wish to see
  error messages, instead set ``error_messages`` to 0.
  Defaults to 20.

error_messages
  Number of messages to preserve for each task group. As most tasks that
  fail repeatedly do so for similar reasons each time, it's not usually
  necessary to keep every message. This controls how many messages are
  kept for each task or task group. The most recent error messages are
  kept. Set to 0 to not include error messages in the e-mails.
  Defaults to 1.

group_by_error_messages
  Quite often, a system or cluster failure will cause many disparate
  task types to fail for the same reason. This can cause a lot of noise
  in the batch e-mails. This cuts down on the noise by listing items
  with identical error messages together. Error messages are compared
  after limiting by ``error_lines``.
  Defaults to true.


[hadoop]
--------

Parameters controlling basic hadoop tasks

command
  Name of command for running hadoop from the command line. Defaults to
  "hadoop"

python_executable
  Name of command for running python from the command line. Defaults to
  "python"

scheduler
  Type of scheduler to use when scheduling hadoop jobs. Can be "fair" or
  "capacity". Defaults to "fair".

streaming_jar
  Path to your streaming jar. Must be specified to run streaming jobs.

version
  Version of hadoop used in your cluster. Can be "cdh3", "chd4", or
  "apache1". Defaults to "cdh4".


[hdfs]
------

Parameters controlling the use of snakebite to speed up hdfs queries.

client
  Client to use for most hadoop commands. Options are "snakebite",
  "snakebite_with_hadoopcli_fallback", "webhdfs" and "hadoopcli". Snakebite is
  much faster, so use of it is encouraged. webhdfs is fast and works with
  Python 3 as well, but has not been used that much in the wild.
  Both snakebite and webhdfs requires you to install it separately on
  the machine. Defaults to "hadoopcli".

client_version
  Optionally specifies hadoop client version for snakebite.

effective_user
  Optionally specifies the effective user for snakebite.

namenode_host
  The hostname of the namenode. Needed for snakebite if
  snakebite_autoconfig is not set.

namenode_port
  The port used by snakebite on the namenode. Needed for snakebite if
  snakebite_autoconfig is not set.

snakebite_autoconfig
  If true, attempts to automatically detect the host and port of the
  namenode for snakebite queries. Defaults to false.

tmp_dir
  Path to where Luigi will put temporary files on hdfs


[hive]
------

Parameters controlling hive tasks

command
  Name of the command used to run hive on the command line. Defaults to
  "hive".

hiverc_location
  Optional path to hive rc file.

metastore_host
  Hostname for metastore.

metastore_port
  Port for hive to connect to metastore host.

release
  If set to "apache", uses a hive client that better handles apache
  hive output. All other values use the standard client Defaults to
  "cdh4".


[kubernetes]
------------

Parameters controlling Kubernetes Job Tasks

auth_method
  Authorization method to access the cluster.
  Options are "kubeconfig_" or "service-account_"

kubeconfig_path
  Path to kubeconfig file, for cluster authentication.
  It defaults to ``~/.kube/config``, which is the default location when
  using minikube_.
  When auth_method is "service-account" this property is ignored.

max_retrials
  Maximum number of retrials in case of job failure.

.. _service-account: http://kubernetes.io/docs/user-guide/kubeconfig-file
.. _kubeconfig: http://kubernetes.io/docs/user-guide/service-accounts
.. _minikube: http://kubernetes.io/docs/getting-started-guides/minikube


[mysql]
-------

Parameters controlling use of MySQL targets

marker_table
  Table in which to store status of table updates. This table will be
  created if it doesn't already exist. Defaults to "table_updates".


[postgres]
----------

Parameters controlling the use of Postgres targets

local_tmp_dir
  Directory in which to temporarily store data before writing to
  postgres. Uses system default if not specified.

marker_table
  Table in which to store status of table updates. This table will be
  created if it doesn't already exist. Defaults to "table_updates".


[redshift]
----------

Parameters controlling the use of Redshift targets

marker_table
  Table in which to store status of table updates. This table will be
  created if it doesn't already exist. Defaults to "table_updates".

.. _resources-config:

[resources]
-----------

This section can contain arbitrary keys. Each of these specifies the
amount of a global resource that the scheduler can allow workers to use.
The scheduler will prevent running jobs with resources specified from
exceeding the counts in this section. Unspecified resources are assumed
to have limit 1. Example resources section for a configuration with 2
hive resources and 1 mysql resource:

.. code:: ini

  [resources]
  hive=2
  mysql=1

Note that it was not necessary to specify the 1 for mysql here, but it
is good practice to do so when you have a fixed set of resources.

.. _retcode-config:

[retcode]
---------

Configure return codes for the Luigi binary. In the case of multiple return
codes that could apply, for example a failing task and missing data, the
*numerically greatest* return code is returned.

We recommend that you copy this set of exit codes to your ``luigi.cfg`` file:

.. code:: ini

  [retcode]
  # The following return codes are the recommended exit codes for Luigi
  # They are in increasing level of severity (for most applications)
  already_running=10
  missing_data=20
  not_run=25
  task_failed=30
  scheduling_error=35
  unhandled_exception=40

already_running
  This can happen in two different cases. Either the local lock file was taken
  at the time the invocation starts up. Or, the central scheduler have reported
  that some tasks could not have been run, because other workers are already
  running the tasks.
missing_data
  For when an :py:class:`~luigi.task.ExternalTask` is not complete, and this
  caused the worker to give up.  As an alternative to fiddling with this, see
  the [worker] keep_alive option.
not_run
  For when a task is not granted run permission by the scheduler. Typically
  because of lack of resources, because the task has been already run by
  another worker or because the attempted task is in DISABLED state.
  Connectivity issues with the central scheduler might also cause this.
  This does not include the cases for which a run is not allowed due to missing
  dependencies (missing_data) or due to the fact that another worker is currently
  running the task (already_running).
task_failed
  For signaling that there were last known to have failed. Typically because
  some exception have been raised.
scheduling_error
  For when a task's ``complete()`` or ``requires()`` method fails with an
  exception, or when the limit number of tasks is reached.
unhandled_exception
  For internal Luigi errors.  Defaults to 4, since this type of error
  probably will not recover over time.

If you customize return codes, prefer to set them in range 128 to 255 to avoid
conflicts. Return codes in range 0 to 127 are reserved for possible future use
by Luigi contributors.

[scalding]
----------

Parameters controlling running of scalding jobs

scala_home
  Home directory for scala on your machine. Defaults to either
  SCALA_HOME or /usr/share/scala if SCALA_HOME is unset.

scalding_home
  Home directory for scalding on your machine. Defaults to either
  SCALDING_HOME or /usr/share/scalding if SCALDING_HOME is unset.

scalding_provided
  Provided directory for scalding on your machine. Defaults to either
  SCALDING_HOME/provided or /usr/share/scalding/provided

scalding_libjars
  Libjars directory for scalding on your machine. Defaults to either
  SCALDING_HOME/libjars or /usr/share/scalding/libjars


.. _scheduler-config:

[scheduler]
-----------

Parameters controlling scheduler behavior

batch_emails
  Whether to send batch e-mails for failures and disables rather than
  sending immediate disable e-mails and just relying on workers to send
  immediate batch e-mails.
  Defaults to false.

disable_hard_timeout
  Hard time limit after which tasks will be disabled by the server if
  they fail again, in seconds. It will disable the task if it fails
  **again** after this amount of time. E.g. if this was set to 600
  (i.e. 10 minutes), and the task first failed at 10:00am, the task would
  be disabled if it failed again any time after 10:10am. Note: This setting
  does not consider the values of the ``retry_count`` or
  ``disable_window`` settings.

retry_count
  Number of times a task can fail within ``disable_window`` before
  the scheduler will automatically disable it. If not set, the scheduler
  will not automatically disable jobs.

disable_persist
  Number of seconds for which an automatic scheduler disable lasts.
  Defaults to 86400 (1 day).

disable_window
  Number of seconds during which ``retry_count`` failures must
  occur in order for an automatic disable by the scheduler. The
  scheduler forgets about disables that have occurred longer ago than
  this amount of time. Defaults to 3600 (1 hour).

max_shown_tasks
  .. versionadded:: 1.0.20

  The maximum number of tasks returned in a task_list api call. This
  will restrict the number of tasks shown in task lists in the
  visualiser. Small values can alleviate frozen browsers when there are
  too many done tasks. This defaults to 100000 (one hundred thousand).

max_graph_nodes
  .. versionadded:: 2.0.0

  The maximum number of nodes returned by a dep_graph or
  inverse_dep_graph api call. Small values can greatly speed up graph
  display in the visualiser by limiting the number of nodes shown. Some
  of the nodes that are not sent to the visualiser will still show up as
  dependencies of nodes that were sent. These nodes are given TRUNCATED
  status.

record_task_history
  If true, stores task history in a database. Defaults to false.

remove_delay
  Number of seconds to wait before removing a task that has no
  stakeholders. Defaults to 600 (10 minutes).

retry_delay
  Number of seconds to wait after a task failure to mark it pending
  again. Defaults to 900 (15 minutes).

state_path
  Path in which to store the Luigi scheduler's state. When the scheduler
  is shut down, its state is stored in this path. The scheduler must be
  shut down cleanly for this to work, usually with a kill command. If
  the kill command includes the -9 flag, the scheduler will not be able
  to save its state. When the scheduler is started, it will load the
  state from this path if it exists. This will restore all scheduled
  jobs and other state from when the scheduler last shut down.

  Sometimes this path must be deleted when restarting the scheduler
  after upgrading Luigi, as old state files can become incompatible
  with the new scheduler. When this happens, all workers should be
  restarted after the scheduler both to become compatible with the
  updated code and to reschedule the jobs that the scheduler has now
  forgotten about.

  This defaults to /var/lib/luigi-server/state.pickle

worker_disconnect_delay
  Number of seconds to wait after a worker has stopped pinging the
  scheduler before removing it and marking all of its running tasks as
  failed. Defaults to 60.

pause_enabled
  If false, disables pause/unpause operations and hides the pause toggle from
  the visualiser.

send_messages
  When true, the scheduler is allowed to send messages to running tasks and
  the central scheduler provides a simple prompt per task to send messages.
  Defaults to true.

metrics_collector
  Optional setting allowing Luigi to use a contribution to collect metrics
  about the pipeline to a third-party. By default this uses the default metric
  collector that acts as a shell and does nothing. The currently available
  options are "datadog", "prometheus" and "custom". If it's custom the
  'metrics_custom_import' needs to be set.

metrics_custom_import
  Optional setting allowing Luigi to import a custom subclass of MetricsCollector
  at runtime. The string should be formatted like "module.sub_module.ClassName".


[sendgrid]
----------

These parameters control sending error e-mails through SendGrid.

apikey
  API key of the SendGrid account.


[smtp]
------

These parameters control the smtp server setup.

host
  Hostname for sending mail through smtp. Defaults to localhost.

local_hostname
  If specified, overrides the FQDN of localhost in the HELO/EHLO
  command.

no_tls
  If true, connects to smtp without TLS. Defaults to false.

password
  Password to log in to your smtp server. Must be specified for
  username to have an effect.

port
  Port number for smtp on smtp_host. Defaults to 0.

ssl
  If true, connects to smtp through SSL. Defaults to false.

timeout
  Sets the number of seconds after which smtp attempts should time out.
  Defaults to 10.

username
  Username to log in to your smtp server, if necessary.


[spark]
-------

Parameters controlling the default execution of :py:class:`~luigi.contrib.spark.SparkSubmitTask` and :py:class:`~luigi.contrib.spark.PySparkTask`:

.. deprecated:: 1.1.1
   :py:class:`~luigi.contrib.spark.SparkJob`, :py:class:`~luigi.contrib.spark.Spark1xJob` and :py:class:`~luigi.contrib.spark.PySpark1xJob`
    are deprecated. Please use :py:class:`~luigi.contrib.spark.SparkSubmitTask` or :py:class:`~luigi.contrib.spark.PySparkTask`.

spark_submit
  Command to run in order to submit spark jobs. Default: ``"spark-submit"``

master
  Master url to use for ``spark_submit``. Example: local[*], spark://masterhost:7077. Default: Spark default (Prior to 1.1.1: yarn-client)

deploy_mode
    Whether to launch the driver programs locally ("client") or on one of the worker machines inside the cluster ("cluster"). Default: Spark default

jars
    Comma-separated list of local jars to include on the driver and executor classpaths. Default: Spark default

packages
    Comma-separated list of packages to link to on the driver and executors

py_files
    Comma-separated list of .zip, .egg, or .py files to place on the ``PYTHONPATH`` for Python apps. Default: Spark default

files
    Comma-separated list of files to be placed in the working directory of each executor. Default: Spark default

conf:
    Arbitrary Spark configuration property in the form Prop=Value|Prop2=Value2. Default: Spark default

properties_file
    Path to a file from which to load extra properties. Default: Spark default

driver_memory
    Memory for driver (e.g. 1000M, 2G). Default: Spark default

driver_java_options
    Extra Java options to pass to the driver. Default: Spark default

driver_library_path
    Extra library path entries to pass to the driver. Default: Spark default

driver_class_path
    Extra class path entries to pass to the driver. Default: Spark default

executor_memory
    Memory per executor (e.g. 1000M, 2G). Default: Spark default

*Configuration for Spark submit jobs on Spark standalone with cluster deploy mode only:*

driver_cores
    Cores for driver. Default: Spark default

supervise
    If given, restarts the driver on failure. Default: Spark default

*Configuration for Spark submit jobs on Spark standalone and Mesos only:*

total_executor_cores
    Total cores for all executors. Default: Spark default

*Configuration for Spark submit jobs on YARN only:*

executor_cores
    Number of cores per executor. Default: Spark default

queue
    The YARN queue to submit to. Default: Spark default

num_executors
    Number of executors to launch. Default: Spark default

archives
    Comma separated list of archives to be extracted into the working directory of each executor. Default: Spark default

hadoop_conf_dir
  Location of the hadoop conf dir. Sets HADOOP_CONF_DIR environment variable
  when running spark. Example: /etc/hadoop/conf

*Extra configuration for PySparkTask jobs:*

py_packages
    Comma-separated list of local packages (in your python path) to be distributed to the cluster.

*Parameters controlling the execution of SparkJob jobs (deprecated):*


[task_history]
--------------

Parameters controlling storage of task history in a database

db_connection
  Connection string for connecting to the task history db using
  sqlalchemy.


[execution_summary]
-------------------

Parameters controlling execution summary of a worker

summary_length
  Maximum number of tasks to show in an execution summary.  If the value is 0,
  then all tasks will be displayed.  Default value is 5.


[webhdfs]
---------

port
  The port to use for webhdfs. The normal namenode port is probably on a
  different port from this one.

user
  Perform file system operations as the specified user instead of $USER.  Since
  this parameter is not honored by any of the other hdfs clients, you should
  think twice before setting this parameter.

client_type
  The type of client to use. Default is the "insecure" client that requires no
  authentication. The other option is the "kerberos" client that uses kerberos
  authentication.

[datadog]
---------

api_key
  The api key found in the account settings of Datadog under the API
  sections.
app_key
  The application key found in the account settings of Datadog under the API
  sections.
default_tags
  Optional settings that adds the tag to all the metrics and events sent to
  Datadog. Default value is "application:luigi".
environment
  Allows you to tweak multiple environment to differentiate between production,
  staging or development metrics within Datadog. Default value is "development".
statsd_host
  The host that has the statsd instance to allow Datadog to send statsd metric. Default value is "localhost".
statsd_port
  The port on the host that allows connection to the statsd host. Defaults value is 8125.
metric_namespace
  Optional prefix to add to the beginning of every metric sent to Datadog.
  Default value is "luigi".

Per Task Retry-Policy
---------------------

Luigi also supports defining ``retry_policy`` per task.

.. code-block:: python

    class GenerateWordsFromHdfs(luigi.Task):

       retry_count = 2

        ...

    class GenerateWordsFromRDBM(luigi.Task):

       retry_count = 5

        ...

    class CountLetters(luigi.Task):

        def requires(self):
            return [GenerateWordsFromHdfs()]

        def run():
            yield GenerateWordsFromRDBM()

        ...

If none of retry-policy fields is defined per task, the field value will be **default** value which is defined in luigi config file.

To make luigi sticks to the given retry-policy, be sure you run luigi worker with ``keep_alive`` config. Please check ``keep_alive`` config in :ref:`worker-config` section.

Retry-Policy Fields
-------------------

The fields below are in retry-policy and they can be defined per task.

* ``retry_count``
* ``disable_hard_timeout``
* ``disable_window``