luigi.contrib.prometheus_metric

Classes

PrometheusMetricsCollector(*args, **kwargs)
prometheus(*args, **kwargs)

class luigi.contrib.prometheus_metric.prometheus(*args, **kwargs)

use_task_family_in_labels

A Parameter whose value is a bool. This parameter has an implicit default value of False. For the command line interface this means that the value is False unless you add "--the-bool-parameter" to your command without giving a parameter value. This is considered implicit parsing (the default). However, in some situations one might want to give the explicit bool value ("--the-bool-parameter true|false"), e.g. when you configure the default value to be True. This is called explicit parsing. When omitting the parameter value, it is still considered True but to avoid ambiguities during argument parsing, make sure to always place bool parameters behind the task family on the command line when using explicit parsing.

You can toggle between the two parsing modes on a per-parameter base via

class MyTask(luigi.Task):
    implicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.IMPLICIT_PARSING)
    explicit_bool = luigi.BoolParameter(parsing=luigi.BoolParameter.EXPLICIT_PARSING)

or globally by

luigi.BoolParameter.parsing = luigi.BoolParameter.EXPLICIT_PARSING

for all bool parameters instantiated after this line.

task_parameters_to_use_in_labels

Parameter whose value is a list.

In the task definition, use

class MyTask(luigi.Task):
  grades = luigi.ListParameter()

    def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

At the command line, use

$ luigi --module my_tasks MyTask --grades <JSON string>

Simple example with two grades:

$ luigi --module my_tasks MyTask --grades '[100,70]'

It is possible to provide a JSON schema that should be validated by the given value:

class MyTask(luigi.Task):
  grades = luigi.ListParameter(
    schema={
      "type": "array",
      "items": {
        "type": "number",
        "minimum": 0,
        "maximum": 10
      },
      "minItems": 1
    }
  )

  def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

Using this schema, the following command will work:

$ luigi --module my_tasks MyTask --numbers '[1, 8.7, 6]'

while these commands will fail because the parameter is not valid:

$ luigi --module my_tasks MyTask --numbers '[]'  # must have at least 1 element
$ luigi --module my_tasks MyTask --numbers '[-999, 999]'  # elements must be in [0, 10]

Finally, the provided schema can be a custom validator:

custom_validator = jsonschema.Draft4Validator(
  schema={
    "type": "array",
    "items": {
      "type": "number",
      "minimum": 0,
      "maximum": 10
    },
    "minItems": 1
  }
)

class MyTask(luigi.Task):
  grades = luigi.ListParameter(schema=custom_validator)

  def run(self):
        sum = 0
        for element in self.grades:
            sum += element
        avg = sum / len(self.grades)

class luigi.contrib.prometheus_metric.PrometheusMetricsCollector(*args, **kwargs)

generate_latest()

handle_task_started(task)

handle_task_failed(task)

handle_task_disabled(task, config)

handle_task_done(task)

configure_http_handler(http_handler)