luigi.contrib.pyspark_runner module

The pyspark program.

This module will be run by spark-submit for PySparkTask jobs.

The first argument is a path to the pickled instance of the PySparkTask, other arguments are the ones returned by PySparkTask.app_options()

class luigi.contrib.pyspark_runner.SparkContextEntryPoint(conf)

Bases: _SparkEntryPoint

sc = None

class luigi.contrib.pyspark_runner.SparkSessionEntryPoint(conf)

Bases: _SparkEntryPoint

spark = None

class luigi.contrib.pyspark_runner.AbstractPySparkRunner(job, *args)

Bases: object

run()

class luigi.contrib.pyspark_runner.PySparkRunner(job, *args)

Bases: AbstractPySparkRunner

class luigi.contrib.pyspark_runner.PySparkSessionRunner(job, *args)

Bases: AbstractPySparkRunner