luigi.contrib.pyspark_runner module

The pyspark program.

This module will be run by spark-submit for PySparkTask jobs.

The first argument is a path to the pickled instance of the PySparkTask, other arguments are the ones returned by PySparkTask.app_options()

class luigi.contrib.pyspark_runner.SparkContextEntryPoint(conf)[source]

Bases: luigi.contrib.pyspark_runner._SparkEntryPoint

sc = None
class luigi.contrib.pyspark_runner.SparkSessionEntryPoint(conf)[source]

Bases: luigi.contrib.pyspark_runner._SparkEntryPoint

spark = None
class luigi.contrib.pyspark_runner.AbstractPySparkRunner(job, *args)[source]

Bases: object

run()[source]
class luigi.contrib.pyspark_runner.PySparkRunner(job, *args)

Bases: luigi.contrib.pyspark_runner.AbstractPySparkRunner

class luigi.contrib.pyspark_runner.PySparkSessionRunner(job, *args)

Bases: luigi.contrib.pyspark_runner.AbstractPySparkRunner