luigi.contrib.pyspark_runner module

The pyspark program.

This module will be run by spark-submit for PySparkTask jobs.

The first argument is a path to the pickled instance of the PySparkTask, other arguments are the ones returned by PySparkTask.app_options()

class luigi.contrib.pyspark_runner.PySparkRunner(job, *args)[source]

Bases: object

run()[source]