luigi.contrib.mrrunner module

Since after Luigi 2.5.0, this is a private module to Luigi. Luigi users should not rely on that importing this module works. Furthermore, “luigi mr streaming” have been greatly superseeded by technoligies like Spark, Hive, etc.

The hadoop runner.

This module contains the main() method which will be used to run the mapper, combiner, or reducer on the Hadoop nodes.

class luigi.contrib.mrrunner.Runner(job=None)

Bases: object

Run the mapper, combiner, or reducer on hadoop nodes.

run(kind, stdin=<open file '<stdin>', mode 'r'>, stdout=<open file '<stdout>', mode 'w'>)

extract_packages_archive()

luigi.contrib.mrrunner.print_exception(exc)

luigi.contrib.mrrunner.main(args=None, stdin=<open file '<stdin>', mode 'r'>, stdout=<open file '<stdout>', mode 'w'>, print_exception=<function print_exception>)

Run either the mapper, combiner, or reducer from the class instance in the file “job-instance.pickle”.

Arguments:

kind – is either map, combiner, or reduce