luigi.contrib.pyspark_runner module¶
The pyspark program.
This module will be run by spark-submit for PySparkTask jobs.
The first argument is a path to the pickled instance of the PySparkTask, other arguments are the ones returned by PySparkTask.app_options()
- class luigi.contrib.pyspark_runner.SparkContextEntryPoint(conf)[source]¶
Bases:
_SparkEntryPoint
- sc = None¶
- class luigi.contrib.pyspark_runner.SparkSessionEntryPoint(conf)[source]¶
Bases:
_SparkEntryPoint
- spark = None¶
- class luigi.contrib.pyspark_runner.PySparkRunner(job, *args)¶
Bases:
AbstractPySparkRunner
- class luigi.contrib.pyspark_runner.PySparkSessionRunner(job, *args)¶
Bases:
AbstractPySparkRunner