luigi.contrib.lsf module¶
- luigi.contrib.lsf.track_job(job_id)[source]¶
Tracking is done by requesting each job and then searching for whether the job has one of the following states: - “RUN”, - “PEND”, - “SSUSP”, - “EXIT” based on the LSF documentation
- class luigi.contrib.lsf.LSFJobTask(*args, **kwargs)[source]¶
Bases:
Task
Takes care of uploading and executing an LSF job
- n_cpu_flag = Insignificant IntParameter (defaults to 2)¶
- resource_flag = Insignificant Parameter (defaults to mem=8192)¶
- memory_flag = Insignificant Parameter (defaults to 8192)¶
- queue_flag = Insignificant Parameter (defaults to queue_name)¶
- runtime_flag = IntParameter (defaults to 60)¶
- job_name_flag = Parameter (defaults to )¶
- poll_time = Insignificant FloatParameter (defaults to 5): specify the wait time to poll bjobs for the job status¶
- save_job_info = BoolParameter (defaults to False)¶
- output = Parameter (defaults to )¶
- extra_bsub_args = Parameter (defaults to )¶
- job_status = None¶
- init_local()[source]¶
Implement any work to setup any internal datastructure etc here. You can add extra input using the requires_local/input_local methods. Anything you set on the object will be pickled and available on the compute nodes.
- run()[source]¶
The procedure: - Pickle the class - Tarball the dependencies - Construct a bsub argument that runs a generic runner function with the path to the pickled class - Runner function loads the class from pickle - Runner class untars the dependencies - Runner function hits the button on the class’s work() method
- class luigi.contrib.lsf.LocalLSFJobTask(*args, **kwargs)[source]¶
Bases:
LSFJobTask
A local version of JobTask, for easier debugging.
- run()[source]¶
The procedure: - Pickle the class - Tarball the dependencies - Construct a bsub argument that runs a generic runner function with the path to the pickled class - Runner function loads the class from pickle - Runner class untars the dependencies - Runner function hits the button on the class’s work() method