luigi.contrib.lsf module

luigi.contrib.lsf.track_job(job_id)

Tracking is done by requesting each job and then searching for whether the job has one of the following states: - “RUN”, - “PEND”, - “SSUSP”, - “EXIT” based on the LSF documentation

luigi.contrib.lsf.kill_job(job_id)

Kill a running LSF job

class luigi.contrib.lsf.LSFJobTask(*args, **kwargs)

Bases: Task

Takes care of uploading and executing an LSF job

n_cpu_flag = Insignificant IntParameter (defaults to 2)

shared_tmp_dir = Insignificant Parameter (defaults to /tmp)

resource_flag = Insignificant Parameter (defaults to mem=8192)

memory_flag = Insignificant Parameter (defaults to 8192)

queue_flag = Insignificant Parameter (defaults to queue_name)

runtime_flag = IntParameter (defaults to 60)

job_name_flag = Parameter (defaults to )

poll_time = Insignificant FloatParameter (defaults to 5): specify the wait time to poll bjobs for the job status

save_job_info = BoolParameter (defaults to False)

output = Parameter (defaults to )

extra_bsub_args = Parameter (defaults to )

job_status = None

fetch_task_failures()

Read in the error file from bsub

fetch_task_output()

Read in the output file

init_local()

Implement any work to setup any internal datastructure etc here. You can add extra input using the requires_local/input_local methods. Anything you set on the object will be pickled and available on the compute nodes.

run()

The procedure: - Pickle the class - Tarball the dependencies - Construct a bsub argument that runs a generic runner function with the path to the pickled class - Runner function loads the class from pickle - Runner class untars the dependencies - Runner function hits the button on the class’s work() method

work()

Subclass this for where you’re doing your actual work.

Why not run(), like other tasks? Because we need run to always be something that the Worker can call, and that’s the real logical place to do LSF scheduling. So, the work will happen in work().

class luigi.contrib.lsf.LocalLSFJobTask(*args, **kwargs)

Bases: LSFJobTask

A local version of JobTask, for easier debugging.

run()

The procedure: - Pickle the class - Tarball the dependencies - Construct a bsub argument that runs a generic runner function with the path to the pickled class - Runner function loads the class from pickle - Runner class untars the dependencies - Runner function hits the button on the class’s work() method