luigi.contrib.ecs module

EC2 Container Service wrapper for Luigi

From the AWS website:

Amazon EC2 Container Service (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances.

To use ECS, you create a taskDefinition JSON that defines the docker run command for one or more containers in a task or service, and then submit this JSON to the API to run the task.

This boto3-powered wrapper allows you to create Luigi Tasks to submit ECS taskDefinition s. You can either pass a dict (mapping directly to the taskDefinition JSON) OR an Amazon Resource Name (arn) for a previously registered taskDefinition.

Requires:

• boto3 package
• Amazon AWS credentials discoverable by boto3 (e.g., by using aws configure from awscli)
• A running ECS cluster (see ECS Get Started)

Written and maintained by Jake Feala (@jfeala) for Outlier Bio (@outlierbio)

class luigi.contrib.ecs.ECSTask(*args, **kwargs)

Bases: luigi.task.Task

Base class for an Amazon EC2 Container Service Task

Amazon ECS requires you to register “tasks”, which are JSON descriptions for how to issue the docker run command. This Luigi Task can either run a pre-registered ECS taskDefinition, OR register the task on the fly from a Python dict.

Parameters:

• task_def_arn –

  pre-registered task definition ARN (Amazon Resource Name), of the form:

  arn:aws:ecs:<region>:<user_id>:task-definition/<family>:<tag>
  

• task_def –

  dict describing task in taskDefinition JSON format, for example:

  task_def = {
      'family': 'hello-world',
      'volumes': [],
      'containerDefinitions': [
          {
              'memory': 1,
              'essential': True,
              'name': 'hello-world',
              'image': 'ubuntu',
              'command': ['/bin/echo', 'hello world']
          }
      ]
  }
  

• cluster – str defining the ECS cluster to use. When this is not defined it will use the default one.

task_def_arn = OptionalParameter (defaults to None)

task_def = OptionalParameter (defaults to None)

cluster = Parameter (defaults to default)

ecs_task_ids

Expose the ECS task ID

command

Command passed to the containers

Override to return list of dicts with keys ‘name’ and ‘command’, describing the container names and commands to pass to the container. Directly corresponds to the overrides parameter of runTask API. For example:

[
    {
        'name': 'myContainer',
        'command': ['/bin/sleep', '60']
    }
]

run()

The task run method, to be overridden in a subclass.

See Task.run