luigi.contrib.ecs module

EC2 Container Service wrapper for Luigi

From the AWS website:

Amazon EC2 Container Service (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances.

To use ECS, you create a taskDefinition JSON that defines the docker run command for one or more containers in a task or service, and then submit this JSON to the API to run the task.

This boto3-powered wrapper allows you to create Luigi Tasks to submit ECS taskDefinition s. You can either pass a dict (mapping directly to the taskDefinition JSON) OR an Amazon Resource Name (arn) for a previously registered taskDefinition.

Requires:

  • boto3 package

  • Amazon AWS credentials discoverable by boto3 (e.g., by using aws configure from awscli)

  • A running ECS cluster (see ECS Get Started)

Written and maintained by Jake Feala (@jfeala) for Outlier Bio (@outlierbio)

class luigi.contrib.ecs.ECSTask(*args, **kwargs)[source]

Bases: Task

Base class for an Amazon EC2 Container Service Task

Amazon ECS requires you to register “tasks”, which are JSON descriptions for how to issue the docker run command. This Luigi Task can either run a pre-registered ECS taskDefinition, OR register the task on the fly from a Python dict.

Parameters:
  • task_def_arn

    pre-registered task definition ARN (Amazon Resource Name), of the form:

    arn:aws:ecs:<region>:<user_id>:task-definition/<family>:<tag>
    

  • task_def

    dict describing task in taskDefinition JSON format, for example:

    task_def = {
        'family': 'hello-world',
        'volumes': [],
        'containerDefinitions': [
            {
                'memory': 1,
                'essential': True,
                'name': 'hello-world',
                'image': 'ubuntu',
                'command': ['/bin/echo', 'hello world']
            }
        ]
    }
    

  • cluster – str defining the ECS cluster to use. When this is not defined it will use the default one.

task_def_arn = OptionalParameter (defaults to None)
task_def = OptionalParameter (defaults to None)
cluster = Parameter (defaults to default)
property ecs_task_ids

Expose the ECS task ID

property command

Command passed to the containers

Override to return list of dicts with keys ‘name’ and ‘command’, describing the container names and commands to pass to the container. These values will be specified in the containerOverrides property of the overrides parameter passed to the runTask API.

Example:

[
    {
        'name': 'myContainer',
        'command': ['/bin/sleep', '60']
    }
]
static update_container_overrides_command(container_overrides, command)[source]

Update a list of container overrides with the specified command.

The specified command will take precedence over any existing commands in container_overrides for the same container name. If no existing command yet exists in container_overrides for the specified command, it will be added.

property combined_overrides

Return single dict combining any provided overrides parameters.

This is used to allow custom overrides parameters to be specified in self.run_task_kwargs while ensuring that the values specified in self.command are honored in containerOverrides.

property run_task_kwargs

Additional keyword arguments to be provided to ECS runTask API.

Override this property in a subclass to provide additional parameters such as network_configuration, launchType, etc.

If the returned dict includes an overrides value with a nested containerOverrides array defining one or more container command values, prior to calling run_task they will be combined with and superseded by any colliding values specified separately in the command property.

Example:

{
    'launchType': 'FARGATE',
    'platformVersion': '1.4.0',
    'networkConfiguration': {
        'awsvpcConfiguration': {
            'subnets': [
                'subnet-01234567890abcdef',
                'subnet-abcdef01234567890'
            ],
            'securityGroups': [
                'sg-abcdef01234567890',
            ],
            'assignPublicIp': 'ENABLED'
        }
    },
    'overrides': {
        'ephemeralStorage': {
            'sizeInGiB': 30
        }
    }
}
run()[source]

The task run method, to be overridden in a subclass.

See Task.run