luigi.contrib.ecs module¶
EC2 Container Service wrapper for Luigi
From the AWS website:
Amazon EC2 Container Service (ECS) is a highly scalable, high performance container management service that supports Docker containers and allows you to easily run applications on a managed cluster of Amazon EC2 instances.
To use ECS, you create a taskDefinition JSON that defines the docker run command for one or more containers in a task or service, and then submit this JSON to the API to run the task.
This boto3-powered wrapper allows you to create Luigi Tasks to submit ECS
taskDefinition
s. You can either pass a dict (mapping directly to the
taskDefinition
JSON) OR an Amazon Resource Name (arn) for a previously
registered taskDefinition
.
Requires:
boto3 package
Amazon AWS credentials discoverable by boto3 (e.g., by using
aws configure
from awscli)A running ECS cluster (see ECS Get Started)
Written and maintained by Jake Feala (@jfeala) for Outlier Bio (@outlierbio)
- class luigi.contrib.ecs.ECSTask(*args, **kwargs)[source]¶
Bases:
Task
Base class for an Amazon EC2 Container Service Task
Amazon ECS requires you to register “tasks”, which are JSON descriptions for how to issue the
docker run
command. This Luigi Task can either run a pre-registered ECS taskDefinition, OR register the task on the fly from a Python dict.- Parameters:
task_def_arn –
pre-registered task definition ARN (Amazon Resource Name), of the form:
arn:aws:ecs:<region>:<user_id>:task-definition/<family>:<tag>
task_def –
dict describing task in taskDefinition JSON format, for example:
task_def = { 'family': 'hello-world', 'volumes': [], 'containerDefinitions': [ { 'memory': 1, 'essential': True, 'name': 'hello-world', 'image': 'ubuntu', 'command': ['/bin/echo', 'hello world'] } ] }
cluster – str defining the ECS cluster to use. When this is not defined it will use the default one.
- task_def_arn = OptionalParameter (defaults to None)¶
- task_def = OptionalParameter (defaults to None)¶
- cluster = Parameter (defaults to default)¶
- property ecs_task_ids¶
Expose the ECS task ID
- property command¶
Command passed to the containers
Override to return list of dicts with keys ‘name’ and ‘command’, describing the container names and commands to pass to the container. These values will be specified in the containerOverrides property of the overrides parameter passed to the runTask API.
Example:
[ { 'name': 'myContainer', 'command': ['/bin/sleep', '60'] } ]
- static update_container_overrides_command(container_overrides, command)[source]¶
Update a list of container overrides with the specified command.
The specified command will take precedence over any existing commands in container_overrides for the same container name. If no existing command yet exists in container_overrides for the specified command, it will be added.
- property combined_overrides¶
Return single dict combining any provided overrides parameters.
This is used to allow custom overrides parameters to be specified in self.run_task_kwargs while ensuring that the values specified in self.command are honored in containerOverrides.
- property run_task_kwargs¶
Additional keyword arguments to be provided to ECS runTask API.
Override this property in a subclass to provide additional parameters such as network_configuration, launchType, etc.
If the returned dict includes an overrides value with a nested containerOverrides array defining one or more container command values, prior to calling run_task they will be combined with and superseded by any colliding values specified separately in the command property.
Example:
{ 'launchType': 'FARGATE', 'platformVersion': '1.4.0', 'networkConfiguration': { 'awsvpcConfiguration': { 'subnets': [ 'subnet-01234567890abcdef', 'subnet-abcdef01234567890' ], 'securityGroups': [ 'sg-abcdef01234567890', ], 'assignPublicIp': 'ENABLED' } }, 'overrides': { 'ephemeralStorage': { 'sizeInGiB': 30 } } }