TL;DR: you have to set an image in your task definition, but you can then override it on your ECSRun run configuration.
View in #prefect-community on Slack
@Scott_Aefsky: Hi folks. I’m trying to get my tasks to log to Cloudwatch through the ECSRun configuration. I’m testing out specifying a yaml config in task_definition_path, and running into an issue. My flows run on different ECR images, so I don’t want to specify the image in the Task Definition file. However, when I leave it out, and specify it in the constructor:
run_config = ECS_RUN(image=ECR_IMAGE, task_definition_path='s3://<path>'
I get an error that container.image should not be empty. My task definition file that I’d like to use is:
networkMode: awsvpc
cpu: 1024
memory: 2048
containerDefinitions:
- name: flow
logConfiguration:
logDriver: awslogs
options:
awslogs-group: test_prefect_log_group
awslogs-region: us-east-1
awslogs-stream-prefix: ecs-prefect
awslogs-create-group: "True"
Is there any way to use a Task Definition file without specifying an image? Or is there another/better way to make sure my ECS tasks get logged? Thanks for any help.
@Kevin_Kho: Even if it’s in the task_definition, it will get overriden by the RunConfig
@Scott_Aefsky: Oh, interesting. So I can put a non-existent image name in the task_definition to get past the error, and the RunConfig will override it?
@Kevin_Kho: I believe so because 3 things are merged before the ECS task is registered. The agent task_definition , the run config task definition, and the run configuration like cpu and memory. The priority is in that order also, so the RunConfig should override. Still though, I am not seeing why you get this error because the default task definition has none, but I guess we must be filling it.
@Scott_Aefsky: Yep, it worked like I wanted with a non-existent image
The full error when I tried to leave the image name out of the task_definition file was:
An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Container.image should not be null or empty.
@Kevin_Kho: Ah ok. Does that work for you then?
@Scott_Aefsky: Yep, I am all set
@Kevin_Kho: Nice!
@Scott_Aefsky: Thanks as always for the quick help, Kevin!