Why the ec2:DescribeVpcs AWS IAM policy is needed on ECSTask even when VPC ID is explicitly provided?

Archive
, ,
1

Problem/User question

Why do I need to give Prefect ECSTask infrastructure block permission ec2:DescribeVpcs, even when I explicitly provide my vpc_id?

Explanation

In the ECSTask infrastructure block, we describe VPCs even when the vpc_id is explicitly provided because we want to provide helpful, high-quality error messages. We need this API call to detect if the VPC exists, and this way, we can provide more explicit error messages helping to distinguish, e.g.:

  • whether VPC exists or not
  • whether VPC has subnets or not

While technically, this could be avoided, it would complicate the code of this infrastructure block and reduce the quality of the error messaging.

Solution

Provide both vpc_id and subnet IDs explicitly. This way, you don’t need to give ECSTask the ec2:DescribeVpcs permission.

"""
pip install prefect -U
pip install prefect-aws
prefect block register -m prefect_aws.ecs
"""
from prefect_aws.ecs import ECSTask, AwsCredentials

ecs = ECSTask(
    aws_credentials=AwsCredentials.load(
        "default"
    ),  # your AWS credentials needed to register the task definition and to run the task from that definition
    image="prefecthq/prefect:2-python3.10",  # example image, could be DockerHub or ECR image
    cpu="256",
    memory="512",
    stream_output=True,
    configure_cloudwatch_logs=True,
    cluster="prefect",
    execution_role_arn="arn:aws:iam::123456789:role/dataflowops_ecs_execution_role",
    task_role_arn="arn:aws:iam::123456789:role/dataflowops_ecs_task_role",
    vpc_id="vpc-0ff32ab58b1c8695a",
    task_customizations=[
        {
            "op": "replace",
            "path": "/networkConfiguration/awsvpcConfiguration/assignPublicIp",
            "value": "DISABLED",
        },
        {
            "op": "add",
            "path": "/networkConfiguration/awsvpcConfiguration/subnets",
            # to only allow specific private subnets from that vpc_id
            "value": ["subnet-052624e5d692fda86"],
        },
    ],
)
ecs.save("default", overwrite=True)