Problem with Prefect + AWS ECS Fargate + GitHub Actions

Archive
1

Hello,

For Prefect 2.0 I am trying to follow the tutorial Prefect + AWS ECS Fargate + GitHub Actions but I found this error

Any idea why it might be?

Thanks in advance!

1 Like
2

Thanks for the issue - it looks like your S3 block doesn’t have credentials attached to it. Can you try setting those explicitly as shown here?

from prefect.filesystems import S3
s3 = S3(
    bucket_path="prefect-orion/prod",
    aws_access_key_id="xxxx",
    aws_secret_access_key="xxxx",
)
print(f'S3 block: {s3.save("prod", overwrite=True)}')
3

Thanks for the fast reply!
Unfortunately I did that and still same error. I also tried to set the AWS keys manually on the block on the Prefect Cloud and did not fix it either :confused:

The error seems to be when adding an object to a bucket (calling the PutObject operation), but in my S3 there is no new bucket created. Should a S3 bucket be created in a previous step before there is an attempt to add an object to it?

Also in your video your S3 block credentials look like this and mine does not show the last characters, does it mean anything?

Thanks!

4

Given that we are already chatting on the GitHub issue, let’s chat there, I’ll close this one.

TL;DR I believe you were trying to use my bucket. To solve it, you need to create your own bucket and provide its name here.

5

Thanks Anna! I closed the issue.
When using delete_all_resources.yml only the AWS resources get deleted. Is there anyway to kill also the Prefect Blocks from a Github action?

6

yup:

and to delete blocks, you can do (change to match your desired block names and types you want to delete):

import subprocess

if __name__ == "__main__":
    blocks = [
        "dbt-cli-profile/default",
        "dbt/attribution",
        "dbt/jaffle-shop",
        "dbt-cloud-credentials/default",
        "snowflake-connector/default",
        "snowflake-credentials/default",
        "snowflake-pandas/default",
        "workspace/default",
        "azure/default",
        "s3/default",
        "gcs/default",
        "docker-container/default",
        "kubernetes-job/default",
        "process/default",
        "slack-webhook/default",
        "github/default",
        "github/dbt-jaffle-shop",
        "github/dbt-attribution",
    ]
    for block in blocks:
        subprocess.run(f"prefect block delete {block}", shell=True)
    block_types = [
        "workspace",
        "snowflake-schema",
        "dbt",
    ]
    for block_type in block_types:
        subprocess.run(f"prefect block type delete {block_type}", shell=True)
1 Like
7

Awesome thanks for the answer!

1 Like