Topics tagged aws

Topic	Replies	Views	Activity
When using ECSRun I'm getting an error: "An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Invalid setting for container 'flow'. At least one of 'memory' or 'memoryReservation' must be specified" Deployments & DevOps prefect-1-0 , aws , ecs , ecs-task-definition	0	197	April 5, 2022
How to pass AWS credentials to my Prefect 1.0 flows? FAQ prefect-1-0 , auth-and-permissions , aws , environment-variable , iam , agent-start-command	0	446	April 2, 2022
Announcing the Terraform Module to Deploy the Prefect Docker Agent on AWS EC2 Tutorials & Resources prefect-1-0 , aws , blog-post , docker-agent , aws-ec2 , agent-setup , terraform , aws-secrets-manager	0	191	March 24, 2022
How to handle ECS errors during flow run deployment? Is it possible to implement a flow-level retry for errors such as this one: "Failed to start [ECS] task for flow run SOME_UUID"? Deployments & DevOps prefect-1-0 , aws , ecs , infrastructure , cpu-memory-resource-allocation , ecs-agent	0	220	March 28, 2022
Additional upload options with S3 Storage and S3 Results Deployments & DevOps prefect-1-0 , storage , aws , s3 , s3-storage , server-side-encryption	7	429	March 24, 2022
How to use DockerFlowRunner with S3 storage? I'm getting an error "botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden" Deployments & DevOps prefect-2-0 , auth-and-permissions , deployment , aws , s3 , gcp , docker-flow-runner , s3-storage , bind-mount , credentials	7	690	March 21, 2022
Launch Week Day 4 is not over yet! We've just released Customer Success Recipes and Prefect 2.0b2! Announcements prefect-2-0 , release-notes , dbt , kubernetes , aws , ci-cd , s3 , docker , secrets , event-driven , airbyte , ecs , infrastructure , kubernetes-agent , docker-agent , docker-flow-runner , network-and-firewall , aws-ec2 , launch-week , agent-setup , linux , terraform , infrastructure-as-code , customer-success-recipes , ecs-agent	0	360	March 17, 2022
How to handle custom resource (CPU vs. GPU) and code dependencies across various tasks and flows? Workflow Orchestration prefect-1-0 , prefect-2-0 , flow-of-flows , kubernetes , aws , orchestrator-pattern , subflows , infrastructure , cpu_request , kubernetes-agent , aws-eks , radar-view , gpu-intensive-workflows , cpu-memory-resource-allocation	2	1035	March 15, 2022
When starting Prefect Server on AWS EC2 using Ubuntu OS, I'm getting an error: "NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff7e3b873d0>: Failed to establish a new connection: [Errno 111] Connection refused" - how to fix this? Deployments & DevOps prefect-1-0 , aws , server , network-and-firewall , aws-ec2 , ubuntu	0	819	March 9, 2022
Multi-cloud setup - data on GCP, agents on AWS EKS, or is it better to deploy one agent in each cloud provider? Deployments & DevOps prefect-1-0 , cloud-1-0 , aws , gcp , agent , infrastructure	0	160	March 3, 2022
When using ECSAgent, I'm getting an error: "An error occurred (ThrottlingException) when calling the DeregisterTaskDefinition operation (reached max retries: 2): Rate exceeded" - how to fix it? FAQ prefect-1-0 , aws , fargate , agent , ecs , throttling	0	160	March 3, 2022
Any plan to add ECSRunTask to AWS Tasks? Integrations prefect-1-0 , aws , fargate , task-library , ecs	0	203	February 22, 2022
How to deal with the long startup time (latency) on flow runs deployed to ECS Fargate? Deployments & DevOps kubernetes , aws , fargate , docker , agent , ecs , infrastructure , kubernetes-run , kubernetes-agent , aws-eks , docker-image , aws-ec2 , image-pull-policy	0	633	March 3, 2022
Building a Data Lake With Prefect Cloud, S3, Glue, and Athena - by Kat Hempstalk Tutorials & Resources prefect-1-0 , aws , blog-post , user-contribution , mysql , data-lake , aws-lambda , aws-ec2	0	348	February 27, 2022
Orchestrating ELT with Prefect, dbt Cloud, and Snowflake (Part 3) Tutorials & Resources snowflake , dbt , kubernetes , aws , automations , state_handlers , resource-manager , blog-post	0	220	January 21, 2022
Orchestrating ELT on Kubernetes with Prefect, dbt, & Snowflake (Part 2) Tutorials & Resources prefect-1-0 , local-development , snowflake , parameters , dbt , elt , kubernetes , aws , ci-cd , blog-post , docker , state-dependencies , secrets , dockerfile , triggers , kubernetes-run , kubernetes-agent , github-storage , ecr , aws-eks , circle-ci	0	561	January 21, 2022
When using the S3Upload task from the task library, I'm getting an error: "AttributeError: 'str' object has no attribute 'bucket'" - how to fix this? Integrations prefect-1-0 , dag-flow-structure , aws , s3 , imperative-api , task-library	0	188	February 14, 2022
What is the correct (and secure) way to pass GITLAB_ACCESS_TOKEN to a KubernetesAgent using Server deployed with Helm chart? Deployments & DevOps prefect-1-0 , kubernetes , aws , server , secrets , helm-chart , environment-variable , user-contribution , gitlab	0	433	February 3, 2022
I am using ECSAgent and sometimes I get an error that says "An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family." Deployments & DevOps run_config , aws , fargate , agent , ecs	1	505	January 30, 2022

When using ECSRun I'm getting an error: "An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Invalid setting for container 'flow'. At least one of 'memory' or 'memoryReservation' must be specified"

Deployments & DevOps

0

197

April 5, 2022

How to pass AWS credentials to my Prefect 1.0 flows?

FAQ

prefect-1-0 , auth-and-permissions , aws , environment-variable , iam , agent-start-command

0

446

April 2, 2022

Announcing the Terraform Module to Deploy the Prefect Docker Agent on AWS EC2

Tutorials & Resources

prefect-1-0 , aws , blog-post , docker-agent , aws-ec2 , agent-setup , terraform , aws-secrets-manager

0

191

March 24, 2022

How to handle ECS errors during flow run deployment? Is it possible to implement a flow-level retry for errors such as this one: "Failed to start [ECS] task for flow run SOME_UUID"?

Deployments & DevOps

prefect-1-0 , aws , ecs , infrastructure , cpu-memory-resource-allocation , ecs-agent

0

220

March 28, 2022

Additional upload options with S3 Storage and S3 Results

Deployments & DevOps

prefect-1-0 , storage , aws , s3 , s3-storage , server-side-encryption

7

429

March 24, 2022

How to use DockerFlowRunner with S3 storage? I'm getting an error "botocore.exceptions.ClientError: An error occurred (403) when calling the HeadObject operation: Forbidden"

Deployments & DevOps

prefect-2-0 , auth-and-permissions , deployment , aws , s3 , gcp , docker-flow-runner , s3-storage , bind-mount , credentials

7

690

March 21, 2022

Launch Week Day 4 is not over yet! We've just released Customer Success Recipes and Prefect 2.0b2!

Announcements

prefect-2-0 , release-notes , dbt , kubernetes , aws , ci-cd , s3 , docker , secrets , event-driven , airbyte , ecs , infrastructure , kubernetes-agent , docker-agent , docker-flow-runner , network-and-firewall , aws-ec2 , launch-week , agent-setup , linux , terraform , infrastructure-as-code , customer-success-recipes , ecs-agent

0

360

March 17, 2022

How to handle custom resource (CPU vs. GPU) and code dependencies across various tasks and flows?

Workflow Orchestration

prefect-1-0 , prefect-2-0 , flow-of-flows , kubernetes , aws , orchestrator-pattern , subflows , infrastructure , cpu_request , kubernetes-agent , aws-eks , radar-view , gpu-intensive-workflows , cpu-memory-resource-allocation

2

1035

March 15, 2022

When starting Prefect Server on AWS EC2 using Ubuntu OS, I'm getting an error: "NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7ff7e3b873d0>: Failed to establish a new connection: [Errno 111] Connection refused" - how to fix this?

Deployments & DevOps

prefect-1-0 , aws , server , network-and-firewall , aws-ec2 , ubuntu

0

819

March 9, 2022

Multi-cloud setup - data on GCP, agents on AWS EKS, or is it better to deploy one agent in each cloud provider?

Deployments & DevOps

prefect-1-0 , cloud-1-0 , aws , gcp , agent , infrastructure

0

160

March 3, 2022

When using ECSAgent, I'm getting an error: "An error occurred (ThrottlingException) when calling the DeregisterTaskDefinition operation (reached max retries: 2): Rate exceeded" - how to fix it?

FAQ

prefect-1-0 , aws , fargate , agent , ecs , throttling

0

160

March 3, 2022

Any plan to add ECSRunTask to AWS Tasks?

Integrations

prefect-1-0 , aws , fargate , task-library , ecs

0

203

February 22, 2022

How to deal with the long startup time (latency) on flow runs deployed to ECS Fargate?

Deployments & DevOps

kubernetes , aws , fargate , docker , agent , ecs , infrastructure , kubernetes-run , kubernetes-agent , aws-eks , docker-image , aws-ec2 , image-pull-policy

0

633

March 3, 2022

Building a Data Lake With Prefect Cloud, S3, Glue, and Athena - by Kat Hempstalk

Tutorials & Resources

prefect-1-0 , aws , blog-post , user-contribution , mysql , data-lake , aws-lambda , aws-ec2

0

348

February 27, 2022

Orchestrating ELT with Prefect, dbt Cloud, and Snowflake (Part 3)

Tutorials & Resources

snowflake , dbt , kubernetes , aws , automations , state_handlers , resource-manager , blog-post

0

220

January 21, 2022

Orchestrating ELT on Kubernetes with Prefect, dbt, & Snowflake (Part 2)

Tutorials & Resources

prefect-1-0 , local-development , snowflake , parameters , dbt , elt , kubernetes , aws , ci-cd , blog-post , docker , state-dependencies , secrets , dockerfile , triggers , kubernetes-run , kubernetes-agent , github-storage , ecr , aws-eks , circle-ci

0

561

January 21, 2022

When using the S3Upload task from the task library, I'm getting an error: "AttributeError: 'str' object has no attribute 'bucket'" - how to fix this?

Integrations

prefect-1-0 , dag-flow-structure , aws , s3 , imperative-api , task-library

0

188

February 14, 2022

What is the correct (and secure) way to pass GITLAB_ACCESS_TOKEN to a KubernetesAgent using Server deployed with Helm chart?

Deployments & DevOps

prefect-1-0 , kubernetes , aws , server , secrets , helm-chart , environment-variable , user-contribution , gitlab