How to Orchestrate ELT pipelines with Prefect, Airbyte and dbt

anna_geller · January 28, 2022, 12:19pm

This tutorial demonstrates how to integrate Prefect, Airbyte, and dbt.

The flow example from the tutorial:

desertaxle/airbyte-prefect-recipe/blob/master/prefect/flow.py

from datetime import timedelta

from prefect import Flow, Parameter, artifacts, task
from prefect.run_configs.local import LocalRun
from prefect.storage.local import Local
from prefect.tasks.airbyte.airbyte import AirbyteConnectionTask
from prefect.tasks.dbt.dbt import DbtShellTask
from prefect.tasks.secrets.base import PrefectSecret
from prefect.tasks.snowflake.snowflake import SnowflakeQuery

sync_airbyte_connection = AirbyteConnectionTask(
    max_retries=3, retry_delay=timedelta(seconds=10)
)
run_dbt = DbtShellTask(
    command="dbt run",
    environment="dev",
    profile_name="github_common_contributors",
    helper_script="cd github_common_contributors",
    set_profiles_envar=False,
    max_retries=3,

This file has been truncated. show original

Prefect Community

How to Orchestrate ELT pipelines with Prefect, Airbyte and dbt

Don't Panic

Read our Privacy Policy