Does the DbtShellTask work with the open-lineage dbt? (dbt-ol run)

[anna_geller]

View in #prefect-community on Slack

@Leon_Kozlowski: Does the DbtShellTask work with the open-lineage dbt? (dbt-ol run)

@Anna_Geller: I’d say no but to be sure you would need to give it a try

cc @alex

@alex: It probably wouldn’t work out of the box since it looks like there are some env variables that dbt-ol expects that the DbtShellTask does not set. You could subclass ShellTask and set the necessary env variables to run dbt-ol

Solution

@Leon_Kozlowski: Cool - I’ll try that
Update, it did work with DbtShellTask
Heres the code sample

with Flow("dbt-transformations") as flow:
    dbt = DbtShellTask(
        return_all=True,
        log_stdout=True,
        log_stderr=True,
        profile_name="transformations",
        environment="dev",
        helper_script="cd transformations",
        profiles_dir="",
    )

    dbt_deps = dbt(task_args={"name": "DbtDeps"}, command="dbt deps")

    dbt_doc_generate = dbt(
        task_args={"name": "DbtDocGenerate"},
        command="dbt docs generate",
        upstream_tasks=[dbt_deps],
    )

    dbt(
        task_args={"name": "DbtOpenLineageRun"},
        command="dbt-ol run",
        upstream_tasks=[dbt_doc_generate],
    )