In this example, I used my local PC where the source of flow definition resides. So does the AKS agent need to be able to pull the flow from my local PC? ( My agent label is same as Flow label)
I am asking this question as I see an error when trying this example out.
Looking up flow metadata… Done
Creating run for flow ‘aks-flowrun’… Done
└── Name: ultramarine-camel
└── UUID: 4cb4d6d0-5ca2-4580-94a1-e11f23c8ba5b
└── Labels: [‘XYZ’]
└── Parameters: {}
└── Context: {}
└── URL: https://cloud.prefect.io/xxx/flow-run/xxx
Watching flow run execution…
└── 07:38:11 | INFO | Entered state : Flow run scheduled.
└── 07:38:20 | INFO | Entered state : Submitted for execution
└── 07:38:21 | INFO | Submitted for execution: Job prefect-job-xxxx
└── 07:38:22 | INFO | Entered state : Failed to load and execute flow run:
ModuleNotFoundError(“No module named ‘C’”)
This won’t work with a remote agent because the Azure VM has no access to files on your local machine. You would need to use remote storage such as Azure blog storage.
And regarding your error message, check this page that provides a more detailed explanation of the issue:
I am able to register the flow and can see the flow in my storage account(encoded).
In Prefect Cloud UI, I can see my flow got registered. However my agent running on AKS cluster is not picking up and running the flow. The labels are same . Am I missing anything?
import prefect
from prefect import task, Flow
from prefect.storage import Azure
from prefect.run_configs import KubernetesRun
STORAGE = Azure(
container="flows",
connection_string_secret="AZURE_STORAGE_CONNECTION_STRING",
# storing as script with Azure would require uploading the file beforehand e.g. as part of CI/CD
stored_as_script=True,
# blob_name=f"flows/{FLOW_NAME}.py",
)
@task
def hello_task():
logger = prefect.context.get("logger")
logger.info("Hello world!")
with Flow("aks-flowrun", run_config=KubernetesRun(labels=["HUMPOC"])) as flow:
hello_task()
if __name__ == "__main__":
flow.register("SriniJ Prefect POC")
I can also see my Kubernetes agent in Prefect Cloud as healthy(checking for flows).
I am running into an issue while sending the agent yaml file in my repsonse. I am getting a message that new users are allowed only 2 links per post.
The agent yaml I deployed to AKS cluster is the ouput i got from this:
prefect agent kubernetes install --rbac --key YOUR_API_KEY --label YOUR_LABEL > agent.yaml
I am expecting the Agent to run the flow and fail. This is what happens when I run the flow directly in Prefect Cloud. I expect this failure because AKS cluster/agent does not have network level access to the storage account.
However I expected the Agent to try running the flow automatically and showing me this error but apparently it never attempts to run my flow.
New update:
I tried with github as storage in my flow:
STORAGE = GitHub(
repo=“PrefectPOC”,
path=f"flows/{FLOW_NAME}.py",
access_token_secret=“GITHUB_ACCESS_TOKEN”,
)
but when i run it from Cloud UI, I get a 404
Failed to load and execute flow run: UnknownObjectException(404, {‘message’: ‘Not Found’, ‘documentation_url’: ‘GitHub REST API - GitHub Docs’}, {‘server’: ‘GitHub.com’, …
My agent is using prefecthq/prefect:1.2.0-python3.7 image.
My flow is using github storage for storage and kubernetes run for flow run.
with Flow(
FLOW_NAME,
storage=STORAGE,
run_config=KubernetesRun(),
A conceptual question.
So does the flow need to mention an image reference(from docker hub or private registry) as well as a storage reference ?
If there is an image reference being provided what is the need to provide the github storage path and access token?
A docker image should have everything that Kubernetes needs to run, right?