Using gitlab block as flow source with Kubernetes Agent on Prefect 2

Help
,
1

A usage question regarding gitlab repository being a block for flow code during deployment…

I have Prefect 2.0 installed on AKS cluster with kubernetes agent.
Prefect gitlab collection is installed using cline

prefect block register -m prefect_gitlab

the block is registered with UI and appears in the output of the command

prefect block ls

with the slug gitlab-repository/gitlab-repo

During deployment run on the queue connected to the agent i see the following exception

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 266, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/deployments.py", line 166, in load_flow_from_flow_run
    storage_block = Block._from_block_document(storage_document)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 548, in _from_block_document
    else cls.get_block_class_from_schema(block_document.block_schema)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 568, in get_block_class_from_schema
    return lookup_type(cls, block_schema_to_key(schema))
  File "/usr/local/lib/python3.10/site-packages/prefect/utilities/dispatch.py", line 186, in lookup_type
    raise KeyError(
KeyError: "No class found for dispatch key 'gitlab-repository' in registry for type 'Block'."

here’s the deployment snippet connected to the block

storage:
  repository: https://....redacted...
  reference: null
  credentials: null
  _block_document_id: f39a13c2-8ce0-4126-9de5-0b2308d44725
  _block_document_name: gitlab-repo
  _is_anonymous: false
  block_type_slug: gitlab-repository
  _block_type_slug: gitlab-repository

Is there something missing in setting up gitlab collection for the server and the agent?

2

here is how you can use it:

3

I followed the registration procedures and the Block is registered with the UI properly.
There is no problem with registering the block.

The exception I posted appears during run where the system is trying to use the Block to read the FLow.
It looks like it does not see the created Block Type (gitlab-repository)

4

For the context, prefect is deployed using these Helm charts

Deployment is done without errors, UI/API/Agent are running properly

5

ahh sorry, I don’t know that much about self-hosting with helm, the README is a great starting point though

6

I will try to see if I can raise it directly with gitlab collection development team in their repository and will post our communication here.

1 Like
7

that’s a great idea!

8

Self-hosting with Docker Swarm, I am getting the same exception. I’ll be watching this thread :eyes:

2 Likes
9

I’ve been trying to see in the code how that magic setting works that was added to the latest release

Basically, it should register modules from the collections - based on the documentation gitlab collection should be added to Orion docker image, trying to play around with debug settings to see if I see that import piece appear in logs. But it was weird that I don’t see gitlab in the list of default modules in UI - maybe it was not included in the Orion build

1 Like
10

Yup, confirmed - gitlab collection is not there in the latest Orion build. So both Orion API and Agent images won’t load that block

when this config value is set

PREFECT_EXTRA_ENTRYPOINTS: prefect_gitlab:GitLabRepository

I get this.

Warning! Failed to load extra entrypoint 'prefect_gitlab:GitLabRepository': ModuleNotFoundError: No module named 'prefect_gitlab'
2 Likes
11

Awesome work there!
Judging from the date of the merge commit, it seems that 2.7.0 is the latest version that can be used until this is fixed.

2 Likes
12

So yeah, we can probably use built in github readonly file system since its using git to create a temp location with git clone.

For gitlab we would need this token format https://oauth2:ACCESS_TOKEN@somegitlab.com/repo/package.git

and if we look at the code here

all we need is to define a secret with value oath2:ACCESS_TOKEN and that’s it.

So my new deployment with github storage works

storage:
  repository: https://....redacted...
  reference: "main"
  access_token:"*****"
  _block_document_id: f39a13c2-8ce0-4126-9de5-0b2308d44725
  _block_document_name: gitlab-repo
  _is_anonymous: false
  block_type_slug: github
  _block_type_slug: github
```
1 Like
13

For what it’s worth, I ran into this issue as well when working with Prefect through Docker. Seems there’s a workaround when using Docker, as you can trigger the package installation via the entrypoint in the docker compose file.

  ## Prefect Agent
  agent:
    image: prefecthq/prefect:2.10.16-python3.11
    restart: always
    entrypoint: ["/opt/prefect/entrypoint.sh", "prefect", "agent", "start", "-q", "default"]
    environment:
      - PREFECT_API_URL=http://server:4200/api
      - EXTRA_PIP_PACKAGES=prefect-gitlab
    profiles: ["agent"]

The first object in the list, "/opt/prefect/entrypoint.sh", triggers the EXTRA_PIP_PACKAGES to be installed. This got me past the issue here, though now I face a connectivity issue. I’m working on that now.

Credit to rpeden for this find.