How to best connect to external resources with a configurable authentification method?

Archive
, , ,
1

For my work, I need to connect to external resources a lot (databases, API, etc.).
I would like to generalize my way of accessing those resources and configuring them.

Right now, I am using .ini configuration files deployed on the server that executes the flows, with the authentication details, url, ports, etc. It works for me but is not very portable.

But I looked at prefect blocks and thought they would do the trick, enabling to set up the authentication details on the UI directly. As I am already doing it for mattermost webhooks notifications, it seems reasonable.

However, I also had a look on context managers and how to use them in prefect 2.O: https://discourse.prefect.io/t/how-to-clean-up-resources-used-in-a-flow/84
And it seemed also like a good pythonic way to connect to an API, however the credentials still need to be supplied somehow.

Which way is generally preferred and in which conditions?
Is an hybrid way the way to go? (a Block subclass that is also a context manager)
Or should we keep both and use them depending on the situation?

1 Like
2

There are many ways how you could approach it. The easiest would be to store sensitive information with a Secret block and implement the actual logic in your code.

3

I understand it is the easiest way.
However as I would like to have my configuration structured, I would rather have custom blocks with all the required fields to authenticate so that the configuration step is guided in a way.
Then I thought I could add a build_connector method for each custom block that returns the corresponding connector object wrapped in a context manager.

4

That’s totally feasible with a custom Block