Running flows in different environments and different versions of python

Archive
1

Is it possible to run flows for different versions of python environments? For example, my prefect is installed in Python 3.9. I’m unable to install it for 3.7 or 3.8 for some reason, documented issue. However, I have lot of code that runs in specific environments using python 3.7, 3.6, 2.7 etc.
I recall few months ago, you were working to make it possible to run flows for different virtual environments. But I’m no longer able to find documentation on that.
Currently, I kick off different environments using bash by changing PATH to point to a specific virtual env, and then kicking off python to run it.
Given that prefect does not work for me in 3.7, then I can not start an agent in 3.7 to run flows in python 3.7, so I’m unsure how to achieve this. In some ways, I need a runner to be separated from code. Like bash is completely separated from python. Python is just one of the things that bash can call.

1 Like
2

What’s your use case for this? Using GitHub Actions, you could specify a matrix strategy and trigger some workflow in different Python environments

3

Thanks Anna. I explained my use case above; I have old code that takes time to migrate to newer versions of python. I thought prefect had or was going to have ability to run code in different virtual environments. I’m just unable to find the documentation for it anymore using prefect. I think an alternative is to use subprocess.run library

I have one workaround, I was just wondering if you have it already built in or planned to do it:

> @flow(name='python 3.7')
> def diff_python():
> 	import subprocess
> 	import os
> 	env = os.environ.copy()
> 	env['PATH'] = open('path_ml37', 'r').readline()
> 	env['PATH'] = env['PATH'].replace(':', ";").replace('/c/', 'c:/').replace('/', '\\')
> 	print('running different version of python')
> 	print(subprocess.run('python --version', env=env, check=True, shell=True))

The output shows that it works; python 3.9 calls and executes python 3.7 without issues, but it is extra code. I guess I can wrap it up into a decorator or similar. I like how prefect likes to take out the negative engineering and make it work much simpler.

19:09:43.576 | INFO | prefect.engine - Created flow run ‘fabulous-bat’ for flow ‘python 3.7’
running different version of python
Python 3.7.9
CompletedProcess(args=‘python --version’, returncode=0)
19:09:43.820 | INFO | Flow run ‘fabulous-bat’ - Finished in state Completed()

1 Like
4

I understand now. The easiest solution would be to create a separate deployment for each of those and ensure that each deployment uses an infrastructure block configured for the respective environment. You could have e.g., 3 Process blocks, or 3 DockerContainer blocks, each configured so that each of them runs in a different Python version.

There are no plans to add any specific virtual environment support because this is something you can configure directly on infrastructure blocks.