See the PDF attached here
https://linen.prefect.io/t/9558407/hello-prefect-community-i-know-this-may-seem-like-small-pota
SETTING UP PREFECT 2 LOCALLY USING .venv
Overview
As we transition to Prefect two it is important for the BS Development team to be able to test flows locally before deploying to Dev/Prod. With that in mind we will be setting up local dev environments within virtual machines in order to emulate the prefect 2 server. The instructions below will show you how to set up your virtual environment for prefect 2 from scratch. All you will need is administrator access while in PowerShell!
Optional but very useful
- Install pip on your windows device
- Install python on your windows device
- run pip install python, any version over 3.7 will do
Install/Create venv
use the commands below to create a venv, if the steps below are not working as intended feel free to refer to this documentation: Python Tools for Managing Virtual Environments - DEV Community
- python -m venv .venv
- cd to the scripts folder
- run .\Activate.ps1 (if this does not work go to file explorer and copy the full file path to the activate.ps1 script like in the image below)
- Once you see a (venv) at the front of your PowerShell window you know you got it working!
activating virtual environment using full file path
Setting up Local Environment
Depending on where you installed your .venv folder your next steps may look slightly different than mine. With that being said, the following steps have worked in multiple scenarios so it should be fine!
you can create your venv in a distinct folder
you can also put it in the same folder as your project workspace
- make sure you are in your activated virtual environment in powershell (you should see the (.venv) like in the screenshot above)
Optional but helpful
- before going any further you may want to confirm everything is present within your venv
- run python --version to confirm you have a version of python installed, hopefully it is above 3.7!
- if it is not 3.7 or higher you will need to upgrade to a version that is, as Prefect 2 requires Python 3.7 or higher
- run pip -V to confirm pip is installed
NOTE**:** if either or both of these packages are not installed you can install them now using the commands listed above, but this may be an indication that you should restart from step one
Continuing Onward
At this point it is time to configure your workspace for Prefect 2! If you have any issues with the following steps you can refer to Prefects documentation here: Installation - Prefect 2 - Coordinating the world’s dataflows
- run pip install -U prefect
- now time to check our work! run pip show prefect hopefully your result looks similar to mine below
the key here is the version being 2.x
- now is a good time to run an additional check on your prefect installation as it can be kind of tricky.
- run prefect version, hopefully the result looks like my screenshot below
take note of the server and server type fields.
- If your result matches the screenshot above, then congrats! you’re perfect. If not however do not worry, we will cover that shortly (the solution is detailed later)
- create a new folder in your filesystem that will act as your local dev environment for prefect 2. You can create this folder either while you are within the .venv or not. For this example, I am using the folder my venv is stored as my development environment, however you could create your workspace in a completely different location than your venv as well.
- blank folder see!
in this example I used the folder that the .venv is stored in as the project folder as well
- once you have created the folder, you can navigate to it in your PowerShell and run the code . command to open the folder in VSCode (see screenshot below)
take note of the interpreter for your environment
- There are two important things to take note of upon first opening the environment.
- The interpreter for your workspace (see red circle in screenshot above)
- the .venv activate.ps1 script being run automatically upon opening the workspace (see underline in screenshot below)
ignore the existing hello_world and requirements.txt for now
DO NOT PANIC
If only one or neither of these things occurred you do not need to backtrack on the instructions, I will explain everything below
- using the PowerShell command prompt in your project environment, activate your .venv that you created manually, this will work best if you use the full file path to the script like in the screenshot above.
- Once you have the green (.venv) showing up in your VSCode PowerShell we are ready to cook
Fixing Python Interpreter issue (screenshot one with the circled interpreter)
- while you are in VSCode, type Ctrl+Shift+P to open up the quick select menu in VSCode
- If it does not appear immediately, type “select interpreter” or “interpreter”. the results should look something like the screenshot below
you want an interpreter that matches your version of python and has .venv
- Once you have selected the proper interpreter you should see the proper interpreter in the bottom left of your VSCode application. NOTE: You may need to close and reopen your project to see this change take place
Fixing .venv not starting up when you open your project environment
- once you have the correct interpreter selected for your project/workspace, try closing and reopening the project from PowerShell. you should see positive results!
Upkeep
- try checking the version of python and prefect just like we did from PowerShell earlier in the instructions. whatever results you saw in that window should match in your project terminal.
- If your prefect version command returned something like the screenshot below, you are not alone
this is a re-creatable issue when installing prefect, i have no idea why
- This is an indication that prefect 2s installation was half baked. luckily, we have the solution.
- add the appropriate requirements.txt file to the project workspace
aiosqlite==0.18.0
alembic==1.9.1
anyio==3.6.2
apprise==1.2.1
arrow==1.2.3
asgi-lifespan==2.0.0
asttokens==2.2.1
asyncpg==0.27.0
attrs==22.2.0
cachetools==5.2.1
certifi==2022.12.7
cffi==1.15.1
cfgv==3.3.1
charset-normalizer==2.1.1
click==8.1.3
cloudpickle==2.2.0
colorama==0.4.6
commonmark==0.9.1
coolname==2.2.0
coverage==7.0.4
croniter==1.3.8
cryptography==39.0.0
cssselect2==0.7.0
dateparser==1.1.5
decorator==5.1.1
defusedxml==0.7.1
distlib==0.3.6
exceptiongroup==1.1.0
execnet==1.9.0
executing==1.2.0
fsspec==2022.11.0
ghp-import==2.1.0
greenlet==2.0.1
griffe==0.25.3
h11==0.14.0
h2==4.1.0
hpack==4.0.0
httpcore==0.16.3
httpx==0.23.3
hyperframe==6.0.1
identify==2.5.12
idna==3.4
iniconfig==2.0.0
ipython==8.8.0
isodate==0.6.1
jmespath==1.0.1
jsonpatch==1.32
jsonpointer==2.3
kubernetes==25.3.0
Mako==1.2.4
Markdown==3.3.7
MarkupSafe==2.1.1
matplotlib-inline==0.1.6
mccabe==0.7.0
mergedeep==1.3.4
moto==4.1.0
msal==1.20.0
msal-extensions==1.0.0
msrest==0.7.1
mypy==0.991
mypy-extensions==0.4.3
nodeenv==1.7.0
numpy==1.24.1
oauthlib==3.2.2
orjson==3.8.4
packaging==23.0
pathspec==0.10.3
pendulum==2.1.2
pickleshare==0.7.5
Pillow==9.4.0
platformdirs==2.6.2
pluggy==1.0.0
portalocker==2.6.0
pre-commit==2.21.0
pure-eval==0.2.2
py-cpuinfo==9.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.10.0
pycparser==2.21
pydantic==1.10.4
pyflakes==3.0.1
Pygments==2.14.0
PyJWT==2.6.0
pymdown-extensions==9.9
pytest==7.2.0
pytest-asyncio==0.20.3
pytest-benchmark==4.0.0
pytest-cov==4.0.0
pytest-env==0.8.1
pytest-timeout==2.1.0
pytest-xdist==3.1.0
python-dateutil==2.8.2
python-slugify==7.0.0
pytkdocs==0.16.1
pywin32==305
PyYAML==6.0
pyyaml_env_tag==0.1
readchar==4.0.3
regex==2022.10.31
requests==2.28.1
requests-oauthlib==1.3.1
responses==0.22.0
respx==0.20.1
rfc3986==1.5.0
rich==13.0.1
rsa==4.9
s3transfer==0.6.0
stack-data==0.6.2
starlette==0.22.0
text-unidecode==1.3
tinycss2==1.2.1
toml==0.10.2
tomli==2.0.1
traitlets==5.8.1
typer==0.7.0
types-toml==0.10.8.1
typing_extensions==4.4.0
tzdata==2022.7
tzlocal==4.2
urllib3==1.26.13
uvicorn==0.20.0
virtualenv==20.17.1
watchdog==2.2.1
watchfiles==0.18.1
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.4.2
Werkzeug==2.2.2
xmltodict==0.13.0
- from within your .venv PowerShell in VSCode, run pip install -r .\requirements.txt
- now check your prefect installation using prefect version, you should see the server and server type reference a SQLite database! congrats!
Tripping over the finish line
if you have made it to this final step then the only thing left to do is try running a flow, good luck!
- within your workspace in VSCode add a new file called “hello_world.py” and paste the following code inside it. This is just a quick and dirty test to confirm all the prefect functionality is working
from prefect import flow, get_run_logger
@flow
def hello(name: str = “world”):
get_run_logger().info(f"Hello {name}!")
if name == “main”:
hello()
- from the command line, run python -m hello_world test your results should look similar to the screenshot below
Note: if this process fails then pull in another developer for a code seance
Miscellaneous Helpful Documentation
- Using Python Environments in Visual Studio Code
- Installation - Prefect 2 - Coordinating the world’s dataflows
- First steps - Prefect 2 - Coordinating the world’s dataflows
- venv — Creation of virtual environments — Python 3.11.1 documentation