Prefect permission denied when run from pool but not from console

Help
1

I have a Prefect flow that saves some data from an FTP site to a windows network drive (Windows 2000 server). If I run this flow from the console via:

conda run -n my_env python -m my_flow

Everything runs without issue. However, if I run from my work pool, I get the error:

PermissionError: [Errno 13] Permission denied: '{MY_NETWORK_LOCATION}\{MY_FILENAME}'

My work pool is being launched from the same conda env as above using:

conda run -n my_env prefect worker start --pool my_pool --work-queue default

The offending line is the file write:

with FTP(host=host) as ftp:
        ftp.login(user=user, passwd=password)
        ftp.retrbinary("RETR " + remote_file, open(local_file, 'wb').write)

where:

  • local_file = {MY_NETWORK_LOCATION}\{MY_FILENAME}
  • remote_file = {MY_FILENAME}

I’m confused as to why it’s running without issue from my command line but not from the work pool. I’ve tried starting the work pool as administrator but that did not fix anything.

UPDATE:

It appears the user account is for the work pool is running as NT AUTHORITY\ANONYMOUS LOGON instead of {MY_DOMAIN}{MY_USER}. I’m not sure why this is.

2

This makes me think it’s an issue with the Python module you’re using for the FTP stuff. Which library are you using?

It’s acting like it’s using trusted credentials (passing your login). Is it set up as a Windows service and failing? What happens locally when you use an incorrect user/pwd… does it succeed anyway?

3

It is not an FTP problem. The file downloads without issue. I tried changing my write location to somewhere local and it wrote to that location without issue. Subsequently the database auth failed with:

Finished in state Failed('Task run encountered an exception InterfaceError: (pyodbc.InterfaceError) ('28000', “[28000] [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for user 'NT AUTHORITY\\ANONYMOUS LOGON'. (18456) (SQLDriverConnect); [28000] [Microsoft][ODBC SQL Server Driver][SQL Server]Login failed for user 'NT AUTHORITY\\ANONYMOUS LOGON'. (18456)”)\n(Background on this error at: https://sqlalche.me/e/14/rvf5)')%27))

This leads me to believe it is a “double hop” problem but I’m not familiar enough with Active Directory to remedy this. I can verify that Prefect (the web app) and all work pools are being started with admin privileges and with my user account. Here is the post I’ve been following about this double hop problem:

4

Is the error you just posted after trying an incorrect user and password?

What I’m getting at is that it might be working locally only because you have permission (because it’s using trusted credentials). And if it’s run as a service and attempts to use the same trusted type of connection… it fails because the system account does not have permissions to the database.

If you change the user and pass to something incorrect… and it still works… then that confirms it isn’t using what you specify in order to succeed locally.

What Python library are you using for the FTP?

5

I’m using windows auth for my DB connection so the connection is trusted:

mssql+pyodbc://{HOST}:{PORT}/{DATABASE}?trusted_connection=yes&driver=SQL+Server

For FTP I’m using ftplib.

If I run the flow from the terminal (not using the work pool) everything works. If I run that same flow from the work pool, the user is set to NT AUTHORITY\ANONYMOUS LOGON and any windows credentials through active directory are not applied.

6

Thanks for replying with more info. I ran into a similar issue using pyodbc trying to connect to a mssql server. I couldn’t figure it out and ended up using pymssql instead which worked. Really helpful, I know. /s

But recently I moved to a new environment and ran into issues again because I didn’t properly set Prefect’s API URL. It seemed to be acting like this double hop issue. Updating it to the local URL made it go away.

Are you self-hosted?

7

Yes actually I am self hosted. That seems encouraging, how did you set the API URL?

8

So I set my prefect api url:

prefect config set PREFECT_API_URL="http://127.0.0.1:4200/api"

Which I got from this link:

I then restarted my prefect server but alas it has not fixed my issue :frowning: