Can I change the location of the agent ( "/tmp/tmp*prefect" to "/home/tmp*prefect") when executing a flow?

Archive
1

Hello,

By default, when i run a deployment from the UI (orion server) and start an agent, the command line “python -m prefect.engine” is executed in a directory at “/tmp/tmp*prefect”. However, I am working on a server where the “/tmp” directory is very limited in both space and access time. Can I change the current directory of the execution to another one on my computer (like /home/tmp*prefect) ?
I use the infrastructure Process to run my flow.

Best regards,

1 Like
2

Could you elaborate? You don’t need to store anything there, it’s only for the duration of a flow run

3

Digging more, I found out that you might be able to change that by setting one of those variables (I’m not sure which one, though):

prefect config set PREFECT_LOCAL_STORAGE_PATH="/your/custom/path"
prefect config set PREFECT_HOME="/your/custom/path"
4

I changed both variables and it didn’t work for me. I run a flow with just “print(os.getcwd())” inside and it still print me “/tmp/tmp3k1tb_3hprefect”

1 Like
5

I work on a server where /tmp is not on the same disk than /home , the size is very limited (about 100 Mo)

1 Like
6

gotcha, I can look at this in 1-2 weeks again

7

Year Really! :thinking:
It is very important problem, for me too, I create Flows for processing big files (~20-30GB) and prefect use tmp dir for store it (I noticed it when Flows was running), prefect cleans after flow executed it is good but I’d like to set the dir for temp files cause boot memory is 50GB only and the boot memory is expensive to use and the better way is to use extra-disk or may be store every data at .prefect/ (which location we already can control)
Now I see just mount extra space to tmp dir

Thank you :slightly_smiling_face:

1 Like
8

That’s a neat idea and definitely viable. Check also prefect settings, there’s a settings variable you can use to adjust which directory is used for that by Prefect

9

You can use the process infra block to set the working directory of an agent for a given deployment:

Process Infra Block:

my_process_infra_block = Process(
    working_dir="flows/subflows"
)
my_process_infra_block.save('my-process-infra-block', overwrite=True)

Flow

@task
def mkdir_task(num):
    os.mkdir("task_created_dir")
    print('sum: ', num + num)
    return num + num

# prefect deployment build temp.py:temp -n dep_temp -ib process/my-process-infra-block -t temp -a
@flow
def temp(num=1):
    result = mkdir_task(num)
    return result

Deployment Build Command:

prefect deployment build temp.py:temp -n dep_temp -ib process/my-process-infra-block -t temp -a

1 Like
10

Also you can just override system env TMPDIR :
export TMPDIR=/your_path