OSError: [Errno 24] "Too many open files" when running a docker agent for some time

Archive
, ,
2

Hi @Skam, great to have you here on Discourse! And thanks for this great write-up of your issue! :clap:

If you are just getting started with Prefect, perhaps it’s easier to start directly with Prefect 2.0?
https://discourse.prefect.io/t/should-i-start-with-prefect-2-0-orion-skipping-prefect-1-0/544

Regarding the error itself, before we jump into things we may try to solve it, I wonder why so many files get opened and not closed. Could you explain your setup a bit more?

  1. What registry are you pulling the images from?
  2. How did you configure credentials to that registry?
  3. Does this error happen on every flow run? Based on what you said it looks like you use some registry credentials that expire after 2-3 days - could this be the issue?

A quick and dirty solution might be to add

DefaultLimitNOFILE=100000

to
/etc/systemd/system.conf

But it would be better to understand why this error is occurring in the first place.