ConcurrentTaskRunner is blocking. Why and how to debug?

I’ve a task that donwload data (http) and upload data (s3 using s3fs) for a given year as argument. The task is synchronuous. I’ve a flow calling this task for all years (2010-2022) and I’d like a concurrent run (which is the default). In general it works for some time and then seems to block. When I CTRL+C, it starts again, I guess because I stop the blocking tasks.

I don’t understand why it blocks (it does not block in Sequential run) and I don’t know how to debug that problem.

Do you have any advice ?

1 Like

Hi and welcome to Discourse! Could you share the code you tried so far? I can try to reproduce and see whether this is something we may configure differently

An idea is to use py-spy top to watch what processes is running when it’s blocking.

1 Like