Do collection tasks count as billed tasks in Prefect Cloud?

View in #prefect-community on Slack

Suresh_R @Suresh_R: Hi, I have a flow like this A -> List -> Dict -> B, Result of A is persisted in S3, Whenever B fails and i try to restart the flow Dict task supplies empty data to B since its result is not persisted. Dict is an internal task which we don’t have any control, How we can overcome this?

Sylvain_HAZARD @Sylvain_HAZARD: Hello !
Collection tasks are used implicitly but they still exist as usual tasks if you need to use them with specific parameters. Check out this doc.

Collection Tasks | Prefect Docs

Suresh_R @Suresh_R: It won’t be counted as task for billing if i use it explicitly right?

Sylvain_HAZARD @Sylvain_HAZARD: Not a Cloud user myself, you might have to wait for a Prefect team member to answer that one, sorry.

Suresh_R @Suresh_R: Ok

Anna_Geller @Anna_Geller: @Sylvain_HAZARD is 100% correct that those tasks are added implicitly. You can avoid those in many cases if you rewrite your tasks a bit. Here is an example: How to avoid tasks such as List, Tuple or Dict in a flow structure?

Regarding billing: those tasks should not count to Billing since they take less than a second to run - see example in the image below using this sample flow:

import random
import time

from prefect import Flow, task

def a_number():
    return random.randint(0, 100)

def get_sum(x):
    return sum(x)

with Flow("Using_Collections") as flow:
    a = a_number()
    b = a_number()
    s = get_sum([a, b])