Orchestrating Machine Learning Workflows with Prefect - by Kevin Kho // MLOps Meetup #94

In this talk, Kevin shows how to orchestrate a full ML hyperparameter grid search pipeline over a Dask cluster. Other tools included will be Pandera, Tune, and Evidently AI.

Table of contents:

[00:00] Introduction to Kevin Kho
[04:34] An Intro to Workflow Orchestration with Prefect
[04:51] Kevin’s background
[05:16] Agenda
[05:44] Workflow Orchestration
[05:48] Machine Learning Pipeline
[06:14] Pipeline dependencies
[07:06] Points of failures
[08:57] Negative Engineering
[10:17] Introducing Prefect
[12:21] Current state of Prefect
[12:33] Prefect Orion
[13:59] Simple problem
[14:14] Motivation
[12:22] Coindesk
[14:49] Setting a Slack web block explicitly
[20:41] Put infinite loop in production
[22:30] Change Dodgecoin to Bitcoin
[23:58] Testing the detective
[24:22] Function as a flow
[32:13] Intrinsic data passing from task to task
[37:28] Open-source tools
[37:42] Pandera Tutorial
[44:39] Tune Tutorial
[49:45] Evidently.AI Tutorial
[56:16] ML Training
[1:01:40] Resources
[1:02:00] Prefect is hiring!
[1:02:46] MLOps Cool Merch!
[1:03:01] Networking event to match with the MLOps Community