S3fs not found despite being present

Help
1

What do you do when you have prefect-ecs agent installed on AWS ECS Fargate with s3fs prefect-aws but still get a task error that it cannot find s3fs

Error log (flow run):

Downloading flow code from storage at ''
03:13:28 PM
prefect.flow_runs

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 234, in get_filesystem_class
    register_implementation(protocol, _import_class(bit["class"]))
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 269, in _import_class
    mod = importlib.import_module(mod)
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 's3fs'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 419, in filesystem
    self._filesystem = fsspec.filesystem(scheme, **self.settings)
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 288, in filesystem
    cls = get_filesystem_class(protocol)
  File "/usr/local/lib/python3.9/site-packages/fsspec/registry.py", line 236, in get_filesystem_class
    raise ImportError(bit["err"]) from e
ImportError: Install s3fs to access S3

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 394, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "/usr/local/lib/python3.9/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/prefect/deployments/deployments.py", line 213, in load_flow_from_flow_run
    await storage_block.get_directory(from_path=deployment.path, local_path=".")
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 494, in get_directory
    return await self.filesystem.get_directory(
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 337, in get_directory
    return self.filesystem.get(from_path, local_path, recursive=True)
  File "/usr/local/lib/python3.9/site-packages/prefect/filesystems.py", line 422, in filesystem
    raise RuntimeError(
RuntimeError: File system created with scheme 's3' from base path 's3://prefect-portal-us-east-1-qa' could not be created. You are likely missing a Python module required to use the given storage protocol.

I have prefect-aws and s3fs installed in the prefect agent as seen from the cloudwatch logs.

[...]
+pip install prefect-aws s3fs
Collecting s3fs
  Obtaining dependency information for s3fs from https://files.pythonhosted.org/packages/36/93/8aed66523d90361211a02dc0435855cc1ef357978decc2b05c8291fc515f/s3fs-2023.9.2-py3-none-any.whl.metadata
  Downloading s3fs-2023.9.2-py3-none-any.whl.metadata (1.6 kB)
Collecting aiobotocore~=2.5.4 (from s3fs)
Collecting fsspec==2023.9.2 (from s3fs)
[...]`

Essentially, I know that the error is the lack of the python package s3fs however, I do not know how to proceed further.