Vertex agent extra loggers

[kzecchini]

I’m trying to add the root logger to vertex agent runs, but I can’t get it to work properly.

I am passing through:

run_config = VertexRun(
    image="my-image", 
    env={"PREFECT__LOGGING__EXTRA_LOGGERS": "['']"},
)

I can see the environment variable submitted on the vertex job. The full submitted vertex job config (redacted potentially sensitive info) is here:

{
  "workerPoolSpecs": [
    {
      "machineSpec": {
        "machineType": "e2-standard-4"
      },
      "replicaCount": "1",
      "diskSpec": {
        "bootDiskType": "pd-ssd",
        "bootDiskSizeGb": 100
      },
      "containerSpec": {
        "imageUri": "my-image",
        "command": [
          "prefect",
          "execute",
          "flow-run"
        ],
        "env": [
          {
            "name": "PREFECT__LOGGING__LEVEL",
            "value": "INFO"
          },
          {
            "name": "PREFECT__LOGGING__EXTRA_LOGGERS",
            "value": "['']"
          },
          {
            "name": "PREFECT__BACKEND",
            "value": "cloud"
          },
          {
            "name": "PREFECT__CLOUD__API",
            "value": "https://api.prefect.io"
          },
          {
            "name": "PREFECT__CLOUD__API_KEY",
            "value": "REDACTED"
          },
          {
            "name": "PREFECT__CLOUD__TENANT_ID",
            "value": "REDACTED"
          },
          {
            "name": "PREFECT__CLOUD__AGENT__LABELS",
            "value": "['vertex']"
          },
          {
            "name": "PREFECT__CLOUD__SEND_FLOW_RUN_LOGS",
            "value": "true"
          },
          {
            "name": "PREFECT__CONTEXT__FLOW_RUN_ID",
            "value": "REDACTED"
          },
          {
            "name": "PREFECT__CONTEXT__FLOW_ID",
            "value": "REDACTED"
          },
          {
            "name": "PREFECT__CLOUD__USE_LOCAL_SECRETS",
            "value": "false"
          },
          {
            "name": "PREFECT__ENGINE__FLOW_RUNNER__DEFAULT_CLASS",
            "value": "prefect.engine.cloud.CloudFlowRunner"
          },
          {
            "name": "PREFECT__ENGINE__TASK_RUNNER__DEFAULT_CLASS",
            "value": "prefect.engine.cloud.CloudTaskRunner"
          },
          {
            "name": "PREFECT__LOGGING__LOG_TO_CLOUD",
            "value": "true"
          },
          {
            "name": "PREFECT__CLOUD__AUTH_TOKEN",
            "value": "REDACTED"
          }
        ]
      }
    }
  ],
  "serviceAccount": REDACTED
}

When I use the same env variable with local run, the logs show up correctly in prefect cloud. For example, this works correctly:

run_config = LocalRun(env={"PREFECT__LOGGING__EXTRA_LOGGERS": "['']"})

It just seems like there is something wrong with the vertex ai logging. Is there a bug? Or is there some configuration/settings that I have wrong?

Any help would be much appreciated. Thanks!

[anna_geller]

First of all, really well-described and tagged issue!

I think that you may be just missing the actual logger name:

export PREFECT__LOGGING__EXTRA_LOGGERS="['your_logger_name']"

Here you can find more details and examples:

How to set up a custom logger to stream logs to Prefect Cloud backend? (extra loggers)

How to set up a custom logger to stream logs to the Prefect Cloud backend? (extra loggers) Many libraries (e.g. boto3) are set up to emit their own logs. Similarly, you may have your own custom logger within your custom modules. Environment variable In order to make those extra loggers stream logs to the Prefect backend (making those logs available in the Prefect UI), add the following environment variable: export PREFECT__LOGGING__EXTRA_LOGGERS="['boto3', 'custom_lib']" Note that you can a…

How to add extra loggers to Prefect 1.0?

This example adds boto3 and botocore loggers so that debug logs from those libraries are displayed in the Prefect UI. Those extra log messages allow to debug issues such as bad requests made to some AWS APIs. First, set those two environment variables on the machine from which you register your flow (e.g. your laptop): export PREFECT__LOGGING__EXTRA_LOGGERS="['boto3', 'botocore']" export PREFECT__LOGGING__LEVEL="DEBUG" Now, when you use boto3 in your flow code and set the level to DEBUG expli…

[kzecchini]

Thanks for the quick reply!

I verified that using PREFECT__LOGGING__EXTRA_LOGGERS="['']"on the Local Agent works perfectly. I see all of my root logs coming through on prefect cloud.

The logs just don’t come through when using the same config on the Vertex runner. I also tried PREFECT__LOGGING__EXTRA_LOGGERS="['root']" on vertex which also does not work.

Any other ideas? Any other info I can provide to help diagnose? I’m confused why it works on local agent and not on vertex.

[kzecchini]

@anna_geller I just tried a test by instantiating a logger:

logger = logging.getLogger("test_logger")

run_config = VertexRun(
    image="my-image",
    env={"PREFECT__LOGGING__EXTRA_LOGGERS": "['test_logger']"}
)

logger.info("test log")

# rest of tasks/flow

When I run on the Vertex agent, this log statement is not shown in the prefect cloud logs.

When I run this on the Local agent with LocalRun(env={"PREFECT__LOGGING__EXTRA_LOGGERS": "['test_logger']"}), I do see the log in the prefect cloud logs.

To me it seems like there could be an issue with the Vertex agent logging unless I have some configuration or settings wrong.

[anna_geller]

I see; it could be an issue with that indeed. But could you explain why do you need extra loggers in the first place? The primary way of adding logs to your flows is by using the Prefect logger from context in your tasks:

prefect.context.logger.info("your log message")

[kzecchini]

The reason I’d like to add the root logger is our team uses proprietary shared libraries/code with pre-existing logging statements.

There are some easy ways to convert some of our classes to inherit from Task, or to create functional tasks on top of our existing code.

But there isn’t a great way to convert the logging since a lot of our shared utils are designed to also be called outside of a prefect context.

[anna_geller]

This is difficult. I can ask the team.

@kzecchini

I also tried PREFECT__LOGGING__EXTRA_LOGGERS="['root']" on vertex which also does not work.

This sounds reasonable, not sure why this isn’t working for you.

How far are you in your Prefect adoption? Do you already have some production resources using Prefect 1.0? I’m asking because this would be easier to configure in Prefect 2.0 as shown here:
https://www.prefect.io/blog/logs-the-prefect-way/

[kzecchini]

Just started to get some things going in prefect the past couple of days, so nothing is in production yet.

I quickly looked over the link you sent - seems like we would still want to use the same environment variable for this? How would that be different in prefect 2.0? I might be missing something there.

We would love to use 2.0, but I had trouble accessing the cloud beta… that’s a separate issue. When I tried to sign up with the prefect cloud email/password I have currently set up for cloud 1.0 it was giving me an error.

[anna_geller]

Prefect 2.0

@kzecchini

We would love to use 2.0, but I had trouble accessing the cloud beta… that’s a separate issue.

Yes, please! Open a separate issue for that, ideally with screenshots of the error.

@kzecchini

I quickly looked over the link you sent - seems like we would still want to use the same environment variable for this?

That’s true, but it’s easier to configure since it can now be set on a profile using settings.

Prefect 1.0

Can you try attaching the logger environment variable directly to your Vertex agent? You can do that using:

prefect agent vertex start --env PREFECT__LOGGING__EXTRA_LOGGERS="['']"

I cross-checked with Michael and setting this on a run config should work, so if it doesn’t work for you and you consider it important, I could open an issue on GitHub for you.

[kzecchini]

Thanks for the quick reply again!

The solution for 1.0 makes a lot of sense - going to give this a try today and report back.

[anna_geller]

Awesome, keep us posted!

[kzecchini]

@anna_geller I did try this option, but it still did not work for me.

I ran the agent with the command:
prefect agent vertex start --label vertex --key $PREFECT_VERTEX_API_KEY --env PREFECT__LOGGING__EXTRA_LOGGERS="['']"

Can we make a github issue for this? It is somewhat important for our team to figure this out. We still do have our logs writing out to GCS stackdriver logging, but it is a little bit of a pain to have to go outside of the prefect UI to go view the logs.

Thanks again for all the help.

[anna_geller]

@kzecchini

Can we make a github issue for this?

Sure, I opened it here

@kzecchini

it is a little bit of a pain to have to go outside of the prefect UI to go view the logs

that’s understandable

You’re very welcome!