You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Ferran Fernández Garrido <ff...@gmail.com> on 2024/05/24 09:01:32 UTC
default_sdk_harness_log_level multi-language support
Hi!
I am trying to adjust the log level for a Beam YAML pipeline that uses
Kafka. Since Kafka is a multi-language transform,
default_sdk_harness_log_level is not currently supported, which means
I can't limit the logs (Kafka can be a bit too verbose sometimes).
This could lead to increased logging costs if you use a cloud service
such as Dataflow to run the pipeline.
Is there any initiative at the moment to support
default_sdk_harness_log_level in multi-language transforms? Maybe some
workaround? I suppose that in terms of Beam YAML, we could pass these
options to the schema providers somehow. However, I don't think that
solution is ideal, as it would not address the issue for other people
using multi-language transforms outside of Beam YAML.
Thanks!
Ferran
Re: default_sdk_harness_log_level multi-language support
Posted by XQ Hu via dev <de...@beam.apache.org>.
Feel free to open a github issue to track this idea.
For Dataflow, the potential workaround is to filter the logs:
https://cloud.google.com/dataflow/docs/guides/filter-logs
On Fri, May 24, 2024 at 5:01 AM Ferran Fernández Garrido <
ffernandez.upc@gmail.com> wrote:
> Hi!
>
> I am trying to adjust the log level for a Beam YAML pipeline that uses
> Kafka. Since Kafka is a multi-language transform,
> default_sdk_harness_log_level is not currently supported, which means
> I can't limit the logs (Kafka can be a bit too verbose sometimes).
> This could lead to increased logging costs if you use a cloud service
> such as Dataflow to run the pipeline.
>
> Is there any initiative at the moment to support
> default_sdk_harness_log_level in multi-language transforms? Maybe some
> workaround? I suppose that in terms of Beam YAML, we could pass these
> options to the schema providers somehow. However, I don't think that
> solution is ideal, as it would not address the issue for other people
> using multi-language transforms outside of Beam YAML.
>
> Thanks!
>
> Ferran
>