You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Vlado Djerek via dev <de...@beam.apache.org> on 2024/03/08 15:02:27 UTC

Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Hi Joey,

We just deployed the 2.54 changes on playground. Can you now check?

Thanks,
Vlado

________________________________
From: Joey Tran <jo...@schrodinger.com>
Sent: Friday, March 8, 2024 2:45:32 PM
To: Andrey Devyatkin
Subject: [EXTERNAL] Re: Hiding logging for beam playground examples

Ah thanks Andrey! I had assumed that since my change was in a release that went out a few weeks ago that Playground would've been redeployed by now. I'll check playground again in an hour or two to see if my changes did anything :)

On Fri, Mar 8, 2024 at 8:28 AM Andrey Devyatkin <an...@akvelon.com>> wrote:
Hi Joey,

Thanks for reaching out! I see that your changes haven't been deploed yet, so I've triggered the corresponding job and Playground will be updated soon.

Thanks,
Andrey

From: Joey Tran <jo...@schrodinger.com>>
Reply to: "dev@beam.apache.org<ma...@beam.apache.org>" <de...@beam.apache.org>>
Date: Thursday, 7 March 2024 at 22:33
To: Valentyn Tymofieiev <va...@google.com>>
Subject: [EXTERNAL] Re: Hiding logging for beam playground examples

I ended up making this change[1] but it looks like Playground is still outputting all the log statements. Could someone help me with this? I suppose it's best to try and actually test this, but I had a hard time getting the environment set up.

[1]https://github.com/apache/beam/pull/29948


















































































"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt

[1] https://github.com/apache/beam/pull/29948#event-11504996254

On Thu, Nov 16, 2023 at 7:42 AM Joey Tran <jo...@schrodinger.com>> wrote:
Good idea though it ended up being a shallow trace
```
  File "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py", line 57, in <module>
    assert False
```
I think I found where the log level is set anyways
```
https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39
```

When I have some time, I'll try doing a local deployment of playground and modifying those log levels

On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev <va...@google.com>> wrote:
I am also not familiar with Playground. I suspect you could try to make it crash and maybe find a stacktrace? Setting logging could like like so: https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110

On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <jo...@schrodinger.com>> wrote:
The motivating example does not use LogElements, just Map(print)
https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function

Some examples of the extraneous logging:
```
2023-09-08 22:46:37,334 [INFO] ==================== <function populate_data_channel_coders at 0x7ff2665e1a20> ====================
2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600
2023-09-08 22:46:37,338 [INFO] Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7ff2664c9870> for environment ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
```

The example code itself doesn't set the log level in some playground code. Does anyone have a pointer to where? I'm not familiar

On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <de...@beam.apache.org>> wrote:
Are the examples using LogElements? https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271

Note that LogElements by default prints to stdout, but can be configured to use a different logger. We could also change the default.

On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <de...@beam.apache.org>> wrote:
+1 to at least setting the log level to higher than info. Some runner
logging (e.g. job started/done) may be useful.

On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>> wrote:
>
> Hi all,
>
> I just had a workshop to demo beam for people at my company and there was a bit of confusion about whether the beam python playground examples were even working and it turned out they just got confused by all the runner logging that is output.
>
> Is this worth keeping? It seems like it'd be a common source of confusion for new users
>
> Cheers,
> Joey

Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Posted by Joey Tran <jo...@schrodinger.com>.
I found the log level setting code [1]. Could we adjust the level here to
ERROR?

[1]
https://github.com/apache/beam/blob/e3fee5156b3515f96dc5ba90ea2fbc6f6be2bd34/playground/backend/internal/preparers/python_preparers.go#L29


On Mon, Mar 18, 2024 at 11:49 AM Vlado Djerek <vl...@akvelon.com>
wrote:

> Hello Joey,
>
> Thanks for waiting. I did try to figure out the interface to adjust the
> logging on direct runner but was not able to find the right change.
> I also see that for Go the default runner is not direct runner but Prism
> https://github.com/apache/beam/issues/24789 and think that the long term
> goal would be that all SDKs use Prism. When they become default for
> playground we can focus on how Prism does logging and change it. It would
> require more work to try do it in direct and that is why I think its best
> to postpone until Prism.
>
> Regards,
> Vlado
>
> ------------------------------
> *From:* Joey Tran <jo...@schrodinger.com>
> *Sent:* Tuesday, March 12, 2024 7:24:42 PM
> *To:* Vlado Djerek
> *Cc:* dev@beam.apache.org
> *Subject:* Re: [EXTERNAL] Re: Hiding logging for beam playground examples
>
> Thanks for the update and work!
>
> On Tue, Mar 12, 2024 at 11:55 AM Vlado Djerek <vl...@akvelon.com>
> wrote:
>
>> Hello Joey,
>>
>> I was doing some testing and the changes i did which are in line of that
>> PR did not work. My latest thought is that the logging is at the runner
>> level and perhaps not even controlled on the Playground setup. Ill continue
>> investigating the issue.
>>
>> Thanks,
>> Vlado
>> ------------------------------
>> *From:* Joey Tran <jo...@schrodinger.com>
>> *Sent:* Tuesday, March 12, 2024 3:55:51 PM
>> *To:* dev@beam.apache.org
>> *Cc:* Vlado Djerek
>> *Subject:* Re: [EXTERNAL] Re: Hiding logging for beam playground examples
>>
>> Any update?
>>
>> If this is the right place, could we instead just hide all logging in
>> beam playground? I was thinking maybe allowing WARNING might make sense but
>> even that might be too much. From what I've seen, all the logging is pretty
>> low level DirectRunner stuff, none of which seems very relevant to a
>> playground user, and it seems unlikely we'd debug issues through playground.
>>
>> Best,
>> Joey
>>
>> On Fri, Mar 8, 2024 at 11:16 AM Vlado Djerek via dev <de...@beam.apache.org>
>> wrote:
>>
>>> Hi all,
>>>
>>> Yeah regarding the build issue think that was fixed with
>>> https://github.com/apache/beam/pull/30497.
>>>
>>> We have deployed 2.54 staging and prod now.
>>> The issue here i think is the place where logger is configured. The PR
>>> you submitted is in playground/infrastructure which I think affects the
>>> logging output of our infra pipelines.
>>>
>>> My guess that its here:
>>> https://github.com/apache/beam/pull/25692/files
>>>
>>> Im waiting on confirmation and going to let you know if its the right
>>> place.
>>>
>>> Thanks,
>>> Vlado
>>> ------------------------------
>>> *From:* Robert Burke <ro...@frantil.com>
>>> *Sent:* Friday, March 8, 2024 4:35:02 PM
>>> *To:* dev
>>> *Subject:* Re: [EXTERNAL] Re: Hiding logging for beam playground
>>> examples
>>>
>>> Thanks!
>>>
>>> The best I can recall, is that there some java dep that wasn't being
>>> pulled properly somehow. But it's very much a needle in a haystack
>>> situation. The logs are thousands of lines of gradle spew. And i was just
>>> following the Beam Release guide for updating the playground.
>>>
>>> As a non-Java person I'm not positioned well to get further in debugging
>>> it, just in reporting it.
>>>
>>> See y'all in 2 weeks!
>>>
>>> On Fri, Mar 8, 2024, 10:19 AM Joey Tran <jo...@schrodinger.com>
>>> wrote:
>>>
>>>> Ah, there is hope still then. Will follow up in a few weeks then. Happy
>>>> vacationing! ☀️
>>>>
>>>> On Fri, Mar 8, 2024 at 10:16 AM Robert Burke <ro...@frantil.com>
>>>> wrote:
>>>>
>>>>> The playground, last i checked, was stuck on 2.52.0 because it's been
>>>>> unable to be built and staged. That issue would need to be resolved first
>>>>>
>>>>> I unfortunately didn't get around to filing it in github or searching
>>>>> for there, as I got distracted by urgent matters and it slipped my mind.
>>>>> Very sorry about that.
>>>>>
>>>>> (Currently I'm away from laptop on vacation, so I can't do that until
>>>>> Monday the 18th at earliest.)
>>>>>
>>>>> On Fri, Mar 8, 2024, 10:05 AM Joey Tran <jo...@schrodinger.com>
>>>>> wrote:
>>>>>
>>>>>> Looks like there's still INFO prints unfortunately
>>>>>> [image: image.png]
>>>>>>
>>>>>> It's possibly my fix just didn't work :'(
>>>>>>
>>>>>> On Fri, Mar 8, 2024 at 10:02 AM Vlado Djerek <
>>>>>> vlado.djerek@akvelon.com> wrote:
>>>>>>
>>>>>>> Hi Joey,
>>>>>>>
>>>>>>> We just deployed the 2.54 changes on playground. Can you now check?
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Vlado
>>>>>>> ------------------------------
>>>>>>> *From:* Joey Tran <jo...@schrodinger.com>
>>>>>>> *Sent:* Friday, March 8, 2024 2:45:32 PM
>>>>>>> *To:* Andrey Devyatkin
>>>>>>> *Subject:* [EXTERNAL] Re: Hiding logging for beam playground
>>>>>>> examples
>>>>>>>
>>>>>>> Ah thanks Andrey! I had assumed that since my change was in a
>>>>>>> release that went out a few weeks ago that Playground would've been
>>>>>>> redeployed by now. I'll check playground again in an hour or two to see if
>>>>>>> my changes did anything :)
>>>>>>>
>>>>>>> On Fri, Mar 8, 2024 at 8:28 AM Andrey Devyatkin <
>>>>>>> andrey.devyatkin@akvelon.com> wrote:
>>>>>>>
>>>>>>>> Hi Joey,
>>>>>>>>
>>>>>>>> Thanks for reaching out! I see that your changes haven't been
>>>>>>>> deploed yet, so I've triggered the corresponding job and Playground will be
>>>>>>>> updated soon.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Andrey
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *From: *Joey Tran <jo...@schrodinger.com>
>>>>>>>> *Reply to: *"dev@beam.apache.org" <de...@beam.apache.org>
>>>>>>>> *Date: *Thursday, 7 March 2024 at 22:33
>>>>>>>> *To: *Valentyn Tymofieiev <va...@google.com>
>>>>>>>> *Subject: *[EXTERNAL] Re: Hiding logging for beam playground
>>>>>>>> examples
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> I ended up making this change[1] but it looks like Playground is
>>>>>>>> still outputting all the log statements. Could someone help me with this? I
>>>>>>>> suppose it's best to try and actually test this, but I had a hard time
>>>>>>>> getting the environment set up.
>>>>>>>>
>>>>>>>> [1]https://github.com/apache/beam/pull/29948
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt
>>>>>>>>
>>>>>>>> [1] https://github.com/apache/beam/pull/29948#event-11504996254
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Thu, Nov 16, 2023 at 7:42 AM Joey Tran <
>>>>>>>> joey.tran@schrodinger.com> wrote:
>>>>>>>>
>>>>>>>> Good idea though it ended up being a shallow trace
>>>>>>>> ```
>>>>>>>>   File
>>>>>>>> "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py",
>>>>>>>> line 57, in <module>
>>>>>>>>     assert False
>>>>>>>> ```
>>>>>>>> I think I found where the log level is set anyways
>>>>>>>> ```
>>>>>>>>
>>>>>>>> https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39
>>>>>>>> ```
>>>>>>>>
>>>>>>>> When I have some time, I'll try doing a local deployment of
>>>>>>>> playground and modifying those log levels
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev <
>>>>>>>> valentyn@google.com> wrote:
>>>>>>>>
>>>>>>>> I am also not familiar with Playground. I suspect you could try to
>>>>>>>> make it crash and maybe find a stacktrace? Setting logging could like like
>>>>>>>> so:
>>>>>>>> https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <
>>>>>>>> joey.tran@schrodinger.com> wrote:
>>>>>>>>
>>>>>>>> The motivating example does not use LogElements, just Map(print)
>>>>>>>>
>>>>>>>> https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function
>>>>>>>>
>>>>>>>> Some examples of the extraneous logging:
>>>>>>>> ```
>>>>>>>> 2023-09-08 22:46:37,334 [INFO] ==================== <function
>>>>>>>> populate_data_channel_coders at 0x7ff2665e1a20> ====================
>>>>>>>> 2023-09-08 22:46:37,336 [INFO] Creating state cache with size
>>>>>>>> 104857600
>>>>>>>> 2023-09-08 22:46:37,338 [INFO] Created Worker handler
>>>>>>>> <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
>>>>>>>> object at 0x7ff2664c9870> for environment
>>>>>>>> ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
>>>>>>>> ```
>>>>>>>>
>>>>>>>> The example code itself doesn't set the log level in some
>>>>>>>> playground code. Does anyone have a pointer to where? I'm not familiar
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <
>>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>>
>>>>>>>> Are the examples using LogElements?
>>>>>>>> https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Note that LogElements by default prints to stdout, but can be
>>>>>>>> configured to use a different logger. We could also change the default.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <
>>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>>
>>>>>>>> +1 to at least setting the log level to higher than info. Some
>>>>>>>> runner
>>>>>>>> logging (e.g. job started/done) may be useful.
>>>>>>>>
>>>>>>>> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <
>>>>>>>> joey.tran@schrodinger.com> wrote:
>>>>>>>> >
>>>>>>>> > Hi all,
>>>>>>>> >
>>>>>>>> > I just had a workshop to demo beam for people at my company and
>>>>>>>> there was a bit of confusion about whether the beam python playground
>>>>>>>> examples were even working and it turned out they just got confused by all
>>>>>>>> the runner logging that is output.
>>>>>>>> >
>>>>>>>> > Is this worth keeping? It seems like it'd be a common source of
>>>>>>>> confusion for new users
>>>>>>>> >
>>>>>>>> > Cheers,
>>>>>>>> > Joey
>>>>>>>>
>>>>>>>>

Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Posted by Vlado Djerek via dev <de...@beam.apache.org>.
Hello Joey,

Thanks for waiting. I did try to figure out the interface to adjust the logging on direct runner but was not able to find the right change.
I also see that for Go the default runner is not direct runner but Prism https://github.com/apache/beam/issues/24789 and think that the long term goal would be that all SDKs use Prism. When they become default for playground we can focus on how Prism does logging and change it. It would require more work to try do it in direct and that is why I think its best to postpone until Prism.

Regards,
Vlado

________________________________
From: Joey Tran <jo...@schrodinger.com>
Sent: Tuesday, March 12, 2024 7:24:42 PM
To: Vlado Djerek
Cc: dev@beam.apache.org
Subject: Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Thanks for the update and work!

On Tue, Mar 12, 2024 at 11:55 AM Vlado Djerek <vl...@akvelon.com>> wrote:

Hello Joey,

I was doing some testing and the changes i did which are in line of that PR did not work. My latest thought is that the logging is at the runner level and perhaps not even controlled on the Playground setup. Ill continue investigating the issue.

Thanks,
Vlado

________________________________
From: Joey Tran <jo...@schrodinger.com>>
Sent: Tuesday, March 12, 2024 3:55:51 PM
To: dev@beam.apache.org<ma...@beam.apache.org>
Cc: Vlado Djerek
Subject: Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Any update?

If this is the right place, could we instead just hide all logging in beam playground? I was thinking maybe allowing WARNING might make sense but even that might be too much. From what I've seen, all the logging is pretty low level DirectRunner stuff, none of which seems very relevant to a playground user, and it seems unlikely we'd debug issues through playground.

Best,
Joey

On Fri, Mar 8, 2024 at 11:16 AM Vlado Djerek via dev <de...@beam.apache.org>> wrote:

Hi all,

Yeah regarding the build issue think that was fixed with https://github.com/apache/beam/pull/30497.

We have deployed 2.54 staging and prod now.
The issue here i think is the place where logger is configured. The PR you submitted is in playground/infrastructure which I think affects the logging output of our infra pipelines.

My guess that its here:
https://github.com/apache/beam/pull/25692/files

Im waiting on confirmation and going to let you know if its the right place.

Thanks,
Vlado

________________________________
From: Robert Burke <ro...@frantil.com>>
Sent: Friday, March 8, 2024 4:35:02 PM
To: dev
Subject: Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Thanks!

The best I can recall, is that there some java dep that wasn't being pulled properly somehow. But it's very much a needle in a haystack situation. The logs are thousands of lines of gradle spew. And i was just following the Beam Release guide for updating the playground.

As a non-Java person I'm not positioned well to get further in debugging it, just in reporting it.

See y'all in 2 weeks!

On Fri, Mar 8, 2024, 10:19 AM Joey Tran <jo...@schrodinger.com>> wrote:
Ah, there is hope still then. Will follow up in a few weeks then. Happy vacationing! ☀️

On Fri, Mar 8, 2024 at 10:16 AM Robert Burke <ro...@frantil.com>> wrote:
The playground, last i checked, was stuck on 2.52.0 because it's been unable to be built and staged. That issue would need to be resolved first

I unfortunately didn't get around to filing it in github or searching for there, as I got distracted by urgent matters and it slipped my mind. Very sorry about that.

(Currently I'm away from laptop on vacation, so I can't do that until Monday the 18th at earliest.)

On Fri, Mar 8, 2024, 10:05 AM Joey Tran <jo...@schrodinger.com>> wrote:
Looks like there's still INFO prints unfortunately
[image.png]

It's possibly my fix just didn't work :'(

On Fri, Mar 8, 2024 at 10:02 AM Vlado Djerek <vl...@akvelon.com>> wrote:

Hi Joey,

We just deployed the 2.54 changes on playground. Can you now check?

Thanks,
Vlado

________________________________
From: Joey Tran <jo...@schrodinger.com>>
Sent: Friday, March 8, 2024 2:45:32 PM
To: Andrey Devyatkin
Subject: [EXTERNAL] Re: Hiding logging for beam playground examples

Ah thanks Andrey! I had assumed that since my change was in a release that went out a few weeks ago that Playground would've been redeployed by now. I'll check playground again in an hour or two to see if my changes did anything :)

On Fri, Mar 8, 2024 at 8:28 AM Andrey Devyatkin <an...@akvelon.com>> wrote:
Hi Joey,

Thanks for reaching out! I see that your changes haven't been deploed yet, so I've triggered the corresponding job and Playground will be updated soon.

Thanks,
Andrey

From: Joey Tran <jo...@schrodinger.com>>
Reply to: "dev@beam.apache.org<ma...@beam.apache.org>" <de...@beam.apache.org>>
Date: Thursday, 7 March 2024 at 22:33
To: Valentyn Tymofieiev <va...@google.com>>
Subject: [EXTERNAL] Re: Hiding logging for beam playground examples

I ended up making this change[1] but it looks like Playground is still outputting all the log statements. Could someone help me with this? I suppose it's best to try and actually test this, but I had a hard time getting the environment set up.

[1]https://github.com/apache/beam/pull/29948


















































































"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt

[1] https://github.com/apache/beam/pull/29948#event-11504996254

On Thu, Nov 16, 2023 at 7:42 AM Joey Tran <jo...@schrodinger.com>> wrote:
Good idea though it ended up being a shallow trace
```
  File "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py", line 57, in <module>
    assert False
```
I think I found where the log level is set anyways
```
https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39
```

When I have some time, I'll try doing a local deployment of playground and modifying those log levels

On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev <va...@google.com>> wrote:
I am also not familiar with Playground. I suspect you could try to make it crash and maybe find a stacktrace? Setting logging could like like so: https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110

On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <jo...@schrodinger.com>> wrote:
The motivating example does not use LogElements, just Map(print)
https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function

Some examples of the extraneous logging:
```
2023-09-08 22:46:37,334 [INFO] ==================== <function populate_data_channel_coders at 0x7ff2665e1a20> ====================
2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600
2023-09-08 22:46:37,338 [INFO] Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7ff2664c9870> for environment ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
```

The example code itself doesn't set the log level in some playground code. Does anyone have a pointer to where? I'm not familiar

On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <de...@beam.apache.org>> wrote:
Are the examples using LogElements? https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271

Note that LogElements by default prints to stdout, but can be configured to use a different logger. We could also change the default.

On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <de...@beam.apache.org>> wrote:
+1 to at least setting the log level to higher than info. Some runner
logging (e.g. job started/done) may be useful.

On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>> wrote:
>
> Hi all,
>
> I just had a workshop to demo beam for people at my company and there was a bit of confusion about whether the beam python playground examples were even working and it turned out they just got confused by all the runner logging that is output.
>
> Is this worth keeping? It seems like it'd be a common source of confusion for new users
>
> Cheers,
> Joey

Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Posted by Joey Tran <jo...@schrodinger.com>.
Thanks for the update and work!

On Tue, Mar 12, 2024 at 11:55 AM Vlado Djerek <vl...@akvelon.com>
wrote:

> Hello Joey,
>
> I was doing some testing and the changes i did which are in line of that
> PR did not work. My latest thought is that the logging is at the runner
> level and perhaps not even controlled on the Playground setup. Ill continue
> investigating the issue.
>
> Thanks,
> Vlado
> ------------------------------
> *From:* Joey Tran <jo...@schrodinger.com>
> *Sent:* Tuesday, March 12, 2024 3:55:51 PM
> *To:* dev@beam.apache.org
> *Cc:* Vlado Djerek
> *Subject:* Re: [EXTERNAL] Re: Hiding logging for beam playground examples
>
> Any update?
>
> If this is the right place, could we instead just hide all logging in beam
> playground? I was thinking maybe allowing WARNING might make sense but even
> that might be too much. From what I've seen, all the logging is pretty low
> level DirectRunner stuff, none of which seems very relevant to a playground
> user, and it seems unlikely we'd debug issues through playground.
>
> Best,
> Joey
>
> On Fri, Mar 8, 2024 at 11:16 AM Vlado Djerek via dev <de...@beam.apache.org>
> wrote:
>
>> Hi all,
>>
>> Yeah regarding the build issue think that was fixed with
>> https://github.com/apache/beam/pull/30497.
>>
>> We have deployed 2.54 staging and prod now.
>> The issue here i think is the place where logger is configured. The PR
>> you submitted is in playground/infrastructure which I think affects the
>> logging output of our infra pipelines.
>>
>> My guess that its here:
>> https://github.com/apache/beam/pull/25692/files
>>
>> Im waiting on confirmation and going to let you know if its the right
>> place.
>>
>> Thanks,
>> Vlado
>> ------------------------------
>> *From:* Robert Burke <ro...@frantil.com>
>> *Sent:* Friday, March 8, 2024 4:35:02 PM
>> *To:* dev
>> *Subject:* Re: [EXTERNAL] Re: Hiding logging for beam playground examples
>>
>> Thanks!
>>
>> The best I can recall, is that there some java dep that wasn't being
>> pulled properly somehow. But it's very much a needle in a haystack
>> situation. The logs are thousands of lines of gradle spew. And i was just
>> following the Beam Release guide for updating the playground.
>>
>> As a non-Java person I'm not positioned well to get further in debugging
>> it, just in reporting it.
>>
>> See y'all in 2 weeks!
>>
>> On Fri, Mar 8, 2024, 10:19 AM Joey Tran <jo...@schrodinger.com>
>> wrote:
>>
>>> Ah, there is hope still then. Will follow up in a few weeks then. Happy
>>> vacationing! ☀️
>>>
>>> On Fri, Mar 8, 2024 at 10:16 AM Robert Burke <ro...@frantil.com> wrote:
>>>
>>>> The playground, last i checked, was stuck on 2.52.0 because it's been
>>>> unable to be built and staged. That issue would need to be resolved first
>>>>
>>>> I unfortunately didn't get around to filing it in github or searching
>>>> for there, as I got distracted by urgent matters and it slipped my mind.
>>>> Very sorry about that.
>>>>
>>>> (Currently I'm away from laptop on vacation, so I can't do that until
>>>> Monday the 18th at earliest.)
>>>>
>>>> On Fri, Mar 8, 2024, 10:05 AM Joey Tran <jo...@schrodinger.com>
>>>> wrote:
>>>>
>>>>> Looks like there's still INFO prints unfortunately
>>>>> [image: image.png]
>>>>>
>>>>> It's possibly my fix just didn't work :'(
>>>>>
>>>>> On Fri, Mar 8, 2024 at 10:02 AM Vlado Djerek <vl...@akvelon.com>
>>>>> wrote:
>>>>>
>>>>>> Hi Joey,
>>>>>>
>>>>>> We just deployed the 2.54 changes on playground. Can you now check?
>>>>>>
>>>>>> Thanks,
>>>>>> Vlado
>>>>>> ------------------------------
>>>>>> *From:* Joey Tran <jo...@schrodinger.com>
>>>>>> *Sent:* Friday, March 8, 2024 2:45:32 PM
>>>>>> *To:* Andrey Devyatkin
>>>>>> *Subject:* [EXTERNAL] Re: Hiding logging for beam playground examples
>>>>>>
>>>>>> Ah thanks Andrey! I had assumed that since my change was in a release
>>>>>> that went out a few weeks ago that Playground would've been redeployed by
>>>>>> now. I'll check playground again in an hour or two to see if my changes did
>>>>>> anything :)
>>>>>>
>>>>>> On Fri, Mar 8, 2024 at 8:28 AM Andrey Devyatkin <
>>>>>> andrey.devyatkin@akvelon.com> wrote:
>>>>>>
>>>>>>> Hi Joey,
>>>>>>>
>>>>>>> Thanks for reaching out! I see that your changes haven't been
>>>>>>> deploed yet, so I've triggered the corresponding job and Playground will be
>>>>>>> updated soon.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Andrey
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *From: *Joey Tran <jo...@schrodinger.com>
>>>>>>> *Reply to: *"dev@beam.apache.org" <de...@beam.apache.org>
>>>>>>> *Date: *Thursday, 7 March 2024 at 22:33
>>>>>>> *To: *Valentyn Tymofieiev <va...@google.com>
>>>>>>> *Subject: *[EXTERNAL] Re: Hiding logging for beam playground
>>>>>>> examples
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I ended up making this change[1] but it looks like Playground is
>>>>>>> still outputting all the log statements. Could someone help me with this? I
>>>>>>> suppose it's best to try and actually test this, but I had a hard time
>>>>>>> getting the environment set up.
>>>>>>>
>>>>>>> [1]https://github.com/apache/beam/pull/29948
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt
>>>>>>>
>>>>>>> [1] https://github.com/apache/beam/pull/29948#event-11504996254
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 16, 2023 at 7:42 AM Joey Tran <jo...@schrodinger.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Good idea though it ended up being a shallow trace
>>>>>>> ```
>>>>>>>   File
>>>>>>> "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py",
>>>>>>> line 57, in <module>
>>>>>>>     assert False
>>>>>>> ```
>>>>>>> I think I found where the log level is set anyways
>>>>>>> ```
>>>>>>>
>>>>>>> https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39
>>>>>>> ```
>>>>>>>
>>>>>>> When I have some time, I'll try doing a local deployment of
>>>>>>> playground and modifying those log levels
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev <
>>>>>>> valentyn@google.com> wrote:
>>>>>>>
>>>>>>> I am also not familiar with Playground. I suspect you could try to
>>>>>>> make it crash and maybe find a stacktrace? Setting logging could like like
>>>>>>> so:
>>>>>>> https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <
>>>>>>> joey.tran@schrodinger.com> wrote:
>>>>>>>
>>>>>>> The motivating example does not use LogElements, just Map(print)
>>>>>>>
>>>>>>> https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function
>>>>>>>
>>>>>>> Some examples of the extraneous logging:
>>>>>>> ```
>>>>>>> 2023-09-08 22:46:37,334 [INFO] ==================== <function
>>>>>>> populate_data_channel_coders at 0x7ff2665e1a20> ====================
>>>>>>> 2023-09-08 22:46:37,336 [INFO] Creating state cache with size
>>>>>>> 104857600
>>>>>>> 2023-09-08 22:46:37,338 [INFO] Created Worker handler
>>>>>>> <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
>>>>>>> object at 0x7ff2664c9870> for environment
>>>>>>> ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
>>>>>>> ```
>>>>>>>
>>>>>>> The example code itself doesn't set the log level in some playground
>>>>>>> code. Does anyone have a pointer to where? I'm not familiar
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <
>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>
>>>>>>> Are the examples using LogElements?
>>>>>>> https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Note that LogElements by default prints to stdout, but can be
>>>>>>> configured to use a different logger. We could also change the default.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <
>>>>>>> dev@beam.apache.org> wrote:
>>>>>>>
>>>>>>> +1 to at least setting the log level to higher than info. Some runner
>>>>>>> logging (e.g. job started/done) may be useful.
>>>>>>>
>>>>>>> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>
>>>>>>> wrote:
>>>>>>> >
>>>>>>> > Hi all,
>>>>>>> >
>>>>>>> > I just had a workshop to demo beam for people at my company and
>>>>>>> there was a bit of confusion about whether the beam python playground
>>>>>>> examples were even working and it turned out they just got confused by all
>>>>>>> the runner logging that is output.
>>>>>>> >
>>>>>>> > Is this worth keeping? It seems like it'd be a common source of
>>>>>>> confusion for new users
>>>>>>> >
>>>>>>> > Cheers,
>>>>>>> > Joey
>>>>>>>
>>>>>>>

Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Posted by Vlado Djerek via dev <de...@beam.apache.org>.
Hello Joey,

I was doing some testing and the changes i did which are in line of that PR did not work. My latest thought is that the logging is at the runner level and perhaps not even controlled on the Playground setup. Ill continue investigating the issue.

Thanks,
Vlado

________________________________
From: Joey Tran <jo...@schrodinger.com>
Sent: Tuesday, March 12, 2024 3:55:51 PM
To: dev@beam.apache.org
Cc: Vlado Djerek
Subject: Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Any update?

If this is the right place, could we instead just hide all logging in beam playground? I was thinking maybe allowing WARNING might make sense but even that might be too much. From what I've seen, all the logging is pretty low level DirectRunner stuff, none of which seems very relevant to a playground user, and it seems unlikely we'd debug issues through playground.

Best,
Joey

On Fri, Mar 8, 2024 at 11:16 AM Vlado Djerek via dev <de...@beam.apache.org>> wrote:

Hi all,

Yeah regarding the build issue think that was fixed with https://github.com/apache/beam/pull/30497.

We have deployed 2.54 staging and prod now.
The issue here i think is the place where logger is configured. The PR you submitted is in playground/infrastructure which I think affects the logging output of our infra pipelines.

My guess that its here:
https://github.com/apache/beam/pull/25692/files

Im waiting on confirmation and going to let you know if its the right place.

Thanks,
Vlado

________________________________
From: Robert Burke <ro...@frantil.com>>
Sent: Friday, March 8, 2024 4:35:02 PM
To: dev
Subject: Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Thanks!

The best I can recall, is that there some java dep that wasn't being pulled properly somehow. But it's very much a needle in a haystack situation. The logs are thousands of lines of gradle spew. And i was just following the Beam Release guide for updating the playground.

As a non-Java person I'm not positioned well to get further in debugging it, just in reporting it.

See y'all in 2 weeks!

On Fri, Mar 8, 2024, 10:19 AM Joey Tran <jo...@schrodinger.com>> wrote:
Ah, there is hope still then. Will follow up in a few weeks then. Happy vacationing! ☀️

On Fri, Mar 8, 2024 at 10:16 AM Robert Burke <ro...@frantil.com>> wrote:
The playground, last i checked, was stuck on 2.52.0 because it's been unable to be built and staged. That issue would need to be resolved first

I unfortunately didn't get around to filing it in github or searching for there, as I got distracted by urgent matters and it slipped my mind. Very sorry about that.

(Currently I'm away from laptop on vacation, so I can't do that until Monday the 18th at earliest.)

On Fri, Mar 8, 2024, 10:05 AM Joey Tran <jo...@schrodinger.com>> wrote:
Looks like there's still INFO prints unfortunately
[image.png]

It's possibly my fix just didn't work :'(

On Fri, Mar 8, 2024 at 10:02 AM Vlado Djerek <vl...@akvelon.com>> wrote:

Hi Joey,

We just deployed the 2.54 changes on playground. Can you now check?

Thanks,
Vlado

________________________________
From: Joey Tran <jo...@schrodinger.com>>
Sent: Friday, March 8, 2024 2:45:32 PM
To: Andrey Devyatkin
Subject: [EXTERNAL] Re: Hiding logging for beam playground examples

Ah thanks Andrey! I had assumed that since my change was in a release that went out a few weeks ago that Playground would've been redeployed by now. I'll check playground again in an hour or two to see if my changes did anything :)

On Fri, Mar 8, 2024 at 8:28 AM Andrey Devyatkin <an...@akvelon.com>> wrote:
Hi Joey,

Thanks for reaching out! I see that your changes haven't been deploed yet, so I've triggered the corresponding job and Playground will be updated soon.

Thanks,
Andrey

From: Joey Tran <jo...@schrodinger.com>>
Reply to: "dev@beam.apache.org<ma...@beam.apache.org>" <de...@beam.apache.org>>
Date: Thursday, 7 March 2024 at 22:33
To: Valentyn Tymofieiev <va...@google.com>>
Subject: [EXTERNAL] Re: Hiding logging for beam playground examples

I ended up making this change[1] but it looks like Playground is still outputting all the log statements. Could someone help me with this? I suppose it's best to try and actually test this, but I had a hard time getting the environment set up.

[1]https://github.com/apache/beam/pull/29948


















































































"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt

[1] https://github.com/apache/beam/pull/29948#event-11504996254

On Thu, Nov 16, 2023 at 7:42 AM Joey Tran <jo...@schrodinger.com>> wrote:
Good idea though it ended up being a shallow trace
```
  File "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py", line 57, in <module>
    assert False
```
I think I found where the log level is set anyways
```
https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39
```

When I have some time, I'll try doing a local deployment of playground and modifying those log levels

On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev <va...@google.com>> wrote:
I am also not familiar with Playground. I suspect you could try to make it crash and maybe find a stacktrace? Setting logging could like like so: https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110

On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <jo...@schrodinger.com>> wrote:
The motivating example does not use LogElements, just Map(print)
https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function

Some examples of the extraneous logging:
```
2023-09-08 22:46:37,334 [INFO] ==================== <function populate_data_channel_coders at 0x7ff2665e1a20> ====================
2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600
2023-09-08 22:46:37,338 [INFO] Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7ff2664c9870> for environment ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
```

The example code itself doesn't set the log level in some playground code. Does anyone have a pointer to where? I'm not familiar

On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <de...@beam.apache.org>> wrote:
Are the examples using LogElements? https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271

Note that LogElements by default prints to stdout, but can be configured to use a different logger. We could also change the default.

On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <de...@beam.apache.org>> wrote:
+1 to at least setting the log level to higher than info. Some runner
logging (e.g. job started/done) may be useful.

On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>> wrote:
>
> Hi all,
>
> I just had a workshop to demo beam for people at my company and there was a bit of confusion about whether the beam python playground examples were even working and it turned out they just got confused by all the runner logging that is output.
>
> Is this worth keeping? It seems like it'd be a common source of confusion for new users
>
> Cheers,
> Joey

Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Posted by Joey Tran <jo...@schrodinger.com>.
Any update?

If this is the right place, could we instead just hide all logging in beam
playground? I was thinking maybe allowing WARNING might make sense but even
that might be too much. From what I've seen, all the logging is pretty low
level DirectRunner stuff, none of which seems very relevant to a playground
user, and it seems unlikely we'd debug issues through playground.

Best,
Joey

On Fri, Mar 8, 2024 at 11:16 AM Vlado Djerek via dev <de...@beam.apache.org>
wrote:

> Hi all,
>
> Yeah regarding the build issue think that was fixed with
> https://github.com/apache/beam/pull/30497.
>
> We have deployed 2.54 staging and prod now.
> The issue here i think is the place where logger is configured. The PR
> you submitted is in playground/infrastructure which I think affects the
> logging output of our infra pipelines.
>
> My guess that its here:
> https://github.com/apache/beam/pull/25692/files
>
> Im waiting on confirmation and going to let you know if its the right
> place.
>
> Thanks,
> Vlado
> ------------------------------
> *From:* Robert Burke <ro...@frantil.com>
> *Sent:* Friday, March 8, 2024 4:35:02 PM
> *To:* dev
> *Subject:* Re: [EXTERNAL] Re: Hiding logging for beam playground examples
>
> Thanks!
>
> The best I can recall, is that there some java dep that wasn't being
> pulled properly somehow. But it's very much a needle in a haystack
> situation. The logs are thousands of lines of gradle spew. And i was just
> following the Beam Release guide for updating the playground.
>
> As a non-Java person I'm not positioned well to get further in debugging
> it, just in reporting it.
>
> See y'all in 2 weeks!
>
> On Fri, Mar 8, 2024, 10:19 AM Joey Tran <jo...@schrodinger.com> wrote:
>
>> Ah, there is hope still then. Will follow up in a few weeks then. Happy
>> vacationing! ☀️
>>
>> On Fri, Mar 8, 2024 at 10:16 AM Robert Burke <ro...@frantil.com> wrote:
>>
>>> The playground, last i checked, was stuck on 2.52.0 because it's been
>>> unable to be built and staged. That issue would need to be resolved first
>>>
>>> I unfortunately didn't get around to filing it in github or searching
>>> for there, as I got distracted by urgent matters and it slipped my mind.
>>> Very sorry about that.
>>>
>>> (Currently I'm away from laptop on vacation, so I can't do that until
>>> Monday the 18th at earliest.)
>>>
>>> On Fri, Mar 8, 2024, 10:05 AM Joey Tran <jo...@schrodinger.com>
>>> wrote:
>>>
>>>> Looks like there's still INFO prints unfortunately
>>>> [image: image.png]
>>>>
>>>> It's possibly my fix just didn't work :'(
>>>>
>>>> On Fri, Mar 8, 2024 at 10:02 AM Vlado Djerek <vl...@akvelon.com>
>>>> wrote:
>>>>
>>>>> Hi Joey,
>>>>>
>>>>> We just deployed the 2.54 changes on playground. Can you now check?
>>>>>
>>>>> Thanks,
>>>>> Vlado
>>>>> ------------------------------
>>>>> *From:* Joey Tran <jo...@schrodinger.com>
>>>>> *Sent:* Friday, March 8, 2024 2:45:32 PM
>>>>> *To:* Andrey Devyatkin
>>>>> *Subject:* [EXTERNAL] Re: Hiding logging for beam playground examples
>>>>>
>>>>> Ah thanks Andrey! I had assumed that since my change was in a release
>>>>> that went out a few weeks ago that Playground would've been redeployed by
>>>>> now. I'll check playground again in an hour or two to see if my changes did
>>>>> anything :)
>>>>>
>>>>> On Fri, Mar 8, 2024 at 8:28 AM Andrey Devyatkin <
>>>>> andrey.devyatkin@akvelon.com> wrote:
>>>>>
>>>>>> Hi Joey,
>>>>>>
>>>>>> Thanks for reaching out! I see that your changes haven't been deploed
>>>>>> yet, so I've triggered the corresponding job and Playground will be updated
>>>>>> soon.
>>>>>>
>>>>>> Thanks,
>>>>>> Andrey
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From: *Joey Tran <jo...@schrodinger.com>
>>>>>> *Reply to: *"dev@beam.apache.org" <de...@beam.apache.org>
>>>>>> *Date: *Thursday, 7 March 2024 at 22:33
>>>>>> *To: *Valentyn Tymofieiev <va...@google.com>
>>>>>> *Subject: *[EXTERNAL] Re: Hiding logging for beam playground examples
>>>>>>
>>>>>>
>>>>>>
>>>>>> I ended up making this change[1] but it looks like Playground is
>>>>>> still outputting all the log statements. Could someone help me with this? I
>>>>>> suppose it's best to try and actually test this, but I had a hard time
>>>>>> getting the environment set up.
>>>>>>
>>>>>> [1]https://github.com/apache/beam/pull/29948
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt
>>>>>>
>>>>>> [1] https://github.com/apache/beam/pull/29948#event-11504996254
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Thu, Nov 16, 2023 at 7:42 AM Joey Tran <jo...@schrodinger.com>
>>>>>> wrote:
>>>>>>
>>>>>> Good idea though it ended up being a shallow trace
>>>>>> ```
>>>>>>   File
>>>>>> "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py",
>>>>>> line 57, in <module>
>>>>>>     assert False
>>>>>> ```
>>>>>> I think I found where the log level is set anyways
>>>>>> ```
>>>>>>
>>>>>> https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39
>>>>>> ```
>>>>>>
>>>>>> When I have some time, I'll try doing a local deployment of
>>>>>> playground and modifying those log levels
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev <
>>>>>> valentyn@google.com> wrote:
>>>>>>
>>>>>> I am also not familiar with Playground. I suspect you could try to
>>>>>> make it crash and maybe find a stacktrace? Setting logging could like like
>>>>>> so:
>>>>>> https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <jo...@schrodinger.com>
>>>>>> wrote:
>>>>>>
>>>>>> The motivating example does not use LogElements, just Map(print)
>>>>>>
>>>>>> https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function
>>>>>>
>>>>>> Some examples of the extraneous logging:
>>>>>> ```
>>>>>> 2023-09-08 22:46:37,334 [INFO] ==================== <function
>>>>>> populate_data_channel_coders at 0x7ff2665e1a20> ====================
>>>>>> 2023-09-08 22:46:37,336 [INFO] Creating state cache with size
>>>>>> 104857600
>>>>>> 2023-09-08 22:46:37,338 [INFO] Created Worker handler
>>>>>> <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
>>>>>> object at 0x7ff2664c9870> for environment
>>>>>> ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
>>>>>> ```
>>>>>>
>>>>>> The example code itself doesn't set the log level in some playground
>>>>>> code. Does anyone have a pointer to where? I'm not familiar
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>> Are the examples using LogElements?
>>>>>> https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271
>>>>>>
>>>>>>
>>>>>>
>>>>>> Note that LogElements by default prints to stdout, but can be
>>>>>> configured to use a different logger. We could also change the default.
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <
>>>>>> dev@beam.apache.org> wrote:
>>>>>>
>>>>>> +1 to at least setting the log level to higher than info. Some runner
>>>>>> logging (e.g. job started/done) may be useful.
>>>>>>
>>>>>> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>
>>>>>> wrote:
>>>>>> >
>>>>>> > Hi all,
>>>>>> >
>>>>>> > I just had a workshop to demo beam for people at my company and
>>>>>> there was a bit of confusion about whether the beam python playground
>>>>>> examples were even working and it turned out they just got confused by all
>>>>>> the runner logging that is output.
>>>>>> >
>>>>>> > Is this worth keeping? It seems like it'd be a common source of
>>>>>> confusion for new users
>>>>>> >
>>>>>> > Cheers,
>>>>>> > Joey
>>>>>>
>>>>>>

Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Posted by Vlado Djerek via dev <de...@beam.apache.org>.
Hi all,

Yeah regarding the build issue think that was fixed with https://github.com/apache/beam/pull/30497.

We have deployed 2.54 staging and prod now.
The issue here i think is the place where logger is configured. The PR you submitted is in playground/infrastructure which I think affects the logging output of our infra pipelines.

My guess that its here:
https://github.com/apache/beam/pull/25692/files

Im waiting on confirmation and going to let you know if its the right place.

Thanks,
Vlado

________________________________
From: Robert Burke <ro...@frantil.com>
Sent: Friday, March 8, 2024 4:35:02 PM
To: dev
Subject: Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Thanks!

The best I can recall, is that there some java dep that wasn't being pulled properly somehow. But it's very much a needle in a haystack situation. The logs are thousands of lines of gradle spew. And i was just following the Beam Release guide for updating the playground.

As a non-Java person I'm not positioned well to get further in debugging it, just in reporting it.

See y'all in 2 weeks!

On Fri, Mar 8, 2024, 10:19 AM Joey Tran <jo...@schrodinger.com>> wrote:
Ah, there is hope still then. Will follow up in a few weeks then. Happy vacationing! ☀️

On Fri, Mar 8, 2024 at 10:16 AM Robert Burke <ro...@frantil.com>> wrote:
The playground, last i checked, was stuck on 2.52.0 because it's been unable to be built and staged. That issue would need to be resolved first

I unfortunately didn't get around to filing it in github or searching for there, as I got distracted by urgent matters and it slipped my mind. Very sorry about that.

(Currently I'm away from laptop on vacation, so I can't do that until Monday the 18th at earliest.)

On Fri, Mar 8, 2024, 10:05 AM Joey Tran <jo...@schrodinger.com>> wrote:
Looks like there's still INFO prints unfortunately
[image.png]

It's possibly my fix just didn't work :'(

On Fri, Mar 8, 2024 at 10:02 AM Vlado Djerek <vl...@akvelon.com>> wrote:

Hi Joey,

We just deployed the 2.54 changes on playground. Can you now check?

Thanks,
Vlado

________________________________
From: Joey Tran <jo...@schrodinger.com>>
Sent: Friday, March 8, 2024 2:45:32 PM
To: Andrey Devyatkin
Subject: [EXTERNAL] Re: Hiding logging for beam playground examples

Ah thanks Andrey! I had assumed that since my change was in a release that went out a few weeks ago that Playground would've been redeployed by now. I'll check playground again in an hour or two to see if my changes did anything :)

On Fri, Mar 8, 2024 at 8:28 AM Andrey Devyatkin <an...@akvelon.com>> wrote:
Hi Joey,

Thanks for reaching out! I see that your changes haven't been deploed yet, so I've triggered the corresponding job and Playground will be updated soon.

Thanks,
Andrey

From: Joey Tran <jo...@schrodinger.com>>
Reply to: "dev@beam.apache.org<ma...@beam.apache.org>" <de...@beam.apache.org>>
Date: Thursday, 7 March 2024 at 22:33
To: Valentyn Tymofieiev <va...@google.com>>
Subject: [EXTERNAL] Re: Hiding logging for beam playground examples

I ended up making this change[1] but it looks like Playground is still outputting all the log statements. Could someone help me with this? I suppose it's best to try and actually test this, but I had a hard time getting the environment set up.

[1]https://github.com/apache/beam/pull/29948


















































































"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt

[1] https://github.com/apache/beam/pull/29948#event-11504996254

On Thu, Nov 16, 2023 at 7:42 AM Joey Tran <jo...@schrodinger.com>> wrote:
Good idea though it ended up being a shallow trace
```
  File "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py", line 57, in <module>
    assert False
```
I think I found where the log level is set anyways
```
https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39
```

When I have some time, I'll try doing a local deployment of playground and modifying those log levels

On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev <va...@google.com>> wrote:
I am also not familiar with Playground. I suspect you could try to make it crash and maybe find a stacktrace? Setting logging could like like so: https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110

On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <jo...@schrodinger.com>> wrote:
The motivating example does not use LogElements, just Map(print)
https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function

Some examples of the extraneous logging:
```
2023-09-08 22:46:37,334 [INFO] ==================== <function populate_data_channel_coders at 0x7ff2665e1a20> ====================
2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600
2023-09-08 22:46:37,338 [INFO] Created Worker handler <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler object at 0x7ff2664c9870> for environment ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
```

The example code itself doesn't set the log level in some playground code. Does anyone have a pointer to where? I'm not familiar

On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <de...@beam.apache.org>> wrote:
Are the examples using LogElements? https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271

Note that LogElements by default prints to stdout, but can be configured to use a different logger. We could also change the default.

On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <de...@beam.apache.org>> wrote:
+1 to at least setting the log level to higher than info. Some runner
logging (e.g. job started/done) may be useful.

On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>> wrote:
>
> Hi all,
>
> I just had a workshop to demo beam for people at my company and there was a bit of confusion about whether the beam python playground examples were even working and it turned out they just got confused by all the runner logging that is output.
>
> Is this worth keeping? It seems like it'd be a common source of confusion for new users
>
> Cheers,
> Joey

Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Posted by Robert Burke <ro...@frantil.com>.
Thanks!

The best I can recall, is that there some java dep that wasn't being pulled
properly somehow. But it's very much a needle in a haystack situation. The
logs are thousands of lines of gradle spew. And i was just following the
Beam Release guide for updating the playground.

As a non-Java person I'm not positioned well to get further in debugging
it, just in reporting it.

See y'all in 2 weeks!

On Fri, Mar 8, 2024, 10:19 AM Joey Tran <jo...@schrodinger.com> wrote:

> Ah, there is hope still then. Will follow up in a few weeks then. Happy
> vacationing! ☀️
>
> On Fri, Mar 8, 2024 at 10:16 AM Robert Burke <ro...@frantil.com> wrote:
>
>> The playground, last i checked, was stuck on 2.52.0 because it's been
>> unable to be built and staged. That issue would need to be resolved first
>>
>> I unfortunately didn't get around to filing it in github or searching for
>> there, as I got distracted by urgent matters and it slipped my mind. Very
>> sorry about that.
>>
>> (Currently I'm away from laptop on vacation, so I can't do that until
>> Monday the 18th at earliest.)
>>
>> On Fri, Mar 8, 2024, 10:05 AM Joey Tran <jo...@schrodinger.com>
>> wrote:
>>
>>> Looks like there's still INFO prints unfortunately
>>> [image: image.png]
>>>
>>> It's possibly my fix just didn't work :'(
>>>
>>> On Fri, Mar 8, 2024 at 10:02 AM Vlado Djerek <vl...@akvelon.com>
>>> wrote:
>>>
>>>> Hi Joey,
>>>>
>>>> We just deployed the 2.54 changes on playground. Can you now check?
>>>>
>>>> Thanks,
>>>> Vlado
>>>> ------------------------------
>>>> *From:* Joey Tran <jo...@schrodinger.com>
>>>> *Sent:* Friday, March 8, 2024 2:45:32 PM
>>>> *To:* Andrey Devyatkin
>>>> *Subject:* [EXTERNAL] Re: Hiding logging for beam playground examples
>>>>
>>>> Ah thanks Andrey! I had assumed that since my change was in a release
>>>> that went out a few weeks ago that Playground would've been redeployed by
>>>> now. I'll check playground again in an hour or two to see if my changes did
>>>> anything :)
>>>>
>>>> On Fri, Mar 8, 2024 at 8:28 AM Andrey Devyatkin <
>>>> andrey.devyatkin@akvelon.com> wrote:
>>>>
>>>>> Hi Joey,
>>>>>
>>>>> Thanks for reaching out! I see that your changes haven't been deploed
>>>>> yet, so I've triggered the corresponding job and Playground will be updated
>>>>> soon.
>>>>>
>>>>> Thanks,
>>>>> Andrey
>>>>>
>>>>>
>>>>>
>>>>> *From: *Joey Tran <jo...@schrodinger.com>
>>>>> *Reply to: *"dev@beam.apache.org" <de...@beam.apache.org>
>>>>> *Date: *Thursday, 7 March 2024 at 22:33
>>>>> *To: *Valentyn Tymofieiev <va...@google.com>
>>>>> *Subject: *[EXTERNAL] Re: Hiding logging for beam playground examples
>>>>>
>>>>>
>>>>>
>>>>> I ended up making this change[1] but it looks like Playground is still
>>>>> outputting all the log statements. Could someone help me with this? I
>>>>> suppose it's best to try and actually test this, but I had a hard time
>>>>> getting the environment set up.
>>>>>
>>>>> [1]https://github.com/apache/beam/pull/29948
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt
>>>>>
>>>>> [1] https://github.com/apache/beam/pull/29948#event-11504996254
>>>>>
>>>>>
>>>>>
>>>>> On Thu, Nov 16, 2023 at 7:42 AM Joey Tran <jo...@schrodinger.com>
>>>>> wrote:
>>>>>
>>>>> Good idea though it ended up being a shallow trace
>>>>> ```
>>>>>   File
>>>>> "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py",
>>>>> line 57, in <module>
>>>>>     assert False
>>>>> ```
>>>>> I think I found where the log level is set anyways
>>>>> ```
>>>>>
>>>>> https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39
>>>>> ```
>>>>>
>>>>> When I have some time, I'll try doing a local deployment of playground
>>>>> and modifying those log levels
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev <
>>>>> valentyn@google.com> wrote:
>>>>>
>>>>> I am also not familiar with Playground. I suspect you could try to
>>>>> make it crash and maybe find a stacktrace? Setting logging could like like
>>>>> so:
>>>>> https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <jo...@schrodinger.com>
>>>>> wrote:
>>>>>
>>>>> The motivating example does not use LogElements, just Map(print)
>>>>>
>>>>> https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function
>>>>>
>>>>> Some examples of the extraneous logging:
>>>>> ```
>>>>> 2023-09-08 22:46:37,334 [INFO] ==================== <function
>>>>> populate_data_channel_coders at 0x7ff2665e1a20> ====================
>>>>> 2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600
>>>>> 2023-09-08 22:46:37,338 [INFO] Created Worker handler
>>>>> <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
>>>>> object at 0x7ff2664c9870> for environment
>>>>> ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
>>>>> ```
>>>>>
>>>>> The example code itself doesn't set the log level in some playground
>>>>> code. Does anyone have a pointer to where? I'm not familiar
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <
>>>>> dev@beam.apache.org> wrote:
>>>>>
>>>>> Are the examples using LogElements?
>>>>> https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271
>>>>>
>>>>>
>>>>>
>>>>> Note that LogElements by default prints to stdout, but can be
>>>>> configured to use a different logger. We could also change the default.
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <
>>>>> dev@beam.apache.org> wrote:
>>>>>
>>>>> +1 to at least setting the log level to higher than info. Some runner
>>>>> logging (e.g. job started/done) may be useful.
>>>>>
>>>>> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>
>>>>> wrote:
>>>>> >
>>>>> > Hi all,
>>>>> >
>>>>> > I just had a workshop to demo beam for people at my company and
>>>>> there was a bit of confusion about whether the beam python playground
>>>>> examples were even working and it turned out they just got confused by all
>>>>> the runner logging that is output.
>>>>> >
>>>>> > Is this worth keeping? It seems like it'd be a common source of
>>>>> confusion for new users
>>>>> >
>>>>> > Cheers,
>>>>> > Joey
>>>>>
>>>>>

Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Posted by Joey Tran <jo...@schrodinger.com>.
Ah, there is hope still then. Will follow up in a few weeks then. Happy
vacationing! ☀️

On Fri, Mar 8, 2024 at 10:16 AM Robert Burke <ro...@frantil.com> wrote:

> The playground, last i checked, was stuck on 2.52.0 because it's been
> unable to be built and staged. That issue would need to be resolved first
>
> I unfortunately didn't get around to filing it in github or searching for
> there, as I got distracted by urgent matters and it slipped my mind. Very
> sorry about that.
>
> (Currently I'm away from laptop on vacation, so I can't do that until
> Monday the 18th at earliest.)
>
> On Fri, Mar 8, 2024, 10:05 AM Joey Tran <jo...@schrodinger.com> wrote:
>
>> Looks like there's still INFO prints unfortunately
>> [image: image.png]
>>
>> It's possibly my fix just didn't work :'(
>>
>> On Fri, Mar 8, 2024 at 10:02 AM Vlado Djerek <vl...@akvelon.com>
>> wrote:
>>
>>> Hi Joey,
>>>
>>> We just deployed the 2.54 changes on playground. Can you now check?
>>>
>>> Thanks,
>>> Vlado
>>> ------------------------------
>>> *From:* Joey Tran <jo...@schrodinger.com>
>>> *Sent:* Friday, March 8, 2024 2:45:32 PM
>>> *To:* Andrey Devyatkin
>>> *Subject:* [EXTERNAL] Re: Hiding logging for beam playground examples
>>>
>>> Ah thanks Andrey! I had assumed that since my change was in a release
>>> that went out a few weeks ago that Playground would've been redeployed by
>>> now. I'll check playground again in an hour or two to see if my changes did
>>> anything :)
>>>
>>> On Fri, Mar 8, 2024 at 8:28 AM Andrey Devyatkin <
>>> andrey.devyatkin@akvelon.com> wrote:
>>>
>>>> Hi Joey,
>>>>
>>>> Thanks for reaching out! I see that your changes haven't been deploed
>>>> yet, so I've triggered the corresponding job and Playground will be updated
>>>> soon.
>>>>
>>>> Thanks,
>>>> Andrey
>>>>
>>>>
>>>>
>>>> *From: *Joey Tran <jo...@schrodinger.com>
>>>> *Reply to: *"dev@beam.apache.org" <de...@beam.apache.org>
>>>> *Date: *Thursday, 7 March 2024 at 22:33
>>>> *To: *Valentyn Tymofieiev <va...@google.com>
>>>> *Subject: *[EXTERNAL] Re: Hiding logging for beam playground examples
>>>>
>>>>
>>>>
>>>> I ended up making this change[1] but it looks like Playground is still
>>>> outputting all the log statements. Could someone help me with this? I
>>>> suppose it's best to try and actually test this, but I had a hard time
>>>> getting the environment set up.
>>>>
>>>> [1]https://github.com/apache/beam/pull/29948
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt
>>>>
>>>> [1] https://github.com/apache/beam/pull/29948#event-11504996254
>>>>
>>>>
>>>>
>>>> On Thu, Nov 16, 2023 at 7:42 AM Joey Tran <jo...@schrodinger.com>
>>>> wrote:
>>>>
>>>> Good idea though it ended up being a shallow trace
>>>> ```
>>>>   File
>>>> "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py",
>>>> line 57, in <module>
>>>>     assert False
>>>> ```
>>>> I think I found where the log level is set anyways
>>>> ```
>>>>
>>>> https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39
>>>> ```
>>>>
>>>> When I have some time, I'll try doing a local deployment of playground
>>>> and modifying those log levels
>>>>
>>>>
>>>>
>>>> On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev <
>>>> valentyn@google.com> wrote:
>>>>
>>>> I am also not familiar with Playground. I suspect you could try to make
>>>> it crash and maybe find a stacktrace? Setting logging could like like so:
>>>> https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110
>>>>
>>>>
>>>>
>>>> On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <jo...@schrodinger.com>
>>>> wrote:
>>>>
>>>> The motivating example does not use LogElements, just Map(print)
>>>>
>>>> https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function
>>>>
>>>> Some examples of the extraneous logging:
>>>> ```
>>>> 2023-09-08 22:46:37,334 [INFO] ==================== <function
>>>> populate_data_channel_coders at 0x7ff2665e1a20> ====================
>>>> 2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600
>>>> 2023-09-08 22:46:37,338 [INFO] Created Worker handler
>>>> <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
>>>> object at 0x7ff2664c9870> for environment
>>>> ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
>>>> ```
>>>>
>>>> The example code itself doesn't set the log level in some playground
>>>> code. Does anyone have a pointer to where? I'm not familiar
>>>>
>>>>
>>>>
>>>> On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>> Are the examples using LogElements?
>>>> https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271
>>>>
>>>>
>>>>
>>>> Note that LogElements by default prints to stdout, but can be
>>>> configured to use a different logger. We could also change the default.
>>>>
>>>>
>>>>
>>>> On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>> +1 to at least setting the log level to higher than info. Some runner
>>>> logging (e.g. job started/done) may be useful.
>>>>
>>>> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>
>>>> wrote:
>>>> >
>>>> > Hi all,
>>>> >
>>>> > I just had a workshop to demo beam for people at my company and there
>>>> was a bit of confusion about whether the beam python playground examples
>>>> were even working and it turned out they just got confused by all the
>>>> runner logging that is output.
>>>> >
>>>> > Is this worth keeping? It seems like it'd be a common source of
>>>> confusion for new users
>>>> >
>>>> > Cheers,
>>>> > Joey
>>>>
>>>>

Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Posted by Robert Burke <ro...@frantil.com>.
The playground, last i checked, was stuck on 2.52.0 because it's been
unable to be built and staged. That issue would need to be resolved first

I unfortunately didn't get around to filing it in github or searching for
there, as I got distracted by urgent matters and it slipped my mind. Very
sorry about that.

(Currently I'm away from laptop on vacation, so I can't do that until
Monday the 18th at earliest.)

On Fri, Mar 8, 2024, 10:05 AM Joey Tran <jo...@schrodinger.com> wrote:

> Looks like there's still INFO prints unfortunately
> [image: image.png]
>
> It's possibly my fix just didn't work :'(
>
> On Fri, Mar 8, 2024 at 10:02 AM Vlado Djerek <vl...@akvelon.com>
> wrote:
>
>> Hi Joey,
>>
>> We just deployed the 2.54 changes on playground. Can you now check?
>>
>> Thanks,
>> Vlado
>> ------------------------------
>> *From:* Joey Tran <jo...@schrodinger.com>
>> *Sent:* Friday, March 8, 2024 2:45:32 PM
>> *To:* Andrey Devyatkin
>> *Subject:* [EXTERNAL] Re: Hiding logging for beam playground examples
>>
>> Ah thanks Andrey! I had assumed that since my change was in a release
>> that went out a few weeks ago that Playground would've been redeployed by
>> now. I'll check playground again in an hour or two to see if my changes did
>> anything :)
>>
>> On Fri, Mar 8, 2024 at 8:28 AM Andrey Devyatkin <
>> andrey.devyatkin@akvelon.com> wrote:
>>
>>> Hi Joey,
>>>
>>> Thanks for reaching out! I see that your changes haven't been deploed
>>> yet, so I've triggered the corresponding job and Playground will be updated
>>> soon.
>>>
>>> Thanks,
>>> Andrey
>>>
>>>
>>>
>>> *From: *Joey Tran <jo...@schrodinger.com>
>>> *Reply to: *"dev@beam.apache.org" <de...@beam.apache.org>
>>> *Date: *Thursday, 7 March 2024 at 22:33
>>> *To: *Valentyn Tymofieiev <va...@google.com>
>>> *Subject: *[EXTERNAL] Re: Hiding logging for beam playground examples
>>>
>>>
>>>
>>> I ended up making this change[1] but it looks like Playground is still
>>> outputting all the log statements. Could someone help me with this? I
>>> suppose it's best to try and actually test this, but I had a hard time
>>> getting the environment set up.
>>>
>>> [1]https://github.com/apache/beam/pull/29948
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt
>>>
>>> [1] https://github.com/apache/beam/pull/29948#event-11504996254
>>>
>>>
>>>
>>> On Thu, Nov 16, 2023 at 7:42 AM Joey Tran <jo...@schrodinger.com>
>>> wrote:
>>>
>>> Good idea though it ended up being a shallow trace
>>> ```
>>>   File
>>> "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py",
>>> line 57, in <module>
>>>     assert False
>>> ```
>>> I think I found where the log level is set anyways
>>> ```
>>>
>>> https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39
>>> ```
>>>
>>> When I have some time, I'll try doing a local deployment of playground
>>> and modifying those log levels
>>>
>>>
>>>
>>> On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev <
>>> valentyn@google.com> wrote:
>>>
>>> I am also not familiar with Playground. I suspect you could try to make
>>> it crash and maybe find a stacktrace? Setting logging could like like so:
>>> https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110
>>>
>>>
>>>
>>> On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <jo...@schrodinger.com>
>>> wrote:
>>>
>>> The motivating example does not use LogElements, just Map(print)
>>>
>>> https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function
>>>
>>> Some examples of the extraneous logging:
>>> ```
>>> 2023-09-08 22:46:37,334 [INFO] ==================== <function
>>> populate_data_channel_coders at 0x7ff2665e1a20> ====================
>>> 2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600
>>> 2023-09-08 22:46:37,338 [INFO] Created Worker handler
>>> <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
>>> object at 0x7ff2664c9870> for environment
>>> ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
>>> ```
>>>
>>> The example code itself doesn't set the log level in some playground
>>> code. Does anyone have a pointer to where? I'm not familiar
>>>
>>>
>>>
>>> On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>> Are the examples using LogElements?
>>> https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271
>>>
>>>
>>>
>>> Note that LogElements by default prints to stdout, but can be configured
>>> to use a different logger. We could also change the default.
>>>
>>>
>>>
>>> On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>> +1 to at least setting the log level to higher than info. Some runner
>>> logging (e.g. job started/done) may be useful.
>>>
>>> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>
>>> wrote:
>>> >
>>> > Hi all,
>>> >
>>> > I just had a workshop to demo beam for people at my company and there
>>> was a bit of confusion about whether the beam python playground examples
>>> were even working and it turned out they just got confused by all the
>>> runner logging that is output.
>>> >
>>> > Is this worth keeping? It seems like it'd be a common source of
>>> confusion for new users
>>> >
>>> > Cheers,
>>> > Joey
>>>
>>>

Re: [EXTERNAL] Re: Hiding logging for beam playground examples

Posted by Joey Tran <jo...@schrodinger.com>.
Looks like there's still INFO prints unfortunately
[image: image.png]

It's possibly my fix just didn't work :'(

On Fri, Mar 8, 2024 at 10:02 AM Vlado Djerek <vl...@akvelon.com>
wrote:

> Hi Joey,
>
> We just deployed the 2.54 changes on playground. Can you now check?
>
> Thanks,
> Vlado
> ------------------------------
> *From:* Joey Tran <jo...@schrodinger.com>
> *Sent:* Friday, March 8, 2024 2:45:32 PM
> *To:* Andrey Devyatkin
> *Subject:* [EXTERNAL] Re: Hiding logging for beam playground examples
>
> Ah thanks Andrey! I had assumed that since my change was in a release that
> went out a few weeks ago that Playground would've been redeployed by now.
> I'll check playground again in an hour or two to see if my changes did
> anything :)
>
> On Fri, Mar 8, 2024 at 8:28 AM Andrey Devyatkin <
> andrey.devyatkin@akvelon.com> wrote:
>
>> Hi Joey,
>>
>> Thanks for reaching out! I see that your changes haven't been deploed
>> yet, so I've triggered the corresponding job and Playground will be updated
>> soon.
>>
>> Thanks,
>> Andrey
>>
>>
>>
>> *From: *Joey Tran <jo...@schrodinger.com>
>> *Reply to: *"dev@beam.apache.org" <de...@beam.apache.org>
>> *Date: *Thursday, 7 March 2024 at 22:33
>> *To: *Valentyn Tymofieiev <va...@google.com>
>> *Subject: *[EXTERNAL] Re: Hiding logging for beam playground examples
>>
>>
>>
>> I ended up making this change[1] but it looks like Playground is still
>> outputting all the log statements. Could someone help me with this? I
>> suppose it's best to try and actually test this, but I had a hard time
>> getting the environment set up.
>>
>> [1]https://github.com/apache/beam/pull/29948
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt
>>
>> [1] https://github.com/apache/beam/pull/29948#event-11504996254
>>
>>
>>
>> On Thu, Nov 16, 2023 at 7:42 AM Joey Tran <jo...@schrodinger.com>
>> wrote:
>>
>> Good idea though it ended up being a shallow trace
>> ```
>>   File
>> "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py",
>> line 57, in <module>
>>     assert False
>> ```
>> I think I found where the log level is set anyways
>> ```
>>
>> https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39
>> ```
>>
>> When I have some time, I'll try doing a local deployment of playground
>> and modifying those log levels
>>
>>
>>
>> On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev <va...@google.com>
>> wrote:
>>
>> I am also not familiar with Playground. I suspect you could try to make
>> it crash and maybe find a stacktrace? Setting logging could like like so:
>> https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110
>>
>>
>>
>> On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <jo...@schrodinger.com>
>> wrote:
>>
>> The motivating example does not use LogElements, just Map(print)
>>
>> https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function
>>
>> Some examples of the extraneous logging:
>> ```
>> 2023-09-08 22:46:37,334 [INFO] ==================== <function
>> populate_data_channel_coders at 0x7ff2665e1a20> ====================
>> 2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600
>> 2023-09-08 22:46:37,338 [INFO] Created Worker handler
>> <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
>> object at 0x7ff2664c9870> for environment
>> ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
>> ```
>>
>> The example code itself doesn't set the log level in some playground
>> code. Does anyone have a pointer to where? I'm not familiar
>>
>>
>>
>> On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <
>> dev@beam.apache.org> wrote:
>>
>> Are the examples using LogElements?
>> https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271
>>
>>
>>
>> Note that LogElements by default prints to stdout, but can be configured
>> to use a different logger. We could also change the default.
>>
>>
>>
>> On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <
>> dev@beam.apache.org> wrote:
>>
>> +1 to at least setting the log level to higher than info. Some runner
>> logging (e.g. job started/done) may be useful.
>>
>> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>
>> wrote:
>> >
>> > Hi all,
>> >
>> > I just had a workshop to demo beam for people at my company and there
>> was a bit of confusion about whether the beam python playground examples
>> were even working and it turned out they just got confused by all the
>> runner logging that is output.
>> >
>> > Is this worth keeping? It seems like it'd be a common source of
>> confusion for new users
>> >
>> > Cheers,
>> > Joey
>>
>>