You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Joey Tran <jo...@schrodinger.com> on 2023/11/14 17:33:57 UTC
Hiding logging for beam playground examples
Hi all,
I just had a workshop to demo beam for people at my company and there was a
bit of confusion about whether the beam python playground examples were
even working and it turned out they just got confused by all the runner
logging that is output.
Is this worth keeping? It seems like it'd be a common source of confusion
for new users
Cheers,
Joey
Re: Hiding logging for beam playground examples
Posted by Joey Tran <jo...@schrodinger.com>.
I ended up making this change[1] but it looks like Playground is still
outputting all the log statements. Could someone help me with this? I
suppose it's best to try and actually test this, but I had a hard time
getting the environment set up.
[1]https://github.com/apache/beam/pull/29948
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt
[1] https://github.com/apache/beam/pull/29948#event-11504996254
On Thu, Nov 16, 2023 at 7:42 AM Joey Tran <jo...@schrodinger.com> wrote:
> Good idea though it ended up being a shallow trace
> ```
> File
> "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py",
> line 57, in <module>
> assert False
> ```
> I think I found where the log level is set anyways
> ```
>
> https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39
> ```
>
> When I have some time, I'll try doing a local deployment of playground and
> modifying those log levels
>
> On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev <va...@google.com>
> wrote:
>
>> I am also not familiar with Playground. I suspect you could try to make
>> it crash and maybe find a stacktrace? Setting logging could like like so:
>> https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110
>>
>> On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <jo...@schrodinger.com>
>> wrote:
>>
>>> The motivating example does not use LogElements, just Map(print)
>>>
>>> https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function
>>>
>>> Some examples of the extraneous logging:
>>> ```
>>> 2023-09-08 22:46:37,334 [INFO] ==================== <function
>>> populate_data_channel_coders at 0x7ff2665e1a20> ====================
>>> 2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600
>>> 2023-09-08 22:46:37,338 [INFO] Created Worker handler
>>> <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
>>> object at 0x7ff2664c9870> for environment
>>> ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
>>> ```
>>>
>>> The example code itself doesn't set the log level in some playground
>>> code. Does anyone have a pointer to where? I'm not familiar
>>>
>>> On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> Are the examples using LogElements?
>>>> https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271
>>>>
>>>> Note that LogElements by default prints to stdout, but can be
>>>> configured to use a different logger. We could also change the default.
>>>>
>>>> On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <
>>>> dev@beam.apache.org> wrote:
>>>>
>>>>> +1 to at least setting the log level to higher than info. Some runner
>>>>> logging (e.g. job started/done) may be useful.
>>>>>
>>>>> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>
>>>>> wrote:
>>>>> >
>>>>> > Hi all,
>>>>> >
>>>>> > I just had a workshop to demo beam for people at my company and
>>>>> there was a bit of confusion about whether the beam python playground
>>>>> examples were even working and it turned out they just got confused by all
>>>>> the runner logging that is output.
>>>>> >
>>>>> > Is this worth keeping? It seems like it'd be a common source of
>>>>> confusion for new users
>>>>> >
>>>>> > Cheers,
>>>>> > Joey
>>>>>
>>>>
Re: Hiding logging for beam playground examples
Posted by Joey Tran <jo...@schrodinger.com>.
Good idea though it ended up being a shallow trace
```
File
"/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py",
line 57, in <module>
assert False
```
I think I found where the log level is set anyways
```
https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39
```
When I have some time, I'll try doing a local deployment of playground and
modifying those log levels
On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev <va...@google.com>
wrote:
> I am also not familiar with Playground. I suspect you could try to make it
> crash and maybe find a stacktrace? Setting logging could like like so:
> https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110
>
> On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <jo...@schrodinger.com>
> wrote:
>
>> The motivating example does not use LogElements, just Map(print)
>>
>> https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function
>>
>> Some examples of the extraneous logging:
>> ```
>> 2023-09-08 22:46:37,334 [INFO] ==================== <function
>> populate_data_channel_coders at 0x7ff2665e1a20> ====================
>> 2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600
>> 2023-09-08 22:46:37,338 [INFO] Created Worker handler
>> <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
>> object at 0x7ff2664c9870> for environment
>> ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
>> ```
>>
>> The example code itself doesn't set the log level in some playground
>> code. Does anyone have a pointer to where? I'm not familiar
>>
>> On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <
>> dev@beam.apache.org> wrote:
>>
>>> Are the examples using LogElements?
>>> https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271
>>>
>>> Note that LogElements by default prints to stdout, but can be configured
>>> to use a different logger. We could also change the default.
>>>
>>> On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <
>>> dev@beam.apache.org> wrote:
>>>
>>>> +1 to at least setting the log level to higher than info. Some runner
>>>> logging (e.g. job started/done) may be useful.
>>>>
>>>> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>
>>>> wrote:
>>>> >
>>>> > Hi all,
>>>> >
>>>> > I just had a workshop to demo beam for people at my company and there
>>>> was a bit of confusion about whether the beam python playground examples
>>>> were even working and it turned out they just got confused by all the
>>>> runner logging that is output.
>>>> >
>>>> > Is this worth keeping? It seems like it'd be a common source of
>>>> confusion for new users
>>>> >
>>>> > Cheers,
>>>> > Joey
>>>>
>>>
Re: Hiding logging for beam playground examples
Posted by Valentyn Tymofieiev via dev <de...@beam.apache.org>.
I am also not familiar with Playground. I suspect you could try to make it
crash and maybe find a stacktrace? Setting logging could like like so:
https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110
On Wed, Nov 15, 2023 at 12:06 PM Joey Tran <jo...@schrodinger.com>
wrote:
> The motivating example does not use LogElements, just Map(print)
>
> https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function
>
> Some examples of the extraneous logging:
> ```
> 2023-09-08 22:46:37,334 [INFO] ==================== <function
> populate_data_channel_coders at 0x7ff2665e1a20> ====================
> 2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600
> 2023-09-08 22:46:37,338 [INFO] Created Worker handler
> <apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
> object at 0x7ff2664c9870> for environment
> ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
> ```
>
> The example code itself doesn't set the log level in some playground code.
> Does anyone have a pointer to where? I'm not familiar
>
> On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <
> dev@beam.apache.org> wrote:
>
>> Are the examples using LogElements?
>> https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271
>>
>> Note that LogElements by default prints to stdout, but can be configured
>> to use a different logger. We could also change the default.
>>
>> On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <
>> dev@beam.apache.org> wrote:
>>
>>> +1 to at least setting the log level to higher than info. Some runner
>>> logging (e.g. job started/done) may be useful.
>>>
>>> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>
>>> wrote:
>>> >
>>> > Hi all,
>>> >
>>> > I just had a workshop to demo beam for people at my company and there
>>> was a bit of confusion about whether the beam python playground examples
>>> were even working and it turned out they just got confused by all the
>>> runner logging that is output.
>>> >
>>> > Is this worth keeping? It seems like it'd be a common source of
>>> confusion for new users
>>> >
>>> > Cheers,
>>> > Joey
>>>
>>
Re: Hiding logging for beam playground examples
Posted by Joey Tran <jo...@schrodinger.com>.
The motivating example does not use LogElements, just Map(print)
https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function
Some examples of the extraneous logging:
```
2023-09-08 22:46:37,334 [INFO] ==================== <function
populate_data_channel_coders at 0x7ff2665e1a20> ====================
2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600
2023-09-08 22:46:37,338 [INFO] Created Worker handler
<apache_beam.runners.portability.fn_api_runner.worker_handlers.EmbeddedWorkerHandler
object at 0x7ff2664c9870> for environment
ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'')
```
The example code itself doesn't set the log level in some playground code.
Does anyone have a pointer to where? I'm not familiar
On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev <
dev@beam.apache.org> wrote:
> Are the examples using LogElements?
> https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271
>
> Note that LogElements by default prints to stdout, but can be configured
> to use a different logger. We could also change the default.
>
> On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <
> dev@beam.apache.org> wrote:
>
>> +1 to at least setting the log level to higher than info. Some runner
>> logging (e.g. job started/done) may be useful.
>>
>> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>
>> wrote:
>> >
>> > Hi all,
>> >
>> > I just had a workshop to demo beam for people at my company and there
>> was a bit of confusion about whether the beam python playground examples
>> were even working and it turned out they just got confused by all the
>> runner logging that is output.
>> >
>> > Is this worth keeping? It seems like it'd be a common source of
>> confusion for new users
>> >
>> > Cheers,
>> > Joey
>>
>
Re: Hiding logging for beam playground examples
Posted by Valentyn Tymofieiev via dev <de...@beam.apache.org>.
Are the examples using LogElements?
https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271
Note that LogElements by default prints to stdout, but can be configured to
use a different logger. We could also change the default.
On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev <de...@beam.apache.org>
wrote:
> +1 to at least setting the log level to higher than info. Some runner
> logging (e.g. job started/done) may be useful.
>
> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com>
> wrote:
> >
> > Hi all,
> >
> > I just had a workshop to demo beam for people at my company and there
> was a bit of confusion about whether the beam python playground examples
> were even working and it turned out they just got confused by all the
> runner logging that is output.
> >
> > Is this worth keeping? It seems like it'd be a common source of
> confusion for new users
> >
> > Cheers,
> > Joey
>
Re: Hiding logging for beam playground examples
Posted by Robert Bradshaw via dev <de...@beam.apache.org>.
+1 to at least setting the log level to higher than info. Some runner
logging (e.g. job started/done) may be useful.
On Tue, Nov 14, 2023 at 9:37 AM Joey Tran <jo...@schrodinger.com> wrote:
>
> Hi all,
>
> I just had a workshop to demo beam for people at my company and there was a bit of confusion about whether the beam python playground examples were even working and it turned out they just got confused by all the runner logging that is output.
>
> Is this worth keeping? It seems like it'd be a common source of confusion for new users
>
> Cheers,
> Joey