Python Precommit significantly flaky

[Pablo Estrada <pa...@google.com>]

Hi all,
the Precommit tests have been flaky for a couple of days due to
ExternalTransform tests. See
https://builds.apache.org/job/beam_PreCommit_Python_Cron/ - and an example
failure:

https://builds.apache.org/job/beam_PreCommit_Python_Cron/2723/

Does somebody know about this?
Best
-P.

Re: Python Precommit significantly flaky

[Brian Hulette <bh...@google.com>]

Thanks to Kyle we captured some additional logging for
https://issues.apache.org/jira/browse/BEAM-9975. I spent a little time
looking at it and found two different issues (see details in the comments):
https://issues.apache.org/jira/browse/BEAM-10006 - PipelineOptions can pick
up definitions from unrelated tests
https://issues.apache.org/jira/browse/BEAM-10007 - PortableRunner doesn't
handle ValueProvider instances when converting pipeline options

10006 needs some more investigation. I _think_ the fix for 10007 is pretty
simple, but I'm not familiar with ValueProvider so I could use some
guidance.

Can anyone help out with either of these issues?


On Wed, May 13, 2020 at 12:26 PM Valentyn Tymofieiev <va...@google.com>
wrote:

> +1. PortableRunnerTest is very flaky - I looked at last 10 precommit-on-PR
> runs and 7 of them had errors in PortableRunnerTests.
>
>

Re: Python Precommit significantly flaky

[Valentyn Tymofieiev <va...@google.com>]

+1. PortableRunnerTest is very flaky - I looked at last 10 precommit-on-PR
runs and 7 of them had errors in PortableRunnerTests.

Re: Python Precommit significantly flaky

[Brian Hulette <bh...@google.com>]

I think the previously mentioned issues are resolved now, but the python
precommit still has some flakes. I filed:
https://issues.apache.org/jira/browse/BEAM-9975 - PortableRunnerTest flake
"ParseError: Unexpected type for Value message."

An example of this is
https://builds.apache.org/job/beam_PreCommit_Python_Cron/2743/

It looks the same failure mode as
https://issues.apache.org/jira/browse/BEAM-9907, but it occurs for
PortableRunnerTest (and sub-classes). In BEAM-9907 it was because a
PipelineOption instance was unintentionally constructed such that it read
arguments from sys.argv, and in a test it parsed arguments intended for the
test framework, not for Beam. I don't _think_ that's what's happening here,
but I'm not sure.

Could someone with more context take a look?

Brian



On Thu, May 7, 2020 at 11:10 AM Ahmet Altay <al...@google.com> wrote:

> I am aware of the following two that contributes to precommit flakiness.
> Someone is working on both.
>
> https://issues.apache.org/jira/browse/BEAM-9907 - ExternalTransform
> https://issues.apache.org/jira/browse/BEAM-9767 - test_streaming_wordcount
> flaky
>
>
> On Thu, May 7, 2020 at 10:54 AM Brian Hulette <bh...@google.com> wrote:
>
>> It looks like this is likely due to my PR for pipeline options in
>> external transforms: https://github.com/apache/beam/pull/11574
>>
>> I'll confirm and roll back if it is.
>>
>> On Thu, May 7, 2020 at 10:51 AM Pablo Estrada <pa...@google.com> wrote:
>>
>>> Hi all,
>>> the Precommit tests have been flaky for a couple of days due to
>>> ExternalTransform tests. See
>>> https://builds.apache.org/job/beam_PreCommit_Python_Cron/ - and an
>>> example failure:
>>>
>>> https://builds.apache.org/job/beam_PreCommit_Python_Cron/2723/
>>>
>>> Does somebody know about this?
>>> Best
>>> -P.
>>>
>>

Re: Python Precommit significantly flaky

[Ahmet Altay <al...@google.com>]

I am aware of the following two that contributes to precommit flakiness.
Someone is working on both.

https://issues.apache.org/jira/browse/BEAM-9907 - ExternalTransform
https://issues.apache.org/jira/browse/BEAM-9767 - test_streaming_wordcount
flaky


On Thu, May 7, 2020 at 10:54 AM Brian Hulette <bh...@google.com> wrote:

> It looks like this is likely due to my PR for pipeline options in external
> transforms: https://github.com/apache/beam/pull/11574
>
> I'll confirm and roll back if it is.
>
> On Thu, May 7, 2020 at 10:51 AM Pablo Estrada <pa...@google.com> wrote:
>
>> Hi all,
>> the Precommit tests have been flaky for a couple of days due to
>> ExternalTransform tests. See
>> https://builds.apache.org/job/beam_PreCommit_Python_Cron/ - and an
>> example failure:
>>
>> https://builds.apache.org/job/beam_PreCommit_Python_Cron/2723/
>>
>> Does somebody know about this?
>> Best
>> -P.
>>
>

Re: Python Precommit significantly flaky

[Brian Hulette <bh...@google.com>]

It looks like this is likely due to my PR for pipeline options in external
transforms: https://github.com/apache/beam/pull/11574

I'll confirm and roll back if it is.

On Thu, May 7, 2020 at 10:51 AM Pablo Estrada <pa...@google.com> wrote:

> Hi all,
> the Precommit tests have been flaky for a couple of days due to
> ExternalTransform tests. See
> https://builds.apache.org/job/beam_PreCommit_Python_Cron/ - and an
> example failure:
>
> https://builds.apache.org/job/beam_PreCommit_Python_Cron/2723/
>
> Does somebody know about this?
> Best
> -P.
>