You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Fernando Morales Martinez <fe...@wizeline.com> on 2021/03/16 23:36:43 UTC
BEAM-11023: tests failing on Spark Structured Streaming runner
Hi team,
it is mentioned in this WI that the tests (GroupByKeyTest
testLargeKeys100MB and testGroupByKeyWithBadEqualsHashCode) stopped working
around five months ago.
I took a look at the PRs prior to that date and couldn't find a report
stating that they were working.
Is there a way to get reports from before June 2020 (the farthest back I
was able to navigate) so I can compare the tests succeeding against them
failing?
Thanks a lot!
- Fernando Morales
--
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*
Re: BEAM-11023: tests failing on Spark Structured Streaming runner
Posted by Fernando Morales Martinez <fe...@wizeline.com>.
Thanks Brian, Ismael!
Benjamín (@benWize) opened PR https://github.com/apache/beam/pull/14275
PostCommit checks are working
https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming_PR/64/
Just a heads up in case someone else from the dev team would like to take a
peak.
- Fernando Morales
On Wed, Mar 17, 2021 at 9:37 AM Ismaël Mejía <ie...@gmail.com> wrote:
> Actually there are many reasons that could have produced this
> regression even if the code of the runner has not changed at all: (1)
> those tests weren't enabled before and now are and they weren't
> passing or (2) the tests were changed or (3) my principal guess: the
> translation strategy of a runners-core library changed and as a side
> effect the tests fail in the runner, maybe the SDF/use_deprecated_read
> changes.
>
>
> On Wed, Mar 17, 2021 at 4:44 PM Brian Hulette <bh...@google.com> wrote:
> >
> > You can look through the history of the PostCommit [1]. We only keep a
> couple weeks of history, but it looks like we have one successful run from
> Sept 10, 2020, marked as "keep forever", that ran on commit
> 57055262e7a6bff447eef2df1e6efcda754939ca. Is that what you're looking for?
> >
> > (Somewhat related, I was under the impression that Jenkins always kept
> the before/after runs around the last state change, but that doesn't seem
> to be the case as the first failure we have is [3])
> >
> > Brian
> >
> > [1]
> https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/
> > [2]
> https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/2049/
> > [3]
> https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/2098/
> >
> > On Tue, Mar 16, 2021 at 4:36 PM Fernando Morales Martinez <
> fernando.morales@wizeline.com> wrote:
> >>
> >> Hi team,
> >> it is mentioned in this WI that the tests (GroupByKeyTest
> testLargeKeys100MB and testGroupByKeyWithBadEqualsHashCode) stopped working
> around five months ago.
> >> I took a look at the PRs prior to that date and couldn't find a report
> stating that they were working.
> >>
> >> Is there a way to get reports from before June 2020 (the farthest back
> I was able to navigate) so I can compare the tests succeeding against them
> failing?
> >>
> >> Thanks a lot!
> >> - Fernando Morales
> >>
> >> This email and its contents (including any attachments) are being sent
> to
> >> you on the condition of confidentiality and may be protected by legal
> >> privilege. Access to this email by anyone other than the intended
> recipient
> >> is unauthorized. If you are not the intended recipient, please
> immediately
> >> notify the sender by replying to this message and delete the material
> >> immediately from your system. Any further use, dissemination,
> distribution
> >> or reproduction of this email is strictly prohibited. Further, no
> >> representation is made with respect to any content contained in this
> email.
>
--
*This email and its contents (including any attachments) are being sent to
you on the condition of confidentiality and may be protected by legal
privilege. Access to this email by anyone other than the intended recipient
is unauthorized. If you are not the intended recipient, please immediately
notify the sender by replying to this message and delete the material
immediately from your system. Any further use, dissemination, distribution
or reproduction of this email is strictly prohibited. Further, no
representation is made with respect to any content contained in this email.*
Re: BEAM-11023: tests failing on Spark Structured Streaming runner
Posted by Ismaël Mejía <ie...@gmail.com>.
Actually there are many reasons that could have produced this
regression even if the code of the runner has not changed at all: (1)
those tests weren't enabled before and now are and they weren't
passing or (2) the tests were changed or (3) my principal guess: the
translation strategy of a runners-core library changed and as a side
effect the tests fail in the runner, maybe the SDF/use_deprecated_read
changes.
On Wed, Mar 17, 2021 at 4:44 PM Brian Hulette <bh...@google.com> wrote:
>
> You can look through the history of the PostCommit [1]. We only keep a couple weeks of history, but it looks like we have one successful run from Sept 10, 2020, marked as "keep forever", that ran on commit 57055262e7a6bff447eef2df1e6efcda754939ca. Is that what you're looking for?
>
> (Somewhat related, I was under the impression that Jenkins always kept the before/after runs around the last state change, but that doesn't seem to be the case as the first failure we have is [3])
>
> Brian
>
> [1] https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/
> [2] https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/2049/
> [3] https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/2098/
>
> On Tue, Mar 16, 2021 at 4:36 PM Fernando Morales Martinez <fe...@wizeline.com> wrote:
>>
>> Hi team,
>> it is mentioned in this WI that the tests (GroupByKeyTest testLargeKeys100MB and testGroupByKeyWithBadEqualsHashCode) stopped working around five months ago.
>> I took a look at the PRs prior to that date and couldn't find a report stating that they were working.
>>
>> Is there a way to get reports from before June 2020 (the farthest back I was able to navigate) so I can compare the tests succeeding against them failing?
>>
>> Thanks a lot!
>> - Fernando Morales
>>
>> This email and its contents (including any attachments) are being sent to
>> you on the condition of confidentiality and may be protected by legal
>> privilege. Access to this email by anyone other than the intended recipient
>> is unauthorized. If you are not the intended recipient, please immediately
>> notify the sender by replying to this message and delete the material
>> immediately from your system. Any further use, dissemination, distribution
>> or reproduction of this email is strictly prohibited. Further, no
>> representation is made with respect to any content contained in this email.
Re: BEAM-11023: tests failing on Spark Structured Streaming runner
Posted by Brian Hulette <bh...@google.com>.
You can look through the history of the PostCommit [1]. We only keep a
couple weeks of history, but it looks like we have one successful run from
Sept 10, 2020, marked as "keep forever", that ran on commit
57055262e7a6bff447eef2df1e6efcda754939ca.
Is that what you're looking for?
(Somewhat related, I was under the impression that Jenkins always kept the
before/after runs around the last state change, but that doesn't seem to be
the case as the first failure we have is [3])
Brian
[1]
https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/
[2]
https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/2049/
[3]
https://ci-beam.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/2098/
On Tue, Mar 16, 2021 at 4:36 PM Fernando Morales Martinez <
fernando.morales@wizeline.com> wrote:
> Hi team,
> it is mentioned in this WI that the tests (GroupByKeyTest
> testLargeKeys100MB and testGroupByKeyWithBadEqualsHashCode) stopped working
> around five months ago.
> I took a look at the PRs prior to that date and couldn't find a report
> stating that they were working.
>
> Is there a way to get reports from before June 2020 (the farthest back I
> was able to navigate) so I can compare the tests succeeding against them
> failing?
>
> Thanks a lot!
> - Fernando Morales
>
>
>
>
>
>
>
>
> *This email and its contents (including any attachments) are being sent
> toyou on the condition of confidentiality and may be protected by
> legalprivilege. Access to this email by anyone other than the intended
> recipientis unauthorized. If you are not the intended recipient, please
> immediatelynotify the sender by replying to this message and delete the
> materialimmediately from your system. Any further use, dissemination,
> distributionor reproduction of this email is strictly prohibited. Further,
> norepresentation is made with respect to any content contained in this
> email.*