You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Mikhail Gryzykhin <mi...@google.com> on 2018/09/06 17:27:30 UTC
Failing: beam_PerformanceTests_Python
Hi everyone,
I see that beam_PerformanceTests_Python
<https://builds.apache.org/job/beam_PerformanceTests_Python/1391/console> job
is failing during last several days.
Corresponding errors:
2 days ago it was failure in beam_integration_benchmark(1/1)
<https://builds.apache.org/job/beam_PerformanceTests_Python/1386/console>
Now it is incompatible packages
<https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console> [
BEAM-5334 <https://issues.apache.org/jira/browse/BEAM-5334>]
Can someone help look into this?
Regards,
--Mikhail
Have feedback <http://go/migryz-feedback>?
Re: Failing: beam_PerformanceTests_Python
Posted by Łukasz Gajowy <lu...@gmail.com>.
I think we should focus on fixing the BEAM-5334
<https://issues.apache.org/jira/browse/BEAM-5334> issue first. I
investigated it a little bit and tried to reproduce it locally on my
machine - no success. The logs say:
"05:18:37 error: [Errno 2] No such file or directory"
but do not specify what is the path of the file or directory which seems
weird to me. Is Jenkins hiding the path for some reason (a bold idea of
mine)? Could someone with direct access to Jenkins check what is happening
(e.g. by running the commands manually on Jenkins executor)?
I also tried to explicitly define pip version to 18.0
<https://github.com/apache/beam/pull/6347/files> because it now uses 10.0.1
to see if this fixes the problem. Surprisingly to me, logs still say that
version 10.0.1 is used
<https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console>.
Fixing this issue is very important because it prevents many Jenkins jobs
from being run. It not only prevents Python performance tests from being
run but all of them - even the Java ones that are also run using the
PerfKitBenchmarker tool or Spark Performance tests.
pt., 7 wrz 2018 o 02:48 Mark Liu <ma...@google.com> napisał(a):
> I created a PR-6344 <https://github.com/apache/beam/pull/6344/files> that
> synced to a point earlier and expected to see Jenkins error 2 days ago, but
> not. Probably the current issue is relevant to Jenkins build environment.
>
> We can also think about moving performance tests to Gradle which seems
> provide a stable way to setup python environment
> <https://github.com/apache/beam/blob/932858ece39fb2477baa19638f0a6b36fb5dee9d/sdks/python/build.gradle#L31>
> since recent beam_PostCommit_Python_Verify
> <https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/>
> keeps green.
>
> Mark
>
> On Thu, Sep 6, 2018 at 1:04 PM Mikhail Gryzykhin <mi...@google.com>
> wrote:
>
>> Thank you for looking into this.
>>
>> --Mikhail
>>
>> Have feedback <http://go/migryz-feedback>?
>>
>>
>> On Thu, Sep 6, 2018 at 11:20 AM Mark Liu <ma...@google.com> wrote:
>>
>>> BEAM-5296 <https://issues.apache.org/jira/browse/BEAM-5296> is tracking
>>> the previous error. Some debugging efforts are made and summarized in
>>> PR-6321
>>> <https://github.com/apache/beam/pull/6321#issuecomment-418339689>.
>>>
>>> But I didn't realize test failed in different error now.
>>>
>>> Mark
>>>
>>> On Thu, Sep 6, 2018 at 10:28 AM Mikhail Gryzykhin <mi...@google.com>
>>> wrote:
>>>
>>>> Hi everyone,
>>>>
>>>> I see that beam_PerformanceTests_Python
>>>> <https://builds.apache.org/job/beam_PerformanceTests_Python/1391/console> job
>>>> is failing during last several days.
>>>>
>>>> Corresponding errors:
>>>> 2 days ago it was failure in beam_integration_benchmark(1/1)
>>>> <https://builds.apache.org/job/beam_PerformanceTests_Python/1386/console>
>>>> Now it is incompatible packages
>>>> <https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console>
>>>> [BEAM-5334 <https://issues.apache.org/jira/browse/BEAM-5334>]
>>>>
>>>> Can someone help look into this?
>>>>
>>>> Regards,
>>>> --Mikhail
>>>>
>>>> Have feedback <http://go/migryz-feedback>?
>>>>
>>>
Re: Failing: beam_PerformanceTests_Python
Posted by Mark Liu <ma...@google.com>.
I came out a PR <https://github.com/apache/beam/pull/6392> which is the
first step to move benchmark execution on Gradle task. Corresponding
Perfkit changes
<https://github.com/markflyhigh/PerfKitBenchmarker/pull/2/files> are also
required.
Changes include:
- new `run_integration_test.sh` script that provide better command line
flags and extend more use cases.
- new Gradle task `beam-sdks-python:integrationTest` that uses the new
script and can be used by Perfkit to execute any integration test without
worry about environment setup.
Hopefully those changes can fix existing benchmark failure, but also it
targets on improving user experience for running python integration tests
and simplify test infrastructure.
Ideas and suggestions are welcome.
Mark
On Sat, Sep 8, 2018 at 1:37 AM Łukasz Gajowy <lu...@gmail.com>
wrote:
> Oh, ok. Sorry. Thank you for the clarification!
>
>
> pt., 7 wrz 2018 o 21:03 Mark Liu <ma...@google.com> napisał(a):
>
>> PostCommit_Python_Verify do run this command which is configured in
>> sdks/python/build.gradle
>> <https://github.com/apache/beam/blob/master/sdks/python/build.gradle#L192> and
>> is a dependency of `:beam-sdks-python:postCommit`
>> <https://github.com/apache/beam/blob/master/sdks/python/build.gradle#L320> which
>> is called by Jenkins in here
>> <https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_Python_Verify.groovy#L36>
>> .
>>
>> On Fri, Sep 7, 2018 at 7:53 AM Łukasz Gajowy <lu...@gmail.com>
>> wrote:
>>
>>>
>>>> We can also think about moving performance tests to Gradle which seems
>>>> provide a stable way to setup python environment
>>>> <https://github.com/apache/beam/blob/932858ece39fb2477baa19638f0a6b36fb5dee9d/sdks/python/build.gradle#L31>
>>>> since recent beam_PostCommit_Python_Verify
>>>> <https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/>
>>>> keeps green.
>>>>
>>>>
>>> Correct me if I'm wrong but PostCommit_Python_Verify doesn't run "pip
>>> install -e sdks/python/[gcp,test]". As this is the line that breaks the
>>> build in other jobs, it may be the reason why the job doesnt fail.
>>>
>>
Re: Failing: beam_PerformanceTests_Python
Posted by Łukasz Gajowy <lu...@gmail.com>.
Oh, ok. Sorry. Thank you for the clarification!
pt., 7 wrz 2018 o 21:03 Mark Liu <ma...@google.com> napisał(a):
> PostCommit_Python_Verify do run this command which is configured in
> sdks/python/build.gradle
> <https://github.com/apache/beam/blob/master/sdks/python/build.gradle#L192> and
> is a dependency of `:beam-sdks-python:postCommit`
> <https://github.com/apache/beam/blob/master/sdks/python/build.gradle#L320> which
> is called by Jenkins in here
> <https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_Python_Verify.groovy#L36>
> .
>
> On Fri, Sep 7, 2018 at 7:53 AM Łukasz Gajowy <lu...@gmail.com>
> wrote:
>
>>
>>> We can also think about moving performance tests to Gradle which seems
>>> provide a stable way to setup python environment
>>> <https://github.com/apache/beam/blob/932858ece39fb2477baa19638f0a6b36fb5dee9d/sdks/python/build.gradle#L31>
>>> since recent beam_PostCommit_Python_Verify
>>> <https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/>
>>> keeps green.
>>>
>>>
>> Correct me if I'm wrong but PostCommit_Python_Verify doesn't run "pip
>> install -e sdks/python/[gcp,test]". As this is the line that breaks the
>> build in other jobs, it may be the reason why the job doesnt fail.
>>
>
Re: Failing: beam_PerformanceTests_Python
Posted by Mark Liu <ma...@google.com>.
PostCommit_Python_Verify do run this command which is configured in
sdks/python/build.gradle
<https://github.com/apache/beam/blob/master/sdks/python/build.gradle#L192> and
is a dependency of `:beam-sdks-python:postCommit`
<https://github.com/apache/beam/blob/master/sdks/python/build.gradle#L320>
which
is called by Jenkins in here
<https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_PostCommit_Python_Verify.groovy#L36>
.
On Fri, Sep 7, 2018 at 7:53 AM Łukasz Gajowy <lu...@gmail.com>
wrote:
>
>> We can also think about moving performance tests to Gradle which seems
>> provide a stable way to setup python environment
>> <https://github.com/apache/beam/blob/932858ece39fb2477baa19638f0a6b36fb5dee9d/sdks/python/build.gradle#L31>
>> since recent beam_PostCommit_Python_Verify
>> <https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/>
>> keeps green.
>>
>>
> Correct me if I'm wrong but PostCommit_Python_Verify doesn't run "pip
> install -e sdks/python/[gcp,test]". As this is the line that breaks the
> build in other jobs, it may be the reason why the job doesnt fail.
>
Re: Failing: beam_PerformanceTests_Python
Posted by Łukasz Gajowy <lu...@gmail.com>.
>
>
> We can also think about moving performance tests to Gradle which seems
> provide a stable way to setup python environment
> <https://github.com/apache/beam/blob/932858ece39fb2477baa19638f0a6b36fb5dee9d/sdks/python/build.gradle#L31>
> since recent beam_PostCommit_Python_Verify
> <https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/>
> keeps green.
>
>
Correct me if I'm wrong but PostCommit_Python_Verify doesn't run "pip
install -e sdks/python/[gcp,test]". As this is the line that breaks the
build in other jobs, it may be the reason why the job doesnt fail.
Re: Failing: beam_PerformanceTests_Python
Posted by Mark Liu <ma...@google.com>.
I created a PR-6344 <https://github.com/apache/beam/pull/6344/files> that
synced to a point earlier and expected to see Jenkins error 2 days ago, but
not. Probably the current issue is relevant to Jenkins build environment.
We can also think about moving performance tests to Gradle which seems
provide a stable way to setup python environment
<https://github.com/apache/beam/blob/932858ece39fb2477baa19638f0a6b36fb5dee9d/sdks/python/build.gradle#L31>
since recent beam_PostCommit_Python_Verify
<https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Python_Verify/>
keeps green.
Mark
On Thu, Sep 6, 2018 at 1:04 PM Mikhail Gryzykhin <mi...@google.com> wrote:
> Thank you for looking into this.
>
> --Mikhail
>
> Have feedback <http://go/migryz-feedback>?
>
>
> On Thu, Sep 6, 2018 at 11:20 AM Mark Liu <ma...@google.com> wrote:
>
>> BEAM-5296 <https://issues.apache.org/jira/browse/BEAM-5296> is tracking
>> the previous error. Some debugging efforts are made and summarized in
>> PR-6321 <https://github.com/apache/beam/pull/6321#issuecomment-418339689>
>> .
>>
>> But I didn't realize test failed in different error now.
>>
>> Mark
>>
>> On Thu, Sep 6, 2018 at 10:28 AM Mikhail Gryzykhin <mi...@google.com>
>> wrote:
>>
>>> Hi everyone,
>>>
>>> I see that beam_PerformanceTests_Python
>>> <https://builds.apache.org/job/beam_PerformanceTests_Python/1391/console> job
>>> is failing during last several days.
>>>
>>> Corresponding errors:
>>> 2 days ago it was failure in beam_integration_benchmark(1/1)
>>> <https://builds.apache.org/job/beam_PerformanceTests_Python/1386/console>
>>> Now it is incompatible packages
>>> <https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console>
>>> [BEAM-5334 <https://issues.apache.org/jira/browse/BEAM-5334>]
>>>
>>> Can someone help look into this?
>>>
>>> Regards,
>>> --Mikhail
>>>
>>> Have feedback <http://go/migryz-feedback>?
>>>
>>
Re: Failing: beam_PerformanceTests_Python
Posted by Mikhail Gryzykhin <mi...@google.com>.
Thank you for looking into this.
--Mikhail
Have feedback <http://go/migryz-feedback>?
On Thu, Sep 6, 2018 at 11:20 AM Mark Liu <ma...@google.com> wrote:
> BEAM-5296 <https://issues.apache.org/jira/browse/BEAM-5296> is tracking
> the previous error. Some debugging efforts are made and summarized in
> PR-6321 <https://github.com/apache/beam/pull/6321#issuecomment-418339689>
> .
>
> But I didn't realize test failed in different error now.
>
> Mark
>
> On Thu, Sep 6, 2018 at 10:28 AM Mikhail Gryzykhin <mi...@google.com>
> wrote:
>
>> Hi everyone,
>>
>> I see that beam_PerformanceTests_Python
>> <https://builds.apache.org/job/beam_PerformanceTests_Python/1391/console> job
>> is failing during last several days.
>>
>> Corresponding errors:
>> 2 days ago it was failure in beam_integration_benchmark(1/1)
>> <https://builds.apache.org/job/beam_PerformanceTests_Python/1386/console>
>> Now it is incompatible packages
>> <https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console>
>> [BEAM-5334 <https://issues.apache.org/jira/browse/BEAM-5334>]
>>
>> Can someone help look into this?
>>
>> Regards,
>> --Mikhail
>>
>> Have feedback <http://go/migryz-feedback>?
>>
>
Re: Failing: beam_PerformanceTests_Python
Posted by Mark Liu <ma...@google.com>.
BEAM-5296 <https://issues.apache.org/jira/browse/BEAM-5296> is tracking the
previous error. Some debugging efforts are made and summarized in PR-6321
<https://github.com/apache/beam/pull/6321#issuecomment-418339689>.
But I didn't realize test failed in different error now.
Mark
On Thu, Sep 6, 2018 at 10:28 AM Mikhail Gryzykhin <mi...@google.com> wrote:
> Hi everyone,
>
> I see that beam_PerformanceTests_Python
> <https://builds.apache.org/job/beam_PerformanceTests_Python/1391/console> job
> is failing during last several days.
>
> Corresponding errors:
> 2 days ago it was failure in beam_integration_benchmark(1/1)
> <https://builds.apache.org/job/beam_PerformanceTests_Python/1386/console>
> Now it is incompatible packages
> <https://builds.apache.org/job/beam_PerformanceTests_Python/1390/console>
> [BEAM-5334 <https://issues.apache.org/jira/browse/BEAM-5334>]
>
> Can someone help look into this?
>
> Regards,
> --Mikhail
>
> Have feedback <http://go/migryz-feedback>?
>