You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Hannah Jiang (Jira)" <ji...@apache.org> on 2020/07/23 23:07:00 UTC
[jira] [Comment Edited] (BEAM-10567) PythonDocker precommit seems
flaky
[ https://issues.apache.org/jira/browse/BEAM-10567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17164017#comment-17164017 ]
Hannah Jiang edited comment on BEAM-10567 at 7/23/20, 11:06 PM:
----------------------------------------------------------------
I went through several recent failures and don't think the failures are from license pull related tasks.
One common pattern I observed is *[Execution failed for task ':sdks:go:resolveBuildDependencies'|https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Commit/1908/console].*
Execution failed for task ':sdks:go:resolveBuildDependencies'.
10:38:06 > Exception in resolution, message is:
10:38:06 Cannot resolve dependency:github.com/coreos/etcd: commit='11214aa33bf5a47d3d9d8dafe0f6b97237dfe921', urls=[https://github.com/coreos/etcd.git, git@github.com:coreos/etcd.git]
10:38:06 Resolution stack is:
10:38:06 +- github.com/apache/beam/sdks/go
Some other errors are
1. [ProtocolError: ("Connection broken: error(104, 'Connection reset by peer')", error(104, 'Connection reset by peer'))
|https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Commit/1746/consoleFull] when downloading tensorflow for Py2.
2. [91mERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device
|https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Commit/1905/console] <- This tends to happen to several consecutive runs until the space is released.
I looked through last 14 failed runs, and around 80% of time, it failed with the :sdks:go:resolveBuildDependencies task. In order to fix the flaky, we should look into this task. No failures are due to license job.
Another way to reduce the flakiness is merging this task to PythonPrecommit, because some tasks are duplicated, like the one mentioned above. By merging these two, we will have one failure, instead of two failures.
The reason PythonDocker task is created is that 1. PythonPrecommit is huge enough, don't want to add more tasks when it's reasonable to separate them. Docker images are not created with PythonPrecommit tests. 2. The job should run with --info option to print error logs from docker image, otherwise, it's hard to debug. PythonPrecommit doesn't run with --info option, so we need to add it, but if we do, it prints out too many logs.
Merging or not is a trade-off.
was (Author: hannahjiang):
I went through several recent failures and don't think the failures are from license pull related tasks.
One common pattern I observed is *[Execution failed for task ':sdks:go:resolveBuildDependencies'|https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Commit/1908/console].*
Execution failed for task ':sdks:go:resolveBuildDependencies'.
10:38:06 > Exception in resolution, message is:
10:38:06 Cannot resolve dependency:github.com/coreos/etcd: commit='11214aa33bf5a47d3d9d8dafe0f6b97237dfe921', urls=[https://github.com/coreos/etcd.git, git@github.com:coreos/etcd.git]
10:38:06 Resolution stack is:
10:38:06 +- github.com/apache/beam/sdks/go
Some other errors are
1. [ProtocolError: ("Connection broken: error(104, 'Connection reset by peer')", error(104, 'Connection reset by peer'))
|https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Commit/1746/consoleFull] when downloading tensorflow for Py2.
2. [91mERROR: Could not install packages due to an EnvironmentError: [Errno 28] No space left on device
|https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Commit/1905/console] <- This tends to happen to several consecutive runs until the space is released.
I looked through last 14 failed runs, and around 80% of time, it failed with the :sdks:go:resolveBuildDependencies task. In order to fix the flaky, we should look into this task. No failures are due to license job.
Another way to reduce the flakiness is merging this task to PythonPrecommit, because some tasks are duplicated, like the one mentioned above. By merging these two, we will have one failure, instead of two failures.
The reason PythonDocker task is created is that 1. PythonPrecommit is huge enough, don't want to add more tasks when it's reasonable to separate them. Docker images are not created with PythonPrecommit tests. 2. The job should run with --info option to print error logs from docker image, otherwise, it's hard to debug. PythonPrecommit doesn't run with --info option, so we need to add it, but if we do, it prints out too many logs.
Merging or not is a trade-off.
> PythonDocker precommit seems flaky
> ----------------------------------
>
> Key: BEAM-10567
> URL: https://issues.apache.org/jira/browse/BEAM-10567
> Project: Beam
> Issue Type: Bug
> Components: test-failures
> Reporter: Udi Meiri
> Assignee: Hannah Jiang
> Priority: P2
>
> I've been getting these failures recently:
> {code}
> 08:53:23 > Task :sdks:python:container:py2:docker FAILED
> 08:53:23 The command '/bin/sh -c pip install -r /tmp/base_image_requirements.txt && python -c "from google.protobuf.internal import api_implementation; assert api_implementation._default_implementation_type == 'cpp'; print ('Verified fast protobuf used.')" && rm -rf /root/.cache/pip' returned a non-zero code: 2
> 08:53:23 :sdks:python:container:py2:docker (Thread[Execution worker for ':' Thread 11,5,main]) completed. Took 2 mins 32.762 secs.
> {code}
> https://ci-beam.apache.org/job/beam_PreCommit_PythonDocker_Commit/1746/consoleFull
--
This message was sent by Atlassian Jira
(v8.3.4#803005)