You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tvalentyn (via GitHub)" <gi...@apache.org> on 2023/09/27 18:56:02 UTC
[GitHub] [beam] tvalentyn opened a new issue, #28703: [Failing Test]: Building a wheel for integration tests sometimes times out
tvalentyn opened a new issue, #28703:
URL: https://github.com/apache/beam/issues/28703
### What happened?
We are hitting subprocess.TimeoutExpired within cibuildwheel:
```
12:48:24 > Task :sdks:python:bdistPy38linux
12:48:24
12:48:24 [notice] A new release of pip available: 22.2.2 -> 23.2.1
12:48:24 [notice] To update, run: pip install --upgrade pip
12:48:24 Processing /project
12:48:24 Preparing metadata (setup.py): started
12:48:24 Preparing metadata (setup.py): finished with status 'done'
12:48:24 Building wheels for collected packages: apache-beam
12:48:24 Building wheel for apache-beam (setup.py): started
12:48:24 Building wheel for apache-beam (setup.py): still running...
12:48:24 Building wheel for apache-beam (setup.py): still running...
12:48:24 Building wheel for apache-beam (setup.py): still running...
12:48:24 Building wheel for apache-beam (setup.py): finished with status 'done'
12:48:24 Created wheel for apache-beam: filename=apache_beam-2.52.0.dev0-cp38-cp38-linux_x86_64.whl size=15364571 sha256=e737d0e77efe4e77279557c8b6226fd9a5d2f36b71343343a00ca1aae340e4ba
12:48:24 Stored in directory: /tmp/pip-ephem-wheel-cache-jf8ssnkj/wheels/d4/3a/c0/4dc152b1840724d5b992a8268bb4ef33fdbe42ffe1429b845c
12:48:24 Successfully built apache-beam
12:48:24 + /opt/python/cp38-cp38/bin/python -c 'import sys, json, glob; json.dump(glob.glob('"'"'/tmp/cibuildwheel/built_wheel/*.whl'"'"'), sys.stdout)'
12:48:24 + rm -rf /tmp/cibuildwheel/repaired_wheel
12:48:25 + mkdir -p /tmp/cibuildwheel/repaired_wheel
12:48:25
12:48:25 ✓ 253.39s
12:48:25 Repairing wheel...
12:48:25
12:48:25 + sh -c 'auditwheel repair -w /tmp/cibuildwheel/repaired_wheel /tmp/cibuildwheel/built_wheel/apache_beam-2.52.0.dev0-cp38-cp38-linux_x86_64.whl'
12:48:25 INFO:auditwheel.main_repair:Repairing apache_beam-2.52.0.dev0-cp38-cp38-linux_x86_64.whl
12:48:25 INFO:auditwheel.wheeltools:Previous filename tags: linux_x86_64
12:48:27 INFO:auditwheel.wheeltools:New filename tags: manylinux_2_17_x86_64, manylinux2014_x86_64
12:48:27 INFO:auditwheel.wheeltools:Previous WHEEL info tags: cp38-cp38-linux_x86_64
12:48:27 INFO:auditwheel.wheeltools:New WHEEL info tags: cp38-cp38-manylinux_2_17_x86_64, cp38-cp38-manylinux2014_x86_64
12:48:27 INFO:auditwheel.main_repair:
12:48:31 Fixed-up wheel written to /tmp/cibuildwheel/repaired_wheel/apache_beam-2.52.0.dev0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
12:48:31 + /opt/python/cp38-cp38/bin/python -c 'import sys, json, glob; json.dump(glob.glob('"'"'/tmp/cibuildwheel/repaired_wheel/*.whl'"'"'), sys.stdout)'
12:48:31 + mkdir -p /output
12:48:31 + mv /tmp/cibuildwheel/repaired_wheel/apache_beam-2.52.0.dev0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl /output
12:48:31
12:48:31 ✓ 6.64s
12:48:31
12:48:31 ✓ cp38-manylinux_x86_64 finished in 270.43s
12:48:31 Copying wheels back to host...
12:48:31
12:48:31
12:48:31 ✓ 0.17s
12:48:31 Traceback (most recent call last):
12:49:01 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Integration_Commit/src/build/gradleenv/1922375555/bin/cibuildwheel", line 8, in <module>
12:49:01 sys.exit(main())
12:49:01 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Integration_Commit/src/build/gradleenv/1922375555/lib/python3.8/site-packages/cibuildwheel/__main__.py", line 129, in main
12:49:01 build_in_directory(args)
12:49:01 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Integration_Commit/src/build/gradleenv/1922375555/lib/python3.8/site-packages/cibuildwheel/__main__.py", line 248, in build_in_directory
12:49:01 cibuildwheel.linux.build(options, tmp_path)
12:49:01 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Integration_Commit/src/build/gradleenv/1922375555/lib/python3.8/site-packages/cibuildwheel/linux.py", line 384, in build
12:49:01 build_in_container(
12:49:01 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Integration_Commit/src/build/gradleenv/1922375555/lib/python3.8/site-packages/cibuildwheel/oci_container.py", line 134, in __exit__
12:49:01 self.process.wait(timeout=30)
12:49:01 File "/usr/lib/python3.8/subprocess.py", line 1083, in wait
12:49:01 return self._wait(timeout=timeout)
12:49:01 File "/usr/lib/python3.8/subprocess.py", line 1798, in _wait
12:49:01 raise TimeoutExpired(self.args, timeout)
12:49:01 subprocess.TimeoutExpired: Command '['docker', 'start', '--attach', '--interactive', 'cibuildwheel-925e8aec-2a6f-4f02-b2e5-22df08e131cc']' timed out after 30 seconds
12:49:01
12:49:13 > Task :sdks:python:bdistPy38linux FAILED
12:49:13
```
We could perhaps add a retry around cibuildwheel call.
### Issue Failure
Failure: Test is flaky
### Issue Priority
Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Building a wheel for integration tests sometimes times out [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #28703:
URL: https://github.com/apache/beam/issues/28703#issuecomment-1847797413
Added a retry logic. Let's reopen if we see it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] jrmccluskey commented on issue #28703: [Failing Test]: Building a wheel for integration tests sometimes times out
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on issue #28703:
URL: https://github.com/apache/beam/issues/28703#issuecomment-1739421565
Reading through the documentation for `cibuildwheel` we don't get a route for configuring any timeouts. I'm a little concerned about just wrapping the `cibuildwheel` calls in a retry though, we'd be adding more runtime to the actions and not necessarily addressing the underlying issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Building a wheel for integration tests sometimes times out [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #28703:
URL: https://github.com/apache/beam/issues/28703#issuecomment-1846246482
Filed https://github.com/pypa/cibuildwheel/issues/1692 for CIBuildWheel folks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Building a wheel for integration tests sometimes times out [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed issue #28703: [Failing Test]: Building a wheel for integration tests sometimes times out
URL: https://github.com/apache/beam/issues/28703
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Building a wheel for integration tests sometimes times out [beam]
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on issue #28703:
URL: https://github.com/apache/beam/issues/28703#issuecomment-1773375182
Is this still observed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #28703: [Failing Test]: Building a wheel for integration tests sometimes times out
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #28703:
URL: https://github.com/apache/beam/issues/28703#issuecomment-1739887242
What if we retry a limited # of times?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Building a wheel for integration tests sometimes times out [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #28703:
URL: https://github.com/apache/beam/issues/28703#issuecomment-1780132134
Yes, still happening:
```
13:46:31
13:46:38 > Task :sdks:python:bdistPy311linux
13:46:38 Traceback (most recent call last):
13:46:38 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2/src/build/gradleenv/1922375555/bin/cibuildwheel", line 8, in <module>
13:46:38 sys.exit(main())
13:46:38 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2/src/build/gradleenv/1922375555/lib/python3.8/site-packages/cibuildwheel/__main__.py", line 129, in main
13:46:38 build_in_directory(args)
13:46:38 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2/src/build/gradleenv/1922375555/lib/python3.8/site-packages/cibuildwheel/__main__.py", line 248, in build_in_directory
13:46:38 cibuildwheel.linux.build(options, tmp_path)
13:46:38 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2/src/build/gradleenv/1922375555/lib/python3.8/site-packages/cibuildwheel/linux.py", line 384, in build
13:46:38 build_in_container(
13:46:38 File "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit@2/src/build/gradleenv/1922375555/lib/python3.8/site-packages/cibuildwheel/oci_container.py", line 134, in __exit__
13:46:38 self.process.wait(timeout=30)
13:46:38 File "/usr/lib/python3.8/subprocess.py", line 1083, in wait
13:46:38 return self._wait(timeout=timeout)
13:46:38 File "/usr/lib/python3.8/subprocess.py", line 1798, in _wait
13:46:38 raise TimeoutExpired(self.args, timeout)
13:46:38 subprocess.TimeoutExpired: Command '['docker', 'start', '--attach', '--interactive', 'cibuildwheel-690819bd-0a6a-47c7-bf5a-4c280b516a4f']' timed out after 30 seconds
13:46:38
13:46:41 > Task :sdks:python:bdistPy311linux FAILED
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: Building a wheel for integration tests sometimes times out [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #28703:
URL: https://github.com/apache/beam/issues/28703#issuecomment-1846219063
I think timeout 30sec is hardcoded in:
https://github.com/pypa/cibuildwheel/blob/886ba0f6628c5234efb2fb16c5628e3125ab3173/cibuildwheel/oci_container.py#L185
Beam code where we can have some retry logic:
https://github.com/apache/beam/blob/318227345264e70ed2241538b8ab42a64af5a282/sdks/python/build.gradle#L96
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org