You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "jrmccluskey (via GitHub)" <gi...@apache.org> on 2023/10/31 14:29:55 UTC
[I] [Failing Test]: PythonPostCommit is Perma-Red [beam]
jrmccluskey opened a new issue, #29214:
URL: https://github.com/apache/beam/issues/29214
### What happened?
The apache_beam/io/external/xlang_kinesisio_it_test.py::CrossLanguageKinesisIOTest::test_kinesis_write test is failing in the Python PostCommit with a consistent error message:
```
botocore.exceptions.ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "http://localhost:32770/".
```
The test is defined here: https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/external/xlang_kinesisio_it_test.py#L94
### Issue Failure
Failure: Test is continually failing
### Issue Priority
Priority: 1 (unhealthy code / failing or flaky postcommit so we cannot be sure the product is healthy)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Perma-Red [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm closed issue #29214: [Failing Test]: PythonPostCommit is Perma-Red
URL: https://github.com/apache/beam/issues/29214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Perma-Red [beam]
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1792947289
Is this still relevant?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Extremely Flaky [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1898239727
Its still flaky, failing differently
https://github.com/apache/beam/actions/runs/7563095321/job/20594841089#step:9:37337
Cloud be the test is somehow broken but i cannot see a pattern right now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Extremely Flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1885182358
Now it is still flaky though with lower frequency: https://github.com/apache/beam/runs/20274778848
There is other flaky test. e.g.
```
apache_beam.examples.fastavro_it_test.FastavroIT
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
Traceback (most recent call last):
File "apache_beam/runners/common.py", line 1435, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 636, in apache_beam.runners.common.SimpleInvoker.invoke_process
File "apache_beam/runners/common.py", line 1611, in apache_beam.runners.common._OutputHandler.handle_process_outputs
File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.10/site-packages/apache_beam/io/filebasedsource.py", line 380, in process
source = self._source_from_file(metadata.path)
File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.10/site-packages/apache_beam/io/filebasedsource.py", line 127, in __init__
self._validate()
File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.10/site-packages/apache_beam/options/value_provider.py", line 193, in _f
return fnc(self, *args, **kwargs)
File "/opt/apache/beam-venv/beam-venv-worker-sdk-0-0/lib/python3.10/site-packages/apache_beam/io/filebasedsource.py", line 190, in _validate
raise IOError('No files found based on the file pattern %s' % pattern)
OSError: No files found based on the file pattern gs://temp-storage-for-end-to-end-tests/py-it-cloud/output/e26d0c72-c41a-43e4-aa51-598e6994b277/fastavro-00003-of-00004
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Extremely Flaky [beam]
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck closed issue #29214: [Failing Test]: PythonPostCommit is Extremely Flaky
URL: https://github.com/apache/beam/issues/29214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Perma-Red [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1802087075
The flow you referenced is green
https://github.com/apache/beam/actions/runs/6797617071/job/18480096798
So yeah, flakiness. Glad to have it sorted we were lucky that MTU issues did not cause bigger problems
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Perma-Red [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1799760456
I did not meant to autoclose its only a part. Sorry im not aware of mechanisms, worked on a lot of different systems :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Extremely Flaky [beam]
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1896777397
There's one week until the 2.54.0 cut and this issue is tagged for that release, if possible/necessary please complete the necessary work before then, or move this to the [2.55.0 Release Milestone](https://github.com/apache/beam/milestone/19).
This one seems like we may need to cherry pick though if additional fixes occur.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Perma-Red [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1799763423
No worries, its actually a GitHub feature - https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Perma-Red [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1792951364
Yep still a problem, @damccorm said there had been a few issues along these lines with the self-hosted runners
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Perma-Red [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1798361948
Im looking into this one, there are issues with runners that soon will be fixed but that is still not the final fix needed for python postcommit, keep you posted
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Perma-Red [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1799051263
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Perma-Red [beam]
Posted by "volatilemolotov (via GitHub)" <gi...@apache.org>.
volatilemolotov commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1801593868
So a scheduled run failed with one actually passing. Other three jobs fail in different places
https://github.com/apache/beam/actions/runs/6794201259/job/18470220889#step:9:37734
Any ideas what is going on? Could it be because of parallel run (I had a full green run in my fork)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Perma-Red [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1802017514
Definitely seems like we've upgraded from permared jobs to test flakiness, so I don't think this is a runner/actions problem anymore. For example https://github.com/apache/beam/actions/runs/6797617071/job/18480096798 already has 3 green jobs (with a 4th still running)
At least some of it is caused by https://github.com/apache/beam/issues/29076 - I see a bunch of failures related to that test in the workflow you linked.
I have https://github.com/apache/beam/pull/29197 to fix that, was holding off on merging since there was a lot going on causing issues, but it might be time to merge. I'm running https://github.com/apache/beam/actions/runs/6799736065 to make sure I'm correctly sickbaying it, but once that runs (assuming its working as expected) I think we should merge the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Perma-Red [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1799746254
@volatilemolotov I don't think you meant to auto-close this with the PR, is that right? If yes, we can reclose after a green signal anyways I guess
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Failing Test]: PythonPostCommit is Extremely Flaky [beam]
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on issue #29214:
URL: https://github.com/apache/beam/issues/29214#issuecomment-1930505433
Closing this one, as all the python postcommits are passing on the release branch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org