You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "github-actions[bot] (via GitHub)" <gi...@apache.org> on 2024/04/13 15:33:30 UTC
[I] The PostCommit TransformService Direct job is flaky [beam]
github-actions[bot] opened a new issue, #30960:
URL: https://github.com/apache/beam/issues/30960
The PostCommit TransformService Direct is failing over 50% of the time
Please visit https://github.com/apache/beam/actions/workflows/beam_PostCommit_TransformService_Direct.yml?query=is%3Afailure+branch%3Amaster to see the logs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-3488092782
Haven't reproduced it 100% locally, but in local it shows the following error:
```
RuntimeError: The grpc package installed is at version 1.65.5, but the generated code in org/apache/beam/model/pipeline/v1/standard_window_fns_pb2_grpc.py depends on grpcio>=1.71.0. Please upgrade your grpc module to grpcio>=1.71.0 or downgrade your generated code using grpcio-tools<=1.65.5.
Resolve mutations for :sdks:python:test-suites:direct:xlang:fnApiJobServerCleanup (Thread[#194,Execution worker Thread 21,5,main]) started.
:sdks:python:test-suites:direct:xlang:fnApiJobServerCleanup (Thread[#194,Execution worker Thread 21,5,main]) started.
```
what I suspect is that the test compiles grpc code under Python 3.9, but then run Beam at Python 3.13, causing similar conflict. This explains why Python 3.12 used to work but the test started failing since Python 3.13
@aIbrahiim @jrmccluskey can we partly revert TransformService test change in #35056 to make it run on Python 3.9 and 3.12, until all Python versions can have a shared, and working grpc ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-4039672172
Stabilized
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-2739047334
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-2295303298
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-3736658818
Fixed by #37270
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-3368553096
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-2301602224
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto closed issue #30960: The PostCommit TransformService Direct job is flaky
URL: https://github.com/apache/beam/issues/30960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-3354814144
Stabilized
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-3406992626
breaking since Oct 3, likely after switching to Python 3.13 (#35056) cc: @jrmccluskey @tvalentyn what could be causing grpc error in Python 3.13 alone?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-4179816594
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto closed issue #30960: The PostCommit TransformService Direct job is flaky
URL: https://github.com/apache/beam/issues/30960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "shahine44 (via GitHub)" <gi...@apache.org>.
shahine44 commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-2301626667
من مبتدی هستم و زیاد چیزی نمی دانم و می خواهم یاد بگیرم
در تاریخ چهارشنبه ۲۱ اوت ۲۰۲۴، ۱۳:۰۴ github-actions[bot] <
***@***.***> نوشت:
> Reopened #30960 <https://github.com/apache/beam/issues/30960>.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/issues/30960#event-13957494517>, or
> unsubscribe
> <https://github.com/notifications/unsubscribe-auth/A5VBXF3REVJOQ5ZJMLVS4N3ZSRNKVAVCNFSM6AAAAABGFRSTSGVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJTHE2TONBZGQ2TCNY>
> .
> You are receiving this because you are subscribed to this thread.Message
> ID: ***@***.***>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-3488186442
I see that as a fault with the test. We can't necessarily control when different python versions will require different dependency versions (and unless the python foundation changes the release cadence for versions we will be trying to support 4-5 python versions simultaneously for the foreseeable future) but we can make sure that our tests match what users get when they install beam at a specific version.
In short, I'd prefer that we have the 3.13 version of the test re-build grpc with python 3.13
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-3407047920
There's a bug in grpc 1.66.0+ that can cause timeouts, only Python 3.13 uses a version beyond this out of necessity (prior releases do not support 3.13.) #36525 includes experiments that are supposed to mitigate this problem and drops them into our dockerfile, which will hopefully make the 3.13 tests mores stable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #30960: The PostCommit TransformService Direct job is flaky
URL: https://github.com/apache/beam/issues/30960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-2302389476
a random bigtableio_it_test test failing a time, looks like the tests having racing condition running in parallel on same machine (port occupied?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-2314820618
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad closed issue #30960: The PostCommit TransformService Direct job is flaky
URL: https://github.com/apache/beam/issues/30960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-3352781238
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-4522938185
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #30960: The PostCommit TransformService Direct job is flaky
URL: https://github.com/apache/beam/issues/30960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-4036212331
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-3407360067
This is most likely an infra issue (tests passed on 3.9 variant but not 3.13) so move off release blocker, however we still need to fix the test as a PostCommit. To investigate One may need to remove the stdout/stderr redirect, or upload the log file at the end of the workflow to see what happened for the transform service @Amar3tto @aIbrahiim
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-2053728139
https://github.com/apache/beam/pull/30816 breaks the build here.
```
#10 43.81 INFO: pip is looking at multiple versions of apache-beam[dataframe,gcp] to determine which version is compatible with other requirements. This could take a while.
#10 43.81 ERROR: Cannot install apache-beam[dataframe,gcp]==2.56.0.dev0 because these package versions have conflicting dependencies.
#10 43.81
#10 43.81 The conflict is caused by:
#10 43.81 apache-beam[dataframe,gcp] 2.56.0.dev0 depends on google-auth-httplib2<0.2.0 and >=0.1.0; extra == "gcp"
#10 43.81 The user requested (constraint) google-auth-httplib2==0.2.0
```
we have ` 'google-auth-httplib2>=0.1.0,<0.2.0'` in https://github.com/apache/beam/blob/master/sdks/python/setup.py#L445 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm closed issue #30960: The PostCommit TransformService Direct job is flaky
URL: https://github.com/apache/beam/issues/30960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed issue #30960: The PostCommit TransformService Direct job is flaky
URL: https://github.com/apache/beam/issues/30960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn closed issue #30960: The PostCommit TransformService Direct job is flaky
URL: https://github.com/apache/beam/issues/30960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm closed issue #30960: The PostCommit TransformService Direct job is flaky
URL: https://github.com/apache/beam/issues/30960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto closed issue #30960: The PostCommit TransformService Direct job is flaky
URL: https://github.com/apache/beam/issues/30960
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-3406971303
Python 3.13 tests failing due to _InactiveRpcError:
```
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.UNKNOWN
details = "Application error processing RPC"
debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Application error processing RPC", grpc_status:2}"
>
self = <apache_beam.io.gcp.bigtableio_it_test.TestWriteToBigtableXlangIT testMethod=test_set_mutation>
...
> self.run_pipeline([row1, row2])
> raise _InactiveRpcError(state) # pytype: disable=not-instantiable
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
E status = StatusCode.UNKNOWN
E details = "Application error processing RPC"
E debug_error_string = "UNKNOWN:Error received from peer {grpc_message:"Application error processing RPC", grpc_status:2}"
E >
../../build/gradleenv/1922375555/lib/python3.13/site-packages/grpc/_channel.py:996: _InactiveRpcError
```
https://github.com/apache/beam/runs/52803653779
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-3730156545
Reopening since the workflow is still flaky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] The PostCommit TransformService Direct job is flaky [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #30960:
URL: https://github.com/apache/beam/issues/30960#issuecomment-3407131696
This is xlang tests Direct runner not involving dockers. I tested locally (command: ` pytest -v -s apache_beam/io/gcp/bigtableio_it_test.py::TestWriteToBigtableXlangIT::test_set_mutation --test-pipeline-options="--runner=TestDirectRunner --project=apache-beam-testing"`). I get the same error while my machine missing docker. Likely this is simply due to transform service not successfully turned up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org