You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 23:35:02 UTC
[GitHub] [beam] damccorm opened a new issue, #21473: PVR_Spark2_Streaming perma-red
damccorm opened a new issue, #21473:
URL: https://github.com/apache/beam/issues/21473
https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark2_Streaming has been failing a variable number of tests for a while.
Last successful run was Dec 28, 2021 (https://ci-beam.apache.org/job/beam_PostCommit_Java_PVR_Spark2_Streaming/1021/), which was approximately coincident with gradle 7 changes.
Imported from Jira [BEAM-13950](https://issues.apache.org/jira/browse/BEAM-13950). Original Jira may contain additional context.
Reported by: bhulette.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] byronellis commented on issue #21473: PVR_Spark2_Streaming perma-red
Posted by GitBox <gi...@apache.org>.
byronellis commented on issue #21473:
URL: https://github.com/apache/beam/issues/21473#issuecomment-1184812873
In local testing increasing the timeout of the streaming service addresses the issue. Not necessarily a fix, more of a bandaid but it's not clear that we can prevent the case where the service gets torn down before results of the test can be collected in another way. May also fix Spark 3 issues, but I haven't tested that yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on issue #21473: PVR_Spark2_Streaming perma-red
Posted by GitBox <gi...@apache.org>.
mosche commented on issue #21473:
URL: https://github.com/apache/beam/issues/21473#issuecomment-1168532090
Note: there was a recent [discussion](https://lists.apache.org/thread/opfhg3xjb9nptv878sygwj9gjx38rmco) to drop support for spark 2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] byronellis commented on issue #21473: PVR_Spark2_Streaming perma-red
Posted by GitBox <gi...@apache.org>.
byronellis commented on issue #21473:
URL: https://github.com/apache/beam/issues/21473#issuecomment-1184665692
TIL that Spark runner tests will fail (locally at least) if you start running low (80% full or so) on storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on issue #21473: PVR_Spark2_Streaming perma-red
Posted by GitBox <gi...@apache.org>.
mosche commented on issue #21473:
URL: https://github.com/apache/beam/issues/21473#issuecomment-1174881132
Tests for Spark 3 are also failing:
```
gradle :runners:spark:3:job-server:validatesPortableRunnerStreaming --fail-fast
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] byronellis commented on issue #21473: PVR_Spark2_Streaming perma-red
Posted by GitBox <gi...@apache.org>.
byronellis commented on issue #21473:
URL: https://github.com/apache/beam/issues/21473#issuecomment-1182628566
Okay, validatesPortabkeRunnerStreaming is also failing some fraction of tests as well...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] byronellis commented on issue #21473: PVR_Spark2_Streaming perma-red
Posted by GitBox <gi...@apache.org>.
byronellis commented on issue #21473:
URL: https://github.com/apache/beam/issues/21473#issuecomment-1183689891
Okay, after basically just running the tests over and over again... I think what must be happening is that Spark is getting torn down before the test actually completes which of course leads to a test failure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] byronellis commented on issue #21473: PVR_Spark2_Streaming perma-red
Posted by GitBox <gi...@apache.org>.
byronellis commented on issue #21473:
URL: https://github.com/apache/beam/issues/21473#issuecomment-1184808721
.take-issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] byronellis commented on issue #21473: PVR_Spark2_Streaming perma-red
Posted by GitBox <gi...@apache.org>.
byronellis commented on issue #21473:
URL: https://github.com/apache/beam/issues/21473#issuecomment-1186033572
Okay, after changing the timeout to be *even longer* it looks like it actually managed to run on Jenkins. I think it can just take awhile for some tests to spin up Spark, do their thing and then spin down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] byronellis commented on issue #21473: PVR_Spark2_Streaming perma-red
Posted by GitBox <gi...@apache.org>.
byronellis commented on issue #21473:
URL: https://github.com/apache/beam/issues/21473#issuecomment-1182610476
I tried running the Spark 2 tests locally and interestingly they seemed to pass (after I ran them under Java 8 rather than Java 11). I'm wondering if there's maybe just a misconfiguration of the Jenkins environment? (120 passed and 6 ignored)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] byronellis commented on issue #21473: PVR_Spark2_Streaming perma-red
Posted by GitBox <gi...@apache.org>.
byronellis commented on issue #21473:
URL: https://github.com/apache/beam/issues/21473#issuecomment-1182611318
Actually, I take that back... that was just for :runners:spark:2:test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org