You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/10/06 15:26:46 UTC
[GitHub] [beam] mosche opened a new pull request, #23522: Downgrade Scala version in Spark job-server to prevent Scala serialization bug
mosche opened a new pull request, #23522:
URL: https://github.com/apache/beam/pull/23522
Downgrade the Scala version of the job-server to match the Scala version of a Spark 3.1.2 cluster to prevent
a Scala bug (InvalidClassException when deserializing WrappedArray)
(fixes #21092)
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23522: Downgrade Scala version in Spark job-server to prevent Scala serialization bug
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23522:
URL: https://github.com/apache/beam/pull/23522#issuecomment-1274309637
Yes, we deprecated it ... but apparently it cannot be used anymore since long.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23522: Downgrade Scala version in Spark job-server to prevent Scala serialization bug
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23522:
URL: https://github.com/apache/beam/pull/23522#issuecomment-1270270184
R: @aromanenko-dev
R: @ibzib
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #23522: Downgrade Scala version in Spark job-server to prevent Scala serialization bug
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #23522:
URL: https://github.com/apache/beam/pull/23522#issuecomment-1270272009
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23522: Downgrade Scala version in Spark job-server to prevent Scala serialization bug
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23522:
URL: https://github.com/apache/beam/pull/23522#issuecomment-1274078219
@aromanenko-dev ping, should be quick to review ... just a small change to force the right Scala version
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23522: Downgrade Scala version in Spark job-server to prevent Scala serialization bug
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23522:
URL: https://github.com/apache/beam/pull/23522#issuecomment-1274280984
Run Python Spark ValidatesRunner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] mosche commented on pull request #23522: Downgrade Scala version in Spark job-server to prevent Scala serialization bug
Posted by GitBox <gi...@apache.org>.
mosche commented on PR #23522:
URL: https://github.com/apache/beam/pull/23522#issuecomment-1274290020
@aromanenko-dev Unfortunately there's no tests that would cover this ... such classpath issues can only be detected if NOT running Spark in local mode.
I just noticed that the Spark 2 job-server / runner is broken as well, what a mess 😠😡
```
java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.type.TypeBindings.emptyBindings()Lcom/fasterxml/jackson/databind/type/TypeBindings;
at org.apache.beam.sdk.options.PipelineOptionsFactory.createBeanProperty(PipelineOptionsFactory.java:1708)
at org.apache.beam.sdk.options.PipelineOptionsFactory.computeDeserializerForMethod(PipelineOptionsFactory.java:1732)
at java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1660)
at org.apache.beam.sdk.options.PipelineOptionsFactory.getDeserializerForMethod(PipelineOptionsFactory.java:1782)
at org.apache.beam.sdk.options.PipelineOptionsFactory.deserializeNode(PipelineOptionsFactory.java:1806)
```
That issue is even worse, I'm pretty convinced this isn't only affecting the portable runner. Attempting to deserialize pipeline options will fail on the cluster using either `PortableRunner` or `SparkRunner` because the expected Jackson API is newer than the one available on the Spark cluster
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev merged pull request #23522: Downgrade Scala version in Spark job-server to prevent Scala serialization bug
Posted by GitBox <gi...@apache.org>.
aromanenko-dev merged PR #23522:
URL: https://github.com/apache/beam/pull/23522
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on pull request #23522: Downgrade Scala version in Spark job-server to prevent Scala serialization bug
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on PR #23522:
URL: https://github.com/apache/beam/pull/23522#issuecomment-1274306306
@mosche Do we still support Spark 2 runner? I believed it was deprecated and finally will be dropped out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org