You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 12:47:21 UTC
[GitHub] [beam] damccorm opened a new issue, #19704: Avro incompatibilities with Spark 2.2 and Spark 2.3
damccorm opened a new issue, #19704:
URL: https://github.com/apache/beam/issues/19704
Much of the code that depends on Avro (notably the wrappers built with [BeamSQL](https://github.com/apache/beam/blob/ae83448597f64474c3f5754d7b8e3f6b02347a6b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java#L34) but also [some](https://github.com/apache/beam/blob/ae83448597f64474c3f5754d7b8e3f6b02347a6b/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java) [connectors](https://github.com/apache/beam/blob/ae83448597f64474c3f5754d7b8e3f6b02347a6b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java#L42)) require a version \> 1.8.x
This library is not present in Spark 2.2 and Spark 2.3 clusters, which are meant to be supported. These pipelines will fail with ClassNotFoundException / MethodNotFoundExceptions.
Spark 2.4**** should be unaffected.
Relocating or vendoring is probably not appropriate, since Avro is frequently exposed in the API through parameters and potentially in generated specific records.
Imported from Jira [BEAM-7979](https://issues.apache.org/jira/browse/BEAM-7979). Original Jira may contain additional context.
Reported by: rskraba.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev commented on issue #19704: Avro incompatibilities with Spark 2.2 and Spark 2.3
Posted by GitBox <gi...@apache.org>.
aromanenko-dev commented on issue #19704:
URL: https://github.com/apache/beam/issues/19704#issuecomment-1290726269
Spark 2.* support is deprecated in Beam and will be completely removed soon.
So, no need to fix this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] aromanenko-dev closed issue #19704: Avro incompatibilities with Spark 2.2 and Spark 2.3
Posted by GitBox <gi...@apache.org>.
aromanenko-dev closed issue #19704: Avro incompatibilities with Spark 2.2 and Spark 2.3
URL: https://github.com/apache/beam/issues/19704
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org