You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Danny McCormick (Jira)" <ji...@apache.org> on 2022/06/04 12:48:00 UTC

[jira] [Commented] (BEAM-7979) Avro incompatibilities with Spark 2.2 and Spark 2.3

    [ https://issues.apache.org/jira/browse/BEAM-7979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17548048#comment-17548048 ] 

Danny McCormick commented on BEAM-7979:
---------------------------------------

This issue has been migrated to https://github.com/apache/beam/issues/19704

> Avro incompatibilities with Spark 2.2 and Spark 2.3
> ---------------------------------------------------
>
>                 Key: BEAM-7979
>                 URL: https://issues.apache.org/jira/browse/BEAM-7979
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-gcp, io-java-parquet, sdk-java-core
>            Reporter: Ryan Skraba
>            Priority: P3
>              Labels: Clarified
>
> Much of the code that depends on Avro (notably the wrappers built with [BeamSQL|https://github.com/apache/beam/blob/ae83448597f64474c3f5754d7b8e3f6b02347a6b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java#L34] but also [some|https://github.com/apache/beam/blob/ae83448597f64474c3f5754d7b8e3f6b02347a6b/sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/ParquetIO.java] [connectors|https://github.com/apache/beam/blob/ae83448597f64474c3f5754d7b8e3f6b02347a6b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java#L42]) require a version > 1.8.x
> This library is not present in Spark 2.2 and Spark 2.3 clusters, which are meant to be supported.  These pipelines will fail with ClassNotFoundException / MethodNotFoundExceptions.
> Spark 2.4+ should be unaffected.
> Relocating or vendoring is probably not appropriate, since Avro is frequently exposed in the API through parameters and potentially in generated specific records.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)