You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "lk-1984 (via GitHub)" <gi...@apache.org> on 2025/03/19 10:20:08 UTC
[I] Iceberg Kafka Connect Sink has AVRO 1.12.0, but Kafka Connect AVRO Converter wants 1.11.4 [iceberg]
lk-1984 opened a new issue, #12571:
URL: https://github.com/apache/iceberg/issues/12571
### Apache Iceberg version
1.8.1 (latest release)
### Query engine
None
### Please describe the bug 🐞
I am trying to setup the converter for AVRO messages. I have built the Iceberg Kafka Connect Sink to read a Kafka topic, and write into Iceberg table.
Now, I have a problem there.
```
2025-03-19 08:35:44 [2025-03-19 06:35:44,643] ERROR [iceberg-sink-connector|task-0] WorkerSinkTask{id=iceberg-sink-connector-0} Task threw an uncaught and unrecoverable exception. Task is being killed and will not recover until manually restarted (org.apache.kafka.connect.runtime.WorkerTask:234)
2025-03-19 08:35:44 java.lang.LinkageError: loader constraint violation: when resolving method 'org.apache.avro.generic.GenericContainer io.confluent.kafka.serializers.GenericContainerWithVersion.container()' the class loader org.apache.kafka.connect.runtime.isolation.PluginClassLoader @36902638 of the current class, io/confluent/connect/avro/AvroConverter, and the class loader 'app' for the method's defining class, io/confluent/kafka/serializers/GenericContainerWithVersion, have different Class objects for the type org/apache/avro/generic/GenericContainer used in the signature (io.confluent.connect.avro.AvroConverter is in unnamed module of loader org.apache.kafka.connect.runtime.isolation.PluginClassLoader @36902638, parent loader org.apache.kafka.connect.runtime.isolation.DelegatingClassLoader @dd8ba08; io.confluent.kafka.serializers.GenericContainerWithVersion is in unnamed module of loader 'app')
```
I have added `kafka-connect-avro-converter-7.9.0.jar` which has the `io.confluent.connect.avro.AvroConverter`.
The `kafka-connect-avro-converter` wants Avro 1.11.4, but Iceberg 1.8.1 has already Avro 1.12.0 for the Iceberg Kafka Connect Sink. I have a feeling that this is causing the error above.
My JARs.
```
./jaxb-impl-2.2.3-1.jar
./jetty-webapp-9.4.53.v20231009.jar
./glue-2.29.45.jar
./curator-client-5.2.0.jar
./threeten-extra-1.7.1.jar
./hive-shims-2.3.9.jar
./annotations-2.29.45.jar
./annotations-4.1.1.4.jar
./httpcore-4.4.16.jar
./hadoop-yarn-common-3.4.1.jar
./gax-httpjson-2.58.0.jar
./kerby-pkix-2.0.3.jar
./http-auth-spi-2.29.45.jar
./jna-5.13.0.jar
./reload4j-1.2.22.jar
./grpc-protobuf-lite-1.68.1.jar
./google-auth-library-credentials-1.30.0.jar
./google-api-client-2.7.0.jar
./http-client-spi-2.29.45.jar
./retries-2.29.45.jar
./netty-all-4.1.100.Final.jar
./grpc-protobuf-1.68.1.jar
./google-auth-library-oauth2-http-1.30.0.jar
./netty-codec-redis-4.1.100.Final.jar
./opencensus-proto-0.2.0.jar
./javolution-5.5.1.jar
./jersey-json-1.22.0.jar
./http-auth-aws-2.29.45.jar
./google-api-services-storage-v1-rev20241008-2.0.0.jar
./jta-1.1.jar
./opentelemetry-context-1.42.1.jar
./lakeformation-2.29.45.jar
./jackson-datatype-jsr310-2.18.2.jar
./sts-2.29.45.jar
./oauth2-oidc-sdk-11.18.jar
./metrics-spi-2.29.45.jar
./aopalliance-1.0.jar
./jackson-module-jaxb-annotations-2.18.2.jar
./websocket-api-9.4.53.v20231009.jar
./datanucleus-api-jdo-4.2.4.jar
./hadoop-mapreduce-client-common-3.4.1.jar
./netty-codec-http-4.1.115.Final.jar
./gson-2.11.0.jar
./iceberg-common-1.8.0-SNAPSHOT.jar
./jackson-annotations-2.18.2.jar
./caffeine-2.9.3.jar
./dnsjava-3.6.1.jar
./jetty-util-ajax-9.4.53.v20231009.jar
./jackson-databind-2.18.2.jar
./netty-buffer-4.1.115.Final.jar
./javax.jdo-3.2.0-m3.jar
./netty-codec-stomp-4.1.100.Final.jar
./json-utils-2.29.45.jar
./grpc-rls-1.68.1.jar
./commons-codec-1.17.1.jar
./netty-transport-native-epoll-4.1.115.Final-linux-x86_64.jar
./dynamodb-2.29.45.jar
./checksums-spi-2.29.45.jar
./google-http-client-1.45.0.jar
./javax.annotation-api-1.3.2.jar
./auth-2.29.45.jar
./slf4j-reload4j-1.7.36.jar
./protobuf-java-util-4.28.3.jar
./netty-common-4.1.115.Final.jar
./kerby-asn1-2.0.3.jar
./woodstox-core-6.7.0.jar
./google-cloud-core-2.48.0.jar
./azure-storage-common-12.28.0.jar
./iceberg-kafka-connect-1.8.0-SNAPSHOT.jar
./http-auth-2.29.45.jar
./commons-cli-1.5.0.jar
./netty-transport-classes-epoll-4.1.115.Final.jar
./zookeeper-jute-3.8.4.jar
./netty-codec-socks-4.1.115.Final.jar
./hadoop-hdfs-client-3.4.1.jar
./netty-codec-dns-4.1.112.Final.jar
./endpoints-spi-2.29.45.jar
./api-common-2.41.0.jar
./regions-2.29.45.jar
./opentelemetry-api-1.42.1.jar
./curator-framework-5.2.0.jar
./annotations-17.0.0.jar
./commons-logging-1.3.0.jar
./identity-spi-2.29.45.jar
./libthrift-0.9.3.jar
./kerby-config-2.0.3.jar
./guice-4.2.3.jar
./parquet-jackson-1.15.0.jar
./hive-common-2.3.9.jar
./netty-codec-smtp-4.1.100.Final.jar
./netty-transport-4.1.115.Final.jar
./antlr-runtime-3.5.2.jar
./reactor-netty-http-1.0.48.jar
./error_prone_annotations-2.35.1.jar
./kerb-util-2.0.3.jar
./netty-handler-proxy-4.1.115.Final.jar
./animal-sniffer-annotations-1.24.jar
./aws-query-protocol-2.29.45.jar
./grpc-core-1.68.1.jar
./hadoop-shaded-guava-1.3.0.jar
./hadoop-hdfs-3.4.1.jar
./netty-codec-mqtt-4.1.100.Final.jar
./iam-2.29.45.jar
./protocol-core-2.29.45.jar
./azure-storage-file-datalake-12.22.0.jar
./opencsv-2.3.jar
./opentelemetry-semconv-1.25.0-alpha.jar
./netty-handler-4.1.115.Final.jar
./asm-9.6.jar
./grpc-alts-1.68.1.jar
./utils-2.29.45.jar
./zookeeper-3.8.4.jar
./azure-json-1.3.0.jar
./datanucleus-core-4.1.17.jar
./grpc-auth-1.68.1.jar
./commons-net-3.9.0.jar
./opencensus-api-0.31.1.jar
./logback-core-1.2.13.jar
./ant-1.9.1.jar
./hadoop-annotations-3.4.1.jar
./hadoop-mapreduce-client-jobclient-3.4.1.jar
./javax.servlet-api-3.1.0.jar
./iceberg-hive-metastore-1.8.0-SNAPSHOT.jar
./jline-3.9.0.jar
./commons-io-2.16.1.jar
./opentelemetry-sdk-trace-1.42.1.jar
./hadoop-shaded-protobuf_3_25-1.3.0.jar
./iceberg-aws-1.8.0-SNAPSHOT.jar
./perfmark-api-0.27.0.jar
./orc-core-1.9.5-nohive.jar
./lang-tag-1.7.jar
./proto-google-iam-v1-1.44.0.jar
./jetty-util-9.4.53.v20231009.jar
./protobuf-java-4.28.3.jar
./aws-json-protocol-2.29.45.jar
./shared-resourcemapping-0.32.0.jar
./azure-storage-blob-12.29.0.jar
./websocket-common-9.4.53.v20231009.jar
./netty-transport-rxtx-4.1.100.Final.jar
./content-type-2.3.jar
./iceberg-api-1.8.0-SNAPSHOT.jar
./aircompressor-2.0.2.jar
./grpc-netty-shaded-1.68.1.jar
./netty-transport-native-kqueue-4.1.115.Final-osx-x86_64.jar
./jcip-annotations-1.0-1.jar
./hadoop-yarn-client-3.4.1.jar
./nimbus-jose-jwt-9.40.jar
./arns-2.29.45.jar
./azure-identity-1.14.2.jar
./orc-core-1.9.5.jar
./jetty-io-9.4.53.v20231009.jar
./checksums-2.29.45.jar
./gax-grpc-2.58.0.jar
./opentelemetry-gcp-resources-1.37.0-alpha.jar
./netty-resolver-dns-native-macos-4.1.112.Final-osx-aarch_64.jar
./netty-codec-haproxy-4.1.100.Final.jar
./kafka-connect-avro-converter-7.9.0.jar
./sso-2.29.45.jar
./google-oauth-client-1.36.0.jar
./zstd-jni-1.5.6-6.jar
./commons-pool-1.6.jar
./commons-compress-1.27.1.jar
./opentelemetry-api-incubator-1.42.1-alpha.jar
./profiles-2.29.45.jar
./hadoop-auth-3.4.1.jar
./netty-tcnative-classes-2.0.69.Final.jar
./failureaccess-1.0.2.jar
./netty-transport-native-kqueue-4.1.115.Final-osx-aarch_64.jar
./netty-codec-xml-4.1.100.Final.jar
./websocket-client-9.4.53.v20231009.jar
./stax2-api-4.2.2.jar
./google-http-client-appengine-1.45.0.jar
./kms-2.29.45.jar
./commons-dbcp-1.4.jar
./netty-handler-ssl-ocsp-4.1.100.Final.jar
./proto-google-common-protos-2.49.0.jar
./iceberg-azure-1.8.0-SNAPSHOT.jar
./RoaringBitmap-1.3.0.jar
./hadoop-aws-3.4.1.jar
./proto-google-cloud-storage-v2-2.45.0-beta.jar
./opentelemetry-sdk-logs-1.42.1.jar
./netty-resolver-dns-native-macos-4.1.112.Final-osx-x86_64.jar
./datanucleus-rdbms-4.1.19.jar
./grpc-api-1.68.1.jar
./derby-10.10.2.0.jar
./grpc-services-1.68.1.jar
./azure-core-http-netty-1.15.7.jar
./azure-xml-1.1.0.jar
./listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar
./opencensus-contrib-http-util-0.31.1.jar
./metrics-core-3.2.4.jar
./jdo-api-3.0.1.jar
./jsp-api-2.1.jar
./kerb-core-2.0.3.jar
./azure-core-1.54.1.jar
./hive-metastore-2.3.9.jar
./opentelemetry-sdk-common-1.42.1.jar
./apache-client-2.29.45.jar
./jaxb-api-2.2.12.jar
./grpc-inprocess-1.68.1.jar
./grpc-stub-1.68.1.jar
./proto-google-cloud-monitoring-v3-3.55.0.jar
./jakarta.activation-api-1.2.2.jar
./j2objc-annotations-3.0.0.jar
./commons-math3-3.6.1.jar
./snappy-java-1.1.10.7.jar
./crt-core-2.29.45.jar
./jna-platform-5.13.0.jar
./netty-transport-sctp-4.1.100.Final.jar
./grpc-opentelemetry-1.68.1.jar
./netty-transport-native-epoll-4.1.115.Final.jar
./orc-shims-1.9.5.jar
./iceberg-orc-1.8.0-SNAPSHOT.jar
./grpc-context-1.68.1.jar
./jackson-jaxrs-json-provider-2.18.2.jar
./netty-resolver-dns-4.1.112.Final.jar
./gapic-google-cloud-storage-v2-2.45.0-beta.jar
./jetty-servlet-9.4.53.v20231009.jar
./slf4j-api-2.0.16.jar
./commons-text-1.11.0.jar
./libfb303-0.9.3.jar
./commons-collections-3.2.2.jar
./kerby-util-2.0.3.jar
./s3-2.29.45.jar
./conscrypt-openjdk-uber-2.5.2.jar
./aws-core-2.29.45.jar
./netty-codec-http2-4.1.115.Final.jar
./detector-resources-support-0.32.0.jar
./joda-time-2.8.1.jar
./jettison-1.5.4.jar
./json-smart-2.5.1.jar
./netty-transport-native-epoll-4.1.115.Final-linux-aarch_64.jar
./checker-qual-3.47.0.jar
./iceberg-kafka-connect-events-1.8.0-SNAPSHOT.jar
./iceberg-gcp-1.8.0-SNAPSHOT.jar
./httpclient5-5.4.1.jar
./httpcore5-5.3.1.jar
./parquet-encoding-1.15.0.jar
./hive-serde-2.3.9.jar
./hadoop-client-3.4.1.jar
./ant-launcher-1.9.1.jar
./hive-shims-common-2.3.9.jar
./bcprov-jdk18on-1.78.1.jar
./s3-transfer-manager-2.31.1.jar
./audience-annotations-0.12.0.jar
./avro-1.12.0.jar
./msal4j-1.17.2.jar
./reactive-streams-1.0.4.jar
./curator-recipes-5.2.0.jar
./jline-2.12.jar
./metrics-jvm-3.1.0.jar
./commons-beanutils-1.9.4.jar
./eventstream-1.0.1.jar
./kerb-crypto-2.0.3.jar
./grpc-googleapis-1.68.1.jar
./aws-crt-0.33.3.jar
./netty-resolver-4.1.115.Final.jar
./opentelemetry-sdk-1.42.1.jar
./sdk-core-2.29.45.jar
./auto-value-annotations-1.11.0.jar
./accessors-smart-2.5.1.jar
./google-cloud-monitoring-3.55.0.jar
./msal4j-persistence-extension-1.3.0.jar
./jsch-0.1.55.jar
./netty-transport-udt-4.1.100.Final.jar
./jetty-security-9.4.53.v20231009.jar
./grpc-util-1.68.1.jar
./netty-codec-memcache-4.1.100.Final.jar
./commons-configuration2-2.10.1.jar
./grpc-google-cloud-storage-v2-2.45.0-beta.jar
./exporter-metrics-0.31.0.jar
./google-cloud-storage-2.45.0.jar
./bonecp-0.8.0.RELEASE.jar
./reactor-core-3.4.41.jar
./google-cloud-core-grpc-2.48.0.jar
./commons-lang-2.6.jar
./logback-classic-1.2.13.jar
./parquet-column-1.15.0.jar
./iceberg-core-1.8.0-SNAPSHOT.jar
./retries-spi-2.29.45.jar
./iceberg-parquet-1.8.0-SNAPSHOT.jar
./google-http-client-jackson2-1.45.0.jar
./hadoop-common-3.4.1.jar
./gax-2.58.0.jar
./httpclient-4.5.13.jar
./netty-nio-client-2.29.45.jar
./netty-transport-classes-kqueue-4.1.115.Final.jar
./jackson-jaxrs-base-2.18.2.jar
./jackson-core-2.18.2.jar
./azure-storage-internal-avro-12.14.0.jar
./commons-lang3-3.16.0.jar
./metrics-json-3.1.0.jar
./netty-transport-native-unix-common-4.1.115.Final.jar
./jetty-http-9.4.53.v20231009.jar
./google-http-client-apache-v2-1.45.0.jar
./threetenbp-1.7.0.jar
./hive-storage-api-2.4.0.jar
./parquet-hadoop-1.15.0.jar
./hadoop-yarn-api-3.4.1.jar
./jsr305-3.0.2.jar
./parquet-common-1.15.0.jar
./third-party-jackson-core-2.29.45.jar
./netty-codec-4.1.115.Final.jar
./netty-resolver-dns-classes-macos-4.1.112.Final.jar
./opentelemetry-sdk-extension-autoconfigure-spi-1.42.1.jar
./aws-xml-protocol-2.29.45.jar
./guava-33.3.1-jre.jar
./iceberg-bundled-guava-1.8.0-SNAPSHOT.jar
./re2j-1.7.jar
./iceberg-data-1.8.0-SNAPSHOT.jar
./jetty-xml-9.4.53.v20231009.jar
./grpc-grpclb-1.68.1.jar
./javax.inject-1.jar
./grpc-s2a-1.68.1.jar
./jetty-client-9.4.53.v20231009.jar
./hadoop-mapreduce-client-core-3.4.1.jar
./google-cloud-core-http-2.48.0.jar
./netty-tcnative-boringssl-static-2.0.69.Final.jar
./hive-shims-0.23-2.3.9.jar
./grpc-xds-1.68.1.jar
./google-http-client-gson-1.45.0.jar
./parquet-avro-1.15.0.jar
./reactor-netty-core-1.0.48.jar
./http-auth-aws-eventstream-2.29.45.jar
./parquet-format-structures-1.15.0.jar
./opentelemetry-sdk-metrics-1.42.1.jar
./hive-shims-scheduler-2.3.9.jar
./httpcore5-h2-5.3.1.jar
./http-auth-aws-crt-2.29.45.jar
./failsafe-3.3.2.jar
```
### Willingness to contribute
- [ ] I can contribute a fix for this bug independently
- [ ] I would be willing to contribute a fix for this bug with guidance from the Iceberg community
- [ ] I cannot contribute a fix for this bug at this time
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Kafka Connect Sink has AVRO 1.12.0, but Kafka Connect AVRO Converter 7.9.0 wants 1.11.4 and therefore Avro Converter does not work [iceberg]
Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 closed issue #12571: Iceberg Kafka Connect Sink has AVRO 1.12.0, but Kafka Connect AVRO Converter 7.9.0 wants 1.11.4 and therefore Avro Converter does not work
URL: https://github.com/apache/iceberg/issues/12571
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Iceberg Kafka Connect Sink has AVRO 1.12.0, but Kafka Connect AVRO Converter 7.9.0 wants 1.11.4 and therefore Avro Converter does not work [iceberg]
Posted by "lk-1984 (via GitHub)" <gi...@apache.org>.
lk-1984 commented on issue #12571:
URL: https://github.com/apache/iceberg/issues/12571#issuecomment-2742481491
This happens as there are two different versions of Avro in two of the JARs.. I removed the 1.12.0 one, and it works.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org