You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Thomas Thornton <tt...@salesforce.com.INVALID> on 2024/11/20 17:49:33 UTC
Data Corruption in Netty 4.1.111.Final (Kafka 3.9.0)
With the completion of KAFKA-17046
<https://issues.apache.org/jira/browse/KAFKA-17046>, the netty version has
been upgraded to 4.1.111.Final in Kafka 3.9.0. This netty version has a
known issue with data corruption, see netty issue
<https://github.com/netty/netty/issues/14126> and grpc-java issue
<https://github.com/grpc/grpc-java/issues/11284>. It is localized to
grpc-java. This causes data corruption in any Kafka application that uses
grpc-java (e.g., debezium-vitess-connector which runs on Kafka Connect). We
should upgrade to a newer version to avoid this data corruption. I
requested Jira access to open a ticket for this.
For anyone who is trying to resolve this, the workaround is manually
removing these 4.1.111.Final netty dependencies in the /kafka/libs
directory and installing another netty version, eg:
```
RUN rm -f /kafka/libs/netty-codec-4.1.111.Final.jar
RUN curl -sfSL -o /kafka/libs/netty-codec-4.1.110.Final.jar
https://repo1.maven.org/maven2/io/netty/netty-codec/4.1.110.Final/netty-codec-4.1.110.Final.jar
```
You need to repeat this for all of the netty packages.
Re: Data Corruption in Netty 4.1.111.Final (Kafka 3.9.0)
Posted by Thomas Thornton <tt...@salesforce.com.INVALID>.
This was fixed with this PR https://github.com/apache/kafka/pull/17860
On Wed, Nov 20, 2024 at 9:49 AM Thomas Thornton <tt...@salesforce.com>
wrote:
> With the completion of KAFKA-17046
> <https://issues.apache.org/jira/browse/KAFKA-17046>, the netty version
> has been upgraded to 4.1.111.Final in Kafka 3.9.0. This netty version has a
> known issue with data corruption, see netty issue
> <https://github.com/netty/netty/issues/14126> and grpc-java issue
> <https://github.com/grpc/grpc-java/issues/11284>. It is localized to
> grpc-java. This causes data corruption in any Kafka application that uses
> grpc-java (e.g., debezium-vitess-connector which runs on Kafka Connect). We
> should upgrade to a newer version to avoid this data corruption. I
> requested Jira access to open a ticket for this.
>
> For anyone who is trying to resolve this, the workaround is manually
> removing these 4.1.111.Final netty dependencies in the /kafka/libs
> directory and installing another netty version, eg:
>
> ```
> RUN rm -f /kafka/libs/netty-codec-4.1.111.Final.jar
> RUN curl -sfSL -o /kafka/libs/netty-codec-4.1.110.Final.jar
> https://repo1.maven.org/maven2/io/netty/netty-codec/4.1.110.Final/netty-codec-4.1.110.Final.jar
> ```
>
> You need to repeat this for all of the netty packages.
>