You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Ning Zhang (Jira)" <ji...@apache.org> on 2020/11/18 05:20:00 UTC
[jira] [Commented] (KAFKA-10728) Mirroring data without
decompressing with MirrorMaker 2.0
[ https://issues.apache.org/jira/browse/KAFKA-10728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17234255#comment-17234255 ]
Ning Zhang commented on KAFKA-10728:
------------------------------------
my first impression is: unless the producer of MM2 is explicitly set to use `uncompressed` with [https://kafka.apache.org/documentation/#compression.type] it will use the default compression value
> Mirroring data without decompressing with MirrorMaker 2.0
> ---------------------------------------------------------
>
> Key: KAFKA-10728
> URL: https://issues.apache.org/jira/browse/KAFKA-10728
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
> Reporter: Eazhilan Nagarajan
> Priority: Major
>
> Hello,
>
> I use MirrorMaker 2.0 to copy data across two Kafka clusters and it's all working fine. Recently we enabled compressing while producing data into any topic which had a very positive impact on the storage and other resources but while mirroring, the data seems to be decompressed at the target Kafka cluster. I tried enabling compression using the below config in MM2, the data at the target cluster is compressed now, the decompress and re-compress continues to happen and it eats up a lot of resources unnecessarily.
>
> {noformat}
> - alias: my-passive-cluster
> authentication:
> passwordSecret:
> password: password
> secretName: passive-cluster-secret
> type: scram-sha-512
> username: user-1
> bootstrapServers: my-passive-cluster.com:443
> config:
> config.storage.replication.factor: 3
> offset.storage.replication.factor: 3
> status.storage.replication.factor: 3
> producer.compression.type: gzip{noformat}
> I found couple of Jira issues talking about it but I don't know if the shallow iterator option is available now.
> https://issues.apache.org/jira/browse/KAFKA-732, https://issues.apache.org/jira/browse/KAFKA-845
>
> Kindly let me if this is currently available or if it'll be available in the future.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)