You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Ryan En <Ry...@hotmail.com> on 2018/05/29 20:28:40 UTC
Mirrormaker producing to only one partition in a topic
Hi, I'm using Kafka version 0.10.2.0 and trying to use Mirrormaker to the messages from one Kafka cluster to another.
The source and target Kafka cluster are pretty much set up the same... replication factor is 3, number of partitions is 3, auto.create.topics.enable is true.
I am finding that in the target Kafka cluster, all the messages produced by Mirrormaker are going into one partition only.
Here are my configs:
mirrormaker producer
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxxxxxx" password="xxxxxxxxxx";
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=xxxxxxxxxxx
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=xxxxxxxxx
ssl.key.password=xxxxxxxx
retries=5000000
max.in.flight.requests.per.connection=1
client.id=xxxxxxxx
compression.type=lz4
bootstrap.servers=<aggregate-broker>:9292
acks=all
mirrormaker consumer
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxxxxxx" password="xxxxxxxxxx";
ssl.truststore.location=/path/to/truststore.jks
ssl.truststore.password=xxxxxxxxxx
ssl.keystore.location=/path/to/keystore.jks
ssl.keystore.password=xxxxxxxxxx
ssl.key.password=xxxxxxxxx
client.id=xxxxxxxxxxxx
partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
auto.commit.interval.ms=1000
auto.offset.reset=earliest
bootstrap.servers=<local-broker>:9192
session.timeout.ms=30000
group.id=xxxxxxxxxxxxxx
enable.auto.commit=false
Any ideas why this may happen?
Thanks in advance
Re: Mirrormaker producing to only one partition in a topic
Posted by Stephen Powis <sp...@salesforce.com>.
Hey Ryan,
I ran into a similar issue and it was how the RoundRobinAssignor/Partitioner
was hashing the keys in my messages. You may want to look at how thats
implemented and see if its causing all of your messages to end up in the
same partition.
For what its worth, this ticket has the implementation I put together to
deal with it: https://issues.apache.org/jira/browse/KAFKA-3333
Stephen
On Wed, May 30, 2018 at 5:28 AM, Ryan En <Ry...@hotmail.com> wrote:
> Hi, I'm using Kafka version 0.10.2.0 and trying to use Mirrormaker to the
> messages from one Kafka cluster to another.
>
>
> The source and target Kafka cluster are pretty much set up the same...
> replication factor is 3, number of partitions is 3,
> auto.create.topics.enable is true.
>
>
> I am finding that in the target Kafka cluster, all the messages produced
> by Mirrormaker are going into one partition only.
>
>
> Here are my configs:
>
>
> mirrormaker producer
>
>
> security.protocol=SASL_SSL
> sasl.mechanism=PLAIN
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
> required username="xxxxxxxx" password="xxxxxxxxxx";
> ssl.truststore.location=/path/to/truststore.jks
> ssl.truststore.password=xxxxxxxxxxx
> ssl.keystore.location=/path/to/keystore.jks
> ssl.keystore.password=xxxxxxxxx
> ssl.key.password=xxxxxxxx
> retries=5000000
> max.in.flight.requests.per.connection=1
> client.id=xxxxxxxx
> compression.type=lz4
> bootstrap.servers=<aggregate-broker>:9292
> acks=all
>
>
> mirrormaker consumer
>
>
> security.protocol=SASL_SSL
> sasl.mechanism=PLAIN
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule
> required username="xxxxxxxx" password="xxxxxxxxxx";
> ssl.truststore.location=/path/to/truststore.jks
> ssl.truststore.password=xxxxxxxxxx
> ssl.keystore.location=/path/to/keystore.jks
> ssl.keystore.password=xxxxxxxxxx
> ssl.key.password=xxxxxxxxx
> client.id=xxxxxxxxxxxx
> partition.assignment.strategy=org.apache.kafka.clients.
> consumer.RoundRobinAssignor
> auto.commit.interval.ms=1000
> auto.offset.reset=earliest
> bootstrap.servers=<local-broker>:9192
> session.timeout.ms=30000
> group.id=xxxxxxxxxxxxxx
> enable.auto.commit=false
>
>
> Any ideas why this may happen?
>
>
> Thanks in advance
>
Re: Mirrormaker producing to only one partition in a topic
Posted by br...@binance.com,
br...@binance.com.
On 2018/05/29 20:28:40, Ryan En <Ry...@hotmail.com> wrote:
> Hi, I'm using Kafka version 0.10.2.0 and trying to use Mirrormaker to the messages from one Kafka cluster to another.
>
>
> The source and target Kafka cluster are pretty much set up the same... replication factor is 3, number of partitions is 3, auto.create.topics.enable is true.
>
>
> I am finding that in the target Kafka cluster, all the messages produced by Mirrormaker are going into one partition only.
>
>
> Here are my configs:
>
>
> mirrormaker producer
>
>
> security.protocol=SASL_SSL
> sasl.mechanism=PLAIN
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxxxxxx" password="xxxxxxxxxx";
> ssl.truststore.location=/path/to/truststore.jks
> ssl.truststore.password=xxxxxxxxxxx
> ssl.keystore.location=/path/to/keystore.jks
> ssl.keystore.password=xxxxxxxxx
> ssl.key.password=xxxxxxxx
> retries=5000000
> max.in.flight.requests.per.connection=1
> client.id=xxxxxxxx
> compression.type=lz4
> bootstrap.servers=<aggregate-broker>:9292
> acks=all
>
>
> mirrormaker consumer
>
>
> security.protocol=SASL_SSL
> sasl.mechanism=PLAIN
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxxxxxx" password="xxxxxxxxxx";
> ssl.truststore.location=/path/to/truststore.jks
> ssl.truststore.password=xxxxxxxxxx
> ssl.keystore.location=/path/to/keystore.jks
> ssl.keystore.password=xxxxxxxxxx
> ssl.key.password=xxxxxxxxx
> client.id=xxxxxxxxxxxx
> partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
> auto.commit.interval.ms=1000
> auto.offset.reset=earliest
> bootstrap.servers=<local-broker>:9192
> session.timeout.ms=30000
> group.id=xxxxxxxxxxxxxx
> enable.auto.commit=false
>
>
> Any ideas why this may happen?
>
>
> Thanks in advance
> i met the same question as yours. have you solved it by adding some configs. very thank for your share
Re: Mirrormaker producing to only one partition in a topic
Posted by br...@binance.com,
br...@binance.com.
have you solved this problem ? how do you solved... tanks reply
On 2018/05/29 20:28:40, Ryan En <Ry...@hotmail.com> wrote:
> Hi, I'm using Kafka version 0.10.2.0 and trying to use Mirrormaker to the messages from one Kafka cluster to another.
>
>
> The source and target Kafka cluster are pretty much set up the same... replication factor is 3, number of partitions is 3, auto.create.topics.enable is true.
>
>
> I am finding that in the target Kafka cluster, all the messages produced by Mirrormaker are going into one partition only.
>
>
> Here are my configs:
>
>
> mirrormaker producer
>
>
> security.protocol=SASL_SSL
> sasl.mechanism=PLAIN
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxxxxxx" password="xxxxxxxxxx";
> ssl.truststore.location=/path/to/truststore.jks
> ssl.truststore.password=xxxxxxxxxxx
> ssl.keystore.location=/path/to/keystore.jks
> ssl.keystore.password=xxxxxxxxx
> ssl.key.password=xxxxxxxx
> retries=5000000
> max.in.flight.requests.per.connection=1
> client.id=xxxxxxxx
> compression.type=lz4
> bootstrap.servers=<aggregate-broker>:9292
> acks=all
>
>
> mirrormaker consumer
>
>
> security.protocol=SASL_SSL
> sasl.mechanism=PLAIN
> sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="xxxxxxxx" password="xxxxxxxxxx";
> ssl.truststore.location=/path/to/truststore.jks
> ssl.truststore.password=xxxxxxxxxx
> ssl.keystore.location=/path/to/keystore.jks
> ssl.keystore.password=xxxxxxxxxx
> ssl.key.password=xxxxxxxxx
> client.id=xxxxxxxxxxxx
> partition.assignment.strategy=org.apache.kafka.clients.consumer.RoundRobinAssignor
> auto.commit.interval.ms=1000
> auto.offset.reset=earliest
> bootstrap.servers=<local-broker>:9192
> session.timeout.ms=30000
> group.id=xxxxxxxxxxxxxx
> enable.auto.commit=false
>
>
> Any ideas why this may happen?
>
>
> Thanks in advance
>