You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by Kamal C <ka...@gmail.com> on 2016/06/08 13:13:52 UTC
Questions about Kafka Scripts
Hi,
A. What is the usage of `kafka-replica-verification.sh` script ? I don't
find any documentations about it in [1] and [2].
I have a topic `test` with 10 partitions. Ran the above script, it
continuously prints the below results.
[kamal@tcltest1 bin]$ sh kafka-replica-verification.sh --time -2
--topic-white-list test --report-interval-ms 1000 --broker-list
localhost:9092
2016-06-07 12:49:13,664: verification process is started.
2016-06-07 12:49:14,632: max lag is 0 for partition [test,6] at offset
39008 among 10 partitions
2016-06-07 12:49:15,633: max lag is 0 for partition [test,6] at offset
39008 among 10 partitions
2016-06-07 12:49:16,634: max lag is 0 for partition [test,6] at offset
39008 among 10 partitions
2016-06-07 12:49:17,635: max lag is 0 for partition [test,6] at offset
39008 among 10 partitions
2016-06-07 12:49:18,636: max lag is 0 for partition [test,6] at offset
39008 among 10 partitions
2016-06-07 12:49:19,637: max lag is 0 for partition [test,6] at offset
39008 among 10 partitions
B. `kafka-replay-log-producer.sh` script throws ConsumerTimeoutException.
What is wrong with the below command ?
[kamal@tcltest1 bin]$ sh kafka-replay-log-producer.sh --broker-list
localhost:9092 --inputtopic test --outputtopic xyz --reporting-interval
1000 --zookeeper localhost:2181
[2016-06-07 12:42:10,925] ERROR consumer thread timing out
(kafka.tools.ReplayLogProducer$ZKConsumerThread)
kafka.consumer.ConsumerTimeoutException
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:69)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
at
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:64)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:56)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at kafka.consumer.KafkaStream.foreach(KafkaStream.scala:25)
at
kafka.tools.ReplayLogProducer$ZKConsumerThread.run(ReplayLogProducer.scala:140)
[1]: http://kafka.apache.org/documentation.html
[2]: https://cwiki.apache.org/confluence/display/KAFKA/System+Tools
Re: Questions about Kafka Scripts
Posted by Gwen Shapira <gw...@confluent.io>.
[A]
Unfortunately, we only documented this in the code:
/**
* For verifying the consistency among replicas.
*
* 1. start a fetcher on every broker.
* 2. each fetcher does the following
* 2.1 issues fetch request
* 2.2 puts the fetched result in a shared buffer
* 2.3 waits for all other fetchers to finish step 2.2
* 2.4 one of the fetchers verifies the consistency of fetched
results among replicas
*
* The consistency verification is up to the high watermark. The tool
reports the
* max lag between the verified offset and the high watermark among
all partitions.
*
* If a broker goes down, the verification of the partitions on that
broker is delayed
* until the broker is up again.
*
* Caveats:
* 1. The tools needs all brokers to be up at startup time.
* 2. The tool doesn't handle out of range offsets.
*
Feel free to contribute an improvement to our official documentation.
[B] Apparently this is the expected behavior if the consumer didn't
see new messages for 10 seconds.
On Thu, Jun 9, 2016 at 7:05 AM, Kamal C <ka...@gmail.com> wrote:
> Hi,
>
> A. What is the usage of `kafka-replica-verification.sh` script ? I don't
> find any documentations about it in [1] and [2].
>
> I have a topic `test` with 10 partitions. Ran the above script, it
> continuously prints the below results.
>
> [kamal@tcltest1 bin]$ sh kafka-replica-verification.sh --time -2
> --topic-white-list test --report-interval-ms 1000 --broker-list
> localhost:9092
> 2016-06-07 12:49:13,664: verification process is started.
> 2016-06-07 12:49:14,632: max lag is 0 for partition [test,6] at offset
> 39008 among 10 partitions
> 2016-06-07 12:49:15,633: max lag is 0 for partition [test,6] at offset
> 39008 among 10 partitions
> 2016-06-07 12:49:16,634: max lag is 0 for partition [test,6] at offset
> 39008 among 10 partitions
> 2016-06-07 12:49:17,635: max lag is 0 for partition [test,6] at offset
> 39008 among 10 partitions
> 2016-06-07 12:49:18,636: max lag is 0 for partition [test,6] at offset
> 39008 among 10 partitions
> 2016-06-07 12:49:19,637: max lag is 0 for partition [test,6] at offset
> 39008 among 10 partitions
>
> B. `kafka-replay-log-producer.sh` script throws ConsumerTimeoutException.
> What is wrong with the below command ?
>
> [kamal@tcltest1 bin]$ sh kafka-replay-log-producer.sh --broker-list
> localhost:9092 --inputtopic test --outputtopic xyz --reporting-interval
> 1000 --zookeeper localhost:2181
> [2016-06-07 12:42:10,925] ERROR consumer thread timing out
> (kafka.tools.ReplayLogProducer$ZKConsumerThread)
> kafka.consumer.ConsumerTimeoutException
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:69)
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
> at
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:64)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:56)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at kafka.consumer.KafkaStream.foreach(KafkaStream.scala:25)
> at
> kafka.tools.ReplayLogProducer$ZKConsumerThread.run(ReplayLogProducer.scala:140)
>
>
> [1]: http://kafka.apache.org/documentation.html
> [2]: https://cwiki.apache.org/confluence/display/KAFKA/System+Tools
Fwd: Questions about Kafka Scripts
Posted by Kamal C <ka...@gmail.com>.
Hi,
A. What is the usage of `kafka-replica-verification.sh` script ? I don't
find any documentations about it in [1] and [2].
I have a topic `test` with 10 partitions. Ran the above script, it
continuously prints the below results.
[kamal@tcltest1 bin]$ sh kafka-replica-verification.sh --time -2
--topic-white-list test --report-interval-ms 1000 --broker-list
localhost:9092
2016-06-07 12:49:13,664: verification process is started.
2016-06-07 12:49:14,632: max lag is 0 for partition [test,6] at offset
39008 among 10 partitions
2016-06-07 12:49:15,633: max lag is 0 for partition [test,6] at offset
39008 among 10 partitions
2016-06-07 12:49:16,634: max lag is 0 for partition [test,6] at offset
39008 among 10 partitions
2016-06-07 12:49:17,635: max lag is 0 for partition [test,6] at offset
39008 among 10 partitions
2016-06-07 12:49:18,636: max lag is 0 for partition [test,6] at offset
39008 among 10 partitions
2016-06-07 12:49:19,637: max lag is 0 for partition [test,6] at offset
39008 among 10 partitions
B. `kafka-replay-log-producer.sh` script throws ConsumerTimeoutException.
What is wrong with the below command ?
[kamal@tcltest1 bin]$ sh kafka-replay-log-producer.sh --broker-list
localhost:9092 --inputtopic test --outputtopic xyz --reporting-interval
1000 --zookeeper localhost:2181
[2016-06-07 12:42:10,925] ERROR consumer thread timing out
(kafka.tools.ReplayLogProducer$ZKConsumerThread)
kafka.consumer.ConsumerTimeoutException
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:69)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
at
kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:64)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:56)
at scala.collection.Iterator$class.foreach(Iterator.scala:893)
at kafka.utils.IteratorTemplate.foreach(IteratorTemplate.scala:30)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at kafka.consumer.KafkaStream.foreach(KafkaStream.scala:25)
at
kafka.tools.ReplayLogProducer$ZKConsumerThread.run(ReplayLogProducer.scala:140)
[1]: http://kafka.apache.org/documentation.html
[2]: https://cwiki.apache.org/confluence/display/KAFKA/System+Tools