Re: Review Request 21878: Patch for KAFKA-1468

[Jay Kreps <bo...@gmail.com>]

> On May 29, 2014, 5:20 p.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/server/RequestPurgatory.scala, line 64
> > <https://reviews.apache.org/r/21878/diff/2/?file=598165#file598165line64>
> >
> >     Why we want to reduce the default purgatory size?

This interval is in terms of requests. Consider a case where you have 1000 partitions on your broker and set your fetch timeout to be 10ms (either for consumers or for replication), then in a period of inactivity you will have 10,000,000 obsolete entries in purgatory and purge them all at once. During this time no other purgatory expiration happens which leads to large latency spikes. 


> On May 29, 2014, 5:20 p.m., Guozhang Wang wrote:
> > core/src/main/scala/kafka/tools/TestEndToEndLatency.scala, line 25
> > <https://reviews.apache.org/r/21878/diff/2/?file=598166#file598166line25>
> >
> >     Can we add some comments here indicating the end-to-end latency is for async producer mode?

Well I'm not sure how relevant the acknowledgement mode is as the consumer can't consume until full acknowledgement. Since the consumer fetch and the producer ack happen in parallel I would expect the difference to be negligible.


- Jay


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21878/#review44266
-----------------------------------------------------------


On May 28, 2014, 11:15 p.m., Jay Kreps wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21878/
> -----------------------------------------------------------
> 
> (Updated May 28, 2014, 11:15 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1468
>     https://issues.apache.org/jira/browse/KAFKA-1468
> 
> 
> Repository: kafka
> 
> 
> Description
> -------
> 
> KAFKA-1468 Misc. improvements from benchmarking.
> 
> 
> Diffs
> -----
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java f1def508153b7361b8720dbcc939e06b0e7e45d3 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java bc4074ec841650eff48dd8aafc1e6c712d8eae7d 
>   clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 3e83ae0fefb7578cccbd8dc0cd9b28f22e0951b2 
>   clients/src/main/java/org/apache/kafka/clients/tools/ProducerPerformance.java eb18739d181662e2dee5dd6a23e0181f4c740139 
>   clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java a2b77226f8c58caf632a0f4665bd4e4cd93e643d 
>   core/src/main/scala/kafka/server/KafkaConfig.scala c7508d5568ec6b94c47d1839aa8695e2dbb76b97 
>   core/src/main/scala/kafka/server/RequestPurgatory.scala c064c5c4cf1191335572da8a2caf5f95dce902c1 
>   core/src/main/scala/kafka/tools/TestEndToEndLatency.scala 37a9ec27abfd0acd6c01f11d987dd719bc9138a5 
> 
> Diff: https://reviews.apache.org/r/21878/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Jay Kreps
> 
>