[GitHub] [druid] zhangyouxun commented on issue #11061: How to improving ingestion performance

[GitBox <gi...@apache.org>]

zhangyouxun commented on issue #11061:
URL: https://github.com/apache/druid/issues/11061#issuecomment-811954109


   > Your task count is 30, does that mean there're 30 partitions of your topic ? And is the ingestion speed 5K/s of per peon ? Even it's 5K/s per peon, it's not a very high speed that it needs 10G heap size for each peon. 4G is enough in our clusters to handle messages at much higher speed.
   > 
   > Since streaming ingestion speed is more relevant to streaming middleware, tuning producer and consumer may help. Are you using Kafka ? And what parameters you have adopted to configure Kafka producers and consumers ?
   
   the num of partitions is 270.
   The maxBytesInMemory is about 300Mb, is it not make full use of heap?
   
   we use Kafka
   max.poll.records = 500
   max.partition.fetch.bytes = 1048576
   fetch.max.bytes = 52428800
   fetch.max.wait.ms = 500
   fetch.min.bytes = 1
   metadata.max.age.ms = 10000
   which parameter needs to be adjusted？


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org