[GitHub] [druid] camelusluo opened a new issue #9378: kafka indexing service qps very low

[GitBox <gi...@apache.org>]

camelusluo opened a new issue #9378: kafka indexing service qps very low
URL: https://github.com/apache/druid/issues/9378
 
 
   our druid cluster consumer kafka data by kafka indexing service, but we find kis qps very low about ~2000/s per task. 
   
   ### Affected Version
   
   druid 0.13.0
   
   ### Description
   vm config
   <pre><code>
   24cores 96G
   </code></pre>
   
   middlemanger config
   <pre><code>
   druid.service=druid/middleManager
   druid.plaintextPort=8091
   
   # Number of tasks per middleManager
   druid.worker.capacity=16
   
   # Task launch parameters
   druid.indexer.runner.javaOpts=-server -Xmx3g -XX:MaxDirectMemorySize=5g -Duser.timezone=UTC+08:00 -Dfile.encoding=UTF-8 -Djava.util.logging.manager=org.apache.logging.log4j.jul.LogManager
   druid.indexer.task.baseTaskDir=var/druid/task
   druid.indexer.task.restoreTasksOnRestart=true
   
   # HTTP server threads
   druid.server.http.numThreads=32
   
   # Processing threads and buffers on Peons
   druid.indexer.fork.property.druid.processing.buffer.sizeBytes=268435456
   druid.indexer.fork.property.druid.processing.numThreads=8
   
   # Hadoop indexing
   druid.indexer.task.hadoopWorkingPath=var/druid/hadoop-tmp
   </code></pre>
   
   kafka partition
   <pre><code>
   24
   </code></pre>
   
   supervisor
   <pre><code>
   {
     "type": "kafka",
     "dataSchema": {
       "dataSource": "stream-report-request",
       "parser": {
         "type": "string",
         "parseSpec": {
           "format": "json",
           "timestampSpec": {
             "column": "processTime",
             "format": "auto"
           },
           "dimensionsSpec": {
             "dimensions": [
               "adGroupId",
               "adPositionId",
               "creativeId",
               "orderId",
               "province",
               "city",
               "dspName",
               "adClass",
               "resourceFreqControlMode",
               "companyId",
               "dspPosId",
               "targetGame",
               "targetScene",
               "targetKeyword",
               "targetRoom",
               "gbCode",
               "filterReason",
               "configId",
               "blockIds",
               "strBlockIds",
               "itemId",
               "pctrBucket"
             ],
             "dimensionExclusions": [],
             "spatialDimensions": []
           }
         }
       },
       "metricsSpec": [{
         "type": "longSum",
         "name": "adRequestCount",
         "fieldName": "adRequestCount"
       },
         {
           "type": "longSum",
           "name": "adRecallCount",
           "fieldName": "adRecallCount"
         },
         {
           "type": "thetaSketch",
           "name": "uid",
           "fieldName": "uid"
         }
       ],
       "granularitySpec": {
         "type": "uniform",
         "segmentGranularity": "ten_minute",
         "queryGranularity": "NONE"
       }
     },
     "tuningConfig": {
       "type": "kafka",
       "maxRowsInMemory": 20000,
       "maxBytesInMemory": -1,
       "intermediatePersistPeriod": "PT10M"
     },
     "ioConfig": {
       "topic": "stream-report-request-topic",
       "consumerProperties": {
         "bootstrap.servers": "",
         "group.id": "consumer_group_druid"
       },
       "useEarliestOffset": true,
       "taskCount": 12,
       "replicas": 1,
       "taskDuration": "PT10M",
       "completionTimeout": "PT10M"
     }
   }
   </code></pre>
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org

[GitHub] [druid] jihoonson commented on issue #9378: kafka indexing service qps very low

[GitBox <gi...@apache.org>]

jihoonson commented on issue #9378: kafka indexing service qps very low
URL: https://github.com/apache/druid/issues/9378#issuecomment-588369288
 
 
   Hi @camelus0211, `maxRowsInMemory` looks too low. Try setting it to a higher value if you can. The default value is 1m in the latest version.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org