[GitHub] [incubator-druid] jon-wei commented on issue #8276: KIS tasks in 0.15.1 RC2 sometimes duplicate rows with the same dimension values

[GitBox <gi...@apache.org>]

jon-wei commented on issue #8276: KIS tasks in 0.15.1 RC2 sometimes duplicate rows with the same dimension values
URL: https://github.com/apache/incubator-druid/issues/8276#issuecomment-520048197
 
 
   With Kafka ingestion, it's possible that rollup is not 100%, due to the incremental publishing, e.g.:
   - Kafka task gets a row like `{"timestamp":"2019-08-09 14:00:00", "dim1":"hello", "metric":1}`
   - Kafka task hits a memory/row/time limit and does an incremental publish
   - If after the incremental publish, another row `{"timestamp":"2019-08-09 14:00:00", "dim1":"hello", "metric":2}` arrives, it won't be rolled up into the first already published segment, and a non-aggregating query would return both of those rows
   
   If you run an exact aggregating query like:
   
   ```
   SELECT accountid, SUM(cpu) 
   FROM "kube-metrics-minute" 
   WHERE accountid='52c7e28eebb26a86fb1505b5' AND  
    "__time" >= TIMESTAMP '2019-08-09 14:00:00' AND "__time" < TIMESTAMP '2019-08-09 16:30:00' AND cluster='ds' AND etype='container' AND namespace='default' AND name='app-alertproc' AND pod='alertproc-deployment-65f59b6b8-gc57m' 
   GROUP BY accountid
   ```
   
   does the sum on CPU return the expected value?
   
   You could compare the results of that query across the minute/second/30m datasources and see if they match, to rule out imperfect rollup from incremental publishes.
   
   
   
   
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org