You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/01/16 09:33:54 UTC
[GitHub] quenlang edited a comment on issue #6853: horrible histogram
post-aggregation
quenlang edited a comment on issue #6853: horrible histogram post-aggregation
URL: https://github.com/apache/incubator-druid/issues/6853#issuecomment-454710643
@jon-wei As the data ingested into druid which was stayed at the memory or local disk of middleManager peon task before handing off to deep storage, so i performed a query which will be sent to peon task. The query like this:
```
{
"queryType": "timeseries",
"dataSource": {
"type": "table",
"name": "sketch_1"
},
"intervals": {
"type": "intervals",
"intervals": [
"2019-01-16T00:30:00/2019-01-17T00:00:00"
]
},
"descending": false,
"virtualColumns": [],
"granularity": {
"type": "all"
},
"aggregations": [
{
"type": "longSum",
"name": "count_total",
"fieldName": "count",
"expression": null
},
{
"type": "approxHistogramFold",
"name": "dns_time_histogram",
"fieldName": "dns_time_histogram",
"resolution": 50,
"numBuckets": 7,
"lowerLimit": 0
}, [13/1830]
{
"type": "quantilesDoublesSketch",
"name": "dns_time_sketch",
"fieldName": "dns_time_sketch",
"k": 128
}
],
"postAggregations": [
{
"type": "customBuckets",
"name": "performance_histogram",
"fieldName": "dns_time_histogram",
"breaks": [0, 200.0, 400.0, 600.0, 800.0, 1000.0, 1200.0, 1400.0, "Infinity"]
},
{
"type" : "quantiles",
"name" : "histogram_quantile",
"fieldName" : "dns_time_histogram",
"probabilities" : [0.50, 0.75, 0.90, 0.95]
},
{
"type" : "quantilesDoublesSketchToHistogram",
"name": "performance_sketch",
"field": {
"type": "fieldAccess",
"fieldName": "dns_time_sketch"
},
"splitPoints" : [200, 400, 600, 800, 1000, 1200, 1400]
},
{
"type" : "quantilesDoublesSketchToQuantiles",
"name": "sketch_quantile",
"field": {
"type": "fieldAccess",
"fieldName": "dns_time_sketch"
},
"fractions" : [0.50, 0.75, 0.90, 0.95]
}
],
"context": {
"skipEmptyBuckets": "true"
}
}
```
In order to compare the accuracy of approximate histogram and quantiles sketch so i defined two aggregation methods in the same query. And the result as below:
```
[ {
"timestamp" : "2019-01-16T02:01:00.000Z",
"result" : {
"performance_histogram" : {
"breaks" : [ 0.0, 200.0, 400.0, 600.0, 800.0, 1000.0, 1200.0, 1400.0, "Infinity" ],
"counts" : [ 5.0, 0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 3.0 ]
},
"histogram_quantile" : {
"probabilities" : [ 0.5, 0.75, 0.9, 0.95 ],
"quantiles" : [ 99.0, 1150.0, 1772.0, 1886.0 ],
"min" : 0.0,
"max" : 2000.0
},
"count_total" : 10,
"dns_time_histogram" : {
"breaks" : [ -333.3333435058594, 0.0, 333.3333435058594, 666.6666870117188, 1000.0, 1333.3333740234375, 1666.666748046875, 2000.0 ],
"counts" : [ 1.0, 5.0, 1.0, 0.0, 0.0, 0.0, 3.0 ]
},
"sketch_quantile" : [ 100.0, 1700.0, 2000.0, 2000.0 ],
"dns_time_sketch" : 10,
"performance_sketch" : [ 6.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 3.0 ]
}
} ]
```
The orignal row data which i wrote to kafka, include 11 rows as below:
```
{"timestamp":1547604098000,"uid":1,"name":"zjk","dns_time":0.0,"count":1}
{"timestamp":1547604098000,"uid":1,"name":"zjk","dns_time":99.0,"count":1}
{"timestamp":1547604158000,"uid":2,"name":"quen","dns_time":100.0,"count":1}
{"timestamp":1547604158000,"uid":2,"name":"quen","dns_time":600.0,"count":1}
{"timestamp":1547604158000,"uid":3,"name":"quen","dns_time":2000.0,"count":1}
{"timestamp":1547604218000,"uid":4,"name":"zjk","dns_time":5.0,"count":1}
{"timestamp":1547604218000,"uid":5,"name":"zjk","dns_time":20.0,"count":1}
{"timestamp":1547604218000,"uid":6,"name":"zjk","dns_time":2.0,"count":1}
{"timestamp":1547604218000,"uid":7,"name":"zjk","dns_time":1772.0,"count":1}
{"timestamp":1547604218000,"uid":8,"name":"zjk","dns_time":1700.0,"count":1}
{"timestamp":1547604218000,"uid":9,"name":"zjk","dns_time":,300.0"count":1n}
```
Also i calculated the quantiles of [0.50, 0.75, 0.90, 0.95] and the histograms of [ 0.0, 200.0, 400.0, 600.0, 800.0, 1000.0, 1200.0, 1400.0, "Infinity" ] by myself. They were [100, 1150, 1772, 1772] and [ 6.0, 0.0, 0.0, 1.0, 0.0, 0.0, 0.0, 3.0 ].
Compared the actual result with the query result, i found the quantile query of approximate histogram was more accurate than quantiles sketch, but the histogram query of quantiles sketch was win.
Can you tell me more about why the the quantile query of approximate histogram was more accurate thanquantiles sketch?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org