You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/10/20 06:02:57 UTC
[GitHub] [druid] minkedong opened a new issue #10519: When I use topN query, the results are very different.
minkedong opened a new issue #10519:
URL: https://github.com/apache/druid/issues/10519
When I use topN query, the results are very different.
### Affected Version
Druid version is 0.16.0
### Description
When I use the following query:
{
"queryType": "topN",
"dataSource": "adx_data",
"intervals": "2020-10-19T00Z/2020-10-21T00Z",
"granularity": "all",
"filter": {
"type": "selector",
"dimension": "bundle",
"value": "1115565187"
},
"dimension": {
"type": "default",
"dimension": "bundle",
"outputName": "bundle"
},
"aggregations": [
{
"name": "rev",
"type": "doubleSum",
"fieldName": "rev"
},
{
"name": "traffic",
"type": "longSum",
"fieldName": "traffic"
}
],
"metric": "rev",
"threshold": 50
}
The corresponding query result is:
{"bundle":"1115565187","rev":1681.6186172404705,"traffic":103772277}
However, when I remove the filter condition and request the query again:
{
"queryType": "topN",
"dataSource": "adx_data",
"intervals": "2020-10-19T00Z/2020-10-21T00Z",
"granularity": "all",
"dimension": {
"type": "default",
"dimension": "bundle",
"outputName": "bundle"
},
"aggregations": [
{
"name": "rev",
"type": "doubleSum",
"fieldName": "rev"
},
{
"name": "traffic",
"type": "longSum",
"fieldName": "traffic"
}
],
"metric": "rev",
"threshold": 50
}
The corresponding query result is:
{"bundle":"1115565187","rev":1681.6186172404705,"traffic":22087792}
{"bundle":"591560124","rev":1255.0023926230278,"traffic":25825}
{"bundle":"com.pixel.art.coloring.color.number","rev":1010.6935608959175,"traffic":59439}
{"bundle":"1502447854","rev":635.8367775218067,"traffic":60758}
{"bundle":"1207472156","rev":631.9766362940973,"traffic":25891452}
...
So, my question is, why the traffic sum value of the same bundle ("1115565187") has changed after removing the filter condition. Referring to the data I had ingestion, the the first result is correct, which makes me very confused, Can anyone help me answer this question, thank you very much。
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] minkedong commented on issue #10519: When I use topN query, the results are very different.
Posted by GitBox <gi...@apache.org>.
minkedong commented on issue #10519:
URL: https://github.com/apache/druid/issues/10519#issuecomment-719326525
> For TopN, query engine collect top 1000(by default) ordered result from each segment, then merge all results to determine the global top 1000. For example, a data source has two segment A,B, for your second query, segment A, B contains value of (bundle, traffic) respectively (1115565187, 20) ranking 49 and (1115565187, 40) ranking 51, because of threshold = 50, query engine only collect result of segment A, result of B drop, which is approximate. But for your first query with filter, segment A, B contains value of (bundle, traffic) respectively (1115565187, 20) ranking 1 and (1115565187, 40) ranking 1 , result collected is accurate. if you need accurate result of your second query, try to increase threshold.
@zoov-w Thank you very much for your answer, Your answer helped me solve the problem and at the same time let me understand the cause of the problem, great!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zoov-w commented on issue #10519: When I use topN query, the results are very different.
Posted by GitBox <gi...@apache.org>.
zoov-w commented on issue #10519:
URL: https://github.com/apache/druid/issues/10519#issuecomment-716283745
For TopN, query engine collect top 1000(by default) ordered result from each segment, then merge all results to determine the global top 1000. For example, a data source has two segment A,B, for your second query, segment A, B contains value of (bundle, traffic) respectively (1115565187, 20) ranking 49 and (1115565187, 40) ranking 51, because of threshold = 50, query engine only collect result of segment A, result of B drop, which is approximate. But for your first query with filter, segment A, B contains value of (bundle, traffic) respectively (1115565187, 20) ranking 1 and (1115565187, 40) ranking 1 , result collected is accurate. if you need accurate result of your second query, try to increase threshold.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] minkedong closed issue #10519: When I use topN query, the results are very different.
Posted by GitBox <gi...@apache.org>.
minkedong closed issue #10519:
URL: https://github.com/apache/druid/issues/10519
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zoov-w commented on issue #10519: When I use topN query, the results are very different.
Posted by GitBox <gi...@apache.org>.
zoov-w commented on issue #10519:
URL: https://github.com/apache/druid/issues/10519#issuecomment-716188986
Hi, maybe you can read this document https://druid.apache.org/docs/latest/querying/topnquery.html#aliasing, If you are wanting the top 100 of a high cardinality, uniformly distributed dimension ordered by some low-cardinality, uniformly distributed dimension, you are potentially going to get aggregates back that are missing data.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org