You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/08/04 19:14:45 UTC
[GitHub] [druid] gulinfu opened a new issue #10238: Perfect rollup is not perfect (maybe due to multi-value dimension?)
gulinfu opened a new issue #10238:
URL: https://github.com/apache/druid/issues/10238
### Affected Version
0.18.1
### Description
ingestion spec:
```
{
"type": "index_parallel",
"spec": {
"dataSchema": {
"dimensionsSpec": {
"dimensions": [
"dimensionA",
"dimensionB",
"dimensionC",
"dimensionD",
"dimensionE",
"dimensionF"
]
},
"metricsSpec": [
{
"fieldName": "firstPartyCookie",
"name": "1stPartyCookie_hll",
"type": "HLLSketchBuild"
},
{
"fieldName": "thirdPartyCookie",
"name": "3rdPartyCookie_hll",
"type": "HLLSketchBuild"
},
{
"fieldName": "viewCount",
"name": "viewCount",
"type": "longSum"
}
],
"granularitySpec": {
"intervals": [
"2020-08-04T15:00:00.000Z/2020-08-04T16:00:00.000Z"
],
"segmentGranularity": "hour",
"queryGranularity": "hour",
"type": "uniform"
},
"timestampSpec": {
"column": "timestamp",
"format": "auto"
},
"dataSource": "xxx_production"
},
"ioConfig": {
"inputSource": {
"prefixes": [
"s3://xxx/2020-08/04/15"
],
"type": "s3"
},
"type": "index_parallel",
"inputFormat": {
"listDelimiter": "\u0001",
"columns": [
"timestamp",
"dimensionA",
"dimensionB",
"dimensionC",
"dimensionD",
"dimensionE",
"dimensionF"
"firstPartyCookie",
"thirdPartyCookie",
"viewCount"
],
"type": "csv"
}
},
"tuningConfig": {
"forceGuaranteedRollup": "true",
"partitionsSpec": {
"numShards": 1,
"type": "hashed"
},
"maxNumConcurrentSubTasks": 7,
"type": "index_parallel"
}
}
}
```
When I do some SQL `select *` query on the ingested druid data, I realize there are some rows with the same dimensions, which means they are not perfectly rolled up.
To validate my assumption, I tried to reindex the existing data source using druid inputSource, and the output segment is about 50% size and 50% number of rows compared to the origin. I did the same SQL query, and I don't see rows with the same dimensions this time. I think it's perfectly rolled up now after reindexing.
reindex spec:
```
{
"type": "index_parallel",
"spec": {
"dataSchema": {
"dataSource": "xxx_rollup_reindex",
"timestampSpec": {
"column": "timestamp",
"format": "auto"
},
"dimensionsSpec": {
"dimensions": [
"dimensionA",
"dimensionB",
"dimensionC",
"dimensionD",
"dimensionE",
"dimensionF"
]
},
"metricsSpec": [
{
"fieldName": "1stPartyCookie_hll",
"name": "1stPartyCookie_hll",
"type": "HLLSketchMerge"
},
{
"fieldName": "3rdPartyCookie_hll",
"name": "3rdPartyCookie_hll",
"type": "HLLSketchMerge"
},
{
"fieldName": "viewCount",
"name": "viewCount",
"type": "longSum"
}
],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "hour",
"queryGranularity": "hour",
"intervals": [
"2020-08-04T15:00:00.000Z/2020-08-04T16:00:00.000Z"
]
}
},
"ioConfig": {
"type": "index_parallel",
"inputSource": {
"type": "druid",
"dataSource": "xxx_production",
"interval": "2020-08-04T15:00:00.000Z/2020-08-04T16:00:00.000Z"
}
},
"tuningConfig": {
"type": "index_parallel",
"maxNumConcurrentSubTasks": 1,
"forceGuaranteedRollup": "true",
"partitionsSpec": {
"numShards": 1,
"type": "hashed"
}
}
}
}
```
One of my dimension is a multi-value dimension, maybe that's the reason? But still, we would expect "forceGuaranteedRollup": "true" to give me a perfect rollup anyway.
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gulinfu commented on issue #10238: Perfect rollup is not perfect (maybe due to multi-value dimension?)
Posted by GitBox <gi...@apache.org>.
gulinfu commented on issue #10238:
URL: https://github.com/apache/druid/issues/10238#issuecomment-668844415
@clintropolis
Thanks for the info! I'll check that out.
BTW 0.19 middleManagers are supposed to work well with other components on 0.18.1 right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on issue #10238: Perfect rollup is not perfect (maybe due to multi-value dimension?)
Posted by GitBox <gi...@apache.org>.
clintropolis commented on issue #10238:
URL: https://github.com/apache/druid/issues/10238#issuecomment-668885966
I think this should be ok to mix versions, nothing is sticking out in the release notes at least.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gulinfu commented on issue #10238: Perfect rollup is not perfect (maybe due to multi-value dimension?)
Posted by GitBox <gi...@apache.org>.
gulinfu commented on issue #10238:
URL: https://github.com/apache/druid/issues/10238#issuecomment-673189745
validated - it's fixed on 0.19.0
thank you!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] gulinfu closed issue #10238: Perfect rollup is not perfect (maybe due to multi-value dimension?)
Posted by GitBox <gi...@apache.org>.
gulinfu closed issue #10238:
URL: https://github.com/apache/druid/issues/10238
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] clintropolis commented on issue #10238: Perfect rollup is not perfect (maybe due to multi-value dimension?)
Posted by GitBox <gi...@apache.org>.
clintropolis commented on issue #10238:
URL: https://github.com/apache/druid/issues/10238#issuecomment-668828648
Hi @gulinfu,
An issue with perfect roll-up of multi-value string dimensions was fixed by #9742, added in 0.19. Any chance you could test to see if this issue still occurs in this version?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org