You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2025/05/19 20:33:21 UTC
[PR] Do not re-generate star-tree when index version changes [pinot]
Jackie-Jiang opened a new pull request, #15841:
URL: https://github.com/apache/pinot/pull/15841
Do not re-generate star-tree index when the index version changes. This is to prevent the overhead of re-generating all star-trees when the default setting changes. This also aligns with the behavior of forward index update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Do not re-generate star-tree when index version changes [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #15841:
URL: https://github.com/apache/pinot/pull/15841#discussion_r2096532399
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/startree/AggregationSpec.java:
##########
@@ -82,6 +82,15 @@ public Map<String, Object> getFunctionParameters() {
return _functionParameters;
}
+ /// Returns `true` if the given [AggregationSpec] does not match the current one, in which case the star-tree index
+ /// needs to be updated, {@code false} otherwise.
+ /// - Update star-tree if the function parameters or compression codec changes.
+ /// - Do not update star-tree when index version or config changes to avoid rebuilding all the star-trees when the
+ /// default index setting changes. This is consistent with the behavior of forward indexes.
+ public boolean shouldModifyStarTree(AggregationSpec that) {
Review Comment:
I see. Actually, I just realized that `config changes` as you mentioned in the comment must be these three configs: "targetDocsPerChunk, targetMaxChunkSize and deriveNumDocsPerChunk".
It may help to add in the comment how to force the version/config changes if wanted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Do not re-generate star-tree when index version changes [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #15841:
URL: https://github.com/apache/pinot/pull/15841#discussion_r2096517726
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/startree/AggregationSpec.java:
##########
@@ -82,6 +82,15 @@ public Map<String, Object> getFunctionParameters() {
return _functionParameters;
}
+ /// Returns `true` if the given [AggregationSpec] does not match the current one, in which case the star-tree index
+ /// needs to be updated, {@code false} otherwise.
+ /// - Update star-tree if the function parameters or compression codec changes.
+ /// - Do not update star-tree when index version or config changes to avoid rebuilding all the star-trees when the
+ /// default index setting changes. This is consistent with the behavior of forward indexes.
+ public boolean shouldModifyStarTree(AggregationSpec that) {
Review Comment:
what about configs like targetDocsPerChunk, targetMaxChunkSize and deriveNumDocsPerChunk?
btw why compressionCodec has to be checked?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Do not re-generate star-tree when index version changes [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #15841:
URL: https://github.com/apache/pinot/pull/15841#discussion_r2096524563
##########
pinot-segment-spi/src/main/java/org/apache/pinot/segment/spi/index/startree/AggregationSpec.java:
##########
@@ -82,6 +82,15 @@ public Map<String, Object> getFunctionParameters() {
return _functionParameters;
}
+ /// Returns `true` if the given [AggregationSpec] does not match the current one, in which case the star-tree index
+ /// needs to be updated, {@code false} otherwise.
+ /// - Update star-tree if the function parameters or compression codec changes.
+ /// - Do not update star-tree when index version or config changes to avoid rebuilding all the star-trees when the
+ /// default index setting changes. This is consistent with the behavior of forward indexes.
+ public boolean shouldModifyStarTree(AggregationSpec that) {
Review Comment:
This is essentially making the behavior consistent with forward index. See `ForwardIndexHandler.shouldChangeRawCompressionType()` for more context.
We can revisit this and potentially adding a flag to control the behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Do not re-generate star-tree when index version changes [pinot]
Posted by "npawar (via GitHub)" <gi...@apache.org>.
npawar merged PR #15841:
URL: https://github.com/apache/pinot/pull/15841
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org