You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "rchien-atvi (via GitHub)" <gi...@apache.org> on 2023/11/27 20:37:03 UTC
[I] StarTreeIndexViewer does not build second (an deeper) child nodes [pinot]
rchien-atvi opened a new issue, #12053:
URL: https://github.com/apache/pinot/issues/12053
I'm trying to use the StarTreeIndexViewer to validate my StarTree setup as I am not seeing the performance I expect (topic for another day). The current way to load via `ImmutableSegmentLoader.load(segmentDir, ReadMode.heap)` does not seem to load child nodes past the first level.
This is backed by few experiments like adjusting [MAX_CHILDREN](https://github.com/apache/pinot/blob/master/pinot-tools/src/main/java/org/apache/pinot/tools/StarTreeIndexViewer.java#L59C28-L59C40) and trying to visualize index in [pinot-recipes](https://github.com/startreedata/pinot-recipes/tree/main/recipes/startree-index). In all scenarios the first dimension split is built properly but as we recurs to n+1 depth, the node is always marked as leaf.
Can anyone confirm if `ImmutableSegmentLoader.Load` is indeed the issue? I'm happy to contribute a fix (if indeed a bug) but would appreciate some tips on starting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] StarTreeIndexViewer does not build second (an deeper) child nodes [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang closed issue #12053: StarTreeIndexViewer does not build second (an deeper) child nodes
URL: https://github.com/apache/pinot/issues/12053
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] StarTreeIndexViewer does not build second (an deeper) child nodes [pinot]
Posted by "rchien-atvi (via GitHub)" <gi...@apache.org>.
rchien-atvi commented on issue #12053:
URL: https://github.com/apache/pinot/issues/12053#issuecomment-1839782949
Thanks @Jackie-Jiang for reply. Since original post I've found [SegmentDumpTool](https://github.com/apache/pinot/blob/master/pinot-tools/src/main/java/org/apache/pinot/tools/SegmentDumpTool.java). Unfortunately using SegmentDumpTool yields similar outcome.
I've tried
- setting `maxLeafRecords` to `2500`
- switching `dimensionsSplitOrder` where first element has lower carnality
- use pinot `0.12.1` instead of `1.0.0`
In cases the results are the same where 2+ level children are not included.
I'll do another test with [pinot-recipes](https://github.com/startreedata/pinot-recipes/tree/main/recipes/startree-index) and share the STree config and generated index files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] StarTreeIndexViewer does not build second (an deeper) child nodes [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #12053:
URL: https://github.com/apache/pinot/issues/12053#issuecomment-1839219115
Can you try tuning the `maxLeafRecords` within your star-tree config? `StarTreeIndexViewer` should be able to visualize the tree properly, and the reason why you don't see the second level could be because there is just no second level because first level already bring down nodes under `maxLeafRecords`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] StarTreeIndexViewer does not build second (an deeper) child nodes [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #12053:
URL: https://github.com/apache/pinot/issues/12053#issuecomment-1839822005
@rchien-atvi You can try setting `maxLeafRecords` to `1` which fully materialize all the combinations to guarantee the second level can be generated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] StarTreeIndexViewer does not build second (an deeper) child nodes [pinot]
Posted by "rchien-atvi (via GitHub)" <gi...@apache.org>.
rchien-atvi commented on issue #12053:
URL: https://github.com/apache/pinot/issues/12053#issuecomment-1843603342
Re-run my test with `maxLeafRecords` to `1` against both test and real dataset. This time I do see children past second level.
For the pinot-recipes, the layout is exactly as I expect.
Thanks @Jackie-Jiang.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org