You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "egalpin (via GitHub)" <gi...@apache.org> on 2023/08/30 22:11:57 UTC
[GitHub] [pinot] egalpin opened a new issue, #11464: Consider leveraging inverted index to augment time range index performance
egalpin opened a new issue, #11464:
URL: https://github.com/apache/pinot/issues/11464
For cases where `range` index is used for timestamps, I have observed that using a complimentary inverted index at great time granularity greatly improves performance. Ex. if I have epochMs values in a field, and I convert those to yyyy-MM-dd (2023-08-30) strings which are stored in a separate column with inverted index enabled, this massively improves query performance since the inverted index can efficiently produce the superset of docs that could match the epochMs range. In my case, I pad the range with an additional day on each side to allow for ignoring timezone conversions.
This might be nice to provide out of the box as a "special" range index where all of this is handled under-the-hood, at user configurable granularity (hour instead of day, for example). Might not be worth the trouble either, as it's definitely doable as-is be self-managing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #11464: Consider leveraging inverted index to augment time range index performance
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #11464:
URL: https://github.com/apache/pinot/issues/11464#issuecomment-1701522158
This can be the next generation timestamp index!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Consider leveraging inverted index to augment time range index performance [pinot]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #11464:
URL: https://github.com/apache/pinot/issues/11464#issuecomment-5824548942
This issue has been automatically marked as stale because it has not had any activity in the last 150 days. It will be closed in 30 days if no further activity occurs. If this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] egalpin commented on issue #11464: Consider leveraging inverted index to augment time range index performance
Posted by "egalpin (via GitHub)" <gi...@apache.org>.
egalpin commented on issue #11464:
URL: https://github.com/apache/pinot/issues/11464#issuecomment-1728415468
Before and after using the above technique for a workload where every query uses a date range filter:
<img width="786" alt="Screenshot 2023-09-20 at 13 54 37" src="https://github.com/apache/pinot/assets/7582748/91d0b10a-a886-4926-b459-a6b6043ee61e">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Consider leveraging inverted index to augment time range index performance [pinot]
Posted by "egalpin (via GitHub)" <gi...@apache.org>.
egalpin commented on issue #11464:
URL: https://github.com/apache/pinot/issues/11464#issuecomment-2546670434
@gortiz do you know if that range index segment pruning has now been implemented? It seems like it may have based on my empirical observations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Consider leveraging inverted index to augment time range index performance [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #11464:
URL: https://github.com/apache/pinot/issues/11464#issuecomment-2099920742
I don't get this. Is there a reason why we cannot use the range index to prune segments? Or it is just that we didn't implement that yet?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org