You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "egalpin (via GitHub)" <gi...@apache.org> on 2024/02/02 23:08:13 UTC
[I] Add QueryOption support for column-level index skipping at query time [pinot]
egalpin opened a new issue, #12355:
URL: https://github.com/apache/pinot/issues/12355
In some cases it may be beneficial to skip the usage of an index despite its existence, such as utilizing many inverted indices over many columns, and combining the resulting bitmaps. It may be faster to use only the more highly selective index (if that can be known at query time) and instead scan those resulting documents to fulfill another filter (as opposed to fulfill that other filter via another index and combining the bitmaps).
Something like `SET indexSkipList='col1:inverted,range;col2:inverted';` (especially if also combined with `SET AndScanReordering = 'True';`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add QueryOption support for column-level index skipping at query time [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #12355:
URL: https://github.com/apache/pinot/issues/12355#issuecomment-1961258406
Thanks for your findings! Anyway, if there is no clean alternative, we can just apply the one you proposed with url arguments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add QueryOption support for column-level index skipping at query time [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz closed issue #12355: Add QueryOption support for column-level index skipping at query time
URL: https://github.com/apache/pinot/issues/12355
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add QueryOption support for column-level index skipping at query time [pinot]
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on issue #12355:
URL: https://github.com/apache/pinot/issues/12355#issuecomment-1959178580
I really like this feature. What I'm not sure about is the syntax. Not because I'm against URL params. But I would like to prioritize consistency here. It doesn't seem we have other query options where the value is a `Map<String, Collection<T>>`, so adding this feature as it is would mean that future options should use this syntax.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add QueryOption support for column-level index skipping at query time [pinot]
Posted by "egalpin (via GitHub)" <gi...@apache.org>.
egalpin commented on issue #12355:
URL: https://github.com/apache/pinot/issues/12355#issuecomment-1924941336
Thoughts on the list scheme? How should the index names be referenced? Re-using the names from the tableSpec looks a little bit odd as it would lead to things like `SET indexSkipList='col1:invertedIndexColumns';`. `fieldConfigList.INDEX_TYPE` might work, but I've forgotten whether that is the new format being moved towards, or the old format being deprecated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add QueryOption support for column-level index skipping at query time [pinot]
Posted by "kishoreg (via GitHub)" <gi...@apache.org>.
kishoreg commented on issue #12355:
URL: https://github.com/apache/pinot/issues/12355#issuecomment-1925038472
+1 to this feature. We have this for StarTree index.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add QueryOption support for column-level index skipping at query time [pinot]
Posted by "egalpin (via GitHub)" <gi...@apache.org>.
egalpin commented on issue #12355:
URL: https://github.com/apache/pinot/issues/12355#issuecomment-1959870319
@gortiz totally makes sense that we should be more careful in cases of setting precedent. Are there any similar features in other DBs that would provide familiar syntax? I'll take a look around
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add QueryOption support for column-level index skipping at query time [pinot]
Posted by "egalpin (via GitHub)" <gi...@apache.org>.
egalpin commented on issue #12355:
URL: https://github.com/apache/pinot/issues/12355#issuecomment-1936752091
After a bit of thought, I'm more in favour of matching the generally-familiar URL params scheme: `indexSkipConfig='col1=inverted,range&col2=inverted'`. Thoughts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [I] Add QueryOption support for column-level index skipping at query time [pinot]
Posted by "egalpin (via GitHub)" <gi...@apache.org>.
egalpin commented on issue #12355:
URL: https://github.com/apache/pinot/issues/12355#issuecomment-1959985260
I did find that Impala[1] uses `:` as a delimiter for list-type options. Used in this issue context, it might look like: `myColumn=inverted:range`. I'll look a bit more for other examples
[1] https://impala.apache.org/docs/build/html/topics/impala_default_hints_insert_statement.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org