You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "jacktengg (via GitHub)" <gi...@apache.org> on 2023/06/05 09:42:15 UTC
[GitHub] [doris] jacktengg opened a new pull request, #20456: [improvement](column reader) lazy load indices
jacktengg opened a new pull request, #20456:
URL: https://github.com/apache/doris/pull/20456
## Proposed changes
Issue Number: close #xxx
Currently when reading column data, all types of indice are read even if they are not actually used, this PR implements lazy load of indices.
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #20456: [improvement](column reader) lazy load indices
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #20456:
URL: https://github.com/apache/doris/pull/20456
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #20456: [improvement](column reader) lazy load indices
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #20456:
URL: https://github.com/apache/doris/pull/20456#discussion_r1218814082
##########
be/src/olap/rowset/segment_v2/column_reader.cpp:
##########
@@ -220,7 +225,8 @@ Status ColumnReader::init() {
}
Status ColumnReader::new_bitmap_index_iterator(BitmapIndexIterator** iterator) {
- RETURN_IF_ERROR(_ensure_index_loaded());
+ bool use_page_cache = !config::disable_storage_page_cache && _index_meta_use_page_cache;
Review Comment:
could we add a private class member use_page_cache? Then we could just use the class member at line 229.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] jacktengg commented on pull request #20456: [improvement](column reader) lazy load indices
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #20456:
URL: https://github.com/apache/doris/pull/20456#issuecomment-1576508766
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] jacktengg commented on pull request #20456: [improvement](column reader) lazy load indices
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #20456:
URL: https://github.com/apache/doris/pull/20456#issuecomment-1576627465
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20456: [improvement](column reader) lazy load indices
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20456:
URL: https://github.com/apache/doris/pull/20456#issuecomment-1576619100
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20456: [improvement](column reader) lazy load indices
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20456:
URL: https://github.com/apache/doris/pull/20456#issuecomment-1576475416
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #20456: [improvement](column reader) lazy load indices
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #20456:
URL: https://github.com/apache/doris/pull/20456#issuecomment-1577861982
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] jacktengg commented on pull request #20456: [improvement](column reader) lazy load indices
Posted by "jacktengg (via GitHub)" <gi...@apache.org>.
jacktengg commented on PR #20456:
URL: https://github.com/apache/doris/pull/20456#issuecomment-1577861233
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #20456: [improvement](column reader) lazy load indices
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #20456:
URL: https://github.com/apache/doris/pull/20456#discussion_r1218813218
##########
be/src/olap/rowset/segment_v2/column_reader.cpp:
##########
@@ -374,6 +378,9 @@ Status ColumnReader::_get_filtered_pages(
const AndBlockColumnPredicate* col_predicates,
const std::vector<const ColumnPredicate*>* delete_predicates,
std::vector<uint32_t>* page_indexes) {
+ bool use_page_cache = !config::disable_storage_page_cache && _index_meta_use_page_cache;
+ _load_zone_map_index(use_page_cache, _opts.kept_in_memory);
Review Comment:
RETURN_IF_ERROR(_load_zone_map_index....)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org