You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2019/12/24 05:20:47 UTC
[GitHub] [incubator-doris] gaodayue opened a new pull request #2547:
[Segment V2] Support lazy materialization read
gaodayue opened a new pull request #2547: [Segment V2] Support lazy materialization read
URL: https://github.com/apache/incubator-doris/pull/2547
Fixes #2545
Current read path of SegmentIterator
----
1. apply short key index and various column indexes to get the row ranges (ordinals of rows) to scan
2. read all return columns according to the row ranges
3. evaluate column predicates on the RowBlockV2 to further prune rows
Problem
----
When the column predicates at step 3 could filter a large proportion of rows in RowBlockV2, most values of non-predicate columns we read at step 2 are thrown away, i.e we did lots of useless work and I/O at step 2.
Lazy materialization read
----
With lazy materialization, the read path changes to
1. apply short key index and various column indexes to get the row ranges (ordinals of rows) to scan (unchanged)
2. **read only predicate columns** according to the row ranges
3. evaluate column predicates on the RowBlockV2 to further prune rows, a selection vector is maintained to indicate the selected rows
4. **read the remaining columns** based on the *selection vector* of RowBlockV2
In this way, we could avoid reading values of non-predicate columns of all rows that can't pass the predicates.
Example
----
```
function: seek(ordinal), read(block_offset, count)
(step 1) row ranges: [0,2),[4,8),[10,11),[15,20)
(step 1) row ordinals: [0 1 4 5 6 7 10 15 16 17 18 19]
(step 2) read of predicate columns: seek(0),read(0,2),seek(4),read(2,4),seek(10),read(6,1),seek(15),read(7,5)
(step 3) selection vector: [3 4 5 6]
(step 3) selected ordinals: [5 6 7 10]
(step 4) read of remaining columns: seek(5),read(3,3),seek(10),read(6,1)
```
Performance evaluation
----
Lazy materialization is particularly useful when column predicates could filter many rows and lots of big metrics (e.g., hll and bitmap type columns) are queried. In our internal test cases on bitmap columns, queries run 20%~120% faster when using lazy materialization.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2547:
[Segment V2] Support lazy materialization read
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2547: [Segment V2] Support lazy materialization read
URL: https://github.com/apache/incubator-doris/pull/2547#discussion_r361136068
##########
File path: be/src/olap/rowset/segment_v2/segment_iterator.cpp
##########
@@ -353,31 +418,56 @@ Status SegmentIterator::_seek_columns(const std::vector<ColumnId>& column_ids, r
return Status::OK();
}
+Status SegmentIterator::_read_columns(const std::vector<ColumnId>& column_ids,
+ RowBlockV2* block,
+ size_t row_offset,
+ size_t nrows) {
+ for (auto cid : column_ids) {
+ auto column_block = block->column_block(cid);
+ ColumnBlockView dst(&column_block, row_offset);
+ size_t rows_read = nrows;
+ RETURN_IF_ERROR(_column_iterators[cid]->next_batch(&rows_read, &dst));
+ block->set_delete_state(column_block.delete_state());
+ DCHECK_EQ(nrows, rows_read);
+ }
+ return Status::OK();
+}
+
Status SegmentIterator::next_batch(RowBlockV2* block) {
SCOPED_RAW_TIMER(&_opts.stats->block_load_ns);
if (UNLIKELY(!_inited)) {
RETURN_IF_ERROR(_init());
+ if (_lazy_materialization_read) {
+ _block_rowids.reserve(block->capacity());
+ }
_inited = true;
}
uint32_t total_read = 0;
uint32_t remaining = block->capacity();
- // trying to fill in block
+ _block_rowids.resize(0);
+ const auto& read_columns = _lazy_materialization_read ? _predicate_columns : block->schema()->column_ids();
+
+ // phase 1: read rows selected by various index (indicated by _row_bitmap) into block
+ // when using lazy-materialization-read, only columns with predicates are read
do {
uint32_t range_from;
uint32_t range_to;
bool has_next_range = _range_iter->next_range(remaining, &range_from, &range_to);
if (!has_next_range) {
break;
}
+ if (_lazy_materialization_read) {
+ for (uint32_t rid = range_from; rid < range_to; rid++) {
+ _block_rowids.push_back(rid);
Review comment:
If _block_rowids is array, compare operation in push_back can be avoid?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2547: [Segment V2] Support lazy materialization read
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2547: [Segment V2] Support lazy materialization read
URL: https://github.com/apache/incubator-doris/pull/2547#discussion_r361125227
##########
File path: be/src/olap/rowset/segment_v2/segment_iterator.cpp
##########
@@ -314,34 +385,28 @@ Status SegmentIterator::_seek_and_peek(rowid_t rowid) {
// please note that usually RowBlockV2.clear() is called to free MemPool memory before reading the next block,
// but here since there won't be too many keys to seek, we don't call RowBlockV2.clear() so that we can use
// a single MemPool for all seeked keys.
- RETURN_IF_ERROR(_next_batch(_seek_block.get(), 0, &num_rows));
+ RETURN_IF_ERROR(_read_columns(_seek_schema->column_ids(), _seek_block.get(), 0, num_rows));
_seek_block->set_num_rows(num_rows);
return Status::OK();
}
-// Trying to read `rows_read` rows into `block` at the given offset.
-// Return the actual number of rows read in `*rows_read`.
-Status SegmentIterator::_next_batch(RowBlockV2* block, size_t row_offset, size_t* rows_read) {
- bool has_read = false;
- size_t first_read = 0;
- for (auto cid : block->schema()->column_ids()) {
- size_t num_rows = has_read ? first_read : *rows_read;
- auto column_block = block->column_block(cid);
- ColumnBlockView dst(&column_block, row_offset);
- RETURN_IF_ERROR(_column_iterators[cid]->next_batch(&num_rows, &dst));
- block->set_delete_state(column_block.delete_state());
- if (!has_read) {
- has_read = true;
- first_read = num_rows;
- } else if (num_rows != first_read) {
- return Status::InternalError(
- Substitute("Read different rows in different columns"
- ", column($0) read $1 vs column($2) read $3",
- block->schema()->column_ids()[0], first_read, cid, num_rows));
+void SegmentIterator::_init_lazy_materialization() {
+ if (!_col_predicates.empty()) {
+ std::set<ColumnId> predicate_columns;
+ for (auto predicate : _col_predicates) {
+ predicate_columns.insert(predicate->column_id());
+ }
+ // when all return columns have predicates, disable lazy materialization to avoid its overhead
+ if (_schema.column_ids().size() > predicate_columns.size()) {
+ _lazy_materialization_read = true;
Review comment:
add a unit test for lazy materialization.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on a change in pull
request #2547: [Segment V2] Support lazy materialization read
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on a change in pull request #2547: [Segment V2] Support lazy materialization read
URL: https://github.com/apache/incubator-doris/pull/2547#discussion_r361124748
##########
File path: be/src/olap/rowset/segment_v2/segment_iterator.cpp
##########
@@ -34,13 +34,71 @@ using strings::Substitute;
namespace doris {
namespace segment_v2 {
+// A fast range iterator for roaring bitmap. Output ranges use closed-open form, like [from, to).
+// Example:
+// input bitmap: [0 1 4 5 6 7 10 15 16 17 18 19]
+// output ranges: [0,2), [4,8), [10,11), [15,20) (when max_range_size=10)
+// output ranges: [0,2), [4,8), [10,11), [15,18), [18,20) (when max_range_size=3)
+class SegmentIterator::BitmapRangeIterator {
+public:
+ explicit BitmapRangeIterator(const Roaring& bitmap) {
+ roaring_init_iterator(&bitmap.roaring, &_iter);
+ _last_val = 0;
+ _buf = new uint32_t[256];
+ _read_next_batch();
+ }
+
+ ~BitmapRangeIterator() {
+ delete[] _buf;
+ }
+
+ bool has_more_range() const { return !_eof; }
+
+ // read next range into [*from, *to) whose size <= max_range_size.
+ // return false when there is no more range.
+ bool next_range(uint32_t max_range_size, uint32_t* from, uint32_t* to) {
+ if (_eof) {
+ return false;
+ }
+ *from = _buf[_buf_pos];
+ uint32_t range_size = 0;
+ do {
+ _last_val = _buf[_buf_pos];
+ _buf_pos++;
+ range_size++;
+ if (_buf_pos == _buf_size) { // read next batch
+ _read_next_batch();
+ }
+ } while (range_size < max_range_size && !_eof && _buf[_buf_pos] == _last_val + 1);
+ *to = *from + range_size;
+ return true;
+ }
+
+private:
+ void _read_next_batch() {
+ uint32_t n = roaring_read_uint32_iterator(&_iter, _buf, kBatchSize);
+ _buf_pos = 0;
+ _buf_size = n;
+ _eof = n == 0;
+ }
+
+ static const uint32_t kBatchSize = 256;
+ roaring_uint32_iterator_t _iter;
+ uint32_t _last_val;
+ uint32_t* _buf = nullptr;
+ uint32_t _buf_pos;
+ uint32_t _buf_size;
+ bool _eof;
Review comment:
init them in contructor
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2547:
[Segment V2] Support lazy materialization read
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2547: [Segment V2] Support lazy materialization read
URL: https://github.com/apache/incubator-doris/pull/2547#discussion_r361301431
##########
File path: be/src/olap/rowset/segment_v2/segment_iterator.cpp
##########
@@ -353,31 +418,56 @@ Status SegmentIterator::_seek_columns(const std::vector<ColumnId>& column_ids, r
return Status::OK();
}
+Status SegmentIterator::_read_columns(const std::vector<ColumnId>& column_ids,
+ RowBlockV2* block,
+ size_t row_offset,
+ size_t nrows) {
+ for (auto cid : column_ids) {
+ auto column_block = block->column_block(cid);
+ ColumnBlockView dst(&column_block, row_offset);
+ size_t rows_read = nrows;
+ RETURN_IF_ERROR(_column_iterators[cid]->next_batch(&rows_read, &dst));
+ block->set_delete_state(column_block.delete_state());
+ DCHECK_EQ(nrows, rows_read);
+ }
+ return Status::OK();
+}
+
Status SegmentIterator::next_batch(RowBlockV2* block) {
SCOPED_RAW_TIMER(&_opts.stats->block_load_ns);
if (UNLIKELY(!_inited)) {
RETURN_IF_ERROR(_init());
+ if (_lazy_materialization_read) {
+ _block_rowids.reserve(block->capacity());
+ }
_inited = true;
}
uint32_t total_read = 0;
uint32_t remaining = block->capacity();
- // trying to fill in block
+ _block_rowids.resize(0);
+ const auto& read_columns = _lazy_materialization_read ? _predicate_columns : block->schema()->column_ids();
+
+ // phase 1: read rows selected by various index (indicated by _row_bitmap) into block
+ // when using lazy-materialization-read, only columns with predicates are read
do {
uint32_t range_from;
uint32_t range_to;
bool has_next_range = _range_iter->next_range(remaining, &range_from, &range_to);
if (!has_next_range) {
break;
}
+ if (_lazy_materialization_read) {
+ for (uint32_t rid = range_from; rid < range_to; rid++) {
+ _block_rowids.push_back(rid);
Review comment:
Yes, this is what I mean.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on a change in pull request
#2547: [Segment V2] Support lazy materialization read
Posted by GitBox <gi...@apache.org>.
gaodayue commented on a change in pull request #2547: [Segment V2] Support lazy materialization read
URL: https://github.com/apache/incubator-doris/pull/2547#discussion_r361248603
##########
File path: be/src/olap/rowset/segment_v2/segment_iterator.cpp
##########
@@ -314,34 +385,28 @@ Status SegmentIterator::_seek_and_peek(rowid_t rowid) {
// please note that usually RowBlockV2.clear() is called to free MemPool memory before reading the next block,
// but here since there won't be too many keys to seek, we don't call RowBlockV2.clear() so that we can use
// a single MemPool for all seeked keys.
- RETURN_IF_ERROR(_next_batch(_seek_block.get(), 0, &num_rows));
+ RETURN_IF_ERROR(_read_columns(_seek_schema->column_ids(), _seek_block.get(), 0, num_rows));
_seek_block->set_num_rows(num_rows);
return Status::OK();
}
-// Trying to read `rows_read` rows into `block` at the given offset.
-// Return the actual number of rows read in `*rows_read`.
-Status SegmentIterator::_next_batch(RowBlockV2* block, size_t row_offset, size_t* rows_read) {
- bool has_read = false;
- size_t first_read = 0;
- for (auto cid : block->schema()->column_ids()) {
- size_t num_rows = has_read ? first_read : *rows_read;
- auto column_block = block->column_block(cid);
- ColumnBlockView dst(&column_block, row_offset);
- RETURN_IF_ERROR(_column_iterators[cid]->next_batch(&num_rows, &dst));
- block->set_delete_state(column_block.delete_state());
- if (!has_read) {
- has_read = true;
- first_read = num_rows;
- } else if (num_rows != first_read) {
- return Status::InternalError(
- Substitute("Read different rows in different columns"
- ", column($0) read $1 vs column($2) read $3",
- block->schema()->column_ids()[0], first_read, cid, num_rows));
+void SegmentIterator::_init_lazy_materialization() {
+ if (!_col_predicates.empty()) {
+ std::set<ColumnId> predicate_columns;
+ for (auto predicate : _col_predicates) {
+ predicate_columns.insert(predicate->column_id());
+ }
+ // when all return columns have predicates, disable lazy materialization to avoid its overhead
+ if (_schema.column_ids().size() > predicate_columns.size()) {
+ _lazy_materialization_read = true;
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on a change in pull request
#2547: [Segment V2] Support lazy materialization read
Posted by GitBox <gi...@apache.org>.
gaodayue commented on a change in pull request #2547: [Segment V2] Support lazy materialization read
URL: https://github.com/apache/incubator-doris/pull/2547#discussion_r361300838
##########
File path: be/src/olap/rowset/segment_v2/segment_iterator.cpp
##########
@@ -353,31 +418,56 @@ Status SegmentIterator::_seek_columns(const std::vector<ColumnId>& column_ids, r
return Status::OK();
}
+Status SegmentIterator::_read_columns(const std::vector<ColumnId>& column_ids,
+ RowBlockV2* block,
+ size_t row_offset,
+ size_t nrows) {
+ for (auto cid : column_ids) {
+ auto column_block = block->column_block(cid);
+ ColumnBlockView dst(&column_block, row_offset);
+ size_t rows_read = nrows;
+ RETURN_IF_ERROR(_column_iterators[cid]->next_batch(&rows_read, &dst));
+ block->set_delete_state(column_block.delete_state());
+ DCHECK_EQ(nrows, rows_read);
+ }
+ return Status::OK();
+}
+
Status SegmentIterator::next_batch(RowBlockV2* block) {
SCOPED_RAW_TIMER(&_opts.stats->block_load_ns);
if (UNLIKELY(!_inited)) {
RETURN_IF_ERROR(_init());
+ if (_lazy_materialization_read) {
+ _block_rowids.reserve(block->capacity());
+ }
_inited = true;
}
uint32_t total_read = 0;
uint32_t remaining = block->capacity();
- // trying to fill in block
+ _block_rowids.resize(0);
+ const auto& read_columns = _lazy_materialization_read ? _predicate_columns : block->schema()->column_ids();
+
+ // phase 1: read rows selected by various index (indicated by _row_bitmap) into block
+ // when using lazy-materialization-read, only columns with predicates are read
do {
uint32_t range_from;
uint32_t range_to;
bool has_next_range = _range_iter->next_range(remaining, &range_from, &range_to);
if (!has_next_range) {
break;
}
+ if (_lazy_materialization_read) {
+ for (uint32_t rid = range_from; rid < range_to; rid++) {
+ _block_rowids.push_back(rid);
Review comment:
OK, you mean replace push_back with something like `_block_rowids[idx++] = rid` to avoid the comparison?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay merged pull request #2547: [Segment V2]
Support lazy materialization read
Posted by GitBox <gi...@apache.org>.
imay merged pull request #2547: [Segment V2] Support lazy materialization read
URL: https://github.com/apache/incubator-doris/pull/2547
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] imay commented on a change in pull request #2547:
[Segment V2] Support lazy materialization read
Posted by GitBox <gi...@apache.org>.
imay commented on a change in pull request #2547: [Segment V2] Support lazy materialization read
URL: https://github.com/apache/incubator-doris/pull/2547#discussion_r361249752
##########
File path: be/src/olap/rowset/segment_v2/segment_iterator.cpp
##########
@@ -353,31 +418,56 @@ Status SegmentIterator::_seek_columns(const std::vector<ColumnId>& column_ids, r
return Status::OK();
}
+Status SegmentIterator::_read_columns(const std::vector<ColumnId>& column_ids,
+ RowBlockV2* block,
+ size_t row_offset,
+ size_t nrows) {
+ for (auto cid : column_ids) {
+ auto column_block = block->column_block(cid);
+ ColumnBlockView dst(&column_block, row_offset);
+ size_t rows_read = nrows;
+ RETURN_IF_ERROR(_column_iterators[cid]->next_batch(&rows_read, &dst));
+ block->set_delete_state(column_block.delete_state());
+ DCHECK_EQ(nrows, rows_read);
+ }
+ return Status::OK();
+}
+
Status SegmentIterator::next_batch(RowBlockV2* block) {
SCOPED_RAW_TIMER(&_opts.stats->block_load_ns);
if (UNLIKELY(!_inited)) {
RETURN_IF_ERROR(_init());
+ if (_lazy_materialization_read) {
+ _block_rowids.reserve(block->capacity());
+ }
_inited = true;
}
uint32_t total_read = 0;
uint32_t remaining = block->capacity();
- // trying to fill in block
+ _block_rowids.resize(0);
+ const auto& read_columns = _lazy_materialization_read ? _predicate_columns : block->schema()->column_ids();
+
+ // phase 1: read rows selected by various index (indicated by _row_bitmap) into block
+ // when using lazy-materialization-read, only columns with predicates are read
do {
uint32_t range_from;
uint32_t range_to;
bool has_next_range = _range_iter->next_range(remaining, &range_from, &range_to);
if (!has_next_range) {
break;
}
+ if (_lazy_materialization_read) {
+ for (uint32_t rid = range_from; rid < range_to; rid++) {
+ _block_rowids.push_back(rid);
Review comment:
push_back should always have a judgment whether the operation is out of bounds. If you use an array instead, you can avoid this operation.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] kangpinghuang commented on issue #2547: [Segment
V2] Support lazy materialization read
Posted by GitBox <gi...@apache.org>.
kangpinghuang commented on issue #2547: [Segment V2] Support lazy materialization read
URL: https://github.com/apache/incubator-doris/pull/2547#issuecomment-568712914
can you add an unit test for lazy materialization?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on a change in pull request
#2547: [Segment V2] Support lazy materialization read
Posted by GitBox <gi...@apache.org>.
gaodayue commented on a change in pull request #2547: [Segment V2] Support lazy materialization read
URL: https://github.com/apache/incubator-doris/pull/2547#discussion_r361248501
##########
File path: be/src/olap/rowset/segment_v2/segment_iterator.cpp
##########
@@ -34,13 +34,71 @@ using strings::Substitute;
namespace doris {
namespace segment_v2 {
+// A fast range iterator for roaring bitmap. Output ranges use closed-open form, like [from, to).
+// Example:
+// input bitmap: [0 1 4 5 6 7 10 15 16 17 18 19]
+// output ranges: [0,2), [4,8), [10,11), [15,20) (when max_range_size=10)
+// output ranges: [0,2), [4,8), [10,11), [15,18), [18,20) (when max_range_size=3)
+class SegmentIterator::BitmapRangeIterator {
+public:
+ explicit BitmapRangeIterator(const Roaring& bitmap) {
+ roaring_init_iterator(&bitmap.roaring, &_iter);
+ _last_val = 0;
+ _buf = new uint32_t[256];
+ _read_next_batch();
+ }
+
+ ~BitmapRangeIterator() {
+ delete[] _buf;
+ }
+
+ bool has_more_range() const { return !_eof; }
+
+ // read next range into [*from, *to) whose size <= max_range_size.
+ // return false when there is no more range.
+ bool next_range(uint32_t max_range_size, uint32_t* from, uint32_t* to) {
+ if (_eof) {
+ return false;
+ }
+ *from = _buf[_buf_pos];
+ uint32_t range_size = 0;
+ do {
+ _last_val = _buf[_buf_pos];
+ _buf_pos++;
+ range_size++;
+ if (_buf_pos == _buf_size) { // read next batch
+ _read_next_batch();
+ }
+ } while (range_size < max_range_size && !_eof && _buf[_buf_pos] == _last_val + 1);
+ *to = *from + range_size;
+ return true;
+ }
+
+private:
+ void _read_next_batch() {
+ uint32_t n = roaring_read_uint32_iterator(&_iter, _buf, kBatchSize);
+ _buf_pos = 0;
+ _buf_size = n;
+ _eof = n == 0;
+ }
+
+ static const uint32_t kBatchSize = 256;
+ roaring_uint32_iterator_t _iter;
+ uint32_t _last_val;
+ uint32_t* _buf = nullptr;
+ uint32_t _buf_pos;
+ uint32_t _buf_size;
+ bool _eof;
Review comment:
ok
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] gaodayue commented on a change in pull request
#2547: [Segment V2] Support lazy materialization read
Posted by GitBox <gi...@apache.org>.
gaodayue commented on a change in pull request #2547: [Segment V2] Support lazy materialization read
URL: https://github.com/apache/incubator-doris/pull/2547#discussion_r361249110
##########
File path: be/src/olap/rowset/segment_v2/segment_iterator.cpp
##########
@@ -353,31 +418,56 @@ Status SegmentIterator::_seek_columns(const std::vector<ColumnId>& column_ids, r
return Status::OK();
}
+Status SegmentIterator::_read_columns(const std::vector<ColumnId>& column_ids,
+ RowBlockV2* block,
+ size_t row_offset,
+ size_t nrows) {
+ for (auto cid : column_ids) {
+ auto column_block = block->column_block(cid);
+ ColumnBlockView dst(&column_block, row_offset);
+ size_t rows_read = nrows;
+ RETURN_IF_ERROR(_column_iterators[cid]->next_batch(&rows_read, &dst));
+ block->set_delete_state(column_block.delete_state());
+ DCHECK_EQ(nrows, rows_read);
+ }
+ return Status::OK();
+}
+
Status SegmentIterator::next_batch(RowBlockV2* block) {
SCOPED_RAW_TIMER(&_opts.stats->block_load_ns);
if (UNLIKELY(!_inited)) {
RETURN_IF_ERROR(_init());
+ if (_lazy_materialization_read) {
+ _block_rowids.reserve(block->capacity());
+ }
_inited = true;
}
uint32_t total_read = 0;
uint32_t remaining = block->capacity();
- // trying to fill in block
+ _block_rowids.resize(0);
+ const auto& read_columns = _lazy_materialization_read ? _predicate_columns : block->schema()->column_ids();
+
+ // phase 1: read rows selected by various index (indicated by _row_bitmap) into block
+ // when using lazy-materialization-read, only columns with predicates are read
do {
uint32_t range_from;
uint32_t range_to;
bool has_next_range = _range_iter->next_range(remaining, &range_from, &range_to);
if (!has_next_range) {
break;
}
+ if (_lazy_materialization_read) {
+ for (uint32_t rid = range_from; rid < range_to; rid++) {
+ _block_rowids.push_back(rid);
Review comment:
Sorry I don't get it. What do you mean?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org