You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/05/11 07:44:02 UTC
[GitHub] [incubator-doris] wangbo opened a new pull request, #9500: [fix](storage) fix core for string predicate in storage layer
wangbo opened a new pull request, #9500:
URL: https://github.com/apache/incubator-doris/pull/9500
# Proposed changes
Issue Number: close #xxx
## Problem Summary:
Describe the overview of changes.
## Checklist(Required)
1. Does it affect the original behavior: (Yes/No/I Don't know)
2. Has unit tests been added: (Yes/No/No Need)
3. Has document been added or modified: (Yes/No/No Need)
4. Does it need to update dependencies: (Yes/No)
5. Are there any changes that cannot be rolled back: (Yes/No)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei merged pull request #9500: [fix](storage) fix core for string predicate in storage layer
Posted by GitBox <gi...@apache.org>.
yiguolei merged PR #9500:
URL: https://github.com/apache/incubator-doris/pull/9500
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #9500: [fix](storage) fix core for string predicate in storage layer
Posted by GitBox <gi...@apache.org>.
yiguolei commented on code in PR #9500:
URL: https://github.com/apache/incubator-doris/pull/9500#discussion_r869996443
##########
be/src/vec/columns/predicate_column.h:
##########
@@ -410,6 +429,9 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
private:
Container data;
+ // manages the memory for slice's data(For string type)
+ std::unique_ptr<MemPool> _pool;
+ bool _is_mem_pool_inited = false;;
Review Comment:
just check _pool == nullptr?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] github-actions[bot] commented on pull request #9500: [fix](storage) fix core for string predicate in storage layer
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #9500:
URL: https://github.com/apache/incubator-doris/pull/9500#issuecomment-1124615417
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] morningman commented on pull request #9500: [fix](storage) fix core for string predicate in storage layer
Posted by GitBox <gi...@apache.org>.
morningman commented on PR #9500:
URL: https://github.com/apache/incubator-doris/pull/9500#issuecomment-1124572308
It seems that we are likely to refer to the memory in the page elsewhere, and the same problem exists?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] yiguolei commented on a diff in pull request #9500: [fix](storage) fix core for string predicate in storage layer
Posted by GitBox <gi...@apache.org>.
yiguolei commented on code in PR #9500:
URL: https://github.com/apache/incubator-doris/pull/9500#discussion_r869997095
##########
be/src/vec/columns/predicate_column.h:
##########
@@ -230,17 +230,36 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
void insert_many_binary_data(char* data_array, uint32_t* len_array,
uint32_t* start_offset_array, size_t num) override {
if constexpr (std::is_same_v<T, StringValue>) {
+ if (!_is_mem_pool_inited) {
+ _pool.reset(new MemPool("PredicateStringColumn"));
+ _is_mem_pool_inited = true;
+ }
+
+ size_t total_mem_size = 0;
+ for (size_t i = 0; i < num; i++) {
+ total_mem_size += len_array[i];
+ }
+
+ char* destination = (char*)_pool->allocate(total_mem_size);
for (size_t i = 0; i < num; i++) {
uint32_t len = len_array[i];
uint32_t start_offset = start_offset_array[i];
- insert_string_value(data_array + start_offset, len);
+ memcpy(destination, data_array + start_offset, len);
+ StringValue sv(destination, len);
+ data.push_back_without_reserve(sv);
+ destination += len;
}
}
}
void insert_default() override { data.push_back(T()); }
- void clear() override { data.clear(); }
+ void clear() override {
+ data.clear();
+ if (_is_mem_pool_inited) {
Review Comment:
Will mempool release the memory when _pool is desconstructed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on a diff in pull request #9500: [fix](storage) fix core for string predicate in storage layer
Posted by GitBox <gi...@apache.org>.
wangbo commented on code in PR #9500:
URL: https://github.com/apache/incubator-doris/pull/9500#discussion_r870052286
##########
be/src/vec/columns/predicate_column.h:
##########
@@ -410,6 +429,9 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
private:
Container data;
+ // manages the memory for slice's data(For string type)
+ std::unique_ptr<MemPool> _pool;
+ bool _is_mem_pool_inited = false;;
Review Comment:
👌🏻
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on a diff in pull request #9500: [fix](storage) fix core for string predicate in storage layer
Posted by GitBox <gi...@apache.org>.
wangbo commented on code in PR #9500:
URL: https://github.com/apache/incubator-doris/pull/9500#discussion_r870050224
##########
be/src/vec/columns/predicate_column.h:
##########
@@ -230,17 +230,36 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
void insert_many_binary_data(char* data_array, uint32_t* len_array,
uint32_t* start_offset_array, size_t num) override {
if constexpr (std::is_same_v<T, StringValue>) {
+ if (!_is_mem_pool_inited) {
+ _pool.reset(new MemPool("PredicateStringColumn"));
+ _is_mem_pool_inited = true;
+ }
+
+ size_t total_mem_size = 0;
+ for (size_t i = 0; i < num; i++) {
+ total_mem_size += len_array[i];
+ }
+
+ char* destination = (char*)_pool->allocate(total_mem_size);
for (size_t i = 0; i < num; i++) {
uint32_t len = len_array[i];
uint32_t start_offset = start_offset_array[i];
- insert_string_value(data_array + start_offset, len);
+ memcpy(destination, data_array + start_offset, len);
+ StringValue sv(destination, len);
+ data.push_back_without_reserve(sv);
+ destination += len;
}
}
}
void insert_default() override { data.push_back(T()); }
- void clear() override { data.clear(); }
+ void clear() override {
+ data.clear();
+ if (_is_mem_pool_inited) {
Review Comment:
This is just for memory reuse.
```
mem_pool.h
/// Makes all allocated chunks available for re-use, but doesn't delete any chunks.
void clear();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] jacktengg commented on a diff in pull request #9500: [fix](storage) fix core for string predicate in storage layer
Posted by GitBox <gi...@apache.org>.
jacktengg commented on code in PR #9500:
URL: https://github.com/apache/incubator-doris/pull/9500#discussion_r870016316
##########
be/src/vec/columns/predicate_column.h:
##########
@@ -230,17 +230,36 @@ class PredicateColumnType final : public COWHelper<IColumn, PredicateColumnType<
void insert_many_binary_data(char* data_array, uint32_t* len_array,
uint32_t* start_offset_array, size_t num) override {
if constexpr (std::is_same_v<T, StringValue>) {
+ if (!_is_mem_pool_inited) {
+ _pool.reset(new MemPool("PredicateStringColumn"));
+ _is_mem_pool_inited = true;
+ }
+
+ size_t total_mem_size = 0;
+ for (size_t i = 0; i < num; i++) {
+ total_mem_size += len_array[i];
+ }
+
+ char* destination = (char*)_pool->allocate(total_mem_size);
for (size_t i = 0; i < num; i++) {
uint32_t len = len_array[i];
uint32_t start_offset = start_offset_array[i];
- insert_string_value(data_array + start_offset, len);
+ memcpy(destination, data_array + start_offset, len);
Review Comment:
Not sure how much performnce will be impacted by allocate and memcpy here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org