You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/09/22 05:57:51 UTC
[GitHub] [doris] Gabriel39 opened a new pull request, #12852: [Improvement](dict) optimize dictionary column
Gabriel39 opened a new pull request, #12852:
URL: https://github.com/apache/doris/pull/12852
# Proposed changes
Issue Number: close #xxx
## Problem summary
Describe your changes.
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zenoyang commented on pull request #12852: [Improvement](dict) optimize dictionary column
Posted by GitBox <gi...@apache.org>.
zenoyang commented on PR #12852:
URL: https://github.com/apache/doris/pull/12852#issuecomment-1255845045
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Gabriel39 commented on a diff in pull request #12852: [Improvement](dict) optimize dictionary column
Posted by GitBox <gi...@apache.org>.
Gabriel39 commented on code in PR #12852:
URL: https://github.com/apache/doris/pull/12852#discussion_r978294410
##########
be/src/vec/columns/column_dictionary.h:
##########
@@ -360,40 +356,50 @@ class ColumnDictionary final : public COWHelper<IColumn, ColumnDictionary<T>> {
if (code >= 0) {
return code;
}
- auto bound = std::upper_bound(_dict_data.begin(), _dict_data.end(), value) -
- _dict_data.begin();
+ auto bound = std::upper_bound(_dict_data->begin(), _dict_data->end(), value) -
+ _dict_data->begin();
return greater ? bound - greater + eq : bound - eq;
}
void find_codes(const phmap::flat_hash_set<StringValue>& values,
std::vector<vectorized::UInt8>& selected) const {
- size_t dict_word_num = _dict_data.size();
+ size_t dict_word_num = _dict_data->size();
selected.resize(dict_word_num);
selected.assign(dict_word_num, false);
- for (const auto& value : values) {
- if (auto it = _inverted_index.find(value); it != _inverted_index.end()) {
- selected[it->second] = true;
+ for (size_t i = 0; i < _dict_data->size(); i++) {
+ if (values.find((*_dict_data)[i]) != values.end()) {
+ selected[i] = true;
}
}
}
void clear() {
- _dict_data.clear();
- _inverted_index.clear();
- _code_convert_table.clear();
Review Comment:
Sorry, this is a mistake and I fixed it.
Btw, I provide a performance test in this PR's description
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zenoyang commented on a diff in pull request #12852: [Improvement](dict) optimize dictionary column
Posted by GitBox <gi...@apache.org>.
zenoyang commented on code in PR #12852:
URL: https://github.com/apache/doris/pull/12852#discussion_r978259350
##########
be/src/vec/columns/column_dictionary.h:
##########
@@ -360,40 +356,50 @@ class ColumnDictionary final : public COWHelper<IColumn, ColumnDictionary<T>> {
if (code >= 0) {
return code;
}
- auto bound = std::upper_bound(_dict_data.begin(), _dict_data.end(), value) -
- _dict_data.begin();
+ auto bound = std::upper_bound(_dict_data->begin(), _dict_data->end(), value) -
+ _dict_data->begin();
return greater ? bound - greater + eq : bound - eq;
}
void find_codes(const phmap::flat_hash_set<StringValue>& values,
std::vector<vectorized::UInt8>& selected) const {
- size_t dict_word_num = _dict_data.size();
+ size_t dict_word_num = _dict_data->size();
selected.resize(dict_word_num);
selected.assign(dict_word_num, false);
- for (const auto& value : values) {
- if (auto it = _inverted_index.find(value); it != _inverted_index.end()) {
- selected[it->second] = true;
+ for (size_t i = 0; i < _dict_data->size(); i++) {
+ if (values.find((*_dict_data)[i]) != values.end()) {
+ selected[i] = true;
}
}
}
void clear() {
- _dict_data.clear();
- _inverted_index.clear();
- _code_convert_table.clear();
Review Comment:
why remove `_code_convert_table.clear()`?
In addition, can you provide the performance improvement data for this optimization?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12852: [Improvement](dict) optimize dictionary column
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #12852:
URL: https://github.com/apache/doris/pull/12852#issuecomment-1256488125
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Gabriel39 commented on a diff in pull request #12852: [Improvement](dict) optimize dictionary column
Posted by GitBox <gi...@apache.org>.
Gabriel39 commented on code in PR #12852:
URL: https://github.com/apache/doris/pull/12852#discussion_r977345927
##########
be/src/vec/columns/column_dictionary.h:
##########
@@ -360,40 +362,58 @@ class ColumnDictionary final : public COWHelper<IColumn, ColumnDictionary<T>> {
if (code >= 0) {
return code;
}
- auto bound = std::upper_bound(_dict_data.begin(), _dict_data.end(), value) -
- _dict_data.begin();
+ auto bound = std::upper_bound(_dict_data->begin(), _dict_data->end(), value) -
+ _dict_data->begin();
return greater ? bound - greater + eq : bound - eq;
}
void find_codes(const phmap::flat_hash_set<StringValue>& values,
std::vector<vectorized::UInt8>& selected) const {
- size_t dict_word_num = _dict_data.size();
+ size_t dict_word_num = _dict_data->size();
selected.resize(dict_word_num);
selected.assign(dict_word_num, false);
- for (const auto& value : values) {
- if (auto it = _inverted_index.find(value); it != _inverted_index.end()) {
- selected[it->second] = true;
+ for (size_t i = 0; i < _dict_data->size(); i++) {
+ if (values.find((*_dict_data)[i]) != values.end()) {
+ selected[i] = true;
}
}
}
void clear() {
- _dict_data.clear();
- _inverted_index.clear();
- _code_convert_table.clear();
+ _dict_data->clear();
_hash_values.clear();
}
void clear_hash_values() { _hash_values.clear(); }
void sort() {
- size_t dict_size = _dict_data.size();
- _code_convert_table.reserve(dict_size);
- std::sort(_dict_data.begin(), _dict_data.end(), _comparator);
+ size_t dict_size = _dict_data->size();
+
+ _perm.resize(dict_size);
+ for (size_t i = 0; i < dict_size; ++i) {
+ _perm[i] = i;
+ }
+
+ struct Comparator {
+ public:
+ Comparator(DictContainer& dict_data) : _dict_data(dict_data) {}
+ bool operator()(const size_t a, const size_t b) const {
+ return _comparator(_dict_data[a], _dict_data[b]);
+ }
+
+ private:
+ StringValue::Comparator _comparator;
+ DictContainer& _dict_data;
+ };
+ Comparator comparator(*_dict_data);
+ std::sort(_perm.begin(), _perm.end(), comparator);
Review Comment:
Done, thanks for your suggestion!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] BiteTheDDDDt commented on a diff in pull request #12852: [Improvement](dict) optimize dictionary column
Posted by GitBox <gi...@apache.org>.
BiteTheDDDDt commented on code in PR #12852:
URL: https://github.com/apache/doris/pull/12852#discussion_r977291889
##########
be/src/vec/columns/column_dictionary.h:
##########
@@ -192,11 +192,13 @@ class ColumnDictionary final : public COWHelper<IColumn, ColumnDictionary<T>> {
Status filter_by_selector(const uint16_t* sel, size_t sel_size, IColumn* col_ptr) override {
auto* res_col = reinterpret_cast<vectorized::ColumnString*>(col_ptr);
+ res_col->get_offsets().reserve(sel_size);
+ res_col->get_chars().reserve(_dict.avg_str_len() * sel_size);
for (size_t i = 0; i < sel_size; i++) {
uint16_t n = sel[i];
auto& code = reinterpret_cast<T&>(_codes[n]);
auto value = _dict.get_value(code);
- res_col->insert_data(value.ptr, value.len);
+ res_col->insert_data_without_reserve(value.ptr, value.len);
Review Comment:
I think `_dict.avg_str_len() * sel_size` may be less than sum length of elements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] adonis0147 commented on a diff in pull request #12852: [Improvement](dict) optimize dictionary column
Posted by GitBox <gi...@apache.org>.
adonis0147 commented on code in PR #12852:
URL: https://github.com/apache/doris/pull/12852#discussion_r977277813
##########
be/src/vec/columns/column_dictionary.h:
##########
@@ -360,40 +362,58 @@ class ColumnDictionary final : public COWHelper<IColumn, ColumnDictionary<T>> {
if (code >= 0) {
return code;
}
- auto bound = std::upper_bound(_dict_data.begin(), _dict_data.end(), value) -
- _dict_data.begin();
+ auto bound = std::upper_bound(_dict_data->begin(), _dict_data->end(), value) -
+ _dict_data->begin();
return greater ? bound - greater + eq : bound - eq;
}
void find_codes(const phmap::flat_hash_set<StringValue>& values,
std::vector<vectorized::UInt8>& selected) const {
- size_t dict_word_num = _dict_data.size();
+ size_t dict_word_num = _dict_data->size();
selected.resize(dict_word_num);
selected.assign(dict_word_num, false);
- for (const auto& value : values) {
- if (auto it = _inverted_index.find(value); it != _inverted_index.end()) {
- selected[it->second] = true;
+ for (size_t i = 0; i < _dict_data->size(); i++) {
+ if (values.find((*_dict_data)[i]) != values.end()) {
+ selected[i] = true;
}
}
}
void clear() {
- _dict_data.clear();
- _inverted_index.clear();
- _code_convert_table.clear();
+ _dict_data->clear();
_hash_values.clear();
}
void clear_hash_values() { _hash_values.clear(); }
void sort() {
- size_t dict_size = _dict_data.size();
- _code_convert_table.reserve(dict_size);
- std::sort(_dict_data.begin(), _dict_data.end(), _comparator);
+ size_t dict_size = _dict_data->size();
+
+ _perm.resize(dict_size);
+ for (size_t i = 0; i < dict_size; ++i) {
+ _perm[i] = i;
+ }
+
+ struct Comparator {
+ public:
+ Comparator(DictContainer& dict_data) : _dict_data(dict_data) {}
+ bool operator()(const size_t a, const size_t b) const {
+ return _comparator(_dict_data[a], _dict_data[b]);
+ }
+
+ private:
+ StringValue::Comparator _comparator;
+ DictContainer& _dict_data;
+ };
+ Comparator comparator(*_dict_data);
+ std::sort(_perm.begin(), _perm.end(), comparator);
Review Comment:
```suggestion
std::sort(_perm.begin(), _perm.end(),
[&dict_data = *_dict_data, &comparator = _comparator](const size_t a,
const size_t b) {
return comparator(dict_data[a], dict_data[b]);
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee merged pull request #12852: [Improvement](dict) optimize dictionary column
Posted by GitBox <gi...@apache.org>.
HappenLee merged PR #12852:
URL: https://github.com/apache/doris/pull/12852
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] Gabriel39 commented on a diff in pull request #12852: [Improvement](dict) optimize dictionary column
Posted by GitBox <gi...@apache.org>.
Gabriel39 commented on code in PR #12852:
URL: https://github.com/apache/doris/pull/12852#discussion_r977308937
##########
be/src/vec/columns/column_dictionary.h:
##########
@@ -192,11 +192,13 @@ class ColumnDictionary final : public COWHelper<IColumn, ColumnDictionary<T>> {
Status filter_by_selector(const uint16_t* sel, size_t sel_size, IColumn* col_ptr) override {
auto* res_col = reinterpret_cast<vectorized::ColumnString*>(col_ptr);
+ res_col->get_offsets().reserve(sel_size);
+ res_col->get_chars().reserve(_dict.avg_str_len() * sel_size);
for (size_t i = 0; i < sel_size; i++) {
uint16_t n = sel[i];
auto& code = reinterpret_cast<T&>(_codes[n]);
auto value = _dict.get_value(code);
- res_col->insert_data(value.ptr, value.len);
+ res_col->insert_data_without_reserve(value.ptr, value.len);
Review Comment:
If so, `chars` in ColumnString will still to reserve a bigger memory block. `_dict.avg_str_len() * sel_size` is just a conservative estimation here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #12852: [Improvement](dict) optimize dictionary column
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #12852:
URL: https://github.com/apache/doris/pull/12852#issuecomment-1256488045
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org