You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "mrhhsg (via GitHub)" <gi...@apache.org> on 2023/02/17 01:29:31 UTC
[GitHub] [doris] mrhhsg opened a new pull request, #16850: [improvement][vec] avoid creating a new column while filtering mutable columns
mrhhsg opened a new pull request, #16850:
URL: https://github.com/apache/doris/pull/16850
# Proposed changes
Currently, when filtering a column, a new column will be created to store the filtering result, which will cause some performance loss
## Problem summary
Describe your changes.
## Checklist(Required)
* [ ] Does it affect the original behavior
* [ ] Has unit tests been added
* [ ] Has document been added or modified
* [ ] Does it need to update dependencies
* [ ] Is this PR support rollback (If NO, please explain WHY)
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16850:
URL: https://github.com/apache/doris/pull/16850#issuecomment-1436421503
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16850:
URL: https://github.com/apache/doris/pull/16850#issuecomment-1437105520
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] HappenLee commented on a diff in pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "HappenLee (via GitHub)" <gi...@apache.org>.
HappenLee commented on code in PR #16850:
URL: https://github.com/apache/doris/pull/16850#discussion_r1111429794
##########
be/src/vec/columns/column_complex.h:
##########
@@ -327,6 +329,36 @@ ColumnPtr ColumnComplexType<T>::filter(const IColumn::Filter& filt,
return res;
}
+template <typename T>
+size_t ColumnComplexType<T>::filter(const IColumn::Filter& filter) {
+ size_t size = data.size();
+ if (size != filter.size()) {
+ LOG(FATAL) << "Size of filter doesn't match size of column.";
+ }
+
+ if (data.size() == 0) {
+ return 0;
+ }
+
+ T* res_data = data.data();
+
+ const UInt8* filter_pos = filter.data();
+ const UInt8* filter_end = filter_pos + size;
+ const T* data_pos = data.data();
+
+ while (filter_pos < filter_end) {
+ if (*filter_pos) {
+ *res_data = *data_pos;
Review Comment:
better do std::move(*data_pos) ?
##########
be/src/vec/columns/column_complex.h:
##########
@@ -327,6 +329,36 @@ ColumnPtr ColumnComplexType<T>::filter(const IColumn::Filter& filt,
return res;
}
+template <typename T>
+size_t ColumnComplexType<T>::filter(const IColumn::Filter& filter) {
+ size_t size = data.size();
+ if (size != filter.size()) {
+ LOG(FATAL) << "Size of filter doesn't match size of column.";
+ }
+
+ if (data.size() == 0) {
+ return 0;
+ }
+
+ T* res_data = data.data();
+
+ const UInt8* filter_pos = filter.data();
+ const UInt8* filter_end = filter_pos + size;
+ const T* data_pos = data.data();
+
+ while (filter_pos < filter_end) {
+ if (*filter_pos) {
+ *res_data = *data_pos;
Review Comment:
better do std::move(*data_pos) ?
##########
be/src/vec/columns/column_complex.h:
##########
@@ -327,6 +329,36 @@ ColumnPtr ColumnComplexType<T>::filter(const IColumn::Filter& filt,
return res;
}
+template <typename T>
+size_t ColumnComplexType<T>::filter(const IColumn::Filter& filter) {
+ size_t size = data.size();
+ if (size != filter.size()) {
+ LOG(FATAL) << "Size of filter doesn't match size of column.";
+ }
+
+ if (data.size() == 0) {
+ return 0;
+ }
+
+ T* res_data = data.data();
+
+ const UInt8* filter_pos = filter.data();
+ const UInt8* filter_end = filter_pos + size;
+ const T* data_pos = data.data();
+
+ while (filter_pos < filter_end) {
+ if (*filter_pos) {
+ *res_data = *data_pos;
Review Comment:
better do std::move(*data_pos) ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16850:
URL: https://github.com/apache/doris/pull/16850#issuecomment-1433977151
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16850:
URL: https://github.com/apache/doris/pull/16850#issuecomment-1433976996
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei merged pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei merged PR #16850:
URL: https://github.com/apache/doris/pull/16850
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16850:
URL: https://github.com/apache/doris/pull/16850#issuecomment-1434271403
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16850:
URL: https://github.com/apache/doris/pull/16850#issuecomment-1437747928
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16850:
URL: https://github.com/apache/doris/pull/16850#issuecomment-1433977695
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei commented on a diff in pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on code in PR #16850:
URL: https://github.com/apache/doris/pull/16850#discussion_r1111365388
##########
be/src/vec/columns/column_decimal.cpp:
##########
@@ -325,6 +325,61 @@ ColumnPtr ColumnDecimal<T>::filter(const IColumn::Filter& filt, ssize_t result_s
return res;
}
+template <typename T>
+size_t ColumnDecimal<T>::filter(const IColumn::Filter& filter) {
+ size_t size = data.size();
+ if (size != filter.size()) {
+ LOG(FATAL) << "Size of filter doesn't match size of column.";
+ }
+
+ const UInt8* filter_pos = filter.data();
+ const UInt8* filter_end = filter_pos + size;
+ const T* data_pos = data.data();
+ T* result_data = data.data();
+
+ /** A slightly more optimized version.
+ * Based on the assumption that often pieces of consecutive values
+ * completely pass or do not pass the filter.
+ * Therefore, we will optimistically check the parts of `SIMD_BYTES` values.
+ */
+ static constexpr size_t SIMD_BYTES = 32;
+ const UInt8* filter_end_sse = filter_pos + size / SIMD_BYTES * SIMD_BYTES;
+
+ while (filter_pos < filter_end_sse) {
+ uint32_t mask = simd::bytes32_mask_to_bits32_mask(filter_pos);
+
+ if (0xFFFFFFFF == mask) {
+ memmove(result_data, data_pos, sizeof(T) * SIMD_BYTES);
+ result_data += SIMD_BYTES;
+ } else {
+ while (mask) {
+ const size_t idx = __builtin_ctzll(mask);
+ *result_data = data_pos[idx];
+ ++result_data;
+ mask = mask & (mask - 1);
+ }
+ }
+
+ filter_pos += SIMD_BYTES;
+ data_pos += SIMD_BYTES;
+ }
+
+ while (filter_pos < filter_end) {
+ if (*filter_pos) {
+ *result_data = *data_pos;
+ ++result_data;
+ }
+
+ ++filter_pos;
+ ++data_pos;
+ }
+
+ const auto result_size = result_data - data.data();
+ data.set_end_ptr(result_data);
+
+ return result_size;
Review Comment:
not call resize？ Not change size?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16850:
URL: https://github.com/apache/doris/pull/16850#issuecomment-1434800647
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mrhhsg commented on a diff in pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on code in PR #16850:
URL: https://github.com/apache/doris/pull/16850#discussion_r1111371859
##########
be/src/vec/columns/column_decimal.cpp:
##########
@@ -325,6 +325,61 @@ ColumnPtr ColumnDecimal<T>::filter(const IColumn::Filter& filt, ssize_t result_s
return res;
}
+template <typename T>
+size_t ColumnDecimal<T>::filter(const IColumn::Filter& filter) {
+ size_t size = data.size();
+ if (size != filter.size()) {
+ LOG(FATAL) << "Size of filter doesn't match size of column.";
+ }
+
+ const UInt8* filter_pos = filter.data();
+ const UInt8* filter_end = filter_pos + size;
+ const T* data_pos = data.data();
+ T* result_data = data.data();
+
+ /** A slightly more optimized version.
+ * Based on the assumption that often pieces of consecutive values
+ * completely pass or do not pass the filter.
+ * Therefore, we will optimistically check the parts of `SIMD_BYTES` values.
+ */
+ static constexpr size_t SIMD_BYTES = 32;
+ const UInt8* filter_end_sse = filter_pos + size / SIMD_BYTES * SIMD_BYTES;
+
+ while (filter_pos < filter_end_sse) {
+ uint32_t mask = simd::bytes32_mask_to_bits32_mask(filter_pos);
+
+ if (0xFFFFFFFF == mask) {
+ memmove(result_data, data_pos, sizeof(T) * SIMD_BYTES);
+ result_data += SIMD_BYTES;
+ } else {
+ while (mask) {
+ const size_t idx = __builtin_ctzll(mask);
+ *result_data = data_pos[idx];
+ ++result_data;
+ mask = mask & (mask - 1);
+ }
+ }
+
+ filter_pos += SIMD_BYTES;
+ data_pos += SIMD_BYTES;
+ }
+
+ while (filter_pos < filter_end) {
+ if (*filter_pos) {
+ *result_data = *data_pos;
+ ++result_data;
+ }
+
+ ++filter_pos;
+ ++data_pos;
+ }
+
+ const auto result_size = result_data - data.data();
+ data.set_end_ptr(result_data);
+
+ return result_size;
Review Comment:
`set_end_ptr` does change the size.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16850:
URL: https://github.com/apache/doris/pull/16850#issuecomment-1437747949
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] mrhhsg commented on pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "mrhhsg (via GitHub)" <gi...@apache.org>.
mrhhsg commented on PR #16850:
URL: https://github.com/apache/doris/pull/16850#issuecomment-1437211532
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] github-actions[bot] commented on pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #16850:
URL: https://github.com/apache/doris/pull/16850#issuecomment-1434033468
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] yiguolei commented on pull request #16850: [improvement](vec) avoid creating a new column while filtering mutable columns
Posted by "yiguolei (via GitHub)" <gi...@apache.org>.
yiguolei commented on PR #16850:
URL: https://github.com/apache/doris/pull/16850#issuecomment-1436488975
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org