You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by "suxiaogang223 (via GitHub)" <gi...@apache.org> on 2024/10/11 01:11:25 UTC
[PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
suxiaogang223 opened a new pull request, #41683:
URL: https://github.com/apache/doris/pull/41683
## Proposed changes
Impl ByteStreamSplitDecoder to decode BYTE_STREAM_SPLIT encoding parquet.
relate pr: https://github.com/apache/arrow/issues/42372
> Apache Parquet does not have any encodings suitable for FP data and the available text compressors (zstd, gzip, etc) do not handle FP data very well.
It is possible to apply a simple data transformation named "stream splitting". Such could be "byte stream splitting" which creates K streams of length N where K is the number of bytes in the data type (4 for floats, 8 for doubles) and N is the number of elements in the sequence.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on code in PR #41683:
URL: https://github.com/apache/doris/pull/41683#discussion_r1796408866
##########
be/src/vec/exec/format/parquet/byte_stream_split_decoder.cpp:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "byte_stream_split_decoder.h"
+
+namespace doris::vectorized {
+void ByteStreamSplitDecoder::set_data(Slice* slice) {
+ Decoder::set_data(slice);
+ DCHECK(_data->get_size() % _type_length == 0);
+ auto byteSplitStreamNum = _data->get_size() / _type_length;
Review Comment:
```suggestion
auto byte_split_stream_num = _data->get_size() / _type_length;
```
Unify the code format. Same suggestion to other variable name
##########
be/src/vec/exec/format/parquet/byte_stream_split_decoder.cpp:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "byte_stream_split_decoder.h"
+
+namespace doris::vectorized {
+void ByteStreamSplitDecoder::set_data(Slice* slice) {
+ Decoder::set_data(slice);
+ DCHECK(_data->get_size() % _type_length == 0);
+ auto byteSplitStreamNum = _data->get_size() / _type_length;
+ _byteSplitStreams = std::vector<char*>(_type_length);
+ for (size_t i = 0; i < _type_length; ++i) {
+ _byteSplitStreams[i] = _data->data + i * byteSplitStreamNum;
+ }
+}
+
+Status ByteStreamSplitDecoder::decode_values(MutableColumnPtr& doris_column, DataTypePtr& data_type,
+ ColumnSelectVector& select_vector,
+ bool is_dict_filter) {
+ if (select_vector.has_filter()) {
+ return _decode_values<true>(doris_column, data_type, select_vector, is_dict_filter);
+ } else {
+ return _decode_values<false>(doris_column, data_type, select_vector, is_dict_filter);
+ }
+}
+
+template <bool has_filter>
+Status ByteStreamSplitDecoder::_decode_values(MutableColumnPtr& doris_column,
+ DataTypePtr& data_type,
+ ColumnSelectVector& select_vector,
+ bool is_dict_filter) {
+ size_t non_null_size = select_vector.num_values() - select_vector.num_nulls();
+ if (UNLIKELY(_offset + non_null_size > _data->size)) {
+ return Status::IOError("Out-of-bounds access in parquet data decoder");
Review Comment:
Print more info like `_offset`, `non_null_size`, `_data->size` in log
##########
be/src/vec/exec/format/parquet/byte_stream_split_decoder.cpp:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "byte_stream_split_decoder.h"
+
+namespace doris::vectorized {
+void ByteStreamSplitDecoder::set_data(Slice* slice) {
+ Decoder::set_data(slice);
+ DCHECK(_data->get_size() % _type_length == 0);
+ auto byteSplitStreamNum = _data->get_size() / _type_length;
Review Comment:
And this should be named `byte_split_stream_length`?
##########
be/src/vec/exec/format/parquet/byte_stream_split_decoder.cpp:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "byte_stream_split_decoder.h"
+
+namespace doris::vectorized {
+void ByteStreamSplitDecoder::set_data(Slice* slice) {
+ Decoder::set_data(slice);
+ DCHECK(_data->get_size() % _type_length == 0);
+ auto byteSplitStreamNum = _data->get_size() / _type_length;
+ _byteSplitStreams = std::vector<char*>(_type_length);
+ for (size_t i = 0; i < _type_length; ++i) {
+ _byteSplitStreams[i] = _data->data + i * byteSplitStreamNum;
+ }
+}
+
+Status ByteStreamSplitDecoder::decode_values(MutableColumnPtr& doris_column, DataTypePtr& data_type,
+ ColumnSelectVector& select_vector,
+ bool is_dict_filter) {
+ if (select_vector.has_filter()) {
+ return _decode_values<true>(doris_column, data_type, select_vector, is_dict_filter);
+ } else {
+ return _decode_values<false>(doris_column, data_type, select_vector, is_dict_filter);
+ }
+}
+
+template <bool has_filter>
+Status ByteStreamSplitDecoder::_decode_values(MutableColumnPtr& doris_column,
+ DataTypePtr& data_type,
+ ColumnSelectVector& select_vector,
+ bool is_dict_filter) {
+ size_t non_null_size = select_vector.num_values() - select_vector.num_nulls();
+ if (UNLIKELY(_offset + non_null_size > _data->size)) {
+ return Status::IOError("Out-of-bounds access in parquet data decoder");
+ }
+
+ size_t primitive_length = remove_nullable(data_type)->get_size_of_value_in_memory();
+ size_t data_index = doris_column->size() * primitive_length;
+ size_t scale_size = (select_vector.num_values() - select_vector.num_filtered()) *
+ (_type_length / primitive_length);
+ doris_column->resize(doris_column->size() + scale_size);
+ char* raw_data = const_cast<char*>(doris_column->get_raw_data().data);
+ ColumnSelectVector::DataReadType read_type;
+
+ while (size_t run_length = select_vector.get_next_run<has_filter>(&read_type)) {
+ switch (read_type) {
+ case ColumnSelectVector::CONTENT: {
+ for (int i = 0; i < run_length; ++i) {
+ // read byte split value from _byteSplitStreams
+ for (size_t j = 0; j < _type_length; ++j) {
+ raw_data[data_index + i * _type_length + j] =
+ _byteSplitStreams[j][_offset / _type_length + i];
+ }
+ }
+ _offset += run_length * _type_length;
+ data_index += run_length * _type_length;
+ break;
+ }
+ case ColumnSelectVector::NULL_DATA: {
+ data_index += run_length * _type_length;
+ break;
+ }
+ case ColumnSelectVector::FILTERED_CONTENT: {
+ _offset += _type_length * run_length;
+ break;
+ }
+ case ColumnSelectVector::FILTERED_NULL: {
+ // do nothing
+ break;
+ }
+ }
+ }
+ return Status::OK();
+}
+
+Status ByteStreamSplitDecoder::skip_values(size_t num_values) {
+ _offset += _type_length * num_values;
+ if (UNLIKELY(_offset > _data->size)) {
+ return Status::IOError("Out-of-bounds access in parquet data decoder");
Review Comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41683:
URL: https://github.com/apache/doris/pull/41683#discussion_r1796572696
##########
be/src/util/byte_stream_split.cpp:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHdst WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "byte_stream_split.h"
+
+#include <vector>
+
+#include "gutil/port.h"
+
+namespace doris {
+
+inline void do_merge_streams(const uint8_t** src_streams, int width, int64_t nvalues,
+ uint8_t* dest) {
+ // Value empirically chosen to provide the best performance on the author's machine
+ constexpr int kBlockSize = 128;
+
+ while (nvalues >= kBlockSize) {
+ for (int stream = 0; stream < width; ++stream) {
+ // Take kBlockSize bytes from the given stream and spread them
+ // to their logical places in destination.
+ const uint8_t* src = src_streams[stream];
+ for (int i = 0; i < kBlockSize; i += 8) {
+ uint64_t v;
+ std::memcpy(&v, src + i, sizeof(v));
+#ifdef IS_LITTLE_ENDIAN
+ dest[stream + i * width] = static_cast<uint8_t>(v);
+ dest[stream + (i + 1) * width] = static_cast<uint8_t>(v >> 8);
+ dest[stream + (i + 2) * width] = static_cast<uint8_t>(v >> 16);
+ dest[stream + (i + 3) * width] = static_cast<uint8_t>(v >> 24);
+ dest[stream + (i + 4) * width] = static_cast<uint8_t>(v >> 32);
+ dest[stream + (i + 5) * width] = static_cast<uint8_t>(v >> 40);
+ dest[stream + (i + 6) * width] = static_cast<uint8_t>(v >> 48);
+ dest[stream + (i + 7) * width] = static_cast<uint8_t>(v >> 56);
+#elif defined IS_BIG_ENDIAN
+ dest[stream + i * width] = static_cast<uint8_t>(v >> 56);
+ dest[stream + (i + 1) * width] = static_cast<uint8_t>(v >> 48);
+ dest[stream + (i + 2) * width] = static_cast<uint8_t>(v >> 40);
+ dest[stream + (i + 3) * width] = static_cast<uint8_t>(v >> 32);
+ dest[stream + (i + 4) * width] = static_cast<uint8_t>(v >> 24);
+ dest[stream + (i + 5) * width] = static_cast<uint8_t>(v >> 16);
+ dest[stream + (i + 6) * width] = static_cast<uint8_t>(v >> 8);
+ dest[stream + (i + 7) * width] = static_cast<uint8_t>(v);
+#endif
+ }
+ src_streams[stream] += kBlockSize;
+ }
+ dest += width * kBlockSize;
+ nvalues -= kBlockSize;
+ }
+
+ // Epilog
+ for (int stream = 0; stream < width; ++stream) {
+ const uint8_t* src = src_streams[stream];
+ for (int64_t i = 0; i < nvalues; ++i) {
+ dest[stream + i * width] = src[i];
+ }
+ }
+}
+
+template <int kNumStreams>
+void byte_stream_split_decode_scalar(const uint8_t* src, int width, int64_t offset,
+ int64_t num_values, int64_t stride, uint8_t* dest) {
+ assert(width == kNumStreams);
+ std::array<const uint8_t*, kNumStreams> src_streams;
Review Comment:
warning: no member named 'array' in namespace 'std' [clang-diagnostic-error]
```cpp
std::array<const uint8_t*, kNumStreams> src_streams;
^
```
##########
be/src/util/byte_stream_split.cpp:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHdst WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "byte_stream_split.h"
+
+#include <vector>
+
+#include "gutil/port.h"
+
+namespace doris {
+
+inline void do_merge_streams(const uint8_t** src_streams, int width, int64_t nvalues,
+ uint8_t* dest) {
+ // Value empirically chosen to provide the best performance on the author's machine
+ constexpr int kBlockSize = 128;
+
+ while (nvalues >= kBlockSize) {
+ for (int stream = 0; stream < width; ++stream) {
+ // Take kBlockSize bytes from the given stream and spread them
+ // to their logical places in destination.
+ const uint8_t* src = src_streams[stream];
+ for (int i = 0; i < kBlockSize; i += 8) {
+ uint64_t v;
+ std::memcpy(&v, src + i, sizeof(v));
Review Comment:
warning: no member named 'memcpy' in namespace 'std'; did you mean simply 'memcpy'? [clang-diagnostic-error]
```suggestion
memcpy(&v, src + i, sizeof(v));
```
<details>
<summary>Additional context</summary>
**/usr/include/string.h:42:** 'memcpy' declared here
```cpp
extern void *memcpy (void *__restrict __dest, const void *__restrict __src,
^
```
</details>
##########
be/src/util/byte_stream_split.cpp:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHdst WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "byte_stream_split.h"
+
+#include <vector>
+
+#include "gutil/port.h"
+
+namespace doris {
+
+inline void do_merge_streams(const uint8_t** src_streams, int width, int64_t nvalues,
+ uint8_t* dest) {
+ // Value empirically chosen to provide the best performance on the author's machine
+ constexpr int kBlockSize = 128;
+
+ while (nvalues >= kBlockSize) {
+ for (int stream = 0; stream < width; ++stream) {
+ // Take kBlockSize bytes from the given stream and spread them
+ // to their logical places in destination.
+ const uint8_t* src = src_streams[stream];
+ for (int i = 0; i < kBlockSize; i += 8) {
+ uint64_t v;
+ std::memcpy(&v, src + i, sizeof(v));
+#ifdef IS_LITTLE_ENDIAN
+ dest[stream + i * width] = static_cast<uint8_t>(v);
+ dest[stream + (i + 1) * width] = static_cast<uint8_t>(v >> 8);
+ dest[stream + (i + 2) * width] = static_cast<uint8_t>(v >> 16);
+ dest[stream + (i + 3) * width] = static_cast<uint8_t>(v >> 24);
+ dest[stream + (i + 4) * width] = static_cast<uint8_t>(v >> 32);
+ dest[stream + (i + 5) * width] = static_cast<uint8_t>(v >> 40);
+ dest[stream + (i + 6) * width] = static_cast<uint8_t>(v >> 48);
+ dest[stream + (i + 7) * width] = static_cast<uint8_t>(v >> 56);
+#elif defined IS_BIG_ENDIAN
+ dest[stream + i * width] = static_cast<uint8_t>(v >> 56);
+ dest[stream + (i + 1) * width] = static_cast<uint8_t>(v >> 48);
+ dest[stream + (i + 2) * width] = static_cast<uint8_t>(v >> 40);
+ dest[stream + (i + 3) * width] = static_cast<uint8_t>(v >> 32);
+ dest[stream + (i + 4) * width] = static_cast<uint8_t>(v >> 24);
+ dest[stream + (i + 5) * width] = static_cast<uint8_t>(v >> 16);
+ dest[stream + (i + 6) * width] = static_cast<uint8_t>(v >> 8);
+ dest[stream + (i + 7) * width] = static_cast<uint8_t>(v);
+#endif
+ }
+ src_streams[stream] += kBlockSize;
+ }
+ dest += width * kBlockSize;
+ nvalues -= kBlockSize;
+ }
+
+ // Epilog
+ for (int stream = 0; stream < width; ++stream) {
+ const uint8_t* src = src_streams[stream];
+ for (int64_t i = 0; i < nvalues; ++i) {
+ dest[stream + i * width] = src[i];
+ }
+ }
+}
+
+template <int kNumStreams>
+void byte_stream_split_decode_scalar(const uint8_t* src, int width, int64_t offset,
+ int64_t num_values, int64_t stride, uint8_t* dest) {
+ assert(width == kNumStreams);
+ std::array<const uint8_t*, kNumStreams> src_streams;
+ for (int stream = 0; stream < kNumStreams; ++stream) {
+ src_streams[stream] = &src[stream * stride + offset];
Review Comment:
warning: use of undeclared identifier 'src_streams' [clang-diagnostic-error]
```cpp
src_streams[stream] = &src[stream * stride + offset];
^
```
##########
be/src/util/byte_stream_split.cpp:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHdst WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "byte_stream_split.h"
+
+#include <vector>
+
+#include "gutil/port.h"
+
+namespace doris {
+
+inline void do_merge_streams(const uint8_t** src_streams, int width, int64_t nvalues,
+ uint8_t* dest) {
+ // Value empirically chosen to provide the best performance on the author's machine
+ constexpr int kBlockSize = 128;
+
+ while (nvalues >= kBlockSize) {
+ for (int stream = 0; stream < width; ++stream) {
+ // Take kBlockSize bytes from the given stream and spread them
+ // to their logical places in destination.
+ const uint8_t* src = src_streams[stream];
+ for (int i = 0; i < kBlockSize; i += 8) {
+ uint64_t v;
+ std::memcpy(&v, src + i, sizeof(v));
+#ifdef IS_LITTLE_ENDIAN
+ dest[stream + i * width] = static_cast<uint8_t>(v);
+ dest[stream + (i + 1) * width] = static_cast<uint8_t>(v >> 8);
+ dest[stream + (i + 2) * width] = static_cast<uint8_t>(v >> 16);
+ dest[stream + (i + 3) * width] = static_cast<uint8_t>(v >> 24);
+ dest[stream + (i + 4) * width] = static_cast<uint8_t>(v >> 32);
+ dest[stream + (i + 5) * width] = static_cast<uint8_t>(v >> 40);
+ dest[stream + (i + 6) * width] = static_cast<uint8_t>(v >> 48);
+ dest[stream + (i + 7) * width] = static_cast<uint8_t>(v >> 56);
+#elif defined IS_BIG_ENDIAN
+ dest[stream + i * width] = static_cast<uint8_t>(v >> 56);
+ dest[stream + (i + 1) * width] = static_cast<uint8_t>(v >> 48);
+ dest[stream + (i + 2) * width] = static_cast<uint8_t>(v >> 40);
+ dest[stream + (i + 3) * width] = static_cast<uint8_t>(v >> 32);
+ dest[stream + (i + 4) * width] = static_cast<uint8_t>(v >> 24);
+ dest[stream + (i + 5) * width] = static_cast<uint8_t>(v >> 16);
+ dest[stream + (i + 6) * width] = static_cast<uint8_t>(v >> 8);
+ dest[stream + (i + 7) * width] = static_cast<uint8_t>(v);
+#endif
+ }
+ src_streams[stream] += kBlockSize;
+ }
+ dest += width * kBlockSize;
+ nvalues -= kBlockSize;
+ }
+
+ // Epilog
+ for (int stream = 0; stream < width; ++stream) {
+ const uint8_t* src = src_streams[stream];
+ for (int64_t i = 0; i < nvalues; ++i) {
+ dest[stream + i * width] = src[i];
+ }
+ }
+}
+
+template <int kNumStreams>
+void byte_stream_split_decode_scalar(const uint8_t* src, int width, int64_t offset,
+ int64_t num_values, int64_t stride, uint8_t* dest) {
+ assert(width == kNumStreams);
+ std::array<const uint8_t*, kNumStreams> src_streams;
Review Comment:
warning: expected expression [clang-diagnostic-error]
```cpp
std::array<const uint8_t*, kNumStreams> src_streams;
^
```
##########
be/src/util/byte_stream_split.cpp:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHdst WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "byte_stream_split.h"
+
+#include <vector>
+
+#include "gutil/port.h"
+
+namespace doris {
+
+inline void do_merge_streams(const uint8_t** src_streams, int width, int64_t nvalues,
+ uint8_t* dest) {
+ // Value empirically chosen to provide the best performance on the author's machine
+ constexpr int kBlockSize = 128;
+
+ while (nvalues >= kBlockSize) {
+ for (int stream = 0; stream < width; ++stream) {
+ // Take kBlockSize bytes from the given stream and spread them
+ // to their logical places in destination.
+ const uint8_t* src = src_streams[stream];
+ for (int i = 0; i < kBlockSize; i += 8) {
+ uint64_t v;
+ std::memcpy(&v, src + i, sizeof(v));
+#ifdef IS_LITTLE_ENDIAN
+ dest[stream + i * width] = static_cast<uint8_t>(v);
+ dest[stream + (i + 1) * width] = static_cast<uint8_t>(v >> 8);
+ dest[stream + (i + 2) * width] = static_cast<uint8_t>(v >> 16);
+ dest[stream + (i + 3) * width] = static_cast<uint8_t>(v >> 24);
+ dest[stream + (i + 4) * width] = static_cast<uint8_t>(v >> 32);
+ dest[stream + (i + 5) * width] = static_cast<uint8_t>(v >> 40);
+ dest[stream + (i + 6) * width] = static_cast<uint8_t>(v >> 48);
+ dest[stream + (i + 7) * width] = static_cast<uint8_t>(v >> 56);
+#elif defined IS_BIG_ENDIAN
+ dest[stream + i * width] = static_cast<uint8_t>(v >> 56);
+ dest[stream + (i + 1) * width] = static_cast<uint8_t>(v >> 48);
+ dest[stream + (i + 2) * width] = static_cast<uint8_t>(v >> 40);
+ dest[stream + (i + 3) * width] = static_cast<uint8_t>(v >> 32);
+ dest[stream + (i + 4) * width] = static_cast<uint8_t>(v >> 24);
+ dest[stream + (i + 5) * width] = static_cast<uint8_t>(v >> 16);
+ dest[stream + (i + 6) * width] = static_cast<uint8_t>(v >> 8);
+ dest[stream + (i + 7) * width] = static_cast<uint8_t>(v);
+#endif
+ }
+ src_streams[stream] += kBlockSize;
+ }
+ dest += width * kBlockSize;
+ nvalues -= kBlockSize;
+ }
+
+ // Epilog
+ for (int stream = 0; stream < width; ++stream) {
+ const uint8_t* src = src_streams[stream];
+ for (int64_t i = 0; i < nvalues; ++i) {
+ dest[stream + i * width] = src[i];
+ }
+ }
+}
+
+template <int kNumStreams>
+void byte_stream_split_decode_scalar(const uint8_t* src, int width, int64_t offset,
+ int64_t num_values, int64_t stride, uint8_t* dest) {
+ assert(width == kNumStreams);
Review Comment:
warning: use of undeclared identifier 'assert' [clang-diagnostic-error]
```cpp
assert(width == kNumStreams);
^
```
##########
be/src/util/byte_stream_split.cpp:
##########
@@ -0,0 +1,115 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHdst WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "byte_stream_split.h"
+
+#include <vector>
+
+#include "gutil/port.h"
+
+namespace doris {
+
+inline void do_merge_streams(const uint8_t** src_streams, int width, int64_t nvalues,
+ uint8_t* dest) {
+ // Value empirically chosen to provide the best performance on the author's machine
+ constexpr int kBlockSize = 128;
+
+ while (nvalues >= kBlockSize) {
+ for (int stream = 0; stream < width; ++stream) {
+ // Take kBlockSize bytes from the given stream and spread them
+ // to their logical places in destination.
+ const uint8_t* src = src_streams[stream];
+ for (int i = 0; i < kBlockSize; i += 8) {
+ uint64_t v;
+ std::memcpy(&v, src + i, sizeof(v));
+#ifdef IS_LITTLE_ENDIAN
+ dest[stream + i * width] = static_cast<uint8_t>(v);
+ dest[stream + (i + 1) * width] = static_cast<uint8_t>(v >> 8);
+ dest[stream + (i + 2) * width] = static_cast<uint8_t>(v >> 16);
+ dest[stream + (i + 3) * width] = static_cast<uint8_t>(v >> 24);
+ dest[stream + (i + 4) * width] = static_cast<uint8_t>(v >> 32);
+ dest[stream + (i + 5) * width] = static_cast<uint8_t>(v >> 40);
+ dest[stream + (i + 6) * width] = static_cast<uint8_t>(v >> 48);
+ dest[stream + (i + 7) * width] = static_cast<uint8_t>(v >> 56);
+#elif defined IS_BIG_ENDIAN
+ dest[stream + i * width] = static_cast<uint8_t>(v >> 56);
+ dest[stream + (i + 1) * width] = static_cast<uint8_t>(v >> 48);
+ dest[stream + (i + 2) * width] = static_cast<uint8_t>(v >> 40);
+ dest[stream + (i + 3) * width] = static_cast<uint8_t>(v >> 32);
+ dest[stream + (i + 4) * width] = static_cast<uint8_t>(v >> 24);
+ dest[stream + (i + 5) * width] = static_cast<uint8_t>(v >> 16);
+ dest[stream + (i + 6) * width] = static_cast<uint8_t>(v >> 8);
+ dest[stream + (i + 7) * width] = static_cast<uint8_t>(v);
+#endif
+ }
+ src_streams[stream] += kBlockSize;
+ }
+ dest += width * kBlockSize;
+ nvalues -= kBlockSize;
+ }
+
+ // Epilog
+ for (int stream = 0; stream < width; ++stream) {
+ const uint8_t* src = src_streams[stream];
+ for (int64_t i = 0; i < nvalues; ++i) {
+ dest[stream + i * width] = src[i];
+ }
+ }
+}
+
+template <int kNumStreams>
+void byte_stream_split_decode_scalar(const uint8_t* src, int width, int64_t offset,
+ int64_t num_values, int64_t stride, uint8_t* dest) {
+ assert(width == kNumStreams);
+ std::array<const uint8_t*, kNumStreams> src_streams;
+ for (int stream = 0; stream < kNumStreams; ++stream) {
+ src_streams[stream] = &src[stream * stride + offset];
+ }
+ do_merge_streams(src_streams.data(), kNumStreams, num_values, dest);
Review Comment:
warning: use of undeclared identifier 'src_streams' [clang-diagnostic-error]
```cpp
do_merge_streams(src_streams.data(), kNumStreams, num_values, dest);
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2409917408
PR approved by at least one committer and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2407538360
<details>
<summary>TPC-DS: <b>Total hot run time: 191612 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1645ea725a237529dfe5f0647c6c9ba4eea116f0, data reload: false
query1 834 380 396 380
query2 6226 2040 1992 1992
query3 8682 198 205 198
query4 33939 23598 23400 23400
query5 3734 476 493 476
query6 280 163 161 161
query7 4190 304 298 298
query8 286 221 226 221
query9 9523 2719 2680 2680
query10 463 279 271 271
query11 17679 15140 15197 15140
query12 153 101 95 95
query13 1594 464 458 458
query14 9631 7568 7605 7568
query15 247 174 179 174
query16 7418 454 435 435
query17 1617 603 617 603
query18 1870 331 328 328
query19 381 154 174 154
query20 121 111 111 111
query21 210 109 106 106
query22 4916 4403 4432 4403
query23 34880 34422 33813 33813
query24 11044 2766 2786 2766
query25 622 412 386 386
query26 967 157 162 157
query27 2240 301 293 293
query28 7482 2406 2391 2391
query29 739 440 433 433
query30 256 149 155 149
query31 1037 777 813 777
query32 99 55 59 55
query33 764 292 285 285
query34 923 497 517 497
query35 890 730 717 717
query36 1092 929 931 929
query37 152 85 87 85
query38 4132 3843 3969 3843
query39 1492 1419 1445 1419
query40 205 96 97 96
query41 48 44 46 44
query42 113 97 94 94
query43 524 485 473 473
query44 1274 803 800 800
query45 192 164 167 164
query46 1149 723 714 714
query47 1919 1839 1840 1839
query48 441 355 350 350
query49 904 412 411 411
query50 817 433 442 433
query51 7368 6889 6943 6889
query52 97 85 87 85
query53 272 182 180 180
query54 1172 491 472 472
query55 81 80 76 76
query56 280 258 290 258
query57 1205 1167 1145 1145
query58 226 225 234 225
query59 3178 3013 3038 3013
query60 293 269 277 269
query61 101 107 100 100
query62 872 659 668 659
query63 219 194 184 184
query64 3967 633 613 613
query65 3314 3248 3198 3198
query66 1321 296 300 296
query67 15936 15705 15716 15705
query68 4394 573 546 546
query69 461 290 319 290
query70 1178 1115 1150 1115
query71 342 268 274 268
query72 7120 3976 3960 3960
query73 758 351 353 351
query74 9502 8982 8914 8914
query75 3394 2684 2642 2642
query76 2903 909 896 896
query77 577 298 286 286
query78 10298 9539 9500 9500
query79 1910 581 602 581
query80 1487 447 447 447
query81 587 242 238 238
query82 699 137 142 137
query83 276 130 137 130
query84 283 79 75 75
query85 1480 302 286 286
query86 415 291 297 291
query87 4513 4273 4343 4273
query88 3766 2452 2394 2394
query89 396 287 286 286
query90 1984 188 185 185
query91 137 108 104 104
query92 65 52 45 45
query93 1270 545 544 544
query94 1060 290 283 283
query95 347 244 245 244
query96 610 280 285 280
query97 3229 3098 3106 3098
query98 225 206 191 191
query99 1528 1300 1313 1300
Total cold run time: 297487 ms
Total hot run time: 191612 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2408988053
TeamCity be ut coverage result:
Function Coverage: 37.42% (9682/25871)
Line Coverage: 28.68% (80309/280063)
Region Coverage: 28.11% (41537/147740)
Branch Coverage: 24.70% (21122/85520)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f67ba7392f1470d524153da38c7f285f2583d12f_f67ba7392f1470d524153da38c7f285f2583d12f/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2408331225
<details>
<summary>TPC-DS: <b>Total hot run time: 191651 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4c47c3e413b11965eadaf04df35fb5ffaf3f451a, data reload: false
query1 974 369 378 369
query2 6510 2087 2080 2080
query3 6708 217 225 217
query4 33605 23361 23631 23361
query5 4317 470 455 455
query6 263 169 168 168
query7 4629 303 301 301
query8 284 224 231 224
query9 9728 2682 2697 2682
query10 462 284 285 284
query11 18098 15301 15123 15123
query12 151 97 93 93
query13 1670 459 435 435
query14 10008 7456 7416 7416
query15 254 166 174 166
query16 8158 456 456 456
query17 1676 576 556 556
query18 2100 305 317 305
query19 368 151 159 151
query20 119 105 108 105
query21 213 103 103 103
query22 4874 4539 4473 4473
query23 34783 34075 34116 34075
query24 11076 2768 2772 2768
query25 662 399 413 399
query26 1180 162 160 160
query27 2830 301 293 293
query28 8056 2443 2422 2422
query29 850 443 440 440
query30 319 156 156 156
query31 1037 774 820 774
query32 91 57 58 57
query33 779 305 288 288
query34 979 500 520 500
query35 872 722 732 722
query36 1116 908 968 908
query37 154 92 86 86
query38 3942 3948 3881 3881
query39 1505 1438 1417 1417
query40 218 98 97 97
query41 50 46 47 46
query42 119 98 99 98
query43 534 485 471 471
query44 1234 807 810 807
query45 196 168 171 168
query46 1145 722 724 722
query47 1947 1853 1859 1853
query48 433 345 340 340
query49 1137 427 426 426
query50 815 402 423 402
query51 7138 6903 6884 6884
query52 98 89 88 88
query53 260 198 186 186
query54 1327 472 486 472
query55 82 77 80 77
query56 297 266 272 266
query57 1300 1174 1141 1141
query58 256 244 256 244
query59 3207 3000 2860 2860
query60 324 286 275 275
query61 126 121 135 121
query62 873 682 684 682
query63 224 191 184 184
query64 5378 612 606 606
query65 3249 3172 3252 3172
query66 1422 316 310 310
query67 15948 15610 15503 15503
query68 4909 559 547 547
query69 453 295 295 295
query70 1111 1126 1105 1105
query71 317 270 268 268
query72 6355 3963 3976 3963
query73 768 344 343 343
query74 9727 8864 8997 8864
query75 3358 2660 2664 2660
query76 2972 932 870 870
query77 431 301 304 301
query78 10536 9742 9674 9674
query79 1835 604 611 604
query80 1702 440 430 430
query81 575 238 245 238
query82 978 139 140 139
query83 246 137 134 134
query84 258 77 76 76
query85 1290 284 279 279
query86 379 280 290 280
query87 4459 4252 4421 4252
query88 3850 2452 2424 2424
query89 405 285 289 285
query90 1859 186 186 186
query91 157 121 104 104
query92 61 46 47 46
query93 1106 529 543 529
query94 964 278 281 278
query95 352 250 251 250
query96 610 288 289 288
query97 3252 3086 3156 3086
query98 211 202 203 202
query99 1568 1317 1281 1281
Total cold run time: 301791 ms
Total hot run time: 191651 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2407437285
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2408950082
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2408986315
<details>
<summary>ClickBench: <b>Total hot run time: 32.81 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f67ba7392f1470d524153da38c7f285f2583d12f, data reload: false
query1 0.03 0.03 0.03
query2 0.07 0.03 0.03
query3 0.23 0.07 0.07
query4 1.65 0.10 0.10
query5 0.52 0.52 0.53
query6 1.13 0.73 0.72
query7 0.02 0.02 0.02
query8 0.04 0.03 0.02
query9 0.56 0.51 0.49
query10 0.54 0.55 0.54
query11 0.14 0.11 0.11
query12 0.13 0.10 0.11
query13 0.61 0.60 0.59
query14 2.85 2.71 2.76
query15 0.90 0.82 0.82
query16 0.39 0.38 0.35
query17 1.00 1.01 1.00
query18 0.20 0.19 0.21
query19 1.86 1.87 2.01
query20 0.01 0.01 0.01
query21 15.35 0.58 0.56
query22 2.61 2.39 1.50
query23 17.13 0.86 0.78
query24 2.63 1.40 1.96
query25 0.33 0.16 0.15
query26 0.40 0.13 0.14
query27 0.04 0.04 0.06
query28 9.90 1.10 1.06
query29 12.62 3.30 3.26
query30 0.24 0.06 0.05
query31 2.90 0.39 0.38
query32 3.25 0.45 0.46
query33 3.01 3.03 3.01
query34 17.20 4.55 4.47
query35 4.58 4.61 4.52
query36 0.64 0.48 0.49
query37 0.08 0.06 0.06
query38 0.04 0.03 0.03
query39 0.03 0.02 0.02
query40 0.15 0.12 0.12
query41 0.07 0.02 0.03
query42 0.03 0.02 0.01
query43 0.04 0.03 0.02
Total cold run time: 106.15 s
Total hot run time: 32.81 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41683:
URL: https://github.com/apache/doris/pull/41683#discussion_r1796971643
##########
be/test/util/byte_stream_split_test.cpp:
##########
@@ -0,0 +1,95 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "util/byte_stream_split.h"
Review Comment:
warning: 'util/byte_stream_split.h' file not found [clang-diagnostic-error]
```cpp
#include "util/byte_stream_split.h"
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2406335623
Thank you for your contribution to Apache Doris.
Don't know what should be done next? See [How to process your PR](https://cwiki.apache.org/confluence/display/DORIS/How+to+process+your+PR)
Since 2024-03-18, the Document has been moved to [doris-website](https://github.com/apache/doris-website).
See [Doris Document](https://cwiki.apache.org/confluence/display/DORIS/Doris+Document).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2408984535
<details>
<summary>TPC-DS: <b>Total hot run time: 191118 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f67ba7392f1470d524153da38c7f285f2583d12f, data reload: false
query1 905 410 428 410
query2 6281 2068 1943 1943
query3 8682 198 202 198
query4 34321 23427 23709 23427
query5 3587 484 464 464
query6 276 164 161 161
query7 4203 299 301 299
query8 273 209 215 209
query9 9508 2682 2691 2682
query10 474 282 277 277
query11 17834 15073 15210 15073
query12 148 100 95 95
query13 1586 481 441 441
query14 9253 7418 7386 7386
query15 276 171 171 171
query16 7981 483 446 446
query17 1693 630 588 588
query18 2160 322 312 312
query19 381 160 164 160
query20 116 118 115 115
query21 214 103 104 103
query22 4611 4728 4453 4453
query23 34889 33588 34080 33588
query24 11202 2798 2821 2798
query25 613 409 403 403
query26 1100 159 161 159
query27 2442 298 300 298
query28 7570 2395 2396 2395
query29 828 448 429 429
query30 261 160 152 152
query31 1021 795 792 792
query32 103 54 52 52
query33 749 315 310 310
query34 899 509 504 504
query35 876 715 726 715
query36 1101 910 960 910
query37 150 83 88 83
query38 4051 3862 3884 3862
query39 1497 1443 1410 1410
query40 203 97 98 97
query41 47 45 46 45
query42 117 99 96 96
query43 530 498 479 479
query44 1308 822 813 813
query45 199 166 169 166
query46 1143 716 691 691
query47 1977 1892 1837 1837
query48 430 339 368 339
query49 912 420 405 405
query50 879 416 428 416
query51 6954 6982 7019 6982
query52 97 89 90 89
query53 258 194 183 183
query54 1152 465 484 465
query55 82 77 102 77
query56 266 257 265 257
query57 1259 1113 1154 1113
query58 222 230 240 230
query59 3202 3207 3078 3078
query60 296 275 261 261
query61 103 100 106 100
query62 875 658 657 657
query63 210 185 187 185
query64 4133 633 603 603
query65 3457 3182 3190 3182
query66 1289 297 316 297
query67 15807 15517 15488 15488
query68 4654 612 570 570
query69 535 301 305 301
query70 1104 1082 1096 1082
query71 360 270 275 270
query72 7030 3928 4082 3928
query73 779 348 361 348
query74 10275 8838 8948 8838
query75 3331 2686 2643 2643
query76 2880 895 884 884
query77 432 323 314 314
query78 10764 9742 9639 9639
query79 1345 590 592 590
query80 1492 436 452 436
query81 583 240 239 239
query82 753 138 133 133
query83 294 135 133 133
query84 283 72 75 72
query85 1342 291 282 282
query86 389 305 309 305
query87 4462 4362 4308 4308
query88 3118 2426 2356 2356
query89 404 284 282 282
query90 2263 184 179 179
query91 140 104 103 103
query92 60 48 49 48
query93 1059 540 541 540
query94 1178 285 278 278
query95 357 255 257 255
query96 613 277 274 274
query97 3253 3096 3143 3096
query98 211 197 193 193
query99 1683 1296 1295 1295
Total cold run time: 298948 ms
Total hot run time: 191118 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2406340538
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "suxiaogang223 (via GitHub)" <gi...@apache.org>.
suxiaogang223 commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2408320354
run buildall
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2408980364
<details>
<summary>TPC-H: <b>Total hot run time: 40815 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f67ba7392f1470d524153da38c7f285f2583d12f, data reload: false
------ Round 1 ----------------------------------
q1 17610 7737 7258 7258
q2 2024 297 288 288
q3 12171 1078 1172 1078
q4 10589 739 746 739
q5 7767 2903 2856 2856
q6 237 152 150 150
q7 998 634 636 634
q8 9364 2010 1872 1872
q9 6554 6483 6431 6431
q10 6983 2256 2290 2256
q11 433 245 250 245
q12 401 216 221 216
q13 17776 3002 2969 2969
q14 250 207 210 207
q15 582 525 525 525
q16 647 578 593 578
q17 979 554 554 554
q18 7233 6678 6821 6678
q19 1350 966 914 914
q20 484 210 197 197
q21 4007 3217 3171 3171
q22 1142 999 1026 999
Total cold run time: 109581 ms
Total hot run time: 40815 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7265 7183 7204 7183
q2 332 230 232 230
q3 2989 2940 2943 2940
q4 2069 1866 1791 1791
q5 5806 5695 5805 5695
q6 236 140 140 140
q7 2267 1830 1826 1826
q8 3372 3635 3442 3442
q9 8935 8919 8904 8904
q10 3583 3575 3555 3555
q11 586 480 500 480
q12 806 674 666 666
q13 9695 3175 3129 3129
q14 310 287 267 267
q15 573 511 514 511
q16 678 634 631 631
q17 1872 1615 1608 1608
q18 8328 7840 7685 7685
q19 1707 1537 1404 1404
q20 2132 1848 1926 1848
q21 5576 5396 5488 5396
q22 1150 1099 1064 1064
Total cold run time: 70267 ms
Total hot run time: 60395 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2406340286
clang-tidy review says "All clean, LGTM! :+1:"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2407549204
<details>
<summary>ClickBench: <b>Total hot run time: 32.87 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1645ea725a237529dfe5f0647c6c9ba4eea116f0, data reload: false
query1 0.03 0.02 0.03
query2 0.06 0.03 0.03
query3 0.22 0.06 0.06
query4 1.64 0.10 0.10
query5 0.51 0.50 0.51
query6 1.13 0.72 0.72
query7 0.02 0.01 0.02
query8 0.03 0.03 0.03
query9 0.56 0.51 0.50
query10 0.56 0.56 0.55
query11 0.14 0.11 0.11
query12 0.14 0.10 0.10
query13 0.60 0.60 0.60
query14 2.71 2.74 2.73
query15 0.89 0.81 0.82
query16 0.39 0.38 0.38
query17 0.99 0.96 1.00
query18 0.20 0.19 0.21
query19 1.91 1.90 1.90
query20 0.02 0.01 0.01
query21 15.36 0.61 0.58
query22 2.19 1.79 1.50
query23 16.81 1.23 0.72
query24 2.72 1.83 1.68
query25 0.35 0.12 0.08
query26 0.50 0.15 0.14
query27 0.04 0.05 0.04
query28 9.70 1.08 1.07
query29 12.59 3.25 3.23
query30 0.25 0.05 0.06
query31 2.86 0.38 0.38
query32 3.28 0.46 0.45
query33 3.02 3.00 2.99
query34 16.87 4.44 4.46
query35 4.47 4.48 4.44
query36 0.67 0.49 0.49
query37 0.08 0.06 0.07
query38 0.04 0.03 0.04
query39 0.03 0.02 0.02
query40 0.15 0.12 0.12
query41 0.08 0.02 0.02
query42 0.03 0.02 0.01
query43 0.03 0.03 0.02
Total cold run time: 104.87 s
Total hot run time: 32.87 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2407513811
<details>
<summary>TPC-H: <b>Total hot run time: 40659 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1645ea725a237529dfe5f0647c6c9ba4eea116f0, data reload: false
------ Round 1 ----------------------------------
q1 17573 7349 7228 7228
q2 2030 275 270 270
q3 12084 1050 1163 1050
q4 10569 727 815 727
q5 7727 2858 2825 2825
q6 237 150 149 149
q7 984 620 603 603
q8 9340 1952 1945 1945
q9 7059 6412 6391 6391
q10 6950 2268 2293 2268
q11 449 244 246 244
q12 404 226 220 220
q13 17761 2990 2998 2990
q14 248 210 211 210
q15 564 525 519 519
q16 642 589 585 585
q17 965 543 563 543
q18 7141 6792 6691 6691
q19 1338 892 902 892
q20 488 199 195 195
q21 4011 3098 3229 3098
q22 1113 1016 1020 1016
Total cold run time: 109677 ms
Total hot run time: 40659 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7276 7186 7190 7186
q2 317 223 226 223
q3 2944 2898 2935 2898
q4 2112 1875 1800 1800
q5 5735 5692 5714 5692
q6 236 144 150 144
q7 2220 1851 1830 1830
q8 3320 3602 3406 3406
q9 8870 8863 8820 8820
q10 3590 3512 3485 3485
q11 585 494 503 494
q12 846 659 637 637
q13 10979 3203 3206 3203
q14 293 272 266 266
q15 566 512 518 512
q16 706 638 661 638
q17 1819 1605 1593 1593
q18 8198 7619 7650 7619
q19 1707 1429 1539 1429
q20 2131 1878 1889 1878
q21 5620 5378 5200 5200
q22 1156 1080 1023 1023
Total cold run time: 71226 ms
Total hot run time: 59976 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2408332840
<details>
<summary>ClickBench: <b>Total hot run time: 32.18 s</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4c47c3e413b11965eadaf04df35fb5ffaf3f451a, data reload: false
query1 0.03 0.03 0.03
query2 0.06 0.03 0.03
query3 0.23 0.06 0.06
query4 1.66 0.10 0.11
query5 0.50 0.52 0.51
query6 1.13 0.72 0.73
query7 0.02 0.01 0.01
query8 0.04 0.02 0.03
query9 0.57 0.50 0.50
query10 0.56 0.55 0.55
query11 0.14 0.11 0.12
query12 0.13 0.10 0.11
query13 0.61 0.60 0.59
query14 2.73 2.75 2.76
query15 0.89 0.84 0.82
query16 0.38 0.39 0.38
query17 1.07 1.10 1.03
query18 0.24 0.22 0.22
query19 1.96 1.76 1.99
query20 0.02 0.01 0.01
query21 15.36 0.58 0.56
query22 3.26 1.63 2.36
query23 17.15 0.86 0.76
query24 3.46 0.79 1.14
query25 0.11 0.14 0.06
query26 0.58 0.15 0.15
query27 0.06 0.05 0.04
query28 10.64 1.10 1.07
query29 12.53 3.25 3.19
query30 0.25 0.06 0.06
query31 3.14 0.39 0.37
query32 3.76 0.45 0.46
query33 3.01 3.02 3.03
query34 17.10 4.47 4.50
query35 4.51 4.50 4.59
query36 0.66 0.47 0.48
query37 0.08 0.06 0.06
query38 0.05 0.03 0.04
query39 0.04 0.02 0.02
query40 0.16 0.13 0.12
query41 0.08 0.02 0.02
query42 0.03 0.02 0.02
query43 0.03 0.02 0.03
Total cold run time: 109.02 s
Total hot run time: 32.18 s
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41683:
URL: https://github.com/apache/doris/pull/41683#discussion_r1796598411
##########
be/src/util/byte_stream_split.cpp:
##########
@@ -0,0 +1,117 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHdst WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "byte_stream_split.h"
+
+#include <array>
+#include <cstring>
+#include <vector>
+
+#include "gutil/port.h"
+
+namespace doris {
+
+inline void do_merge_streams(const uint8_t** src_streams, int width, int64_t nvalues,
+ uint8_t* dest) {
+ // Value empirically chosen to provide the best performance on the author's machine
+ constexpr int kBlockSize = 128;
+
+ while (nvalues >= kBlockSize) {
+ for (int stream = 0; stream < width; ++stream) {
+ // Take kBlockSize bytes from the given stream and spread them
+ // to their logical places in destination.
+ const uint8_t* src = src_streams[stream];
+ for (int i = 0; i < kBlockSize; i += 8) {
+ uint64_t v;
+ std::memcpy(&v, src + i, sizeof(v));
+#ifdef IS_LITTLE_ENDIAN
+ dest[stream + i * width] = static_cast<uint8_t>(v);
+ dest[stream + (i + 1) * width] = static_cast<uint8_t>(v >> 8);
+ dest[stream + (i + 2) * width] = static_cast<uint8_t>(v >> 16);
+ dest[stream + (i + 3) * width] = static_cast<uint8_t>(v >> 24);
+ dest[stream + (i + 4) * width] = static_cast<uint8_t>(v >> 32);
+ dest[stream + (i + 5) * width] = static_cast<uint8_t>(v >> 40);
+ dest[stream + (i + 6) * width] = static_cast<uint8_t>(v >> 48);
+ dest[stream + (i + 7) * width] = static_cast<uint8_t>(v >> 56);
+#elif defined IS_BIG_ENDIAN
+ dest[stream + i * width] = static_cast<uint8_t>(v >> 56);
+ dest[stream + (i + 1) * width] = static_cast<uint8_t>(v >> 48);
+ dest[stream + (i + 2) * width] = static_cast<uint8_t>(v >> 40);
+ dest[stream + (i + 3) * width] = static_cast<uint8_t>(v >> 32);
+ dest[stream + (i + 4) * width] = static_cast<uint8_t>(v >> 24);
+ dest[stream + (i + 5) * width] = static_cast<uint8_t>(v >> 16);
+ dest[stream + (i + 6) * width] = static_cast<uint8_t>(v >> 8);
+ dest[stream + (i + 7) * width] = static_cast<uint8_t>(v);
+#endif
+ }
+ src_streams[stream] += kBlockSize;
+ }
+ dest += width * kBlockSize;
+ nvalues -= kBlockSize;
+ }
+
+ // Epilog
+ for (int stream = 0; stream < width; ++stream) {
+ const uint8_t* src = src_streams[stream];
+ for (int64_t i = 0; i < nvalues; ++i) {
+ dest[stream + i * width] = src[i];
+ }
+ }
+}
+
+template <int kNumStreams>
+void byte_stream_split_decode_scalar(const uint8_t* src, int width, int64_t offset,
+ int64_t num_values, int64_t stride, uint8_t* dest) {
+ DCHECK(width == kNumStreams);
Review Comment:
warning: use of undeclared identifier 'DCHECK' [clang-diagnostic-error]
```cpp
DCHECK(width == kNumStreams);
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on code in PR #41683:
URL: https://github.com/apache/doris/pull/41683#discussion_r1796606914
##########
be/src/util/byte_stream_split.cpp:
##########
@@ -0,0 +1,119 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHdst WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "byte_stream_split.h"
+
+#include <glog/logging.h>
Review Comment:
warning: 'glog/logging.h' file not found [clang-diagnostic-error]
```cpp
#include <glog/logging.h>
^
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2409917440
PR approved by anyone and no changes requested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "morningman (via GitHub)" <gi...@apache.org>.
morningman merged PR #41683:
URL: https://github.com/apache/doris/pull/41683
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2408327227
<details>
<summary>TPC-H: <b>Total hot run time: 40984 ms</b></summary>
```
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4c47c3e413b11965eadaf04df35fb5ffaf3f451a, data reload: false
------ Round 1 ----------------------------------
q1 18183 7500 7362 7362
q2 2447 185 179 179
q3 11775 1157 1182 1157
q4 11326 768 750 750
q5 7789 2901 2801 2801
q6 247 151 155 151
q7 996 645 619 619
q8 9357 1932 1912 1912
q9 6610 6436 6428 6428
q10 6947 2274 2295 2274
q11 447 254 241 241
q12 403 231 218 218
q13 17787 2969 2971 2969
q14 236 207 209 207
q15 588 531 510 510
q16 637 580 574 574
q17 971 558 612 558
q18 7076 6663 6746 6663
q19 1343 1019 969 969
q20 469 203 201 201
q21 3952 3245 3242 3242
q22 1110 1013 999 999
Total cold run time: 110696 ms
Total hot run time: 40984 ms
----- Round 2, with runtime_filter_mode=off -----
q1 7288 7275 7280 7275
q2 330 229 220 220
q3 2943 2732 2783 2732
q4 1964 1759 1711 1711
q5 5452 5489 5494 5489
q6 226 144 140 140
q7 2128 1716 1722 1716
q8 3236 3391 3416 3391
q9 8526 8491 8508 8491
q10 3468 3452 3421 3421
q11 576 473 483 473
q12 800 615 585 585
q13 5593 2976 3019 2976
q14 287 254 275 254
q15 576 518 516 516
q16 663 627 632 627
q17 1796 1565 1558 1558
q18 7819 7469 7330 7330
q19 1658 1511 1463 1463
q20 2057 1829 1785 1785
q21 5242 5084 5178 5084
q22 1121 1054 987 987
Total cold run time: 63749 ms
Total hot run time: 58224 ms
```
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2408330117
TeamCity be ut coverage result:
Function Coverage: 37.40% (9662/25837)
Line Coverage: 28.67% (80205/279726)
Region Coverage: 28.09% (41459/147573)
Branch Coverage: 24.71% (21129/85518)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4c47c3e413b11965eadaf04df35fb5ffaf3f451a_4c47c3e413b11965eadaf04df35fb5ffaf3f451a/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
Re: [PR] [enhance](parquet) Support BYTE_STREAM_SPLIT encoding for parquet reader [doris]
Posted by "doris-robot (via GitHub)" <gi...@apache.org>.
doris-robot commented on PR #41683:
URL: https://github.com/apache/doris/pull/41683#issuecomment-2407561803
TeamCity be ut coverage result:
Function Coverage: 37.39% (9660/25837)
Line Coverage: 28.65% (80149/279726)
Region Coverage: 28.09% (41451/147573)
Branch Coverage: 24.70% (21120/85518)
Coverage Report: http://coverage.selectdb-in.cc/coverage/1645ea725a237529dfe5f0647c6c9ba4eea116f0_1645ea725a237529dfe5f0647c6c9ba4eea116f0/report/index.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org