You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2023/01/12 20:26:11 UTC
[GitHub] [arrow] felipecrv opened a new pull request, #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
felipecrv opened a new pull request, #33641:
URL: https://github.com/apache/arrow/pull/33641
This PR gathers work from multiple PRs that can be closed after this one is merged:
- Closes #13752
- Closes #13754
And the issues associated with those PRs can also be closed:
- Fixes #20350
- Fixes #32543
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1084592097
##########
cpp/src/arrow/ipc/writer.cc:
##########
@@ -526,6 +526,10 @@ class RecordBatchSerializer {
return VisitType(*array.indices());
}
+ Status Visit(const RunEndEncodedArray& type) {
+ return Status::NotImplemented("run-end encoded array in IPC");
+ }
Review Comment:
Yes. This placeholder needs to be done so compilation can succeed and progress can be made. That makes turning this into small PRs a bit tricky as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1092501101
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
Review Comment:
I don't know if it's possible to produce REEs with children containing different offsets, so disregard the "might differ" comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109133067
##########
cpp/src/arrow/scalar.cc:
##########
@@ -583,6 +608,17 @@ Result<std::shared_ptr<Scalar>> StructScalar::field(FieldRef ref) const {
}
}
+RunEndEncodedScalar::RunEndEncodedScalar(std::shared_ptr<Scalar> value,
+ std::shared_ptr<DataType> type)
+ : Scalar{std::move(type), value->is_valid}, value{std::move(value)} {}
Review Comment:
I looked at other Scalars now and they check indirectly by performing some checked casts, so I will add a check here too. :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1091317586
##########
cpp/src/arrow/compute/kernels/vector_run_end_encode_test.cc:
##########
@@ -0,0 +1,196 @@
+// Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
No, not strictly required. When I probe for the next round of reviews I will push with a smaller set of commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1092168865
##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +195,7 @@ int GetNumBuffers(const DataType& type) {
case Type::NA:
case Type::STRUCT:
case Type::FIXED_SIZE_LIST:
+ case Type::RUN_END_ENCODED:
return 1;
Review Comment:
> Also you'll notice that IsValid and IsNull are non-virtual methods.
And making them `virtual` would have performance impacts on everything. This would be a different API just in case kernels need it. More of a `compute` concern than something to be part of the `ArrayData` interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109144639
##########
cpp/src/arrow/array/builder_run_end.cc:
##########
@@ -0,0 +1,331 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/builder_run_end.h"
+#include "arrow/array/builder_primitive.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <utility>
+#include <vector>
+
+#include "arrow/scalar.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace internal {
+
+RunCompressorBuilder::RunCompressorBuilder(MemoryPool* pool,
+ std::shared_ptr<ArrayBuilder> inner_builder,
+ std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), inner_builder_(std::move(inner_builder)) {}
+
+RunCompressorBuilder::~RunCompressorBuilder() = default;
+
+void RunCompressorBuilder::Reset() {
+ current_run_length_ = 0;
+ current_value_.reset();
+ inner_builder_->Reset();
+ UpdateDimensions();
+}
+
+Status RunCompressorBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(inner_builder_->Resize(capacity));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendNulls(int64_t length) {
+ if (ARROW_PREDICT_FALSE(length == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new NULL run
+ DCHECK_EQ(current_value_, NULLPTR);
+ current_run_length_ = length;
+ } else if (current_value_ == NULLPTR) {
+ // Extend the currently open NULL run
+ current_run_length_ += length;
+ } else {
+ // Close then non-NULL run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(inner_builder_->AppendScalar(*current_value_));
+ UpdateDimensions();
+ // Open a new NULL run
+ current_value_.reset();
+ current_run_length_ = length;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to a run-compressed array.");
+}
+
+Status RunCompressorBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (ARROW_PREDICT_FALSE(n_repeats == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ } else if ((current_value_ == NULLPTR && !scalar.is_valid) ||
+ (current_value_ != NULLPTR && current_value_->Equals(scalar))) {
+ // Extend the currently open run
+ current_run_length_ += n_repeats;
+ } else {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendScalars(const ScalarVector& scalars) {
+ if (scalars.empty()) {
+ return Status::OK();
+ }
+ RETURN_NOT_OK(ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendRunCompressedArraySlice(
+ const ArraySpan& run_compressed_array, int64_t offset, int64_t length) {
+ DCHECK(!has_open_run());
+ RETURN_NOT_OK(inner_builder_->AppendArraySlice(run_compressed_array, offset, length));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishCurrentRun() {
+ if (current_run_length_ > 0) {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Clear the current run
+ current_value_.reset();
+ current_run_length_ = 0;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ ARROW_RETURN_NOT_OK(FinishCurrentRun());
+ return inner_builder_->FinishInternal(out);
+}
+
+} // namespace internal
+
+// ----------------------------------------------------------------------
+// RunEndEncodedBuilder
+
+RunEndEncodedBuilder::ValueRunBuilder::ValueRunBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& value_builder,
+ const std::shared_ptr<DataType>& value_type, RunEndEncodedBuilder& ree_builder)
+ : RunCompressorBuilder(pool, std::move(value_builder), std::move(value_type)),
+ ree_builder_(ree_builder) {}
+
+RunEndEncodedBuilder::RunEndEncodedBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder, std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), type_(internal::checked_pointer_cast<RunEndEncodedType>(type)) {
+ auto value_run_builder =
+ std::make_shared<ValueRunBuilder>(pool, value_builder, type_->value_type(), *this);
+ value_run_builder_ = value_run_builder.get();
+ children_ = {run_end_builder, std::move(value_run_builder)};
+ UpdateDimensions(0, 0);
+ null_count_ = 0;
+}
+
+Status RunEndEncodedBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(value_run_builder_->ResizePhysical(capacity));
+ RETURN_NOT_OK(run_end_builder().Resize(capacity));
+ UpdateDimensions(committed_logical_length_, 0);
+ return Status::OK();
+}
+
+void RunEndEncodedBuilder::Reset() {
+ value_run_builder_->Reset();
+ run_end_builder().Reset();
+ UpdateDimensions(0, 0);
+}
+
+Status RunEndEncodedBuilder::AppendNulls(int64_t length) {
+ RETURN_NOT_OK(value_run_builder_->AppendNulls(length));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to run-end encoded array.");
+}
+
+Status RunEndEncodedBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (scalar.type->id() == Type::RUN_END_ENCODED) {
+ return AppendScalar(*internal::checked_cast<const RunEndEncodedScalar&>(scalar).value,
+ n_repeats);
+ }
+ RETURN_NOT_OK(value_run_builder_->AppendScalar(scalar, n_repeats));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendScalars(const ScalarVector& scalars) {
+ RETURN_NOT_OK(this->ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendArray(const ArraySpan& to_append) {
+ DCHECK_GT(to_append.length, 0);
+ DCHECK(!value_run_builder_->has_open_run());
+
+ ree_util::RunEndEncodedArraySpan<RunEndsType> ree_span(to_append);
+ const int64_t physical_offset = ree_span.PhysicalIndex(0);
+ const int64_t physical_length =
+ ree_span.PhysicalIndex(ree_span.length() - 1) + 1 - physical_offset;
+
+ RETURN_NOT_OK(ReservePhysical(physical_length));
+
+ // Append all the run ends from to_append
+ const auto end = ree_span.end();
+ for (auto it = ree_span.iterator(0, physical_offset); it != end; ++it) {
+ const int64_t run_end = committed_logical_length_ + it.run_length();
+ RETURN_NOT_OK(DoAppendRunEnd<RunEndsType>(run_end));
+ UpdateDimensions(run_end, 0);
+ }
+
+ // Append all the values directly
+ RETURN_NOT_OK(value_run_builder_->AppendRunCompressedArraySlice(
+ ree_util::ValuesArray(to_append), physical_offset, physical_length));
+
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendArraySlice(const ArraySpan& array, int64_t offset,
+ int64_t length) {
+ ARROW_DCHECK(offset + length <= array.length);
+ ARROW_DCHECK(array.type->Equals(type_));
+
+ // Ensure any open run is closed before appending the array slice.
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+
+ if (length == 0) {
+ return Status::OK();
+ }
+
+ ArraySpan to_append = array;
+ to_append.SetSlice(array.offset + offset, length);
+
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendArray<int16_t>(to_append));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendArray<int32_t>(to_append));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendArray<int64_t>(to_append));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+
+ return Status::OK();
+}
+
+std::shared_ptr<DataType> RunEndEncodedBuilder::type() const { return type_; }
+
+Status RunEndEncodedBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ // Finish the values array before so we can close the current run and append
+ // the last run-end.
+ std::shared_ptr<ArrayData> values_data;
+ RETURN_NOT_OK(value_run_builder_->FinishInternal(&values_data));
+ auto values_array = MakeArray(values_data);
+
+ ARROW_ASSIGN_OR_RAISE(auto run_ends_array, run_end_builder().Finish());
+
+ ARROW_ASSIGN_OR_RAISE(auto ree_array,
+ RunEndEncodedArray::Make(length_, run_ends_array, values_array));
+ *out = std::move(ree_array->data());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::FinishCurrentRun() {
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+ UpdateDimensions(length_, 0);
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendRunEnd(int64_t run_end) {
+ constexpr auto max = std::numeric_limits<RunEndsType>::max();
+ if (ARROW_PREDICT_FALSE(run_end > max)) {
+ return Status::Invalid("Run end value must fit on run ends type but ", run_end, " > ",
+ max, ".");
+ }
+ return internal::checked_cast<typename CTypeTraits<RunEndsType>::BuilderType*>(
+ children_[0].get())
+ ->Append(static_cast<RunEndsType>(run_end));
+}
+
+Status RunEndEncodedBuilder::AppendRunEnd(int64_t run_end) {
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendRunEnd<int16_t>(run_end));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendRunEnd<int32_t>(run_end));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendRunEnd<int64_t>(run_end));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::CloseRun(const std::shared_ptr<const Scalar>& value,
+ int64_t run_length) {
+ // TODO(felipecrv): gracefully fragment runs bigger than INT32_MAX
+ if (ARROW_PREDICT_FALSE(run_length > std::numeric_limits<int32_t>::max())) {
Review Comment:
Hmm, fair enough, as long as it's documented.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109143495
##########
cpp/src/arrow/util/ree_util_test.cc:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array.h"
+#include "arrow/builder.h"
+#include "arrow/compute/api_vector.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/type_traits.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace ree_util {
+
+template <typename RunEndsType>
+struct ReeUtilTest : public ::testing::Test {
+ // Re-implementation of FindPhysicalIndex that uses trivial linear-search
+ // instead of the more efficient implementation.
+ int64_t FindPhysicalIndexTestImpl(const RunEndsType* run_ends, int64_t run_ends_size,
+ int64_t i, int64_t absolute_offset = 0) {
+ for (int64_t j = 0; j < run_ends_size; j++) {
+ if (absolute_offset + i < run_ends[j]) {
+ return j;
+ }
+ }
+ return run_ends_size;
+ }
+};
+TYPED_TEST_SUITE_P(ReeUtilTest);
+
+TYPED_TEST_P(ReeUtilTest, PhysicalOffset) {
+ using RE = TypeParam; // Run-end type
+ const RE run_ends1[] = {1};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends1, 1, 0, 0), 0);
+ const RE run_ends124[] = {1, 2, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 1, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 2, 0), 2);
+ const RE run_ends234[] = {2, 3, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 1, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 2, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 3, 0), 2);
+ const RE run_ends246[] = {2, 4, 6};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends246, 3, 3, 0), 1);
+
+ // Out-of-range logical offset should return run_ends size
Review Comment:
~The returned value here or in `FindPhysicalIndex` implementation?~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #33641:
URL: https://github.com/apache/arrow/pull/33641#issuecomment-1380958379
:warning: GitHub issue #32104 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #33641:
URL: https://github.com/apache/arrow/pull/33641#issuecomment-1385775692
:warning: GitHub issue #32104 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1092500261
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
Review Comment:
`data.offset` and `data.length` are the logical `offset` and `length`. The physical `offset` into the `run_ends` is `run_ends.offset` + what the offset of the first `run_end` that is > `data.offset` (this is determined by binary-search).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1093302332
##########
cpp/src/arrow/array/array_encoded.h:
##########
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Array accessor classes run-end encoded arrays
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/array/array_base.h"
+#include "arrow/array/data.h"
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/type.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow {
+
+/// \addtogroup encoded-arrays
+///
+/// @{
+
+// ----------------------------------------------------------------------
+// RunEndEncoded
+
+/// \brief Array type for run-end encoded data
+class ARROW_EXPORT RunEndEncodedArray : public Array {
+ public:
+ using TypeClass = RunEndEncodedType;
+
+ explicit RunEndEncodedArray(const std::shared_ptr<ArrayData>& data);
+
+ RunEndEncodedArray(const std::shared_ptr<DataType>& type, int64_t logical_length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values, int64_t offset = 0);
+
+ /// \brief Construct a RunEndEncodedArray from values and run ends arrays
+ ///
+ /// The data type is automatically inferred from the arguments.
+ /// The run_ends and values arrays must have the same length.
+ static Result<std::shared_ptr<RunEndEncodedArray>> Make(
+ const std::shared_ptr<Array>& run_ends, const std::shared_ptr<Array>& values,
+ int64_t logical_length, int64_t offset = 0);
+
+ /// \brief Returns an array holding the logical indexes of each run-end
+ ///
+ /// The physical offset to the array is applied.
+ std::shared_ptr<Array> run_ends_array() const;
+
+ /// \brief Returns an array holding the values of each run
+ ///
+ /// The physical offset to the array is applied.
+ std::shared_ptr<Array> values_array() const;
+
+ /// \brief Find the physical offset of the REE array
+ ///
+ /// This function uses binary-search, so it has a O(log N) cost.
+ int64_t FindPhysicalOffset() const;
+
+ /// \brief Find the physical length of the REE array
+ ///
+ /// Avoid calling this function if the physical length can be estabilished in
+ /// some other way. This function uses binary-search, so it has a O(log N)
+ /// cost.
Review Comment:
Now I get it. "physical length" is the number of run ends and values that you need from the child array to fully represent the logical range of values from `offset` to `length`. I will add better explanations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109073532
##########
cpp/src/arrow/scalar.cc:
##########
@@ -583,6 +608,17 @@ Result<std::shared_ptr<Scalar>> StructScalar::field(FieldRef ref) const {
}
}
+RunEndEncodedScalar::RunEndEncodedScalar(std::shared_ptr<Scalar> value,
+ std::shared_ptr<DataType> type)
+ : Scalar{std::move(type), value->is_valid}, value{std::move(value)} {}
Review Comment:
I didn't see this check in other scalars, so I assumed this is supposed to be handled in another layer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109036220
##########
cpp/src/arrow/type.h:
##########
@@ -1223,6 +1223,35 @@ class ARROW_EXPORT DenseUnionType : public UnionType {
std::string name() const override { return "dense_union"; }
};
+/// \brief Type class for run-end encoded data
+class ARROW_EXPORT RunEndEncodedType : public NestedType {
+ public:
+ static constexpr Type::type type_id = Type::RUN_END_ENCODED;
+
+ static constexpr const char* type_name() { return "run_end_encoded"; }
+
+ explicit RunEndEncodedType(std::shared_ptr<DataType> run_end_type,
+ std::shared_ptr<DataType> value_type);
+
+ DataTypeLayout layout() const override {
+ // NOTE(felipecrv): his can be removed once we ensure existing code does not
+ // assume the existence of at least one buffer.
Review Comment:
The current wording sounds like you intend to change it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109135741
##########
cpp/src/arrow/array/builder_run_end.cc:
##########
@@ -0,0 +1,331 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/builder_run_end.h"
+#include "arrow/array/builder_primitive.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <utility>
+#include <vector>
+
+#include "arrow/scalar.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace internal {
+
+RunCompressorBuilder::RunCompressorBuilder(MemoryPool* pool,
+ std::shared_ptr<ArrayBuilder> inner_builder,
+ std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), inner_builder_(std::move(inner_builder)) {}
+
+RunCompressorBuilder::~RunCompressorBuilder() = default;
+
+void RunCompressorBuilder::Reset() {
+ current_run_length_ = 0;
+ current_value_.reset();
+ inner_builder_->Reset();
+ UpdateDimensions();
+}
+
+Status RunCompressorBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(inner_builder_->Resize(capacity));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendNulls(int64_t length) {
+ if (ARROW_PREDICT_FALSE(length == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new NULL run
+ DCHECK_EQ(current_value_, NULLPTR);
+ current_run_length_ = length;
+ } else if (current_value_ == NULLPTR) {
+ // Extend the currently open NULL run
+ current_run_length_ += length;
+ } else {
+ // Close then non-NULL run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(inner_builder_->AppendScalar(*current_value_));
+ UpdateDimensions();
+ // Open a new NULL run
+ current_value_.reset();
+ current_run_length_ = length;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to a run-compressed array.");
+}
+
+Status RunCompressorBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (ARROW_PREDICT_FALSE(n_repeats == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ } else if ((current_value_ == NULLPTR && !scalar.is_valid) ||
+ (current_value_ != NULLPTR && current_value_->Equals(scalar))) {
+ // Extend the currently open run
+ current_run_length_ += n_repeats;
+ } else {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendScalars(const ScalarVector& scalars) {
+ if (scalars.empty()) {
+ return Status::OK();
+ }
+ RETURN_NOT_OK(ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendRunCompressedArraySlice(
+ const ArraySpan& run_compressed_array, int64_t offset, int64_t length) {
+ DCHECK(!has_open_run());
+ RETURN_NOT_OK(inner_builder_->AppendArraySlice(run_compressed_array, offset, length));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishCurrentRun() {
+ if (current_run_length_ > 0) {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Clear the current run
+ current_value_.reset();
+ current_run_length_ = 0;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ ARROW_RETURN_NOT_OK(FinishCurrentRun());
+ return inner_builder_->FinishInternal(out);
+}
+
+} // namespace internal
+
+// ----------------------------------------------------------------------
+// RunEndEncodedBuilder
+
+RunEndEncodedBuilder::ValueRunBuilder::ValueRunBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& value_builder,
+ const std::shared_ptr<DataType>& value_type, RunEndEncodedBuilder& ree_builder)
+ : RunCompressorBuilder(pool, std::move(value_builder), std::move(value_type)),
+ ree_builder_(ree_builder) {}
+
+RunEndEncodedBuilder::RunEndEncodedBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder, std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), type_(internal::checked_pointer_cast<RunEndEncodedType>(type)) {
+ auto value_run_builder =
+ std::make_shared<ValueRunBuilder>(pool, value_builder, type_->value_type(), *this);
+ value_run_builder_ = value_run_builder.get();
+ children_ = {run_end_builder, std::move(value_run_builder)};
+ UpdateDimensions(0, 0);
+ null_count_ = 0;
+}
+
+Status RunEndEncodedBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(value_run_builder_->ResizePhysical(capacity));
+ RETURN_NOT_OK(run_end_builder().Resize(capacity));
+ UpdateDimensions(committed_logical_length_, 0);
+ return Status::OK();
+}
+
+void RunEndEncodedBuilder::Reset() {
+ value_run_builder_->Reset();
+ run_end_builder().Reset();
+ UpdateDimensions(0, 0);
+}
+
+Status RunEndEncodedBuilder::AppendNulls(int64_t length) {
+ RETURN_NOT_OK(value_run_builder_->AppendNulls(length));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to run-end encoded array.");
+}
+
+Status RunEndEncodedBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (scalar.type->id() == Type::RUN_END_ENCODED) {
+ return AppendScalar(*internal::checked_cast<const RunEndEncodedScalar&>(scalar).value,
+ n_repeats);
+ }
+ RETURN_NOT_OK(value_run_builder_->AppendScalar(scalar, n_repeats));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendScalars(const ScalarVector& scalars) {
+ RETURN_NOT_OK(this->ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendArray(const ArraySpan& to_append) {
+ DCHECK_GT(to_append.length, 0);
+ DCHECK(!value_run_builder_->has_open_run());
+
+ ree_util::RunEndEncodedArraySpan<RunEndsType> ree_span(to_append);
+ const int64_t physical_offset = ree_span.PhysicalIndex(0);
+ const int64_t physical_length =
+ ree_span.PhysicalIndex(ree_span.length() - 1) + 1 - physical_offset;
+
+ RETURN_NOT_OK(ReservePhysical(physical_length));
+
+ // Append all the run ends from to_append
+ const auto end = ree_span.end();
+ for (auto it = ree_span.iterator(0, physical_offset); it != end; ++it) {
+ const int64_t run_end = committed_logical_length_ + it.run_length();
+ RETURN_NOT_OK(DoAppendRunEnd<RunEndsType>(run_end));
+ UpdateDimensions(run_end, 0);
+ }
+
+ // Append all the values directly
+ RETURN_NOT_OK(value_run_builder_->AppendRunCompressedArraySlice(
+ ree_util::ValuesArray(to_append), physical_offset, physical_length));
+
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendArraySlice(const ArraySpan& array, int64_t offset,
+ int64_t length) {
+ ARROW_DCHECK(offset + length <= array.length);
+ ARROW_DCHECK(array.type->Equals(type_));
+
+ // Ensure any open run is closed before appending the array slice.
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+
+ if (length == 0) {
+ return Status::OK();
+ }
+
+ ArraySpan to_append = array;
+ to_append.SetSlice(array.offset + offset, length);
+
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendArray<int16_t>(to_append));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendArray<int32_t>(to_append));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendArray<int64_t>(to_append));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+
+ return Status::OK();
+}
+
+std::shared_ptr<DataType> RunEndEncodedBuilder::type() const { return type_; }
+
+Status RunEndEncodedBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ // Finish the values array before so we can close the current run and append
+ // the last run-end.
+ std::shared_ptr<ArrayData> values_data;
+ RETURN_NOT_OK(value_run_builder_->FinishInternal(&values_data));
+ auto values_array = MakeArray(values_data);
+
+ ARROW_ASSIGN_OR_RAISE(auto run_ends_array, run_end_builder().Finish());
+
+ ARROW_ASSIGN_OR_RAISE(auto ree_array,
+ RunEndEncodedArray::Make(length_, run_ends_array, values_array));
+ *out = std::move(ree_array->data());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::FinishCurrentRun() {
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+ UpdateDimensions(length_, 0);
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendRunEnd(int64_t run_end) {
+ constexpr auto max = std::numeric_limits<RunEndsType>::max();
+ if (ARROW_PREDICT_FALSE(run_end > max)) {
+ return Status::Invalid("Run end value must fit on run ends type but ", run_end, " > ",
+ max, ".");
+ }
+ return internal::checked_cast<typename CTypeTraits<RunEndsType>::BuilderType*>(
+ children_[0].get())
+ ->Append(static_cast<RunEndsType>(run_end));
+}
+
+Status RunEndEncodedBuilder::AppendRunEnd(int64_t run_end) {
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendRunEnd<int16_t>(run_end));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendRunEnd<int32_t>(run_end));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendRunEnd<int64_t>(run_end));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::CloseRun(const std::shared_ptr<const Scalar>& value,
+ int64_t run_length) {
+ // TODO(felipecrv): gracefully fragment runs bigger than INT32_MAX
+ if (ARROW_PREDICT_FALSE(run_length > std::numeric_limits<int32_t>::max())) {
Review Comment:
This depends on us deciding if we want to produce huge runs. I think limiting the size of runs (not the length of logical arrays of course) is going to prevent lots of multi-language integration pains at a very low storage/memory cost. ~2B elements runs compressed to a single value is great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109135947
##########
cpp/src/arrow/array/builder_run_end.cc:
##########
@@ -0,0 +1,331 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/builder_run_end.h"
+#include "arrow/array/builder_primitive.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <utility>
+#include <vector>
+
+#include "arrow/scalar.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace internal {
+
+RunCompressorBuilder::RunCompressorBuilder(MemoryPool* pool,
+ std::shared_ptr<ArrayBuilder> inner_builder,
+ std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), inner_builder_(std::move(inner_builder)) {}
+
+RunCompressorBuilder::~RunCompressorBuilder() = default;
+
+void RunCompressorBuilder::Reset() {
+ current_run_length_ = 0;
+ current_value_.reset();
+ inner_builder_->Reset();
+ UpdateDimensions();
+}
+
+Status RunCompressorBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(inner_builder_->Resize(capacity));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendNulls(int64_t length) {
+ if (ARROW_PREDICT_FALSE(length == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new NULL run
+ DCHECK_EQ(current_value_, NULLPTR);
+ current_run_length_ = length;
+ } else if (current_value_ == NULLPTR) {
+ // Extend the currently open NULL run
+ current_run_length_ += length;
+ } else {
+ // Close then non-NULL run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(inner_builder_->AppendScalar(*current_value_));
+ UpdateDimensions();
+ // Open a new NULL run
+ current_value_.reset();
+ current_run_length_ = length;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to a run-compressed array.");
+}
+
+Status RunCompressorBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (ARROW_PREDICT_FALSE(n_repeats == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ } else if ((current_value_ == NULLPTR && !scalar.is_valid) ||
+ (current_value_ != NULLPTR && current_value_->Equals(scalar))) {
+ // Extend the currently open run
+ current_run_length_ += n_repeats;
+ } else {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendScalars(const ScalarVector& scalars) {
+ if (scalars.empty()) {
+ return Status::OK();
+ }
+ RETURN_NOT_OK(ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendRunCompressedArraySlice(
+ const ArraySpan& run_compressed_array, int64_t offset, int64_t length) {
+ DCHECK(!has_open_run());
+ RETURN_NOT_OK(inner_builder_->AppendArraySlice(run_compressed_array, offset, length));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishCurrentRun() {
+ if (current_run_length_ > 0) {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Clear the current run
+ current_value_.reset();
+ current_run_length_ = 0;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ ARROW_RETURN_NOT_OK(FinishCurrentRun());
+ return inner_builder_->FinishInternal(out);
+}
+
+} // namespace internal
+
+// ----------------------------------------------------------------------
+// RunEndEncodedBuilder
+
+RunEndEncodedBuilder::ValueRunBuilder::ValueRunBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& value_builder,
+ const std::shared_ptr<DataType>& value_type, RunEndEncodedBuilder& ree_builder)
+ : RunCompressorBuilder(pool, std::move(value_builder), std::move(value_type)),
+ ree_builder_(ree_builder) {}
+
+RunEndEncodedBuilder::RunEndEncodedBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder, std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), type_(internal::checked_pointer_cast<RunEndEncodedType>(type)) {
+ auto value_run_builder =
+ std::make_shared<ValueRunBuilder>(pool, value_builder, type_->value_type(), *this);
+ value_run_builder_ = value_run_builder.get();
+ children_ = {run_end_builder, std::move(value_run_builder)};
+ UpdateDimensions(0, 0);
+ null_count_ = 0;
+}
+
+Status RunEndEncodedBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(value_run_builder_->ResizePhysical(capacity));
+ RETURN_NOT_OK(run_end_builder().Resize(capacity));
+ UpdateDimensions(committed_logical_length_, 0);
+ return Status::OK();
+}
+
+void RunEndEncodedBuilder::Reset() {
+ value_run_builder_->Reset();
+ run_end_builder().Reset();
+ UpdateDimensions(0, 0);
+}
+
+Status RunEndEncodedBuilder::AppendNulls(int64_t length) {
+ RETURN_NOT_OK(value_run_builder_->AppendNulls(length));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to run-end encoded array.");
+}
+
+Status RunEndEncodedBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (scalar.type->id() == Type::RUN_END_ENCODED) {
+ return AppendScalar(*internal::checked_cast<const RunEndEncodedScalar&>(scalar).value,
+ n_repeats);
+ }
+ RETURN_NOT_OK(value_run_builder_->AppendScalar(scalar, n_repeats));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendScalars(const ScalarVector& scalars) {
+ RETURN_NOT_OK(this->ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendArray(const ArraySpan& to_append) {
+ DCHECK_GT(to_append.length, 0);
+ DCHECK(!value_run_builder_->has_open_run());
+
+ ree_util::RunEndEncodedArraySpan<RunEndsType> ree_span(to_append);
+ const int64_t physical_offset = ree_span.PhysicalIndex(0);
+ const int64_t physical_length =
+ ree_span.PhysicalIndex(ree_span.length() - 1) + 1 - physical_offset;
+
+ RETURN_NOT_OK(ReservePhysical(physical_length));
+
+ // Append all the run ends from to_append
+ const auto end = ree_span.end();
+ for (auto it = ree_span.iterator(0, physical_offset); it != end; ++it) {
+ const int64_t run_end = committed_logical_length_ + it.run_length();
+ RETURN_NOT_OK(DoAppendRunEnd<RunEndsType>(run_end));
+ UpdateDimensions(run_end, 0);
+ }
+
+ // Append all the values directly
+ RETURN_NOT_OK(value_run_builder_->AppendRunCompressedArraySlice(
+ ree_util::ValuesArray(to_append), physical_offset, physical_length));
+
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendArraySlice(const ArraySpan& array, int64_t offset,
+ int64_t length) {
+ ARROW_DCHECK(offset + length <= array.length);
+ ARROW_DCHECK(array.type->Equals(type_));
+
+ // Ensure any open run is closed before appending the array slice.
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+
+ if (length == 0) {
+ return Status::OK();
+ }
+
+ ArraySpan to_append = array;
+ to_append.SetSlice(array.offset + offset, length);
+
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendArray<int16_t>(to_append));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendArray<int32_t>(to_append));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendArray<int64_t>(to_append));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+
+ return Status::OK();
+}
+
+std::shared_ptr<DataType> RunEndEncodedBuilder::type() const { return type_; }
+
+Status RunEndEncodedBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ // Finish the values array before so we can close the current run and append
+ // the last run-end.
+ std::shared_ptr<ArrayData> values_data;
+ RETURN_NOT_OK(value_run_builder_->FinishInternal(&values_data));
+ auto values_array = MakeArray(values_data);
+
+ ARROW_ASSIGN_OR_RAISE(auto run_ends_array, run_end_builder().Finish());
+
+ ARROW_ASSIGN_OR_RAISE(auto ree_array,
+ RunEndEncodedArray::Make(length_, run_ends_array, values_array));
+ *out = std::move(ree_array->data());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::FinishCurrentRun() {
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+ UpdateDimensions(length_, 0);
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendRunEnd(int64_t run_end) {
+ constexpr auto max = std::numeric_limits<RunEndsType>::max();
+ if (ARROW_PREDICT_FALSE(run_end > max)) {
+ return Status::Invalid("Run end value must fit on run ends type but ", run_end, " > ",
+ max, ".");
+ }
+ return internal::checked_cast<typename CTypeTraits<RunEndsType>::BuilderType*>(
+ children_[0].get())
+ ->Append(static_cast<RunEndsType>(run_end));
+}
+
+Status RunEndEncodedBuilder::AppendRunEnd(int64_t run_end) {
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendRunEnd<int16_t>(run_end));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendRunEnd<int32_t>(run_end));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendRunEnd<int64_t>(run_end));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::CloseRun(const std::shared_ptr<const Scalar>& value,
+ int64_t run_length) {
+ // TODO(felipecrv): gracefully fragment runs bigger than INT32_MAX
+ if (ARROW_PREDICT_FALSE(run_length > std::numeric_limits<int32_t>::max())) {
+ return Status::Invalid(
+ "Run-length of run-encoded arrays must fit in a 32-bit signed integer.");
+ }
+ const int64_t run_end = committed_logical_length_ + run_length;
Review Comment:
Good catch. Will do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109139081
##########
cpp/src/arrow/util/ree_util.h:
##########
@@ -0,0 +1,351 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cassert>
+#include <cstdint>
+
+#include "arrow/array/data.h"
+#include "arrow/type_traits.h"
+#include "arrow/util/macros.h"
+
+namespace arrow {
+namespace ree_util {
+
+/// \brief Get the child array holding the run ends from an REE array
+inline const ArraySpan& RunEndsArray(const ArraySpan& span) { return span.child_data[0]; }
+
+/// \brief Get the child array holding the data values from an REE array
+inline const ArraySpan& ValuesArray(const ArraySpan& span) { return span.child_data[1]; }
+
+/// \brief Get a pointer to run ends values of an REE array
+template <typename RunEndsType>
+const RunEndsType* RunEnds(const ArraySpan& span) {
+ assert(RunEndsArray(span).type->id() == CTypeTraits<RunEndsType>::ArrowType::type_id);
+ return RunEndsArray(span).GetValues<RunEndsType>(1);
+}
+
+namespace internal {
+
+/// \brief Uses binary-search to find the physical offset given a logical offset
+/// and run-end values
+///
+/// \return the physical offset or run_ends_size if the physical offset is not
+/// found in run_ends
+template <typename RunEndsType>
+int64_t FindPhysicalIndex(const RunEndsType* run_ends, int64_t run_ends_size, int64_t i,
+ int64_t absolute_offset) {
+ auto it = std::upper_bound(run_ends, run_ends + run_ends_size, absolute_offset + i);
+ int64_t result = std::distance(run_ends, it);
+ assert(result <= run_ends_size);
+ return result;
+}
+
+/// \brief Uses binary-search to calculate the number of physical values (and
+/// run-ends) necessary to represent the logical range of values from
+/// offset to length
+template <typename RunEndsType>
+int64_t FindPhysicalLength(int64_t length, int64_t offset, const RunEndsType* run_ends,
+ int64_t run_ends_size) {
Review Comment:
I agree. I will fix this now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade merged pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade merged PR #33641:
URL: https://github.com/apache/arrow/pull/33641
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109077870
##########
cpp/src/arrow/array/concatenate.cc:
##########
@@ -436,6 +438,28 @@ class ConcatenateImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
+ int64_t physical_length = 0;
+ for (const auto& input : in_) {
+ if (internal::AddWithOverflow(physical_length,
+ ree_util::FindPhysicalLength(ArraySpan(*input)),
+ &physical_length) ||
+ // Make sure we can safely downcast to int32_t
Review Comment:
And in any case, shouldn't we do the right check for int16_t? What if you concatenate two int16_t arrays with a single run of length 65535?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109019065
##########
cpp/src/arrow/array/array_base.cc:
##########
@@ -165,7 +175,7 @@ struct ScalarFromArraySlotImpl {
array_.length());
}
- if (array_.IsNull(index_)) {
+ if (array_.type()->id() != Type::RUN_END_ENCODED && array_.IsNull(index_)) {
Review Comment:
I think the check is much cheaper than making `IsNull` virtually-dispatched.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on PR #33641:
URL: https://github.com/apache/arrow/pull/33641#issuecomment-1434792065
I kicked the builds, let's see.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1073869283
##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +195,7 @@ int GetNumBuffers(const DataType& type) {
case Type::NA:
case Type::STRUCT:
case Type::FIXED_SIZE_LIST:
+ case Type::RUN_END_ENCODED:
return 1;
Review Comment:
Hmm, why do we have the dummy in the first place?
I suppose either you have the dummy and you put the exceptions in IPC/C Data, or you don't have the dummy and you put the exceptions in things that assume the presence of a validity buffer. I would rather not have the dummy, but without looking at the code, maybe that introduces too many exceptions elsewhere.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1091309874
##########
cpp/src/arrow/array/builder_encoded.cc:
##########
@@ -0,0 +1,213 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/builder_encoded.h"
+#include "arrow/array/builder_primitive.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <utility>
+#include <vector>
+
+#include "arrow/scalar.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedBuilder
+
+RunEndEncodedBuilder::RunEndEncodedBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder, std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), type_(internal::checked_pointer_cast<RunEndEncodedType>(type)) {
+ children_ = {std::move(run_end_builder), std::move(value_builder)};
+}
+
+Status RunEndEncodedBuilder::Resize(int64_t) {
+ return Status::NotImplemented(
Review Comment:
I'm completely rewriting the builder class and this is implemented as `ResizePhysical()` -- in the worst-case, an array that can fit `N` runs can fit `N` logical elements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1092500261
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
Review Comment:
`data.offset` and `data.length` are the logical `offset` and `length`. The physical `offset` into the `run_ends` is `run_ends.offset` + the offset of the first `run_end` that is `> data.offset` (this is determined by binary-search).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1104731628
##########
cpp/src/arrow/array/array_run_end.cc:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/array_run_end.h"
+#include "arrow/array/util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedArray
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<ArrayData>& data) {
+ this->SetData(data);
+}
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<DataType>& type,
+ int64_t length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values,
+ int64_t offset) {
+ this->SetData(ArrayData::Make(type, length,
+ /*buffers=*/{NULLPTR},
+ /*child_data=*/{run_ends->data(), values->data()},
+ /*null_count=*/0, offset));
Review Comment:
`SetData` itself checks against `RUN_END_ENCODED`
```cpp
void RunEndEncodedArray::SetData(const std::shared_ptr<ArrayData>& data) {
ARROW_CHECK_EQ(data->type->id(), Type::RUN_END_ENCODED);
...
```
I looked at other arrays with parameterized types and the prevalent style seems to be: check only the top-level type id and leave the deeper checks to `Validate`.
```cpp
ListArray::ListArray(std::shared_ptr<ArrayData> data) { SetData(std::move(data)); }
LargeListArray::LargeListArray(const std::shared_ptr<ArrayData>& data) { SetData(data); }
ListArray::ListArray(std::shared_ptr<DataType> type, int64_t length,
std::shared_ptr<Buffer> value_offsets, std::shared_ptr<Array> values,
std::shared_ptr<Buffer> null_bitmap, int64_t null_count,
int64_t offset) {
ARROW_CHECK_EQ(type->id(), Type::LIST);
auto internal_data = ArrayData::Make(
std::move(type), length,
BufferVector{std::move(null_bitmap), std::move(value_offsets)}, null_count, offset);
internal_data->child_data.emplace_back(values->data());
SetData(std::move(internal_data));
}
void ListArray::SetData(const std::shared_ptr<ArrayData>& data) {
internal::SetListData(this, data);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1104718606
##########
cpp/src/arrow/array/array_dict_test.cc:
##########
@@ -1117,12 +1117,12 @@ TEST(TestDictionary, Validate) {
arr = std::make_shared<DictionaryArray>(dict_type, indices, MakeArray(invalid_data));
ASSERT_RAISES(Invalid, arr->ValidateFull());
- ASSERT_DEATH(
- {
- std::shared_ptr<Array> null_dict_arr =
- std::make_shared<DictionaryArray>(dict_type, indices, nullptr);
- },
- "");
+ // ASSERT_DEATH(
+ // {
+ // std::shared_ptr<Array> null_dict_arr =
+ // std::make_shared<DictionaryArray>(dict_type, indices, nullptr);
+ // },
+ // "");
Review Comment:
It's part of a single commit I pushed to investigate a CI issue
```
Author: Felipe Oliveira Carvalho <fe...@gmail.com>
AuthorDate: Wed Feb 8 10:13:17 2023 -0300
Commit: Felipe Oliveira Carvalho <fe...@gmail.com>
CommitDate: Mon Feb 13 13:24:47 2023 -0300
TEMPORARY: Comment-out ASSERT_DEATH
diff --git a/cpp/src/arrow/array/array_dict_test.cc b/cpp/src/arrow/array/array_dict_test.cc
index bfa732f16..d1e109c90 100644
--- a/cpp/src/arrow/array/array_dict_test.cc
+++ b/cpp/src/arrow/array/array_dict_test.cc
@@ -1117,12 +1117,12 @@ TEST(TestDictionary, Validate) {
arr = std::make_shared<DictionaryArray>(dict_type, indices, MakeArray(invalid_data));
ASSERT_RAISES(Invalid, arr->ValidateFull());
- ASSERT_DEATH(
- {
- std::shared_ptr<Array> null_dict_arr =
- std::make_shared<DictionaryArray>(dict_type, indices, nullptr);
- },
- "");
+ // ASSERT_DEATH(
+ // {
+ // std::shared_ptr<Array> null_dict_arr =
+ // std::make_shared<DictionaryArray>(dict_type, indices, nullptr);
+ // },
+ // "");
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1104734748
##########
cpp/src/arrow/array/array_run_end.cc:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/array_run_end.h"
+#include "arrow/array/util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedArray
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<ArrayData>& data) {
+ this->SetData(data);
+}
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<DataType>& type,
+ int64_t length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values,
+ int64_t offset) {
+ this->SetData(ArrayData::Make(type, length,
+ /*buffers=*/{NULLPTR},
+ /*child_data=*/{run_ends->data(), values->data()},
+ /*null_count=*/0, offset));
Review Comment:
Hmm. Now that I looked again at `ListArray` I see that it does check the parameterized type. I will add more checks here, then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1073827629
##########
cpp/src/arrow/array/array_encoded.h:
##########
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Array accessor classes run-end encoded arrays
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/array/array_base.h"
+#include "arrow/array/data.h"
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/type.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow {
+
+/// \addtogroup encoded-arrays
+///
+/// @{
+
+// ----------------------------------------------------------------------
+// RunEndEncoded
+
+/// \brief Array type for run-end encoded data
+class ARROW_EXPORT RunEndEncodedArray : public Array {
+ public:
+ using TypeClass = RunEndEncodedType;
+
+ explicit RunEndEncodedArray(const std::shared_ptr<ArrayData>& data);
+
+ RunEndEncodedArray(const std::shared_ptr<DataType>& type, int64_t logical_length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values, int64_t offset = 0);
+
+ /// \brief Construct a RunEndEncodedArray from values and run ends arrays
+ ///
+ /// The data type is automatically inferred from the arguments.
+ /// The run_ends and values arrays must have the same length.
+ static Result<std::shared_ptr<RunEndEncodedArray>> Make(
+ const std::shared_ptr<Array>& run_ends, const std::shared_ptr<Array>& values,
+ int64_t logical_length, int64_t offset = 0);
+
+ /// \brief Returns an array holding the logical indexes of each run-end
+ ///
+ /// The physical offset to the array is applied.
+ std::shared_ptr<Array> run_ends_array() const;
+
+ /// \brief Returns an array holding the values of each run
+ ///
+ /// The physical offset to the array is applied.
+ std::shared_ptr<Array> values_array() const;
+
+ /// \brief Find the physical offset of the REE array
Review Comment:
AFAIU it's a translation from the array `offiset` into the `run_ends` array offset.
Given `run_ends: [2, 4, 10]`, 0, 1, and 2 translate into `0`; 3, and 4 into `1`, 5-10 translate into 2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1085625034
##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +195,7 @@ int GetNumBuffers(const DataType& type) {
case Type::NA:
case Type::STRUCT:
case Type::FIXED_SIZE_LIST:
+ case Type::RUN_END_ENCODED:
return 1;
Review Comment:
I think most of the code in Acero is used to the idea of a validity buffer not existing if the null count is 0. So I think the easier path would be to have 0 buffers. But I'm pretty sure I can work around it if we decide it's easier elsewhere to have one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109012404
##########
cpp/src/arrow/testing/json_internal.cc:
##########
@@ -437,6 +437,10 @@ class SchemaWriter {
Status Visit(const DictionaryType& type) { return VisitType(*type.value_type()); }
+ Status Visit(const RunEndEncodedType& type) {
+ return Status::NotImplemented(type.name());
Review Comment:
Sure, hence why I asked about followups
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109143495
##########
cpp/src/arrow/util/ree_util_test.cc:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array.h"
+#include "arrow/builder.h"
+#include "arrow/compute/api_vector.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/type_traits.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace ree_util {
+
+template <typename RunEndsType>
+struct ReeUtilTest : public ::testing::Test {
+ // Re-implementation of FindPhysicalIndex that uses trivial linear-search
+ // instead of the more efficient implementation.
+ int64_t FindPhysicalIndexTestImpl(const RunEndsType* run_ends, int64_t run_ends_size,
+ int64_t i, int64_t absolute_offset = 0) {
+ for (int64_t j = 0; j < run_ends_size; j++) {
+ if (absolute_offset + i < run_ends[j]) {
+ return j;
+ }
+ }
+ return run_ends_size;
+ }
+};
+TYPED_TEST_SUITE_P(ReeUtilTest);
+
+TYPED_TEST_P(ReeUtilTest, PhysicalOffset) {
+ using RE = TypeParam; // Run-end type
+ const RE run_ends1[] = {1};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends1, 1, 0, 0), 0);
+ const RE run_ends124[] = {1, 2, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 1, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 2, 0), 2);
+ const RE run_ends234[] = {2, 3, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 1, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 2, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 3, 0), 2);
+ const RE run_ends246[] = {2, 4, 6};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends246, 3, 3, 0), 1);
+
+ // Out-of-range logical offset should return run_ends size
Review Comment:
The returned value here or in `FindPhysicalIndex` implementation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109144547
##########
cpp/src/arrow/util/ree_util_test.cc:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array.h"
+#include "arrow/builder.h"
+#include "arrow/compute/api_vector.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/type_traits.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace ree_util {
+
+template <typename RunEndsType>
+struct ReeUtilTest : public ::testing::Test {
+ // Re-implementation of FindPhysicalIndex that uses trivial linear-search
+ // instead of the more efficient implementation.
+ int64_t FindPhysicalIndexTestImpl(const RunEndsType* run_ends, int64_t run_ends_size,
+ int64_t i, int64_t absolute_offset = 0) {
+ for (int64_t j = 0; j < run_ends_size; j++) {
+ if (absolute_offset + i < run_ends[j]) {
+ return j;
+ }
+ }
+ return run_ends_size;
+ }
+};
+TYPED_TEST_SUITE_P(ReeUtilTest);
+
+TYPED_TEST_P(ReeUtilTest, PhysicalOffset) {
+ using RE = TypeParam; // Run-end type
+ const RE run_ends1[] = {1};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends1, 1, 0, 0), 0);
+ const RE run_ends124[] = {1, 2, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 1, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 2, 0), 2);
+ const RE run_ends234[] = {2, 3, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 1, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 2, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 3, 0), 2);
+ const RE run_ends246[] = {2, 4, 6};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends246, 3, 3, 0), 1);
+
+ // Out-of-range logical offset should return run_ends size
Review Comment:
I meant in the implementation, sorry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1093807063
##########
cpp/src/arrow/array/array_encoded.h:
##########
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Array accessor classes run-end encoded arrays
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/array/array_base.h"
+#include "arrow/array/data.h"
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/type.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow {
+
+/// \addtogroup encoded-arrays
+///
+/// @{
+
+// ----------------------------------------------------------------------
+// RunEndEncoded
+
+/// \brief Array type for run-end encoded data
+class ARROW_EXPORT RunEndEncodedArray : public Array {
+ public:
+ using TypeClass = RunEndEncodedType;
+
+ explicit RunEndEncodedArray(const std::shared_ptr<ArrayData>& data);
+
+ RunEndEncodedArray(const std::shared_ptr<DataType>& type, int64_t logical_length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values, int64_t offset = 0);
+
+ /// \brief Construct a RunEndEncodedArray from values and run ends arrays
+ ///
+ /// The data type is automatically inferred from the arguments.
+ /// The run_ends and values arrays must have the same length.
+ static Result<std::shared_ptr<RunEndEncodedArray>> Make(
+ const std::shared_ptr<Array>& run_ends, const std::shared_ptr<Array>& values,
+ int64_t logical_length, int64_t offset = 0);
+
+ /// \brief Returns an array holding the logical indexes of each run-end
+ ///
+ /// The physical offset to the array is applied.
+ std::shared_ptr<Array> run_ends_array() const;
+
+ /// \brief Returns an array holding the values of each run
+ ///
+ /// The physical offset to the array is applied.
+ std::shared_ptr<Array> values_array() const;
+
+ /// \brief Find the physical offset of the REE array
Review Comment:
I'm expanding these comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1093925895
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
+ if (full_validation && run_ends_data.length != 0) {
+ const int64_t run_ends_length = ree_util::RunEndsArray(span).length;
+ int64_t last_run_end = 0;
+ int64_t physical_offset = 0;
+ int64_t physical_end = 0;
+ for (int64_t index = 0; index < run_ends_length; index++) {
+ int64_t run_end = run_ends[index];
+ if (run_end < 1) {
+ return Status::Invalid(
+ "Run ends array invalid: All run ends must be a positive integer but run "
+ "end ",
+ index, " is ", run_end);
+ }
+ if (run_end <= last_run_end) {
+ return Status::Invalid(
+ "Run ends array invalid: Each run end must be greater than the prevous "
+ "one, but run end ",
+ index, " is ", run_end, " and run end ", index - 1, " is ", last_run_end);
+ }
+ if (run_end > data.offset && last_run_end <= data.offset) {
+ physical_offset = index;
+ }
+ if (run_end >= data.offset + data.length &&
+ last_run_end < data.offset + data.length) {
+ physical_end = index + 1;
+ }
+ last_run_end = run_end;
+ }
+ ARROW_DCHECK_EQ(physical_offset, ree_util::FindPhysicalOffset(span));
+ ARROW_DCHECK_EQ(physical_end, physical_offset + ree_util::FindPhysicalLength(span));
+ if (values_data.length < physical_end) {
+ return Status::Invalid(
+ "Values array needs at least ", physical_end,
+ " elements to hold the runs described by the run ends array, but only has ",
+ values_data.length);
+ }
Review Comment:
Agreed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1091347280
##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +195,7 @@ int GetNumBuffers(const DataType& type) {
case Type::NA:
case Type::STRUCT:
case Type::FIXED_SIZE_LIST:
+ case Type::RUN_END_ENCODED:
return 1;
Review Comment:
From looking at @zeroshade PR, I got reminded that the child `values` array will have a validity bitmap. That, of course doesn't map to logical validity of the REE elements, but allows for an `IsValid(i)` at the top-level array that can be implemented at the cost of binary searching for the physical index of `i` and checking the validity bitmap of the child `values` array. Maybe that can make integration easier and allows Acero to improve how it deals with REEs more incrementally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1090863816
##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +195,7 @@ int GetNumBuffers(const DataType& type) {
case Type::NA:
case Type::STRUCT:
case Type::FIXED_SIZE_LIST:
+ case Type::RUN_END_ENCODED:
return 1;
Review Comment:
I think REEs make thinking about nullability so different than normal that a dummy buffer won't really make things much easier. That's my impression without have much knowledge about all the codepaths that arrays go through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on PR #33641:
URL: https://github.com/apache/arrow/pull/33641#issuecomment-1415977367
@lidavidm @wjones127 @westonpace @zeroshade @bkietz
Before this being fully ready for review, I want to:
1. add the Scalar type and remove the `VisitTypeInline` and
2. I want to rewrite the comparator code
3. more tests
but feel free to skim the code and give higher level architectural feedback before a more detailed review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1073580583
##########
cpp/src/arrow/type_traits.h:
##########
@@ -1177,6 +1197,7 @@ constexpr bool is_nested(Type::type type_id) {
case Type::STRUCT:
case Type::SPARSE_UNION:
case Type::DENSE_UNION:
+ case Type::RUN_END_ENCODED:
Review Comment:
From a quick glance, I think this is mostly used as 'physically nested' in the codebase, so this is OK. I don't think we have the right abstractions to differentiate between physical/logical and deal with encodings, e.g. the way we handle dictionaries is often to just decode them, and kernel implementations are very much often lost in the weeds of encoding details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1073816052
##########
cpp/src/arrow/CMakeLists.txt:
##########
@@ -140,13 +140,16 @@ set(ARROW_SRCS
array/array_binary.cc
array/array_decimal.cc
array/array_dict.cc
+ array/array_encoded.h
Review Comment:
```suggestion
- array/array_encoded.h
```
##########
cpp/src/arrow/CMakeLists.txt:
##########
@@ -140,13 +140,16 @@ set(ARROW_SRCS
array/array_binary.cc
array/array_decimal.cc
array/array_dict.cc
+ array/array_encoded.h
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1073512429
##########
cpp/src/arrow/compute/kernels/vector_run_end_encode_test.cc:
##########
@@ -0,0 +1,196 @@
+// Licensed to the Apache Software Foundation (ASF) under one
Review Comment:
Are the additional kernels (encode/decode, filter) required for this PR? Otherwise, it would be better to file them as separate PRs.
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
Review Comment:
We certainly shouldn't fail here (see similar code paths).
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
Review Comment:
I don't understand why this is needed, can you explain?
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
Review Comment:
```suggestion
return Status::Invalid("Run-end-encoded array has non-zero length ", data.length,
```
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
+ if (full_validation && run_ends_data.length != 0) {
Review Comment:
`run_ends_data.length == 0` is already excluded above
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
+ if (full_validation && run_ends_data.length != 0) {
+ const int64_t run_ends_length = ree_util::RunEndsArray(span).length;
+ int64_t last_run_end = 0;
+ int64_t physical_offset = 0;
+ int64_t physical_end = 0;
+ for (int64_t index = 0; index < run_ends_length; index++) {
+ int64_t run_end = run_ends[index];
+ if (run_end < 1) {
+ return Status::Invalid(
+ "Run ends array invalid: All run ends must be a positive integer but run "
+ "end ",
+ index, " is ", run_end);
+ }
+ if (run_end <= last_run_end) {
+ return Status::Invalid(
+ "Run ends array invalid: Each run end must be greater than the prevous "
+ "one, but run end ",
+ index, " is ", run_end, " and run end ", index - 1, " is ", last_run_end);
+ }
+ if (run_end > data.offset && last_run_end <= data.offset) {
+ physical_offset = index;
+ }
+ if (run_end >= data.offset + data.length &&
+ last_run_end < data.offset + data.length) {
+ physical_end = index + 1;
+ }
+ last_run_end = run_end;
+ }
+ ARROW_DCHECK_EQ(physical_offset, ree_util::FindPhysicalOffset(span));
+ ARROW_DCHECK_EQ(physical_end, physical_offset + ree_util::FindPhysicalLength(span));
+ if (values_data.length < physical_end) {
+ return Status::Invalid(
+ "Values array needs at least ", physical_end,
+ " elements to hold the runs described by the run ends array, but only has ",
+ values_data.length);
+ }
Review Comment:
This should not be needed in this loop. We're checking here that the run ends are monotonically increasing. We can check out of the loop that the last run end doesn't exceed the child array length.
##########
cpp/src/arrow/array/array_encoded.cc:
##########
@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/array_encoded.h"
+#include "arrow/array/util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedArray
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<ArrayData>& data) {
+ ARROW_CHECK_EQ(data->type->id(), Type::RUN_END_ENCODED);
+ SetData(data);
+}
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<DataType>& type,
+ int64_t logical_length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values,
+ int64_t offset) {
+ ARROW_CHECK_EQ(type->id(), Type::RUN_END_ENCODED);
+ SetData(ArrayData::Make(type, logical_length,
+ /*buffers=*/{NULLPTR},
+ /*child_data=*/{run_ends->data(), values->data()}, 0, offset));
+}
+
+Result<std::shared_ptr<RunEndEncodedArray>> RunEndEncodedArray::Make(
+ const std::shared_ptr<Array>& run_ends, const std::shared_ptr<Array>& values,
+ int64_t logical_length, int64_t offset) {
+ auto run_ends_type = run_ends->type();
+ auto values_type = values->type();
+ if (!RunEndEncodedType::RunEndsTypeValid(*run_ends_type)) {
+ return Status::Invalid("Run ends array must be int16, int32 or int64 type");
+ }
+ if (run_ends->null_count() != 0) {
+ return Status::Invalid("Run ends array cannot contain null values");
+ }
+
+ return std::make_shared<RunEndEncodedArray>(
+ run_end_encoded(std::move(run_ends_type), std::move(values_type)), logical_length,
+ run_ends, values, offset);
+}
+
+std::shared_ptr<Array> RunEndEncodedArray::run_ends_array() const {
+ return MakeArray(data()->child_data[0]);
+}
+
+std::shared_ptr<Array> RunEndEncodedArray::values_array() const {
+ return MakeArray(data()->child_data[1]);
+}
Review Comment:
These could be cached on the instance instead of calling `MakeArray` every time.
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
Review Comment:
```suggestion
"Offset + length of a run-end-encoded array must fit in a value of the run ends type ",
```
##########
cpp/src/arrow/array/array_encoded.h:
##########
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Array accessor classes run-end encoded arrays
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/array/array_base.h"
+#include "arrow/array/data.h"
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/type.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow {
+
+/// \addtogroup encoded-arrays
+///
+/// @{
+
+// ----------------------------------------------------------------------
+// RunEndEncoded
+
+/// \brief Array type for run-end encoded data
+class ARROW_EXPORT RunEndEncodedArray : public Array {
+ public:
+ using TypeClass = RunEndEncodedType;
+
+ explicit RunEndEncodedArray(const std::shared_ptr<ArrayData>& data);
+
+ RunEndEncodedArray(const std::shared_ptr<DataType>& type, int64_t logical_length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values, int64_t offset = 0);
+
+ /// \brief Construct a RunEndEncodedArray from values and run ends arrays
+ ///
+ /// The data type is automatically inferred from the arguments.
+ /// The run_ends and values arrays must have the same length.
+ static Result<std::shared_ptr<RunEndEncodedArray>> Make(
+ const std::shared_ptr<Array>& run_ends, const std::shared_ptr<Array>& values,
+ int64_t logical_length, int64_t offset = 0);
+
+ /// \brief Returns an array holding the logical indexes of each run-end
+ ///
+ /// The physical offset to the array is applied.
+ std::shared_ptr<Array> run_ends_array() const;
+
+ /// \brief Returns an array holding the values of each run
+ ///
+ /// The physical offset to the array is applied.
+ std::shared_ptr<Array> values_array() const;
+
+ /// \brief Find the physical offset of the REE array
Review Comment:
I don't understand what this means. What is "the physical offset"? Is it different from the array `offset`?
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
Review Comment:
I would call `run_ends_data.GetNullCount` here, because the null count might be set to -1 (meaning not yet computed).
##########
cpp/src/arrow/array/array_encoded.h:
##########
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Array accessor classes run-end encoded arrays
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/array/array_base.h"
+#include "arrow/array/data.h"
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/type.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow {
+
+/// \addtogroup encoded-arrays
+///
+/// @{
+
+// ----------------------------------------------------------------------
+// RunEndEncoded
+
+/// \brief Array type for run-end encoded data
+class ARROW_EXPORT RunEndEncodedArray : public Array {
+ public:
+ using TypeClass = RunEndEncodedType;
+
+ explicit RunEndEncodedArray(const std::shared_ptr<ArrayData>& data);
+
+ RunEndEncodedArray(const std::shared_ptr<DataType>& type, int64_t logical_length,
Review Comment:
What is the "logical length"? The number of runs, or the accumulated run length? Please describe parameters in docstrings, when they are non-obvious.
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
Review Comment:
What should be clarified is how the `offset` of a run-end-encoded array works: is it a logical offset or an offset into the run ends?
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
Review Comment:
Are you comparing pointer values here?
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
Review Comment:
```suggestion
return Status::Invalid("Null count must be 0 for run-end-encoded array, but was ",
```
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
+ if (full_validation && run_ends_data.length != 0) {
+ const int64_t run_ends_length = ree_util::RunEndsArray(span).length;
+ int64_t last_run_end = 0;
+ int64_t physical_offset = 0;
+ int64_t physical_end = 0;
+ for (int64_t index = 0; index < run_ends_length; index++) {
+ int64_t run_end = run_ends[index];
+ if (run_end < 1) {
+ return Status::Invalid(
+ "Run ends array invalid: All run ends must be a positive integer but run "
+ "end ",
+ index, " is ", run_end);
+ }
+ if (run_end <= last_run_end) {
+ return Status::Invalid(
+ "Run ends array invalid: Each run end must be greater than the prevous "
+ "one, but run end ",
+ index, " is ", run_end, " and run end ", index - 1, " is ", last_run_end);
+ }
+ if (run_end > data.offset && last_run_end <= data.offset) {
+ physical_offset = index;
+ }
+ if (run_end >= data.offset + data.length &&
+ last_run_end < data.offset + data.length) {
+ physical_end = index + 1;
+ }
Review Comment:
These conditions seem weird. You should add explanatory comments.
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
+ if (full_validation && run_ends_data.length != 0) {
+ const int64_t run_ends_length = ree_util::RunEndsArray(span).length;
+ int64_t last_run_end = 0;
+ int64_t physical_offset = 0;
+ int64_t physical_end = 0;
+ for (int64_t index = 0; index < run_ends_length; index++) {
+ int64_t run_end = run_ends[index];
+ if (run_end < 1) {
+ return Status::Invalid(
+ "Run ends array invalid: All run ends must be a positive integer but run "
+ "end ",
+ index, " is ", run_end);
+ }
+ if (run_end <= last_run_end) {
+ return Status::Invalid(
+ "Run ends array invalid: Each run end must be greater than the prevous "
+ "one, but run end ",
+ index, " is ", run_end, " and run end ", index - 1, " is ", last_run_end);
+ }
+ if (run_end > data.offset && last_run_end <= data.offset) {
+ physical_offset = index;
+ }
+ if (run_end >= data.offset + data.length &&
+ last_run_end < data.offset + data.length) {
+ physical_end = index + 1;
+ }
+ last_run_end = run_end;
+ }
+ ARROW_DCHECK_EQ(physical_offset, ree_util::FindPhysicalOffset(span));
+ ARROW_DCHECK_EQ(physical_end, physical_offset + ree_util::FindPhysicalLength(span));
Review Comment:
That's gonna be quite costly at runtime. Are you sure you need this?
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
+ if (full_validation && run_ends_data.length != 0) {
+ const int64_t run_ends_length = ree_util::RunEndsArray(span).length;
Review Comment:
Can we consolidate this with `RunEnds<RunEndsType>` above?
##########
cpp/src/arrow/util/ree_util.h:
##########
@@ -0,0 +1,280 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+
+#include "arrow/array/data.h"
+#include "arrow/type_traits.h"
+#include "arrow/util/logging.h"
Review Comment:
We shouldn't include `logging.h` in public headers.
##########
cpp/src/arrow/array/array_encoded.h:
##########
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Array accessor classes run-end encoded arrays
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/array/array_base.h"
+#include "arrow/array/data.h"
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/type.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow {
+
+/// \addtogroup encoded-arrays
+///
+/// @{
+
+// ----------------------------------------------------------------------
+// RunEndEncoded
+
+/// \brief Array type for run-end encoded data
+class ARROW_EXPORT RunEndEncodedArray : public Array {
+ public:
+ using TypeClass = RunEndEncodedType;
+
+ explicit RunEndEncodedArray(const std::shared_ptr<ArrayData>& data);
+
+ RunEndEncodedArray(const std::shared_ptr<DataType>& type, int64_t logical_length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values, int64_t offset = 0);
+
+ /// \brief Construct a RunEndEncodedArray from values and run ends arrays
+ ///
+ /// The data type is automatically inferred from the arguments.
+ /// The run_ends and values arrays must have the same length.
+ static Result<std::shared_ptr<RunEndEncodedArray>> Make(
+ const std::shared_ptr<Array>& run_ends, const std::shared_ptr<Array>& values,
+ int64_t logical_length, int64_t offset = 0);
+
+ /// \brief Returns an array holding the logical indexes of each run-end
+ ///
+ /// The physical offset to the array is applied.
+ std::shared_ptr<Array> run_ends_array() const;
+
+ /// \brief Returns an array holding the values of each run
+ ///
+ /// The physical offset to the array is applied.
+ std::shared_ptr<Array> values_array() const;
+
+ /// \brief Find the physical offset of the REE array
+ ///
+ /// This function uses binary-search, so it has a O(log N) cost.
+ int64_t FindPhysicalOffset() const;
+
+ /// \brief Find the physical length of the REE array
+ ///
+ /// Avoid calling this function if the physical length can be estabilished in
+ /// some other way. This function uses binary-search, so it has a O(log N)
+ /// cost.
Review Comment:
What does it have a O(log N) cost? What do you call "physical length"? The number of runs? Something else?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1104686299
##########
cpp/src/arrow/array/array_dict_test.cc:
##########
@@ -1117,12 +1117,12 @@ TEST(TestDictionary, Validate) {
arr = std::make_shared<DictionaryArray>(dict_type, indices, MakeArray(invalid_data));
ASSERT_RAISES(Invalid, arr->ValidateFull());
- ASSERT_DEATH(
- {
- std::shared_ptr<Array> null_dict_arr =
- std::make_shared<DictionaryArray>(dict_type, indices, nullptr);
- },
- "");
+ // ASSERT_DEATH(
+ // {
+ // std::shared_ptr<Array> null_dict_arr =
+ // std::make_shared<DictionaryArray>(dict_type, indices, nullptr);
+ // },
+ // "");
Review Comment:
did you mean to leave this commented?
##########
cpp/src/arrow/array/array_run_end.cc:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/array_run_end.h"
+#include "arrow/array/util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedArray
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<ArrayData>& data) {
+ this->SetData(data);
+}
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<DataType>& type,
+ int64_t length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values,
+ int64_t offset) {
+ this->SetData(ArrayData::Make(type, length,
+ /*buffers=*/{NULLPTR},
+ /*child_data=*/{run_ends->data(), values->data()},
+ /*null_count=*/0, offset));
Review Comment:
Should we add a couple `DCHECK_EQ`'s to ensure the passed in type is, in fact, a `run_end_encoded` type and that the run-end type and values type matches the types of the passed in arrays?
##########
cpp/src/arrow/array/array_run_end.cc:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/array_run_end.h"
+#include "arrow/array/util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedArray
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<ArrayData>& data) {
+ this->SetData(data);
+}
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<DataType>& type,
+ int64_t length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values,
+ int64_t offset) {
+ this->SetData(ArrayData::Make(type, length,
+ /*buffers=*/{NULLPTR},
+ /*child_data=*/{run_ends->data(), values->data()},
+ /*null_count=*/0, offset));
+}
+
+Result<std::shared_ptr<RunEndEncodedArray>> RunEndEncodedArray::Make(
+ int64_t logical_length, const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values, int64_t logical_offset) {
+ auto run_end_type = run_ends->type();
+ auto values_type = values->type();
+ if (!RunEndEncodedType::RunEndTypeValid(*run_end_type)) {
+ return Status::Invalid("Run end type must be int16, int32 or int64");
+ }
+ if (run_ends->null_count() != 0) {
+ return Status::Invalid("Run ends array cannot contain null values");
+ }
+
+ return std::make_shared<RunEndEncodedArray>(
+ run_end_encoded(std::move(run_end_type), std::move(values_type)), logical_length,
+ run_ends, values, logical_offset);
+}
+
+void RunEndEncodedArray::SetData(const std::shared_ptr<ArrayData>& data) {
+ ARROW_CHECK_EQ(data->type->id(), Type::RUN_END_ENCODED);
+ DCHECK_EQ(data->child_data.size(), 2);
+
+ // A non-zero number of logical values in this array (offset + length) implies
+ // a non-zero number of runs and values.
+ DCHECK(data->offset + data->length == 0 || data->child_data[0]->length > 0);
+ DCHECK(data->offset + data->length == 0 || data->child_data[1]->length > 0);
Review Comment:
possibly add a DCHECK that `child_data[1]->length >= child_data[0]->length` ?
##########
cpp/src/arrow/array/array_run_end.cc:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/array_run_end.h"
+#include "arrow/array/util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedArray
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<ArrayData>& data) {
+ this->SetData(data);
+}
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<DataType>& type,
+ int64_t length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values,
+ int64_t offset) {
+ this->SetData(ArrayData::Make(type, length,
+ /*buffers=*/{NULLPTR},
+ /*child_data=*/{run_ends->data(), values->data()},
+ /*null_count=*/0, offset));
+}
+
+Result<std::shared_ptr<RunEndEncodedArray>> RunEndEncodedArray::Make(
+ int64_t logical_length, const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values, int64_t logical_offset) {
+ auto run_end_type = run_ends->type();
+ auto values_type = values->type();
+ if (!RunEndEncodedType::RunEndTypeValid(*run_end_type)) {
+ return Status::Invalid("Run end type must be int16, int32 or int64");
+ }
+ if (run_ends->null_count() != 0) {
+ return Status::Invalid("Run ends array cannot contain null values");
+ }
Review Comment:
should we return an error if `values->length < run_ends->length` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1104828235
##########
cpp/src/arrow/array/array_run_end.cc:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/array_run_end.h"
+#include "arrow/array/util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedArray
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<ArrayData>& data) {
+ this->SetData(data);
+}
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<DataType>& type,
+ int64_t length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values,
+ int64_t offset) {
+ this->SetData(ArrayData::Make(type, length,
+ /*buffers=*/{NULLPTR},
+ /*child_data=*/{run_ends->data(), values->data()},
+ /*null_count=*/0, offset));
+}
+
+Result<std::shared_ptr<RunEndEncodedArray>> RunEndEncodedArray::Make(
+ int64_t logical_length, const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values, int64_t logical_offset) {
+ auto run_end_type = run_ends->type();
+ auto values_type = values->type();
+ if (!RunEndEncodedType::RunEndTypeValid(*run_end_type)) {
+ return Status::Invalid("Run end type must be int16, int32 or int64");
+ }
+ if (run_ends->null_count() != 0) {
+ return Status::Invalid("Run ends array cannot contain null values");
+ }
Review Comment:
Done. Fixing other issues before another push.
##########
cpp/src/arrow/array/array_run_end.cc:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/array_run_end.h"
+#include "arrow/array/util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedArray
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<ArrayData>& data) {
+ this->SetData(data);
+}
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<DataType>& type,
+ int64_t length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values,
+ int64_t offset) {
+ this->SetData(ArrayData::Make(type, length,
+ /*buffers=*/{NULLPTR},
+ /*child_data=*/{run_ends->data(), values->data()},
+ /*null_count=*/0, offset));
+}
+
+Result<std::shared_ptr<RunEndEncodedArray>> RunEndEncodedArray::Make(
+ int64_t logical_length, const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values, int64_t logical_offset) {
+ auto run_end_type = run_ends->type();
+ auto values_type = values->type();
+ if (!RunEndEncodedType::RunEndTypeValid(*run_end_type)) {
+ return Status::Invalid("Run end type must be int16, int32 or int64");
+ }
+ if (run_ends->null_count() != 0) {
+ return Status::Invalid("Run ends array cannot contain null values");
+ }
+
+ return std::make_shared<RunEndEncodedArray>(
+ run_end_encoded(std::move(run_end_type), std::move(values_type)), logical_length,
+ run_ends, values, logical_offset);
+}
+
+void RunEndEncodedArray::SetData(const std::shared_ptr<ArrayData>& data) {
+ ARROW_CHECK_EQ(data->type->id(), Type::RUN_END_ENCODED);
+ DCHECK_EQ(data->child_data.size(), 2);
+
+ // A non-zero number of logical values in this array (offset + length) implies
+ // a non-zero number of runs and values.
+ DCHECK(data->offset + data->length == 0 || data->child_data[0]->length > 0);
+ DCHECK(data->offset + data->length == 0 || data->child_data[1]->length > 0);
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109071723
##########
cpp/src/arrow/array/concatenate.cc:
##########
@@ -436,6 +438,28 @@ class ConcatenateImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
+ int64_t physical_length = 0;
+ for (const auto& input : in_) {
+ if (internal::AddWithOverflow(physical_length,
+ ree_util::FindPhysicalLength(ArraySpan(*input)),
+ &physical_length) ||
+ // Make sure we can safely downcast to int32_t
Review Comment:
This is from @zagto and I assumed it was due to https://arrow.apache.org/docs/dev/format/Columnar.html#array-lengths
But note that run-ends are cumulative sums of the run-lengths, so summing the lengths of `INT_MAX` runs wouldn't really fit in 32-bit integers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109077152
##########
cpp/src/arrow/array/concatenate.cc:
##########
@@ -436,6 +438,28 @@ class ConcatenateImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
+ int64_t physical_length = 0;
+ for (const auto& input : in_) {
+ if (internal::AddWithOverflow(physical_length,
+ ree_util::FindPhysicalLength(ArraySpan(*input)),
+ &physical_length) ||
+ // Make sure we can safely downcast to int32_t
Review Comment:
Hmm, I don't think we make this check for any other types though (if anything, I think we have tests that check very long arrays)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109078258
##########
cpp/src/arrow/scalar.cc:
##########
@@ -583,6 +608,17 @@ Result<std::shared_ptr<Scalar>> StructScalar::field(FieldRef ref) const {
}
}
+RunEndEncodedScalar::RunEndEncodedScalar(std::shared_ptr<Scalar> value,
+ std::shared_ptr<DataType> type)
+ : Scalar{std::move(type), value->is_valid}, value{std::move(value)} {}
Review Comment:
Hmm, I don't think it is - basically it's here or in validate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109824268
##########
cpp/src/arrow/array/builder_run_end.cc:
##########
@@ -0,0 +1,331 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/builder_run_end.h"
+#include "arrow/array/builder_primitive.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <utility>
+#include <vector>
+
+#include "arrow/scalar.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace internal {
+
+RunCompressorBuilder::RunCompressorBuilder(MemoryPool* pool,
+ std::shared_ptr<ArrayBuilder> inner_builder,
+ std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), inner_builder_(std::move(inner_builder)) {}
+
+RunCompressorBuilder::~RunCompressorBuilder() = default;
+
+void RunCompressorBuilder::Reset() {
+ current_run_length_ = 0;
+ current_value_.reset();
+ inner_builder_->Reset();
+ UpdateDimensions();
+}
+
+Status RunCompressorBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(inner_builder_->Resize(capacity));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendNulls(int64_t length) {
+ if (ARROW_PREDICT_FALSE(length == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new NULL run
+ DCHECK_EQ(current_value_, NULLPTR);
+ current_run_length_ = length;
+ } else if (current_value_ == NULLPTR) {
+ // Extend the currently open NULL run
+ current_run_length_ += length;
+ } else {
+ // Close then non-NULL run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(inner_builder_->AppendScalar(*current_value_));
+ UpdateDimensions();
+ // Open a new NULL run
+ current_value_.reset();
+ current_run_length_ = length;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to a run-compressed array.");
+}
+
+Status RunCompressorBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (ARROW_PREDICT_FALSE(n_repeats == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ } else if ((current_value_ == NULLPTR && !scalar.is_valid) ||
+ (current_value_ != NULLPTR && current_value_->Equals(scalar))) {
+ // Extend the currently open run
+ current_run_length_ += n_repeats;
+ } else {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendScalars(const ScalarVector& scalars) {
+ if (scalars.empty()) {
+ return Status::OK();
+ }
+ RETURN_NOT_OK(ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendRunCompressedArraySlice(
+ const ArraySpan& run_compressed_array, int64_t offset, int64_t length) {
+ DCHECK(!has_open_run());
+ RETURN_NOT_OK(inner_builder_->AppendArraySlice(run_compressed_array, offset, length));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishCurrentRun() {
+ if (current_run_length_ > 0) {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Clear the current run
+ current_value_.reset();
+ current_run_length_ = 0;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ ARROW_RETURN_NOT_OK(FinishCurrentRun());
+ return inner_builder_->FinishInternal(out);
+}
+
+} // namespace internal
+
+// ----------------------------------------------------------------------
+// RunEndEncodedBuilder
+
+RunEndEncodedBuilder::ValueRunBuilder::ValueRunBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& value_builder,
+ const std::shared_ptr<DataType>& value_type, RunEndEncodedBuilder& ree_builder)
+ : RunCompressorBuilder(pool, std::move(value_builder), std::move(value_type)),
+ ree_builder_(ree_builder) {}
+
+RunEndEncodedBuilder::RunEndEncodedBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder, std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), type_(internal::checked_pointer_cast<RunEndEncodedType>(type)) {
+ auto value_run_builder =
+ std::make_shared<ValueRunBuilder>(pool, value_builder, type_->value_type(), *this);
+ value_run_builder_ = value_run_builder.get();
+ children_ = {run_end_builder, std::move(value_run_builder)};
+ UpdateDimensions(0, 0);
+ null_count_ = 0;
+}
+
+Status RunEndEncodedBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(value_run_builder_->ResizePhysical(capacity));
+ RETURN_NOT_OK(run_end_builder().Resize(capacity));
+ UpdateDimensions(committed_logical_length_, 0);
+ return Status::OK();
+}
+
+void RunEndEncodedBuilder::Reset() {
+ value_run_builder_->Reset();
+ run_end_builder().Reset();
+ UpdateDimensions(0, 0);
+}
+
+Status RunEndEncodedBuilder::AppendNulls(int64_t length) {
+ RETURN_NOT_OK(value_run_builder_->AppendNulls(length));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to run-end encoded array.");
+}
+
+Status RunEndEncodedBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (scalar.type->id() == Type::RUN_END_ENCODED) {
+ return AppendScalar(*internal::checked_cast<const RunEndEncodedScalar&>(scalar).value,
+ n_repeats);
+ }
+ RETURN_NOT_OK(value_run_builder_->AppendScalar(scalar, n_repeats));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendScalars(const ScalarVector& scalars) {
+ RETURN_NOT_OK(this->ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendArray(const ArraySpan& to_append) {
+ DCHECK_GT(to_append.length, 0);
+ DCHECK(!value_run_builder_->has_open_run());
+
+ ree_util::RunEndEncodedArraySpan<RunEndsType> ree_span(to_append);
+ const int64_t physical_offset = ree_span.PhysicalIndex(0);
+ const int64_t physical_length =
+ ree_span.PhysicalIndex(ree_span.length() - 1) + 1 - physical_offset;
+
+ RETURN_NOT_OK(ReservePhysical(physical_length));
+
+ // Append all the run ends from to_append
+ const auto end = ree_span.end();
+ for (auto it = ree_span.iterator(0, physical_offset); it != end; ++it) {
+ const int64_t run_end = committed_logical_length_ + it.run_length();
+ RETURN_NOT_OK(DoAppendRunEnd<RunEndsType>(run_end));
+ UpdateDimensions(run_end, 0);
+ }
+
+ // Append all the values directly
+ RETURN_NOT_OK(value_run_builder_->AppendRunCompressedArraySlice(
+ ree_util::ValuesArray(to_append), physical_offset, physical_length));
+
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendArraySlice(const ArraySpan& array, int64_t offset,
+ int64_t length) {
+ ARROW_DCHECK(offset + length <= array.length);
+ ARROW_DCHECK(array.type->Equals(type_));
+
+ // Ensure any open run is closed before appending the array slice.
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+
+ if (length == 0) {
+ return Status::OK();
+ }
+
+ ArraySpan to_append = array;
+ to_append.SetSlice(array.offset + offset, length);
+
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendArray<int16_t>(to_append));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendArray<int32_t>(to_append));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendArray<int64_t>(to_append));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+
+ return Status::OK();
+}
+
+std::shared_ptr<DataType> RunEndEncodedBuilder::type() const { return type_; }
+
+Status RunEndEncodedBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ // Finish the values array before so we can close the current run and append
+ // the last run-end.
+ std::shared_ptr<ArrayData> values_data;
+ RETURN_NOT_OK(value_run_builder_->FinishInternal(&values_data));
+ auto values_array = MakeArray(values_data);
+
+ ARROW_ASSIGN_OR_RAISE(auto run_ends_array, run_end_builder().Finish());
+
+ ARROW_ASSIGN_OR_RAISE(auto ree_array,
+ RunEndEncodedArray::Make(length_, run_ends_array, values_array));
+ *out = std::move(ree_array->data());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::FinishCurrentRun() {
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+ UpdateDimensions(length_, 0);
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendRunEnd(int64_t run_end) {
+ constexpr auto max = std::numeric_limits<RunEndsType>::max();
+ if (ARROW_PREDICT_FALSE(run_end > max)) {
+ return Status::Invalid("Run end value must fit on run ends type but ", run_end, " > ",
+ max, ".");
+ }
+ return internal::checked_cast<typename CTypeTraits<RunEndsType>::BuilderType*>(
+ children_[0].get())
+ ->Append(static_cast<RunEndsType>(run_end));
+}
+
+Status RunEndEncodedBuilder::AppendRunEnd(int64_t run_end) {
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendRunEnd<int16_t>(run_end));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendRunEnd<int32_t>(run_end));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendRunEnd<int64_t>(run_end));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::CloseRun(const std::shared_ptr<const Scalar>& value,
+ int64_t run_length) {
+ // TODO(felipecrv): gracefully fragment runs bigger than INT32_MAX
+ if (ARROW_PREDICT_FALSE(run_length > std::numeric_limits<int32_t>::max())) {
Review Comment:
Done https://github.com/apache/arrow/issues/34237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1108960323
##########
cpp/src/arrow/type.h:
##########
@@ -1223,6 +1223,35 @@ class ARROW_EXPORT DenseUnionType : public UnionType {
std::string name() const override { return "dense_union"; }
};
+/// \brief Type class for run-end encoded data
+class ARROW_EXPORT RunEndEncodedType : public NestedType {
+ public:
+ static constexpr Type::type type_id = Type::RUN_END_ENCODED;
+
+ static constexpr const char* type_name() { return "run_end_encoded"; }
+
+ explicit RunEndEncodedType(std::shared_ptr<DataType> run_end_type,
+ std::shared_ptr<DataType> value_type);
+
+ DataTypeLayout layout() const override {
+ // NOTE(felipecrv): his can be removed once we ensure existing code does not
+ // assume the existence of at least one buffer.
Review Comment:
Should we file a task for it (and make this a `TODO(apache/arrow#NNN)`)?
##########
cpp/src/arrow/util/ree_util.h:
##########
@@ -0,0 +1,351 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cassert>
+#include <cstdint>
+
+#include "arrow/array/data.h"
+#include "arrow/type_traits.h"
+#include "arrow/util/macros.h"
+
+namespace arrow {
+namespace ree_util {
+
+/// \brief Get the child array holding the run ends from an REE array
+inline const ArraySpan& RunEndsArray(const ArraySpan& span) { return span.child_data[0]; }
+
+/// \brief Get the child array holding the data values from an REE array
+inline const ArraySpan& ValuesArray(const ArraySpan& span) { return span.child_data[1]; }
+
+/// \brief Get a pointer to run ends values of an REE array
+template <typename RunEndsType>
+const RunEndsType* RunEnds(const ArraySpan& span) {
+ assert(RunEndsArray(span).type->id() == CTypeTraits<RunEndsType>::ArrowType::type_id);
+ return RunEndsArray(span).GetValues<RunEndsType>(1);
+}
+
+namespace internal {
+
+/// \brief Uses binary-search to find the physical offset given a logical offset
+/// and run-end values
+///
+/// \return the physical offset or run_ends_size if the physical offset is not
+/// found in run_ends
+template <typename RunEndsType>
+int64_t FindPhysicalIndex(const RunEndsType* run_ends, int64_t run_ends_size, int64_t i,
+ int64_t absolute_offset) {
+ auto it = std::upper_bound(run_ends, run_ends + run_ends_size, absolute_offset + i);
+ int64_t result = std::distance(run_ends, it);
+ assert(result <= run_ends_size);
+ return result;
+}
+
+/// \brief Uses binary-search to calculate the number of physical values (and
+/// run-ends) necessary to represent the logical range of values from
+/// offset to length
+template <typename RunEndsType>
+int64_t FindPhysicalLength(int64_t length, int64_t offset, const RunEndsType* run_ends,
+ int64_t run_ends_size) {
Review Comment:
nit: it's a bit confusing that two similar functions have all their parameters in a different order
##########
cpp/src/arrow/scalar.cc:
##########
@@ -583,6 +608,17 @@ Result<std::shared_ptr<Scalar>> StructScalar::field(FieldRef ref) const {
}
}
+RunEndEncodedScalar::RunEndEncodedScalar(std::shared_ptr<Scalar> value,
+ std::shared_ptr<DataType> type)
+ : Scalar{std::move(type), value->is_valid}, value{std::move(value)} {}
Review Comment:
should we have a DCHECK here (e.g. that the value type is of the right type, that `type` is actually an REE type)?
##########
cpp/src/arrow/testing/json_internal.cc:
##########
@@ -437,6 +437,10 @@ class SchemaWriter {
Status Visit(const DictionaryType& type) { return VisitType(*type.value_type()); }
+ Status Visit(const RunEndEncodedType& type) {
+ return Status::NotImplemented(type.name());
Review Comment:
Would be good to put up a follow-up to implement this since this is used for other tests.
##########
cpp/src/arrow/array/array_base.cc:
##########
@@ -165,7 +175,7 @@ struct ScalarFromArraySlotImpl {
array_.length());
}
- if (array_.IsNull(index_)) {
+ if (array_.type()->id() != Type::RUN_END_ENCODED && array_.IsNull(index_)) {
Review Comment:
Hmm, I suppose this is unavoidable.
##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -375,8 +375,9 @@ TEST_F(TestArray, TestMakeArrayOfNull) {
ASSERT_EQ(array->type(), type);
ASSERT_OK(array->ValidateFull());
ASSERT_EQ(array->length(), length);
- if (is_union(type->id())) {
- // For unions, MakeArrayOfNull places the nulls in the children
+ if (is_union(type->id()) || type->id() == Type::RUN_END_ENCODED) {
+ // For unions and run-end encoded, MakeArrayOfNull places the nulls in the
+ // children
Review Comment:
Hmm.
Since this pattern repeats a bit: factor out a helper predicate to test it? It seems we now have a class of 'no top level null' types (union, REE)
##########
cpp/src/arrow/util/ree_util_test.cc:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array.h"
+#include "arrow/builder.h"
+#include "arrow/compute/api_vector.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/type_traits.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace ree_util {
+
+template <typename RunEndsType>
+struct ReeUtilTest : public ::testing::Test {
+ // Re-implementation of FindPhysicalIndex that uses trivial linear-search
+ // instead of the more efficient implementation.
+ int64_t FindPhysicalIndexTestImpl(const RunEndsType* run_ends, int64_t run_ends_size,
+ int64_t i, int64_t absolute_offset = 0) {
+ for (int64_t j = 0; j < run_ends_size; j++) {
+ if (absolute_offset + i < run_ends[j]) {
+ return j;
+ }
+ }
+ return run_ends_size;
+ }
+};
+TYPED_TEST_SUITE_P(ReeUtilTest);
+
+TYPED_TEST_P(ReeUtilTest, PhysicalOffset) {
+ using RE = TypeParam; // Run-end type
+ const RE run_ends1[] = {1};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends1, 1, 0, 0), 0);
+ const RE run_ends124[] = {1, 2, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 1, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 2, 0), 2);
+ const RE run_ends234[] = {2, 3, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 1, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 2, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 3, 0), 2);
+ const RE run_ends246[] = {2, 4, 6};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends246, 3, 3, 0), 1);
+
+ // Out-of-range logical offset should return run_ends size
Review Comment:
Should we assert that the offset is nonnegative?
##########
cpp/src/arrow/array/concatenate.cc:
##########
@@ -436,6 +438,28 @@ class ConcatenateImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
+ int64_t physical_length = 0;
+ for (const auto& input : in_) {
+ if (internal::AddWithOverflow(physical_length,
+ ree_util::FindPhysicalLength(ArraySpan(*input)),
+ &physical_length) ||
+ // Make sure we can safely downcast to int32_t
Review Comment:
Hmm, why do we have to downcast to int32_t here? Shouldn't it be based on the offsets type?
##########
cpp/src/arrow/array/builder_run_end.cc:
##########
@@ -0,0 +1,331 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/builder_run_end.h"
+#include "arrow/array/builder_primitive.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <utility>
+#include <vector>
+
+#include "arrow/scalar.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace internal {
+
+RunCompressorBuilder::RunCompressorBuilder(MemoryPool* pool,
+ std::shared_ptr<ArrayBuilder> inner_builder,
+ std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), inner_builder_(std::move(inner_builder)) {}
+
+RunCompressorBuilder::~RunCompressorBuilder() = default;
+
+void RunCompressorBuilder::Reset() {
+ current_run_length_ = 0;
+ current_value_.reset();
+ inner_builder_->Reset();
+ UpdateDimensions();
+}
+
+Status RunCompressorBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(inner_builder_->Resize(capacity));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendNulls(int64_t length) {
+ if (ARROW_PREDICT_FALSE(length == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new NULL run
+ DCHECK_EQ(current_value_, NULLPTR);
+ current_run_length_ = length;
+ } else if (current_value_ == NULLPTR) {
+ // Extend the currently open NULL run
+ current_run_length_ += length;
+ } else {
+ // Close then non-NULL run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(inner_builder_->AppendScalar(*current_value_));
+ UpdateDimensions();
+ // Open a new NULL run
+ current_value_.reset();
+ current_run_length_ = length;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to a run-compressed array.");
+}
+
+Status RunCompressorBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (ARROW_PREDICT_FALSE(n_repeats == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ } else if ((current_value_ == NULLPTR && !scalar.is_valid) ||
+ (current_value_ != NULLPTR && current_value_->Equals(scalar))) {
+ // Extend the currently open run
+ current_run_length_ += n_repeats;
+ } else {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendScalars(const ScalarVector& scalars) {
+ if (scalars.empty()) {
+ return Status::OK();
+ }
+ RETURN_NOT_OK(ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendRunCompressedArraySlice(
+ const ArraySpan& run_compressed_array, int64_t offset, int64_t length) {
+ DCHECK(!has_open_run());
+ RETURN_NOT_OK(inner_builder_->AppendArraySlice(run_compressed_array, offset, length));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishCurrentRun() {
+ if (current_run_length_ > 0) {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Clear the current run
+ current_value_.reset();
+ current_run_length_ = 0;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ ARROW_RETURN_NOT_OK(FinishCurrentRun());
+ return inner_builder_->FinishInternal(out);
+}
+
+} // namespace internal
+
+// ----------------------------------------------------------------------
+// RunEndEncodedBuilder
+
+RunEndEncodedBuilder::ValueRunBuilder::ValueRunBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& value_builder,
+ const std::shared_ptr<DataType>& value_type, RunEndEncodedBuilder& ree_builder)
+ : RunCompressorBuilder(pool, std::move(value_builder), std::move(value_type)),
+ ree_builder_(ree_builder) {}
+
+RunEndEncodedBuilder::RunEndEncodedBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder, std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), type_(internal::checked_pointer_cast<RunEndEncodedType>(type)) {
+ auto value_run_builder =
+ std::make_shared<ValueRunBuilder>(pool, value_builder, type_->value_type(), *this);
+ value_run_builder_ = value_run_builder.get();
+ children_ = {run_end_builder, std::move(value_run_builder)};
+ UpdateDimensions(0, 0);
+ null_count_ = 0;
+}
+
+Status RunEndEncodedBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(value_run_builder_->ResizePhysical(capacity));
+ RETURN_NOT_OK(run_end_builder().Resize(capacity));
+ UpdateDimensions(committed_logical_length_, 0);
+ return Status::OK();
+}
+
+void RunEndEncodedBuilder::Reset() {
+ value_run_builder_->Reset();
+ run_end_builder().Reset();
+ UpdateDimensions(0, 0);
+}
+
+Status RunEndEncodedBuilder::AppendNulls(int64_t length) {
+ RETURN_NOT_OK(value_run_builder_->AppendNulls(length));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to run-end encoded array.");
+}
+
+Status RunEndEncodedBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (scalar.type->id() == Type::RUN_END_ENCODED) {
+ return AppendScalar(*internal::checked_cast<const RunEndEncodedScalar&>(scalar).value,
+ n_repeats);
+ }
+ RETURN_NOT_OK(value_run_builder_->AppendScalar(scalar, n_repeats));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendScalars(const ScalarVector& scalars) {
+ RETURN_NOT_OK(this->ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendArray(const ArraySpan& to_append) {
+ DCHECK_GT(to_append.length, 0);
+ DCHECK(!value_run_builder_->has_open_run());
+
+ ree_util::RunEndEncodedArraySpan<RunEndsType> ree_span(to_append);
+ const int64_t physical_offset = ree_span.PhysicalIndex(0);
+ const int64_t physical_length =
+ ree_span.PhysicalIndex(ree_span.length() - 1) + 1 - physical_offset;
+
+ RETURN_NOT_OK(ReservePhysical(physical_length));
+
+ // Append all the run ends from to_append
+ const auto end = ree_span.end();
+ for (auto it = ree_span.iterator(0, physical_offset); it != end; ++it) {
+ const int64_t run_end = committed_logical_length_ + it.run_length();
+ RETURN_NOT_OK(DoAppendRunEnd<RunEndsType>(run_end));
+ UpdateDimensions(run_end, 0);
+ }
+
+ // Append all the values directly
+ RETURN_NOT_OK(value_run_builder_->AppendRunCompressedArraySlice(
+ ree_util::ValuesArray(to_append), physical_offset, physical_length));
+
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendArraySlice(const ArraySpan& array, int64_t offset,
+ int64_t length) {
+ ARROW_DCHECK(offset + length <= array.length);
+ ARROW_DCHECK(array.type->Equals(type_));
+
+ // Ensure any open run is closed before appending the array slice.
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+
+ if (length == 0) {
+ return Status::OK();
+ }
+
+ ArraySpan to_append = array;
+ to_append.SetSlice(array.offset + offset, length);
+
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendArray<int16_t>(to_append));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendArray<int32_t>(to_append));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendArray<int64_t>(to_append));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+
+ return Status::OK();
+}
+
+std::shared_ptr<DataType> RunEndEncodedBuilder::type() const { return type_; }
+
+Status RunEndEncodedBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ // Finish the values array before so we can close the current run and append
+ // the last run-end.
+ std::shared_ptr<ArrayData> values_data;
+ RETURN_NOT_OK(value_run_builder_->FinishInternal(&values_data));
+ auto values_array = MakeArray(values_data);
+
+ ARROW_ASSIGN_OR_RAISE(auto run_ends_array, run_end_builder().Finish());
+
+ ARROW_ASSIGN_OR_RAISE(auto ree_array,
+ RunEndEncodedArray::Make(length_, run_ends_array, values_array));
+ *out = std::move(ree_array->data());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::FinishCurrentRun() {
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+ UpdateDimensions(length_, 0);
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendRunEnd(int64_t run_end) {
+ constexpr auto max = std::numeric_limits<RunEndsType>::max();
+ if (ARROW_PREDICT_FALSE(run_end > max)) {
+ return Status::Invalid("Run end value must fit on run ends type but ", run_end, " > ",
+ max, ".");
+ }
+ return internal::checked_cast<typename CTypeTraits<RunEndsType>::BuilderType*>(
+ children_[0].get())
+ ->Append(static_cast<RunEndsType>(run_end));
+}
+
+Status RunEndEncodedBuilder::AppendRunEnd(int64_t run_end) {
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendRunEnd<int16_t>(run_end));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendRunEnd<int32_t>(run_end));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendRunEnd<int64_t>(run_end));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::CloseRun(const std::shared_ptr<const Scalar>& value,
+ int64_t run_length) {
+ // TODO(felipecrv): gracefully fragment runs bigger than INT32_MAX
+ if (ARROW_PREDICT_FALSE(run_length > std::numeric_limits<int32_t>::max())) {
+ return Status::Invalid(
+ "Run-length of run-encoded arrays must fit in a 32-bit signed integer.");
+ }
+ const int64_t run_end = committed_logical_length_ + run_length;
Review Comment:
I suppose, check for overflow?
##########
cpp/src/arrow/array/builder_run_end.cc:
##########
@@ -0,0 +1,331 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/builder_run_end.h"
+#include "arrow/array/builder_primitive.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <utility>
+#include <vector>
+
+#include "arrow/scalar.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace internal {
+
+RunCompressorBuilder::RunCompressorBuilder(MemoryPool* pool,
+ std::shared_ptr<ArrayBuilder> inner_builder,
+ std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), inner_builder_(std::move(inner_builder)) {}
+
+RunCompressorBuilder::~RunCompressorBuilder() = default;
+
+void RunCompressorBuilder::Reset() {
+ current_run_length_ = 0;
+ current_value_.reset();
+ inner_builder_->Reset();
+ UpdateDimensions();
+}
+
+Status RunCompressorBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(inner_builder_->Resize(capacity));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendNulls(int64_t length) {
+ if (ARROW_PREDICT_FALSE(length == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new NULL run
+ DCHECK_EQ(current_value_, NULLPTR);
+ current_run_length_ = length;
+ } else if (current_value_ == NULLPTR) {
+ // Extend the currently open NULL run
+ current_run_length_ += length;
+ } else {
+ // Close then non-NULL run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(inner_builder_->AppendScalar(*current_value_));
+ UpdateDimensions();
+ // Open a new NULL run
+ current_value_.reset();
+ current_run_length_ = length;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to a run-compressed array.");
+}
+
+Status RunCompressorBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (ARROW_PREDICT_FALSE(n_repeats == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ } else if ((current_value_ == NULLPTR && !scalar.is_valid) ||
+ (current_value_ != NULLPTR && current_value_->Equals(scalar))) {
+ // Extend the currently open run
+ current_run_length_ += n_repeats;
+ } else {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendScalars(const ScalarVector& scalars) {
+ if (scalars.empty()) {
+ return Status::OK();
+ }
+ RETURN_NOT_OK(ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendRunCompressedArraySlice(
+ const ArraySpan& run_compressed_array, int64_t offset, int64_t length) {
+ DCHECK(!has_open_run());
+ RETURN_NOT_OK(inner_builder_->AppendArraySlice(run_compressed_array, offset, length));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishCurrentRun() {
+ if (current_run_length_ > 0) {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Clear the current run
+ current_value_.reset();
+ current_run_length_ = 0;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ ARROW_RETURN_NOT_OK(FinishCurrentRun());
+ return inner_builder_->FinishInternal(out);
+}
+
+} // namespace internal
+
+// ----------------------------------------------------------------------
+// RunEndEncodedBuilder
+
+RunEndEncodedBuilder::ValueRunBuilder::ValueRunBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& value_builder,
+ const std::shared_ptr<DataType>& value_type, RunEndEncodedBuilder& ree_builder)
+ : RunCompressorBuilder(pool, std::move(value_builder), std::move(value_type)),
+ ree_builder_(ree_builder) {}
+
+RunEndEncodedBuilder::RunEndEncodedBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder, std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), type_(internal::checked_pointer_cast<RunEndEncodedType>(type)) {
+ auto value_run_builder =
+ std::make_shared<ValueRunBuilder>(pool, value_builder, type_->value_type(), *this);
+ value_run_builder_ = value_run_builder.get();
+ children_ = {run_end_builder, std::move(value_run_builder)};
+ UpdateDimensions(0, 0);
+ null_count_ = 0;
+}
+
+Status RunEndEncodedBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(value_run_builder_->ResizePhysical(capacity));
+ RETURN_NOT_OK(run_end_builder().Resize(capacity));
+ UpdateDimensions(committed_logical_length_, 0);
+ return Status::OK();
+}
+
+void RunEndEncodedBuilder::Reset() {
+ value_run_builder_->Reset();
+ run_end_builder().Reset();
+ UpdateDimensions(0, 0);
+}
+
+Status RunEndEncodedBuilder::AppendNulls(int64_t length) {
+ RETURN_NOT_OK(value_run_builder_->AppendNulls(length));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to run-end encoded array.");
+}
+
+Status RunEndEncodedBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (scalar.type->id() == Type::RUN_END_ENCODED) {
+ return AppendScalar(*internal::checked_cast<const RunEndEncodedScalar&>(scalar).value,
+ n_repeats);
+ }
+ RETURN_NOT_OK(value_run_builder_->AppendScalar(scalar, n_repeats));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendScalars(const ScalarVector& scalars) {
+ RETURN_NOT_OK(this->ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendArray(const ArraySpan& to_append) {
+ DCHECK_GT(to_append.length, 0);
+ DCHECK(!value_run_builder_->has_open_run());
+
+ ree_util::RunEndEncodedArraySpan<RunEndsType> ree_span(to_append);
+ const int64_t physical_offset = ree_span.PhysicalIndex(0);
+ const int64_t physical_length =
+ ree_span.PhysicalIndex(ree_span.length() - 1) + 1 - physical_offset;
+
+ RETURN_NOT_OK(ReservePhysical(physical_length));
+
+ // Append all the run ends from to_append
+ const auto end = ree_span.end();
+ for (auto it = ree_span.iterator(0, physical_offset); it != end; ++it) {
+ const int64_t run_end = committed_logical_length_ + it.run_length();
+ RETURN_NOT_OK(DoAppendRunEnd<RunEndsType>(run_end));
+ UpdateDimensions(run_end, 0);
+ }
+
+ // Append all the values directly
+ RETURN_NOT_OK(value_run_builder_->AppendRunCompressedArraySlice(
+ ree_util::ValuesArray(to_append), physical_offset, physical_length));
+
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendArraySlice(const ArraySpan& array, int64_t offset,
+ int64_t length) {
+ ARROW_DCHECK(offset + length <= array.length);
+ ARROW_DCHECK(array.type->Equals(type_));
+
+ // Ensure any open run is closed before appending the array slice.
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+
+ if (length == 0) {
+ return Status::OK();
+ }
+
+ ArraySpan to_append = array;
+ to_append.SetSlice(array.offset + offset, length);
+
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendArray<int16_t>(to_append));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendArray<int32_t>(to_append));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendArray<int64_t>(to_append));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+
+ return Status::OK();
+}
+
+std::shared_ptr<DataType> RunEndEncodedBuilder::type() const { return type_; }
+
+Status RunEndEncodedBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ // Finish the values array before so we can close the current run and append
+ // the last run-end.
+ std::shared_ptr<ArrayData> values_data;
+ RETURN_NOT_OK(value_run_builder_->FinishInternal(&values_data));
+ auto values_array = MakeArray(values_data);
+
+ ARROW_ASSIGN_OR_RAISE(auto run_ends_array, run_end_builder().Finish());
+
+ ARROW_ASSIGN_OR_RAISE(auto ree_array,
+ RunEndEncodedArray::Make(length_, run_ends_array, values_array));
+ *out = std::move(ree_array->data());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::FinishCurrentRun() {
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+ UpdateDimensions(length_, 0);
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendRunEnd(int64_t run_end) {
+ constexpr auto max = std::numeric_limits<RunEndsType>::max();
+ if (ARROW_PREDICT_FALSE(run_end > max)) {
+ return Status::Invalid("Run end value must fit on run ends type but ", run_end, " > ",
+ max, ".");
+ }
+ return internal::checked_cast<typename CTypeTraits<RunEndsType>::BuilderType*>(
+ children_[0].get())
+ ->Append(static_cast<RunEndsType>(run_end));
+}
+
+Status RunEndEncodedBuilder::AppendRunEnd(int64_t run_end) {
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendRunEnd<int16_t>(run_end));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendRunEnd<int32_t>(run_end));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendRunEnd<int64_t>(run_end));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::CloseRun(const std::shared_ptr<const Scalar>& value,
+ int64_t run_length) {
+ // TODO(felipecrv): gracefully fragment runs bigger than INT32_MAX
+ if (ARROW_PREDICT_FALSE(run_length > std::numeric_limits<int32_t>::max())) {
Review Comment:
Shouldn't this depend on the offset type?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1090866068
##########
cpp/src/arrow/compare.cc:
##########
@@ -700,6 +734,15 @@ class TypeEqualsVisitor {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& left) {
+ const auto& right = checked_cast<const RunEndEncodedType&>(right_);
+ // NOTE(felipecrv): the type of the run_ends might differ, but that doesn't
+ // change the logical meaning of the type so only the encoded type is
+ // compared.
Review Comment:
I will make it more strict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1093210043
##########
cpp/src/arrow/array/array_encoded.h:
##########
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Array accessor classes run-end encoded arrays
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/array/array_base.h"
+#include "arrow/array/data.h"
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/type.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow {
+
+/// \addtogroup encoded-arrays
+///
+/// @{
+
+// ----------------------------------------------------------------------
+// RunEndEncoded
+
+/// \brief Array type for run-end encoded data
+class ARROW_EXPORT RunEndEncodedArray : public Array {
+ public:
+ using TypeClass = RunEndEncodedType;
+
+ explicit RunEndEncodedArray(const std::shared_ptr<ArrayData>& data);
+
+ RunEndEncodedArray(const std::shared_ptr<DataType>& type, int64_t logical_length,
Review Comment:
I'm renaming it back to `length` and adding more explanations after I've internalized all the patterns in `array/data.h`.
In `Make` I'm calling both `logical_length` and `logical_offset` as the offset is also not to be added directly to the offsets of the children.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1092149774
##########
cpp/src/arrow/compute/kernel.h:
##########
@@ -141,6 +141,14 @@ ARROW_EXPORT std::shared_ptr<TypeMatcher> FixedSizeBinaryLike();
// Type)
ARROW_EXPORT std::shared_ptr<TypeMatcher> Primitive();
+/// \brief Match run-end encoded types that encode specific plain data types
+///
+/// @param[in] run_ends_type_matcher a matcher that is applied to the run_ends field
+/// @param[in] encoded_type_matcher a matcher that is applied to the values field
+ARROW_EXPORT std::shared_ptr<TypeMatcher> RunEndEncoded(
+ std::shared_ptr<TypeMatcher> run_ends_type_matcher,
Review Comment:
Not really. Instances of this are passing `Primitive()`. I fixed `Matches` to match the integer types more strictly now. And as you said, kernel implementations will template on them on the three possible types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33641:
URL: https://github.com/apache/arrow/pull/33641#issuecomment-1412828518
:warning: GitHub issue #32104 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] zeroshade commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109011486
##########
cpp/src/arrow/testing/json_internal.cc:
##########
@@ -437,6 +437,10 @@ class SchemaWriter {
Status Visit(const DictionaryType& type) { return VisitType(*type.value_type()); }
+ Status Visit(const RunEndEncodedType& type) {
+ return Status::NotImplemented(type.name());
Review Comment:
There's an issue/task to implement this, it's just not being implemented in *this* PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on PR #33641:
URL: https://github.com/apache/arrow/pull/33641#issuecomment-1432958662
@zeroshade this is now passing all the builds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109159427
##########
cpp/src/arrow/util/ree_util_test.cc:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array.h"
+#include "arrow/builder.h"
+#include "arrow/compute/api_vector.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/type_traits.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace ree_util {
+
+template <typename RunEndsType>
+struct ReeUtilTest : public ::testing::Test {
+ // Re-implementation of FindPhysicalIndex that uses trivial linear-search
+ // instead of the more efficient implementation.
+ int64_t FindPhysicalIndexTestImpl(const RunEndsType* run_ends, int64_t run_ends_size,
+ int64_t i, int64_t absolute_offset = 0) {
+ for (int64_t j = 0; j < run_ends_size; j++) {
+ if (absolute_offset + i < run_ends[j]) {
+ return j;
+ }
+ }
+ return run_ends_size;
+ }
+};
+TYPED_TEST_SUITE_P(ReeUtilTest);
+
+TYPED_TEST_P(ReeUtilTest, PhysicalOffset) {
+ using RE = TypeParam; // Run-end type
+ const RE run_ends1[] = {1};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends1, 1, 0, 0), 0);
+ const RE run_ends124[] = {1, 2, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 1, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 2, 0), 2);
+ const RE run_ends234[] = {2, 3, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 1, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 2, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 3, 0), 2);
+ const RE run_ends246[] = {2, 4, 6};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends246, 3, 3, 0), 1);
+
+ // Out-of-range logical offset should return run_ends size
Review Comment:
That of course doesn't work. :/ I will add a debug check to `FindPhysicalIndex` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109154637
##########
cpp/src/arrow/array/concatenate.cc:
##########
@@ -436,6 +438,28 @@ class ConcatenateImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
+ int64_t physical_length = 0;
+ for (const auto& input : in_) {
+ if (internal::AddWithOverflow(physical_length,
+ ree_util::FindPhysicalLength(ArraySpan(*input)),
+ &physical_length) ||
+ // Make sure we can safely downcast to int32_t
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on PR #33641:
URL: https://github.com/apache/arrow/pull/33641#issuecomment-1434769254
@lidavidm should I rebase and force push so this CI issues goes away or this can be merged as is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1084399237
##########
cpp/src/arrow/type.h:
##########
@@ -1196,6 +1196,49 @@ class ARROW_EXPORT DenseUnionType : public UnionType {
std::string name() const override { return "dense_union"; }
};
+/// \brief Base class for types using a special-encoding (e.g. run-end-encoding)
Review Comment:
Removing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1084591187
##########
cpp/src/arrow/pretty_print.cc:
##########
@@ -41,6 +41,7 @@
#include "arrow/util/formatting.h"
#include "arrow/util/int_util_overflow.h"
#include "arrow/util/key_value_metadata.h"
+#include "arrow/util/ree_util.h"
Review Comment:
Removing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1073830197
##########
cpp/src/arrow/array/array_encoded.h:
##########
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Array accessor classes run-end encoded arrays
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/array/array_base.h"
+#include "arrow/array/data.h"
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/type.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow {
+
+/// \addtogroup encoded-arrays
+///
+/// @{
+
+// ----------------------------------------------------------------------
+// RunEndEncoded
+
+/// \brief Array type for run-end encoded data
+class ARROW_EXPORT RunEndEncodedArray : public Array {
+ public:
+ using TypeClass = RunEndEncodedType;
+
+ explicit RunEndEncodedArray(const std::shared_ptr<ArrayData>& data);
+
+ RunEndEncodedArray(const std::shared_ptr<DataType>& type, int64_t logical_length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values, int64_t offset = 0);
+
+ /// \brief Construct a RunEndEncodedArray from values and run ends arrays
+ ///
+ /// The data type is automatically inferred from the arguments.
+ /// The run_ends and values arrays must have the same length.
+ static Result<std::shared_ptr<RunEndEncodedArray>> Make(
+ const std::shared_ptr<Array>& run_ends, const std::shared_ptr<Array>& values,
+ int64_t logical_length, int64_t offset = 0);
+
+ /// \brief Returns an array holding the logical indexes of each run-end
+ ///
+ /// The physical offset to the array is applied.
+ std::shared_ptr<Array> run_ends_array() const;
+
+ /// \brief Returns an array holding the values of each run
+ ///
+ /// The physical offset to the array is applied.
+ std::shared_ptr<Array> values_array() const;
+
+ /// \brief Find the physical offset of the REE array
+ ///
+ /// This function uses binary-search, so it has a O(log N) cost.
+ int64_t FindPhysicalOffset() const;
+
+ /// \brief Find the physical length of the REE array
+ ///
+ /// Avoid calling this function if the physical length can be estabilished in
+ /// some other way. This function uses binary-search, so it has a O(log N)
+ /// cost.
Review Comment:
"physical length" is the number of integers in the `run_ends` array. I don't understand why this had to be implemented with a search.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109146695
##########
cpp/src/arrow/util/ree_util_test.cc:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array.h"
+#include "arrow/builder.h"
+#include "arrow/compute/api_vector.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/type_traits.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace ree_util {
+
+template <typename RunEndsType>
+struct ReeUtilTest : public ::testing::Test {
+ // Re-implementation of FindPhysicalIndex that uses trivial linear-search
+ // instead of the more efficient implementation.
+ int64_t FindPhysicalIndexTestImpl(const RunEndsType* run_ends, int64_t run_ends_size,
+ int64_t i, int64_t absolute_offset = 0) {
+ for (int64_t j = 0; j < run_ends_size; j++) {
+ if (absolute_offset + i < run_ends[j]) {
+ return j;
+ }
+ }
+ return run_ends_size;
+ }
+};
+TYPED_TEST_SUITE_P(ReeUtilTest);
+
+TYPED_TEST_P(ReeUtilTest, PhysicalOffset) {
+ using RE = TypeParam; // Run-end type
+ const RE run_ends1[] = {1};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends1, 1, 0, 0), 0);
+ const RE run_ends124[] = {1, 2, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 1, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 2, 0), 2);
+ const RE run_ends234[] = {2, 3, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 1, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 2, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 3, 0), 2);
+ const RE run_ends246[] = {2, 4, 6};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends246, 3, 3, 0), 1);
+
+ // Out-of-range logical offset should return run_ends size
Review Comment:
I added test cases with negative `i`. I think that's what you meant :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109143609
##########
cpp/src/arrow/util/ree_util_test.cc:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include "arrow/array.h"
+#include "arrow/builder.h"
+#include "arrow/compute/api_vector.h"
+#include "arrow/testing/gtest_util.h"
+#include "arrow/type_traits.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace ree_util {
+
+template <typename RunEndsType>
+struct ReeUtilTest : public ::testing::Test {
+ // Re-implementation of FindPhysicalIndex that uses trivial linear-search
+ // instead of the more efficient implementation.
+ int64_t FindPhysicalIndexTestImpl(const RunEndsType* run_ends, int64_t run_ends_size,
+ int64_t i, int64_t absolute_offset = 0) {
+ for (int64_t j = 0; j < run_ends_size; j++) {
+ if (absolute_offset + i < run_ends[j]) {
+ return j;
+ }
+ }
+ return run_ends_size;
+ }
+};
+TYPED_TEST_SUITE_P(ReeUtilTest);
+
+TYPED_TEST_P(ReeUtilTest, PhysicalOffset) {
+ using RE = TypeParam; // Run-end type
+ const RE run_ends1[] = {1};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends1, 1, 0, 0), 0);
+ const RE run_ends124[] = {1, 2, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 1, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends124, 3, 2, 0), 2);
+ const RE run_ends234[] = {2, 3, 4};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 0, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 1, 0), 0);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 2, 0), 1);
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends234, 3, 3, 0), 2);
+ const RE run_ends246[] = {2, 4, 6};
+ ASSERT_EQ(internal::FindPhysicalIndex(run_ends246, 3, 3, 0), 1);
+
+ // Out-of-range logical offset should return run_ends size
Review Comment:
I will add it here and push a commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109163880
##########
cpp/src/arrow/array/builder_run_end.cc:
##########
@@ -0,0 +1,331 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/builder_run_end.h"
+#include "arrow/array/builder_primitive.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <utility>
+#include <vector>
+
+#include "arrow/scalar.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace internal {
+
+RunCompressorBuilder::RunCompressorBuilder(MemoryPool* pool,
+ std::shared_ptr<ArrayBuilder> inner_builder,
+ std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), inner_builder_(std::move(inner_builder)) {}
+
+RunCompressorBuilder::~RunCompressorBuilder() = default;
+
+void RunCompressorBuilder::Reset() {
+ current_run_length_ = 0;
+ current_value_.reset();
+ inner_builder_->Reset();
+ UpdateDimensions();
+}
+
+Status RunCompressorBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(inner_builder_->Resize(capacity));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendNulls(int64_t length) {
+ if (ARROW_PREDICT_FALSE(length == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new NULL run
+ DCHECK_EQ(current_value_, NULLPTR);
+ current_run_length_ = length;
+ } else if (current_value_ == NULLPTR) {
+ // Extend the currently open NULL run
+ current_run_length_ += length;
+ } else {
+ // Close then non-NULL run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(inner_builder_->AppendScalar(*current_value_));
+ UpdateDimensions();
+ // Open a new NULL run
+ current_value_.reset();
+ current_run_length_ = length;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to a run-compressed array.");
+}
+
+Status RunCompressorBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (ARROW_PREDICT_FALSE(n_repeats == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ } else if ((current_value_ == NULLPTR && !scalar.is_valid) ||
+ (current_value_ != NULLPTR && current_value_->Equals(scalar))) {
+ // Extend the currently open run
+ current_run_length_ += n_repeats;
+ } else {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendScalars(const ScalarVector& scalars) {
+ if (scalars.empty()) {
+ return Status::OK();
+ }
+ RETURN_NOT_OK(ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendRunCompressedArraySlice(
+ const ArraySpan& run_compressed_array, int64_t offset, int64_t length) {
+ DCHECK(!has_open_run());
+ RETURN_NOT_OK(inner_builder_->AppendArraySlice(run_compressed_array, offset, length));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishCurrentRun() {
+ if (current_run_length_ > 0) {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Clear the current run
+ current_value_.reset();
+ current_run_length_ = 0;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ ARROW_RETURN_NOT_OK(FinishCurrentRun());
+ return inner_builder_->FinishInternal(out);
+}
+
+} // namespace internal
+
+// ----------------------------------------------------------------------
+// RunEndEncodedBuilder
+
+RunEndEncodedBuilder::ValueRunBuilder::ValueRunBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& value_builder,
+ const std::shared_ptr<DataType>& value_type, RunEndEncodedBuilder& ree_builder)
+ : RunCompressorBuilder(pool, std::move(value_builder), std::move(value_type)),
+ ree_builder_(ree_builder) {}
+
+RunEndEncodedBuilder::RunEndEncodedBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder, std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), type_(internal::checked_pointer_cast<RunEndEncodedType>(type)) {
+ auto value_run_builder =
+ std::make_shared<ValueRunBuilder>(pool, value_builder, type_->value_type(), *this);
+ value_run_builder_ = value_run_builder.get();
+ children_ = {run_end_builder, std::move(value_run_builder)};
+ UpdateDimensions(0, 0);
+ null_count_ = 0;
+}
+
+Status RunEndEncodedBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(value_run_builder_->ResizePhysical(capacity));
+ RETURN_NOT_OK(run_end_builder().Resize(capacity));
+ UpdateDimensions(committed_logical_length_, 0);
+ return Status::OK();
+}
+
+void RunEndEncodedBuilder::Reset() {
+ value_run_builder_->Reset();
+ run_end_builder().Reset();
+ UpdateDimensions(0, 0);
+}
+
+Status RunEndEncodedBuilder::AppendNulls(int64_t length) {
+ RETURN_NOT_OK(value_run_builder_->AppendNulls(length));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to run-end encoded array.");
+}
+
+Status RunEndEncodedBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (scalar.type->id() == Type::RUN_END_ENCODED) {
+ return AppendScalar(*internal::checked_cast<const RunEndEncodedScalar&>(scalar).value,
+ n_repeats);
+ }
+ RETURN_NOT_OK(value_run_builder_->AppendScalar(scalar, n_repeats));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendScalars(const ScalarVector& scalars) {
+ RETURN_NOT_OK(this->ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendArray(const ArraySpan& to_append) {
+ DCHECK_GT(to_append.length, 0);
+ DCHECK(!value_run_builder_->has_open_run());
+
+ ree_util::RunEndEncodedArraySpan<RunEndsType> ree_span(to_append);
+ const int64_t physical_offset = ree_span.PhysicalIndex(0);
+ const int64_t physical_length =
+ ree_span.PhysicalIndex(ree_span.length() - 1) + 1 - physical_offset;
+
+ RETURN_NOT_OK(ReservePhysical(physical_length));
+
+ // Append all the run ends from to_append
+ const auto end = ree_span.end();
+ for (auto it = ree_span.iterator(0, physical_offset); it != end; ++it) {
+ const int64_t run_end = committed_logical_length_ + it.run_length();
+ RETURN_NOT_OK(DoAppendRunEnd<RunEndsType>(run_end));
+ UpdateDimensions(run_end, 0);
+ }
+
+ // Append all the values directly
+ RETURN_NOT_OK(value_run_builder_->AppendRunCompressedArraySlice(
+ ree_util::ValuesArray(to_append), physical_offset, physical_length));
+
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendArraySlice(const ArraySpan& array, int64_t offset,
+ int64_t length) {
+ ARROW_DCHECK(offset + length <= array.length);
+ ARROW_DCHECK(array.type->Equals(type_));
+
+ // Ensure any open run is closed before appending the array slice.
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+
+ if (length == 0) {
+ return Status::OK();
+ }
+
+ ArraySpan to_append = array;
+ to_append.SetSlice(array.offset + offset, length);
+
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendArray<int16_t>(to_append));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendArray<int32_t>(to_append));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendArray<int64_t>(to_append));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+
+ return Status::OK();
+}
+
+std::shared_ptr<DataType> RunEndEncodedBuilder::type() const { return type_; }
+
+Status RunEndEncodedBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ // Finish the values array before so we can close the current run and append
+ // the last run-end.
+ std::shared_ptr<ArrayData> values_data;
+ RETURN_NOT_OK(value_run_builder_->FinishInternal(&values_data));
+ auto values_array = MakeArray(values_data);
+
+ ARROW_ASSIGN_OR_RAISE(auto run_ends_array, run_end_builder().Finish());
+
+ ARROW_ASSIGN_OR_RAISE(auto ree_array,
+ RunEndEncodedArray::Make(length_, run_ends_array, values_array));
+ *out = std::move(ree_array->data());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::FinishCurrentRun() {
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+ UpdateDimensions(length_, 0);
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendRunEnd(int64_t run_end) {
+ constexpr auto max = std::numeric_limits<RunEndsType>::max();
+ if (ARROW_PREDICT_FALSE(run_end > max)) {
+ return Status::Invalid("Run end value must fit on run ends type but ", run_end, " > ",
+ max, ".");
+ }
+ return internal::checked_cast<typename CTypeTraits<RunEndsType>::BuilderType*>(
+ children_[0].get())
+ ->Append(static_cast<RunEndsType>(run_end));
+}
+
+Status RunEndEncodedBuilder::AppendRunEnd(int64_t run_end) {
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendRunEnd<int16_t>(run_end));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendRunEnd<int32_t>(run_end));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendRunEnd<int64_t>(run_end));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::CloseRun(const std::shared_ptr<const Scalar>& value,
+ int64_t run_length) {
+ // TODO(felipecrv): gracefully fragment runs bigger than INT32_MAX
+ if (ARROW_PREDICT_FALSE(run_length > std::numeric_limits<int32_t>::max())) {
Review Comment:
I will create an issue to open the discussion up to more people.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109017018
##########
cpp/src/arrow/testing/json_internal.cc:
##########
@@ -437,6 +437,10 @@ class SchemaWriter {
Status Visit(const DictionaryType& type) { return VisitType(*type.value_type()); }
+ Status Visit(const RunEndEncodedType& type) {
+ return Status::NotImplemented(type.name());
Review Comment:
Look, I'm not saying this is a blocker, if there's a ticket I just want to know there is one
##########
cpp/src/arrow/testing/json_internal.cc:
##########
@@ -437,6 +437,10 @@ class SchemaWriter {
Status Visit(const DictionaryType& type) { return VisitType(*type.value_type()); }
+ Status Visit(const RunEndEncodedType& type) {
+ return Status::NotImplemented(type.name());
Review Comment:
Look, I'm not saying this is a blocker, if there's a ticket already I just want to know there is one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109025907
##########
cpp/src/arrow/testing/json_internal.cc:
##########
@@ -437,6 +437,10 @@ class SchemaWriter {
Status Visit(const DictionaryType& type) { return VisitType(*type.value_type()); }
+ Status Visit(const RunEndEncodedType& type) {
+ return Status::NotImplemented(type.name());
Review Comment:
There is an umbrella issue to fix all the leftovers from this PR https://github.com/apache/arrow/issues/34176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109019543
##########
cpp/src/arrow/array/array_base.cc:
##########
@@ -165,7 +175,7 @@ struct ScalarFromArraySlotImpl {
array_.length());
}
- if (array_.IsNull(index_)) {
+ if (array_.type()->id() != Type::RUN_END_ENCODED && array_.IsNull(index_)) {
Review Comment:
Right, it's just an observation...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #33641:
URL: https://github.com/apache/arrow/pull/33641#issuecomment-1435092781
Benchmark runs are scheduled for baseline = 157b8f555bdd5bf2fafc875b8c983c8c51a3c936 and contender = 1264e40918ba95f7c0b1725e296ada25b1385b3d. 1264e40918ba95f7c0b1725e296ada25b1385b3d is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/ad02c07ac33a4d96bc0e3e3b39dae669...51e2bc82e6854fbaa2410b3b388f516f/)
[Finished :arrow_down:0.58% :arrow_up:0.03%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/f473c95ec80c44fca03a2478d754cd15...3a87b15df78e4314be33046eeb484393/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/c3bfdc668cd64b719944247d086bb1db...7a86f65e48b84b899e207635e93897ad/)
[Finished :arrow_down:0.41% :arrow_up:0.03%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/a8f5faecfbb34056b580f09de1eb9b9b...6ec40e5b818842b58da675cd9617a078/)
Buildkite builds:
[Finished] [`1264e409` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2392)
[Finished] [`1264e409` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2422)
[Finished] [`1264e409` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2390)
[Finished] [`1264e409` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2414)
[Finished] [`157b8f55` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2391)
[Finished] [`157b8f55` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2421)
[Finished] [`157b8f55` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2389)
[Finished] [`157b8f55` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2413)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1084546854
##########
cpp/src/arrow/util/ree_util.h:
##########
@@ -0,0 +1,280 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <algorithm>
+#include <cstdint>
+
+#include "arrow/array/data.h"
+#include "arrow/type_traits.h"
+#include "arrow/util/logging.h"
Review Comment:
Oh, I noticed that you use `assert` in headers instead of `DCHECK` when the function can't be moved to the `.cc`. I will do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1084587760
##########
cpp/src/arrow/pretty_print.cc:
##########
@@ -377,6 +378,22 @@ class ArrayPrinter : public PrettyPrinter {
return PrettyPrint(*array.indices(), ChildOptions(true), sink_);
}
+ Status Visit(const RunEndEncodedArray& array) {
+ Newline();
+ Indent();
+ Write("-- run ends array (offset: ");
+ Write(std::to_string(array.offset()));
+ Write(", logical length: ");
+ Write(std::to_string(array.length()));
+ Write(")\n");
Review Comment:
Simplifying this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1092437941
##########
cpp/src/arrow/array/concatenate.cc:
##########
@@ -436,6 +438,23 @@ class ConcatenateImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
Review Comment:
Very likely. I didn't add extra tests from what Tobias' PRs already had. Adding tests is part of my review of this code, so soon to come.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1091316932
##########
cpp/src/arrow/array/builder_encoded.cc:
##########
@@ -0,0 +1,213 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/builder_encoded.h"
+#include "arrow/array/builder_primitive.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <utility>
+#include <vector>
+
+#include "arrow/scalar.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedBuilder
+
+RunEndEncodedBuilder::RunEndEncodedBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder, std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), type_(internal::checked_pointer_cast<RunEndEncodedType>(type)) {
+ children_ = {std::move(run_end_builder), std::move(value_builder)};
+}
+
+Status RunEndEncodedBuilder::Resize(int64_t) {
+ return Status::NotImplemented(
+ "Resizing an REE for a given number of logical elements is not possible, since the "
+ "physical length will vary depending on the contents. To allocate memory for a "
+ "certain number of runs, use ResizePhysical.");
+}
+
+void RunEndEncodedBuilder::Reset() {
+ capacity_ = length_ = null_count_ = 0;
+ value_builder().Reset();
+ run_end_builder().Reset();
+ current_value_.reset();
+ run_start_ = 0;
+}
+
+Status RunEndEncodedBuilder::AppendNull() { return AppendNulls(1); }
+
+Status RunEndEncodedBuilder::AppendNulls(int64_t length) {
+ if (length == 0) {
+ return Status::OK();
+ }
+ if (current_value_) {
+ RETURN_NOT_OK(FinishRun());
+ current_value_ = {};
+ }
+ return AddLength(length);
+}
+
+Status RunEndEncodedBuilder::AppendEmptyValue() { return AppendNull(); }
+
+Status RunEndEncodedBuilder::AppendEmptyValues(int64_t length) {
+ return AppendNulls(length);
+}
+
+Status RunEndEncodedBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (n_repeats == 0) {
+ return Status::OK();
+ }
+ if (!scalar.is_valid) {
+ return AppendNulls(n_repeats);
+ }
+ if (!current_value_ || !current_value_->Equals(scalar)) {
+ RETURN_NOT_OK(FinishRun());
+ current_value_ = scalar.shared_from_this();
+ }
+ return AddLength(n_repeats);
+}
+
+Status RunEndEncodedBuilder::AppendScalars(const ScalarVector& scalars) {
+ for (auto scalar : scalars) {
+ RETURN_NOT_OK(AppendScalar(*scalar, 1));
+ }
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendArray(const ArraySpan& to_append) {
+ const int64_t physical_offset = ree_util::FindPhysicalOffset(to_append);
+ int64_t physical_length = 0;
+ for (auto it = ree_util::MergedRunsIterator<RunEndsType>(to_append);
+ it != ree_util::MergedRunsIterator(); ++it) {
+ physical_length++;
+ length_ += it.run_length();
+ RETURN_NOT_OK(DoAppendRunEnd<RunEndsType>());
+ }
+ return value_builder().AppendArraySlice(ree_util::ValuesArray(to_append),
+ physical_offset, physical_length);
+}
+
+Status RunEndEncodedBuilder::AppendArraySlice(const ArraySpan& array, int64_t offset,
+ int64_t length) {
+ // Finish eventual runs started using AppendScalars() and others before. We don't
+ // attempt to merge them with the runs from the appended array slice for now
+ RETURN_NOT_OK(FinishRun());
+
+ ARROW_DCHECK(offset + length <= array.length);
+ ARROW_DCHECK(array.type->Equals(type_));
+ // Create a slice of the slice for the part we actually want to add
+ ArraySpan to_append = array;
+ to_append.SetSlice(array.offset + offset, length);
+
+ switch (type_->run_ends_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendArray<int16_t>(to_append));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendArray<int32_t>(to_append));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendArray<int64_t>(to_append));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_ends_type());
+ }
+
+ // next run is not merged either
+ run_start_ = length_;
+ return Status::OK();
+}
+
+std::shared_ptr<DataType> RunEndEncodedBuilder::type() const { return type_; }
+
+Status RunEndEncodedBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ ARROW_ASSIGN_OR_RAISE(auto run_ends_array, run_end_builder().Finish());
+ ARROW_ASSIGN_OR_RAISE(auto values_array, value_builder().Finish());
+ ARROW_ASSIGN_OR_RAISE(auto ree_array,
+ RunEndEncodedArray::Make(run_ends_array, values_array, length_));
+ *out = std::move(ree_array->data());
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendRunEnd() {
+ return internal::checked_cast<typename CTypeTraits<RunEndsType>::BuilderType*>(
+ children_[0].get())
+ ->Append(static_cast<RunEndsType>(length_));
+}
+
+Status RunEndEncodedBuilder::FinishRun() {
+ if (length_ - run_start_ == 0) {
+ return Status::OK();
+ }
+ if (current_value_) {
+ RETURN_NOT_OK(value_builder().AppendScalar(*current_value_));
+ } else {
+ RETURN_NOT_OK(value_builder().AppendNull());
+ }
+ switch (type_->run_ends_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendRunEnd<int16_t>());
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendRunEnd<int32_t>());
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendRunEnd<int64_t>());
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_ends_type());
+ }
+ current_value_.reset();
+ run_start_ = 0;
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::ResizePhyiscal(int64_t capacity) {
+ RETURN_NOT_OK(value_builder().Resize(capacity));
+ return run_end_builder().Resize(capacity);
Review Comment:
How else would it be? For every `i < physical_length` `run_ends[i]` is the end of the run of `values[i]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1092501101
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
Review Comment:
I don't know if it's possible to produce REEs with children containing different offsets, so disregard the "might differ" comment above.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1090866068
##########
cpp/src/arrow/compare.cc:
##########
@@ -700,6 +734,15 @@ class TypeEqualsVisitor {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& left) {
+ const auto& right = checked_cast<const RunEndEncodedType&>(right_);
+ // NOTE(felipecrv): the type of the run_ends might differ, but that doesn't
+ // change the logical meaning of the type so only the encoded type is
+ // compared.
Review Comment:
Thanks. I will make it more strict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1091312173
##########
cpp/src/arrow/type.h:
##########
@@ -1196,6 +1196,49 @@ class ARROW_EXPORT DenseUnionType : public UnionType {
std::string name() const override { return "dense_union"; }
};
+/// \brief Base class for types using a special-encoding (e.g. run-end-encoding)
+///
+/// This allows users to check if arrays are logically compatible even when the
+/// physical encoding of the types differs.
+class ARROW_EXPORT EncodedType : public NestedType {
+ public:
+ EncodedType(Type::type id, std::shared_ptr<DataType> encoded_type)
+ : NestedType(id), encoded_type_{std::move(encoded_type)} {}
+
+ const std::shared_ptr<DataType>& encoded_type() const { return encoded_type_; }
Review Comment:
Renamed to `value_type` everywhere. And `values_array()` is now just `values()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1091313806
##########
cpp/src/arrow/scalar.cc:
##########
@@ -1149,7 +1149,7 @@ Result<std::shared_ptr<Scalar>> Scalar::CastTo(std::shared_ptr<DataType> to) con
if (is_valid) {
out->is_valid = true;
ToTypeVisitor unpack_to_type{*this, to, out.get()};
- RETURN_NOT_OK(VisitTypeInline(*to, &unpack_to_type));
+ RETURN_NOT_OK(VisitScalarTypeInline(*to, &unpack_to_type));
Review Comment:
That's what I've been working on for a few days now. I'm also adding JSON converters in the process and rewriting the array builder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1093933950
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -413,6 +414,20 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
+ switch (type.run_ends_type()->id()) {
+ case Type::INT16:
+ return ValidateRunEndEncoded<int16_t>(type);
+ case Type::INT32:
+ return ValidateRunEndEncoded<int32_t>(type);
+ case Type::INT64:
+ return ValidateRunEndEncoded<int64_t>(type);
+ default:
+ return Status::Invalid("Run ends type must be int16, int32 or int64, but got: ",
+ type.run_ends_type());
Review Comment:
Added a `->ToString()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1073852207
##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +195,7 @@ int GetNumBuffers(const DataType& type) {
case Type::NA:
case Type::STRUCT:
case Type::FIXED_SIZE_LIST:
+ case Type::RUN_END_ENCODED:
return 1;
Review Comment:
Well, note that this function does not reflect the number of buffers in the IPC layout, but the number of buffers _in Arrow C++_.
We have special code to work around it in the IPC layer:
https://github.com/apache/arrow/blob/359f28ba9d62a5e8456d92dfbe5b16b790019edd/cpp/src/arrow/ipc/reader.cc#L330-L335
https://github.com/apache/arrow/blob/359f28ba9d62a5e8456d92dfbe5b16b790019edd/cpp/src/arrow/ipc/writer.cc#L157-L169
So we could choose to also have a dummy null bitmap entry (in Arrow C++) for run-end encoded arrays, or not.
@lidavidm @westonpace What are your opinions on this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1073843729
##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +195,7 @@ int GetNumBuffers(const DataType& type) {
case Type::NA:
case Type::STRUCT:
case Type::FIXED_SIZE_LIST:
+ case Type::RUN_END_ENCODED:
return 1;
Review Comment:
You're right. I believe it should be 0.
https://arrow.apache.org/docs/dev/format/Columnar.html#buffer-listing-for-each-layout
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1073904290
##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +195,7 @@ int GetNumBuffers(const DataType& type) {
case Type::NA:
case Type::STRUCT:
case Type::FIXED_SIZE_LIST:
+ case Type::RUN_END_ENCODED:
return 1;
Review Comment:
The dummy in null arrays might be for historical reasons, and was kept for compatibility in Arrow C++.
See https://issues.apache.org/jira/browse/ARROW-6379
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #33641:
URL: https://github.com/apache/arrow/pull/33641#issuecomment-1380958347
* Closes: #32104
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1084398775
##########
cpp/src/arrow/type.h:
##########
@@ -2103,6 +2146,7 @@ static inline bool HasValidityBitmap(Type::type id) {
case Type::NA:
case Type::DENSE_UNION:
case Type::SPARSE_UNION:
+ case Type::RUN_END_ENCODED:
Review Comment:
Adding a commit for this. Might cherry-pick into separate PR later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1084510905
##########
cpp/src/arrow/array/array_encoded.cc:
##########
@@ -0,0 +1,79 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/array_encoded.h"
+#include "arrow/array/util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedArray
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<ArrayData>& data) {
+ ARROW_CHECK_EQ(data->type->id(), Type::RUN_END_ENCODED);
+ SetData(data);
+}
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<DataType>& type,
+ int64_t logical_length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values,
+ int64_t offset) {
+ ARROW_CHECK_EQ(type->id(), Type::RUN_END_ENCODED);
+ SetData(ArrayData::Make(type, logical_length,
+ /*buffers=*/{NULLPTR},
+ /*child_data=*/{run_ends->data(), values->data()}, 0, offset));
+}
+
+Result<std::shared_ptr<RunEndEncodedArray>> RunEndEncodedArray::Make(
+ const std::shared_ptr<Array>& run_ends, const std::shared_ptr<Array>& values,
+ int64_t logical_length, int64_t offset) {
+ auto run_ends_type = run_ends->type();
+ auto values_type = values->type();
+ if (!RunEndEncodedType::RunEndsTypeValid(*run_ends_type)) {
+ return Status::Invalid("Run ends array must be int16, int32 or int64 type");
+ }
+ if (run_ends->null_count() != 0) {
+ return Status::Invalid("Run ends array cannot contain null values");
+ }
+
+ return std::make_shared<RunEndEncodedArray>(
+ run_end_encoded(std::move(run_ends_type), std::move(values_type)), logical_length,
+ run_ends, values, offset);
+}
+
+std::shared_ptr<Array> RunEndEncodedArray::run_ends_array() const {
+ return MakeArray(data()->child_data[0]);
+}
+
+std::shared_ptr<Array> RunEndEncodedArray::values_array() const {
+ return MakeArray(data()->child_data[1]);
+}
Review Comment:
Pushing soon.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109017564
##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -375,8 +375,9 @@ TEST_F(TestArray, TestMakeArrayOfNull) {
ASSERT_EQ(array->type(), type);
ASSERT_OK(array->ValidateFull());
ASSERT_EQ(array->length(), length);
- if (is_union(type->id())) {
- // For unions, MakeArrayOfNull places the nulls in the children
+ if (is_union(type->id()) || type->id() == Type::RUN_END_ENCODED) {
+ // For unions and run-end encoded, MakeArrayOfNull places the nulls in the
+ // children
Review Comment:
I thought about this, but couldn't come up with a good name for this abstract notion. `is_no_top_level_null` ???
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109132558
##########
cpp/src/arrow/array/concatenate.cc:
##########
@@ -436,6 +438,28 @@ class ConcatenateImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
+ int64_t physical_length = 0;
+ for (const auto& input : in_) {
+ if (internal::AddWithOverflow(physical_length,
+ ree_util::FindPhysicalLength(ArraySpan(*input)),
+ &physical_length) ||
+ // Make sure we can safely downcast to int32_t
Review Comment:
The implementation of `AppendArraySlice` should check for all of that.
Can I just remove the physical length sum check, then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109178468
##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -375,8 +375,9 @@ TEST_F(TestArray, TestMakeArrayOfNull) {
ASSERT_EQ(array->type(), type);
ASSERT_OK(array->ValidateFull());
ASSERT_EQ(array->length(), length);
- if (is_union(type->id())) {
- // For unions, MakeArrayOfNull places the nulls in the children
+ if (is_union(type->id()) || type->id() == Type::RUN_END_ENCODED) {
+ // For unions and run-end encoded, MakeArrayOfNull places the nulls in the
+ // children
Review Comment:
Thanks for reminding me. Added a commit with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109018547
##########
cpp/src/arrow/array/array_base.cc:
##########
@@ -165,7 +175,7 @@ struct ScalarFromArraySlotImpl {
array_.length());
}
- if (array_.IsNull(index_)) {
+ if (array_.type()->id() != Type::RUN_END_ENCODED && array_.IsNull(index_)) {
Review Comment:
what do you mean?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1104828832
##########
cpp/src/arrow/array/array_run_end.cc:
##########
@@ -0,0 +1,88 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/array_run_end.h"
+#include "arrow/array/util.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedArray
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<ArrayData>& data) {
+ this->SetData(data);
+}
+
+RunEndEncodedArray::RunEndEncodedArray(const std::shared_ptr<DataType>& type,
+ int64_t length,
+ const std::shared_ptr<Array>& run_ends,
+ const std::shared_ptr<Array>& values,
+ int64_t offset) {
+ this->SetData(ArrayData::Make(type, length,
+ /*buffers=*/{NULLPTR},
+ /*child_data=*/{run_ends->data(), values->data()},
+ /*null_count=*/0, offset));
Review Comment:
Added checks. Fixing other issues before pushing again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1073818108
##########
cpp/src/arrow/array.h:
##########
@@ -34,10 +34,15 @@
/// @{
/// @}
+/// \defgroup encoded-arrays Concrete classes for encoded arrays
Review Comment:
```suggestion
/// \defgroup run-end-encoded-arrays Concrete classes for run-end encoded arrays
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1084395416
##########
cpp/src/arrow/type_test.cc:
##########
@@ -1927,6 +1927,38 @@ TEST(TypesTest, TestDecimalEquals) {
AssertTypeNotEqual(t5, t10);
}
+TEST(TypesTest, TestRunEndEncodedType) {
+ auto int8_make_shared = std::make_shared<RunEndEncodedType>(int32(), list(int8()));
+ auto int8_factory = run_end_encoded(int32(), list(int8()));
+ auto int32_factory = run_end_encoded(int32(), list(int32()));
Review Comment:
renaming: `type_{factory->ree}`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1084592097
##########
cpp/src/arrow/ipc/writer.cc:
##########
@@ -526,6 +526,10 @@ class RecordBatchSerializer {
return VisitType(*array.indices());
}
+ Status Visit(const RunEndEncodedArray& type) {
+ return Status::NotImplemented("run-end encoded array in IPC");
+ }
Review Comment:
Yes. It needs to be done so compilation can succeed and progress can be made. That makes turning this into small PRs a bit tricky as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109145007
##########
cpp/src/arrow/array/array_test.cc:
##########
@@ -375,8 +375,9 @@ TEST_F(TestArray, TestMakeArrayOfNull) {
ASSERT_EQ(array->type(), type);
ASSERT_OK(array->ValidateFull());
ASSERT_EQ(array->length(), length);
- if (is_union(type->id())) {
- // For unions, MakeArrayOfNull places the nulls in the children
+ if (is_union(type->id()) || type->id() == Type::RUN_END_ENCODED) {
+ // For unions and run-end encoded, MakeArrayOfNull places the nulls in the
+ // children
Review Comment:
I thought we had HasValidityBitmap or similar?
##########
cpp/src/arrow/testing/json_internal.cc:
##########
@@ -437,6 +437,10 @@ class SchemaWriter {
Status Visit(const DictionaryType& type) { return VisitType(*type.value_type()); }
+ Status Visit(const RunEndEncodedType& type) {
+ return Status::NotImplemented(type.name());
Review Comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] lidavidm commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "lidavidm (via GitHub)" <gi...@apache.org>.
lidavidm commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109144872
##########
cpp/src/arrow/array/concatenate.cc:
##########
@@ -436,6 +438,28 @@ class ConcatenateImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
+ int64_t physical_length = 0;
+ for (const auto& input : in_) {
+ if (internal::AddWithOverflow(physical_length,
+ ree_util::FindPhysicalLength(ArraySpan(*input)),
+ &physical_length) ||
+ // Make sure we can safely downcast to int32_t
Review Comment:
Ok, if there's already a check, maybe note that this one is redundant and remove it (is there a test covering that though?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109025593
##########
cpp/src/arrow/type.h:
##########
@@ -1223,6 +1223,35 @@ class ARROW_EXPORT DenseUnionType : public UnionType {
std::string name() const override { return "dense_union"; }
};
+/// \brief Type class for run-end encoded data
+class ARROW_EXPORT RunEndEncodedType : public NestedType {
+ public:
+ static constexpr Type::type type_id = Type::RUN_END_ENCODED;
+
+ static constexpr const char* type_name() { return "run_end_encoded"; }
+
+ explicit RunEndEncodedType(std::shared_ptr<DataType> run_end_type,
+ std::shared_ptr<DataType> value_type);
+
+ DataTypeLayout layout() const override {
+ // NOTE(felipecrv): his can be removed once we ensure existing code does not
+ // assume the existence of at least one buffer.
Review Comment:
This seems like a huge change that would break compatibility in many places. That's why I opted for a `NOTE`. I could reword it as a comment without hinting that this will actually change someday.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109158832
##########
cpp/src/arrow/type.h:
##########
@@ -1223,6 +1223,35 @@ class ARROW_EXPORT DenseUnionType : public UnionType {
std::string name() const override { return "dense_union"; }
};
+/// \brief Type class for run-end encoded data
+class ARROW_EXPORT RunEndEncodedType : public NestedType {
+ public:
+ static constexpr Type::type type_id = Type::RUN_END_ENCODED;
+
+ static constexpr const char* type_name() { return "run_end_encoded"; }
+
+ explicit RunEndEncodedType(std::shared_ptr<DataType> run_end_type,
+ std::shared_ptr<DataType> value_type);
+
+ DataTypeLayout layout() const override {
+ // NOTE(felipecrv): his can be removed once we ensure existing code does not
+ // assume the existence of at least one buffer.
Review Comment:
Changed it to a comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109178038
##########
cpp/src/arrow/array/builder_run_end.cc:
##########
@@ -0,0 +1,331 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/builder_run_end.h"
+#include "arrow/array/builder_primitive.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <utility>
+#include <vector>
+
+#include "arrow/scalar.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+namespace internal {
+
+RunCompressorBuilder::RunCompressorBuilder(MemoryPool* pool,
+ std::shared_ptr<ArrayBuilder> inner_builder,
+ std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), inner_builder_(std::move(inner_builder)) {}
+
+RunCompressorBuilder::~RunCompressorBuilder() = default;
+
+void RunCompressorBuilder::Reset() {
+ current_run_length_ = 0;
+ current_value_.reset();
+ inner_builder_->Reset();
+ UpdateDimensions();
+}
+
+Status RunCompressorBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(inner_builder_->Resize(capacity));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendNulls(int64_t length) {
+ if (ARROW_PREDICT_FALSE(length == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new NULL run
+ DCHECK_EQ(current_value_, NULLPTR);
+ current_run_length_ = length;
+ } else if (current_value_ == NULLPTR) {
+ // Extend the currently open NULL run
+ current_run_length_ += length;
+ } else {
+ // Close then non-NULL run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(inner_builder_->AppendScalar(*current_value_));
+ UpdateDimensions();
+ // Open a new NULL run
+ current_value_.reset();
+ current_run_length_ = length;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to a run-compressed array.");
+}
+
+Status RunCompressorBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (ARROW_PREDICT_FALSE(n_repeats == 0)) {
+ return Status::OK();
+ }
+ if (ARROW_PREDICT_FALSE(current_run_length_ == 0)) {
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ } else if ((current_value_ == NULLPTR && !scalar.is_valid) ||
+ (current_value_ != NULLPTR && current_value_->Equals(scalar))) {
+ // Extend the currently open run
+ current_run_length_ += n_repeats;
+ } else {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Open a new run
+ current_value_ = scalar.is_valid ? scalar.shared_from_this() : NULLPTR;
+ current_run_length_ = n_repeats;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendScalars(const ScalarVector& scalars) {
+ if (scalars.empty()) {
+ return Status::OK();
+ }
+ RETURN_NOT_OK(ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::AppendRunCompressedArraySlice(
+ const ArraySpan& run_compressed_array, int64_t offset, int64_t length) {
+ DCHECK(!has_open_run());
+ RETURN_NOT_OK(inner_builder_->AppendArraySlice(run_compressed_array, offset, length));
+ UpdateDimensions();
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishCurrentRun() {
+ if (current_run_length_ > 0) {
+ // Close the current run
+ ARROW_RETURN_NOT_OK(WillCloseRun(current_value_, current_run_length_));
+ ARROW_RETURN_NOT_OK(current_value_ ? inner_builder_->AppendScalar(*current_value_)
+ : inner_builder_->AppendNull());
+ UpdateDimensions();
+ // Clear the current run
+ current_value_.reset();
+ current_run_length_ = 0;
+ }
+ return Status::OK();
+}
+
+Status RunCompressorBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ ARROW_RETURN_NOT_OK(FinishCurrentRun());
+ return inner_builder_->FinishInternal(out);
+}
+
+} // namespace internal
+
+// ----------------------------------------------------------------------
+// RunEndEncodedBuilder
+
+RunEndEncodedBuilder::ValueRunBuilder::ValueRunBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& value_builder,
+ const std::shared_ptr<DataType>& value_type, RunEndEncodedBuilder& ree_builder)
+ : RunCompressorBuilder(pool, std::move(value_builder), std::move(value_type)),
+ ree_builder_(ree_builder) {}
+
+RunEndEncodedBuilder::RunEndEncodedBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder, std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), type_(internal::checked_pointer_cast<RunEndEncodedType>(type)) {
+ auto value_run_builder =
+ std::make_shared<ValueRunBuilder>(pool, value_builder, type_->value_type(), *this);
+ value_run_builder_ = value_run_builder.get();
+ children_ = {run_end_builder, std::move(value_run_builder)};
+ UpdateDimensions(0, 0);
+ null_count_ = 0;
+}
+
+Status RunEndEncodedBuilder::ResizePhysical(int64_t capacity) {
+ RETURN_NOT_OK(value_run_builder_->ResizePhysical(capacity));
+ RETURN_NOT_OK(run_end_builder().Resize(capacity));
+ UpdateDimensions(committed_logical_length_, 0);
+ return Status::OK();
+}
+
+void RunEndEncodedBuilder::Reset() {
+ value_run_builder_->Reset();
+ run_end_builder().Reset();
+ UpdateDimensions(0, 0);
+}
+
+Status RunEndEncodedBuilder::AppendNulls(int64_t length) {
+ RETURN_NOT_OK(value_run_builder_->AppendNulls(length));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendEmptyValues(int64_t length) {
+ return Status::NotImplemented("Append empty values to run-end encoded array.");
+}
+
+Status RunEndEncodedBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (scalar.type->id() == Type::RUN_END_ENCODED) {
+ return AppendScalar(*internal::checked_cast<const RunEndEncodedScalar&>(scalar).value,
+ n_repeats);
+ }
+ RETURN_NOT_OK(value_run_builder_->AppendScalar(scalar, n_repeats));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendScalars(const ScalarVector& scalars) {
+ RETURN_NOT_OK(this->ArrayBuilder::AppendScalars(scalars));
+ UpdateDimensions(committed_logical_length_, value_run_builder_->open_run_length());
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendArray(const ArraySpan& to_append) {
+ DCHECK_GT(to_append.length, 0);
+ DCHECK(!value_run_builder_->has_open_run());
+
+ ree_util::RunEndEncodedArraySpan<RunEndsType> ree_span(to_append);
+ const int64_t physical_offset = ree_span.PhysicalIndex(0);
+ const int64_t physical_length =
+ ree_span.PhysicalIndex(ree_span.length() - 1) + 1 - physical_offset;
+
+ RETURN_NOT_OK(ReservePhysical(physical_length));
+
+ // Append all the run ends from to_append
+ const auto end = ree_span.end();
+ for (auto it = ree_span.iterator(0, physical_offset); it != end; ++it) {
+ const int64_t run_end = committed_logical_length_ + it.run_length();
+ RETURN_NOT_OK(DoAppendRunEnd<RunEndsType>(run_end));
+ UpdateDimensions(run_end, 0);
+ }
+
+ // Append all the values directly
+ RETURN_NOT_OK(value_run_builder_->AppendRunCompressedArraySlice(
+ ree_util::ValuesArray(to_append), physical_offset, physical_length));
+
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::AppendArraySlice(const ArraySpan& array, int64_t offset,
+ int64_t length) {
+ ARROW_DCHECK(offset + length <= array.length);
+ ARROW_DCHECK(array.type->Equals(type_));
+
+ // Ensure any open run is closed before appending the array slice.
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+
+ if (length == 0) {
+ return Status::OK();
+ }
+
+ ArraySpan to_append = array;
+ to_append.SetSlice(array.offset + offset, length);
+
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendArray<int16_t>(to_append));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendArray<int32_t>(to_append));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendArray<int64_t>(to_append));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+
+ return Status::OK();
+}
+
+std::shared_ptr<DataType> RunEndEncodedBuilder::type() const { return type_; }
+
+Status RunEndEncodedBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ // Finish the values array before so we can close the current run and append
+ // the last run-end.
+ std::shared_ptr<ArrayData> values_data;
+ RETURN_NOT_OK(value_run_builder_->FinishInternal(&values_data));
+ auto values_array = MakeArray(values_data);
+
+ ARROW_ASSIGN_OR_RAISE(auto run_ends_array, run_end_builder().Finish());
+
+ ARROW_ASSIGN_OR_RAISE(auto ree_array,
+ RunEndEncodedArray::Make(length_, run_ends_array, values_array));
+ *out = std::move(ree_array->data());
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::FinishCurrentRun() {
+ RETURN_NOT_OK(value_run_builder_->FinishCurrentRun());
+ UpdateDimensions(length_, 0);
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendRunEnd(int64_t run_end) {
+ constexpr auto max = std::numeric_limits<RunEndsType>::max();
+ if (ARROW_PREDICT_FALSE(run_end > max)) {
+ return Status::Invalid("Run end value must fit on run ends type but ", run_end, " > ",
+ max, ".");
+ }
+ return internal::checked_cast<typename CTypeTraits<RunEndsType>::BuilderType*>(
+ children_[0].get())
+ ->Append(static_cast<RunEndsType>(run_end));
+}
+
+Status RunEndEncodedBuilder::AppendRunEnd(int64_t run_end) {
+ switch (type_->run_end_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendRunEnd<int16_t>(run_end));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendRunEnd<int32_t>(run_end));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendRunEnd<int64_t>(run_end));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_end_type());
+ }
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::CloseRun(const std::shared_ptr<const Scalar>& value,
+ int64_t run_length) {
+ // TODO(felipecrv): gracefully fragment runs bigger than INT32_MAX
+ if (ARROW_PREDICT_FALSE(run_length > std::numeric_limits<int32_t>::max())) {
Review Comment:
Thanks for reminding me. Added a commit with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1109016451
##########
cpp/src/arrow/testing/json_internal.cc:
##########
@@ -437,6 +437,10 @@ class SchemaWriter {
Status Visit(const DictionaryType& type) { return VisitType(*type.value_type()); }
+ Status Visit(const RunEndEncodedType& type) {
+ return Status::NotImplemented(type.name());
Review Comment:
What do you mean by follow-up? I am going to refer back to this PR and fix all these `NotImplemented` in follow-up PRs but I would like to merge this PR that is getting big.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1073822032
##########
cpp/src/arrow/array/array_encoded.h:
##########
@@ -0,0 +1,91 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+// Array accessor classes run-end encoded arrays
+
+#pragma once
+
+#include <cstdint>
+#include <memory>
+#include <string>
+#include <utility>
+#include <vector>
+
+#include "arrow/array/array_base.h"
+#include "arrow/array/data.h"
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/type.h"
+#include "arrow/type_fwd.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/macros.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow {
+
+/// \addtogroup encoded-arrays
+///
+/// @{
+
+// ----------------------------------------------------------------------
+// RunEndEncoded
+
+/// \brief Array type for run-end encoded data
+class ARROW_EXPORT RunEndEncodedArray : public Array {
+ public:
+ using TypeClass = RunEndEncodedType;
+
+ explicit RunEndEncodedArray(const std::shared_ptr<ArrayData>& data);
+
+ RunEndEncodedArray(const std::shared_ptr<DataType>& type, int64_t logical_length,
Review Comment:
"logical length" is the accumulated run-length, but I don't think the current code is consistent about it. There is a lot of confusion about logical length and physical length in many places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by GitBox <gi...@apache.org>.
pitrou commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1073318297
##########
cpp/src/arrow/CMakeLists.txt:
##########
@@ -140,13 +140,16 @@ set(ARROW_SRCS
array/array_binary.cc
array/array_decimal.cc
array/array_dict.cc
+ array/array_encoded.h
+ array/array_encoded.cc
Review Comment:
1) `.h` files needn't appear here.
2) can we name this something else? "encoded" is much too vague. For example `array_run_end.cc`
##########
cpp/src/arrow/type_test.cc:
##########
@@ -1927,6 +1927,38 @@ TEST(TypesTest, TestDecimalEquals) {
AssertTypeNotEqual(t5, t10);
}
+TEST(TypesTest, TestRunEndEncodedType) {
+ auto int8_make_shared = std::make_shared<RunEndEncodedType>(int32(), list(int8()));
+ auto int8_factory = run_end_encoded(int32(), list(int8()));
+ auto int32_factory = run_end_encoded(int32(), list(int32()));
Review Comment:
Why "factory"?
##########
cpp/src/arrow/type_traits.h:
##########
@@ -1177,6 +1197,7 @@ constexpr bool is_nested(Type::type type_id) {
case Type::STRUCT:
case Type::SPARSE_UNION:
case Type::DENSE_UNION:
+ case Type::RUN_END_ENCODED:
Review Comment:
I'm not sure this is right? These types are semantically (logically) nested. Run-end encoded arrays are only physically nested. I don't know which choice makes the most sense and/or requires the least special-casing.
@lidavidm @bkietz What do you think?
##########
cpp/src/arrow/type.h:
##########
@@ -1196,6 +1196,49 @@ class ARROW_EXPORT DenseUnionType : public UnionType {
std::string name() const override { return "dense_union"; }
};
+/// \brief Base class for types using a special-encoding (e.g. run-end-encoding)
+///
+/// This allows users to check if arrays are logically compatible even when the
+/// physical encoding of the types differs.
+class ARROW_EXPORT EncodedType : public NestedType {
+ public:
+ EncodedType(Type::type id, std::shared_ptr<DataType> encoded_type)
+ : NestedType(id), encoded_type_{std::move(encoded_type)} {}
+
+ const std::shared_ptr<DataType>& encoded_type() const { return encoded_type_; }
Review Comment:
Note that `DictionaryType` uses `value_type` not `encoded_type`. It would also match the child name which is "values".
##########
cpp/src/arrow/pretty_print.cc:
##########
@@ -377,6 +378,22 @@ class ArrayPrinter : public PrettyPrinter {
return PrettyPrint(*array.indices(), ChildOptions(true), sink_);
}
+ Status Visit(const RunEndEncodedArray& array) {
+ Newline();
+ Indent();
+ Write("-- run ends array (offset: ");
+ Write(std::to_string(array.offset()));
+ Write(", logical length: ");
+ Write(std::to_string(array.length()));
+ Write(")\n");
Review Comment:
Why write out the offset and length? We don't do this for dictionary indices, for example.
##########
cpp/src/arrow/type.h:
##########
@@ -1196,6 +1196,49 @@ class ARROW_EXPORT DenseUnionType : public UnionType {
std::string name() const override { return "dense_union"; }
};
+/// \brief Base class for types using a special-encoding (e.g. run-end-encoding)
Review Comment:
Uh, I'm not sure that's useful, but it sounds confusing (especially as this doesn't seem to cover dictionary types, which are also "encoded").
##########
cpp/src/arrow/compute/kernel.h:
##########
@@ -141,6 +141,14 @@ ARROW_EXPORT std::shared_ptr<TypeMatcher> FixedSizeBinaryLike();
// Type)
ARROW_EXPORT std::shared_ptr<TypeMatcher> Primitive();
+/// \brief Match run-end encoded types that encode specific plain data types
+///
+/// @param[in] run_ends_type_matcher a matcher that is applied to the run_ends field
+/// @param[in] encoded_type_matcher a matcher that is applied to the values field
+ARROW_EXPORT std::shared_ptr<TypeMatcher> RunEndEncoded(
+ std::shared_ptr<TypeMatcher> run_ends_type_matcher,
Review Comment:
Is it generally useful to pass a run ends matcher? The implementation will probably be the same, just templated by run ends type.
##########
cpp/src/arrow/scalar.cc:
##########
@@ -1149,7 +1149,7 @@ Result<std::shared_ptr<Scalar>> Scalar::CastTo(std::shared_ptr<DataType> to) con
if (is_valid) {
out->is_valid = true;
ToTypeVisitor unpack_to_type{*this, to, out.get()};
- RETURN_NOT_OK(VisitTypeInline(*to, &unpack_to_type));
+ RETURN_NOT_OK(VisitScalarTypeInline(*to, &unpack_to_type));
Review Comment:
Hmm, we should definitely define a `RunEndEncodedScalar` instead of hacking our way around its lack, IMHO.
##########
cpp/src/arrow/pretty_print.cc:
##########
@@ -41,6 +41,7 @@
#include "arrow/util/formatting.h"
#include "arrow/util/int_util_overflow.h"
#include "arrow/util/key_value_metadata.h"
+#include "arrow/util/ree_util.h"
Review Comment:
Doesn't seem required?
##########
cpp/src/arrow/array.h:
##########
@@ -34,10 +34,15 @@
/// @{
/// @}
+/// \defgroup encoded-arrays Concrete classes for encoded arrays
Review Comment:
"run-end-encoded arrays"
##########
cpp/src/arrow/array/concatenate.cc:
##########
@@ -436,6 +438,23 @@ class ConcatenateImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
Review Comment:
These changes seem to lack tests, don't they?
##########
cpp/src/arrow/type.h:
##########
@@ -2103,6 +2146,7 @@ static inline bool HasValidityBitmap(Type::type id) {
case Type::NA:
case Type::DENSE_UNION:
case Type::SPARSE_UNION:
+ case Type::RUN_END_ENCODED:
Review Comment:
Unrelated, but could make this function `constexpr` I think.
##########
cpp/src/arrow/compare.cc:
##########
@@ -700,6 +734,15 @@ class TypeEqualsVisitor {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& left) {
+ const auto& right = checked_cast<const RunEndEncodedType&>(right_);
+ // NOTE(felipecrv): the type of the run_ends might differ, but that doesn't
+ // change the logical meaning of the type so only the encoded type is
+ // compared.
Review Comment:
That's inconsistent with how dictionary equality works (see above), and would also break `CompareRunEndEncoded` above.
##########
cpp/src/arrow/testing/json_internal.cc:
##########
@@ -437,6 +437,10 @@ class SchemaWriter {
Status Visit(const DictionaryType& type) { return VisitType(*type.value_type()); }
+ Status Visit(const RunEndEncodedType& type) {
+ return Status::NotImplemented("run-end encoded type in JSON");
Review Comment:
Why not use `Status::NotImplemented(type.name())` as below?
##########
cpp/src/arrow/compare.cc:
##########
@@ -459,6 +473,26 @@ class RangeDataEqualsImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status CompareRunEndEncoded() {
+ auto left_span = ArraySpan(left_);
+ auto right_span = ArraySpan(right_);
+ left_span.SetSlice(left_.offset + left_start_idx_, range_length_);
+ right_span.SetSlice(right_.offset + right_start_idx_, range_length_);
+ for (auto it = ree_util::MergedRunsIterator<RunEndsType, RunEndsType>(left_span,
+ right_span);
Review Comment:
That only works if both run-ends types are identical, which is inconsistent with type equality definition below.
##########
cpp/src/arrow/array/builder_encoded.h:
##########
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <utility>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/array/builder_base.h"
+
+namespace arrow {
+
+/// \addtogroup encoded-builders
+///
+/// @{
+
+// ----------------------------------------------------------------------
+// RunEndEncoded builder
+
+class ARROW_EXPORT RunEndEncodedBuilder : public ArrayBuilder {
+ public:
+ RunEndEncodedBuilder(MemoryPool* pool,
+ const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder,
+ std::shared_ptr<DataType> type);
+
+ Status FinishInternal(std::shared_ptr<ArrayData>* out) final;
+ /// \cond FALSE
+ using ArrayBuilder::Finish;
+ /// \endcond
+
+ Status Resize(int64_t capacity) final;
+ void Reset() final;
+
+ Status Finish(std::shared_ptr<RunEndEncodedArray>* out) { return FinishTyped(out); }
+ Status AppendNull() final;
+ Status AppendNulls(int64_t length) final;
+ Status AppendEmptyValue() final;
+ Status AppendEmptyValues(int64_t length) final;
+ Status AppendScalar(const Scalar& scalar, int64_t n_repeats) final;
+ Status AppendScalars(const ScalarVector& scalars) final;
+ Status AppendArraySlice(const ArraySpan& array, int64_t offset, int64_t length) final;
+ std::shared_ptr<DataType> type() const final;
+
+ /// \brief Allocate enough memory for a given number of runs. Like Resize on non-REE
+ /// builders, it does not account for variable size data.
Review Comment:
This is not clear. Why "physical"? What does the capacity mean (number of offsets? number of values?)?
Also, generally a Reserve facility is more useful than a Resize one.
##########
cpp/src/arrow/array/builder_encoded.cc:
##########
@@ -0,0 +1,213 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/builder_encoded.h"
+#include "arrow/array/builder_primitive.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <utility>
+#include <vector>
+
+#include "arrow/scalar.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedBuilder
+
+RunEndEncodedBuilder::RunEndEncodedBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder, std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), type_(internal::checked_pointer_cast<RunEndEncodedType>(type)) {
+ children_ = {std::move(run_end_builder), std::move(value_builder)};
+}
+
+Status RunEndEncodedBuilder::Resize(int64_t) {
+ return Status::NotImplemented(
Review Comment:
This seems rather hostile. If we don't know how to resize, let's please not expose a method for it.
##########
cpp/src/arrow/visitor_generate.h:
##########
@@ -35,34 +35,39 @@ namespace arrow {
ACTION(Float); \
ACTION(Double)
-#define ARROW_GENERATE_FOR_ALL_TYPES(ACTION) \
- ACTION(Null); \
- ACTION(Boolean); \
- ARROW_GENERATE_FOR_ALL_NUMERIC_TYPES(ACTION); \
- ACTION(String); \
- ACTION(Binary); \
- ACTION(LargeString); \
- ACTION(LargeBinary); \
- ACTION(FixedSizeBinary); \
- ACTION(Duration); \
- ACTION(Date32); \
- ACTION(Date64); \
- ACTION(Timestamp); \
- ACTION(Time32); \
- ACTION(Time64); \
- ACTION(MonthDayNanoInterval); \
- ACTION(MonthInterval); \
- ACTION(DayTimeInterval); \
- ACTION(Decimal128); \
- ACTION(Decimal256); \
- ACTION(List); \
- ACTION(LargeList); \
- ACTION(Map); \
- ACTION(FixedSizeList); \
- ACTION(Struct); \
- ACTION(SparseUnion); \
- ACTION(DenseUnion); \
- ACTION(Dictionary); \
+// all types that can exist as a Scalar
Review Comment:
As I said in another comment, we should add `RunEndEncodedScalar` instead of hacking around it, IMHO.
##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +195,7 @@ int GetNumBuffers(const DataType& type) {
case Type::NA:
case Type::STRUCT:
case Type::FIXED_SIZE_LIST:
+ case Type::RUN_END_ENCODED:
return 1;
Review Comment:
Why 1? Run-end encoded arrays have no top-level buffers (no null bitmap), do they? Or do we reserve an empty null bitmap slot?
##########
cpp/src/arrow/array/concatenate.cc:
##########
@@ -436,6 +438,23 @@ class ConcatenateImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
+ int64_t physical_length = 0;
+ for (auto input : in_) {
Review Comment:
Needn't copy the `shared_ptr`
```suggestion
for (const auto& input : in_) {
```
##########
cpp/src/arrow/array/builder_encoded.cc:
##########
@@ -0,0 +1,213 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/array/builder_encoded.h"
+#include "arrow/array/builder_primitive.h"
+
+#include <cstddef>
+#include <cstdint>
+#include <utility>
+#include <vector>
+
+#include "arrow/scalar.h"
+#include "arrow/util/checked_cast.h"
+#include "arrow/util/logging.h"
+#include "arrow/util/ree_util.h"
+
+namespace arrow {
+
+// ----------------------------------------------------------------------
+// RunEndEncodedBuilder
+
+RunEndEncodedBuilder::RunEndEncodedBuilder(
+ MemoryPool* pool, const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder, std::shared_ptr<DataType> type)
+ : ArrayBuilder(pool), type_(internal::checked_pointer_cast<RunEndEncodedType>(type)) {
+ children_ = {std::move(run_end_builder), std::move(value_builder)};
+}
+
+Status RunEndEncodedBuilder::Resize(int64_t) {
+ return Status::NotImplemented(
+ "Resizing an REE for a given number of logical elements is not possible, since the "
+ "physical length will vary depending on the contents. To allocate memory for a "
+ "certain number of runs, use ResizePhysical.");
+}
+
+void RunEndEncodedBuilder::Reset() {
+ capacity_ = length_ = null_count_ = 0;
+ value_builder().Reset();
+ run_end_builder().Reset();
+ current_value_.reset();
+ run_start_ = 0;
+}
+
+Status RunEndEncodedBuilder::AppendNull() { return AppendNulls(1); }
+
+Status RunEndEncodedBuilder::AppendNulls(int64_t length) {
+ if (length == 0) {
+ return Status::OK();
+ }
+ if (current_value_) {
+ RETURN_NOT_OK(FinishRun());
+ current_value_ = {};
+ }
+ return AddLength(length);
+}
+
+Status RunEndEncodedBuilder::AppendEmptyValue() { return AppendNull(); }
+
+Status RunEndEncodedBuilder::AppendEmptyValues(int64_t length) {
+ return AppendNulls(length);
+}
+
+Status RunEndEncodedBuilder::AppendScalar(const Scalar& scalar, int64_t n_repeats) {
+ if (n_repeats == 0) {
+ return Status::OK();
+ }
+ if (!scalar.is_valid) {
+ return AppendNulls(n_repeats);
+ }
+ if (!current_value_ || !current_value_->Equals(scalar)) {
+ RETURN_NOT_OK(FinishRun());
+ current_value_ = scalar.shared_from_this();
+ }
+ return AddLength(n_repeats);
+}
+
+Status RunEndEncodedBuilder::AppendScalars(const ScalarVector& scalars) {
+ for (auto scalar : scalars) {
+ RETURN_NOT_OK(AppendScalar(*scalar, 1));
+ }
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendArray(const ArraySpan& to_append) {
+ const int64_t physical_offset = ree_util::FindPhysicalOffset(to_append);
+ int64_t physical_length = 0;
+ for (auto it = ree_util::MergedRunsIterator<RunEndsType>(to_append);
+ it != ree_util::MergedRunsIterator(); ++it) {
+ physical_length++;
+ length_ += it.run_length();
+ RETURN_NOT_OK(DoAppendRunEnd<RunEndsType>());
+ }
+ return value_builder().AppendArraySlice(ree_util::ValuesArray(to_append),
+ physical_offset, physical_length);
+}
+
+Status RunEndEncodedBuilder::AppendArraySlice(const ArraySpan& array, int64_t offset,
+ int64_t length) {
+ // Finish eventual runs started using AppendScalars() and others before. We don't
+ // attempt to merge them with the runs from the appended array slice for now
+ RETURN_NOT_OK(FinishRun());
+
+ ARROW_DCHECK(offset + length <= array.length);
+ ARROW_DCHECK(array.type->Equals(type_));
+ // Create a slice of the slice for the part we actually want to add
+ ArraySpan to_append = array;
+ to_append.SetSlice(array.offset + offset, length);
+
+ switch (type_->run_ends_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendArray<int16_t>(to_append));
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendArray<int32_t>(to_append));
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendArray<int64_t>(to_append));
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_ends_type());
+ }
+
+ // next run is not merged either
+ run_start_ = length_;
+ return Status::OK();
+}
+
+std::shared_ptr<DataType> RunEndEncodedBuilder::type() const { return type_; }
+
+Status RunEndEncodedBuilder::FinishInternal(std::shared_ptr<ArrayData>* out) {
+ ARROW_ASSIGN_OR_RAISE(auto run_ends_array, run_end_builder().Finish());
+ ARROW_ASSIGN_OR_RAISE(auto values_array, value_builder().Finish());
+ ARROW_ASSIGN_OR_RAISE(auto ree_array,
+ RunEndEncodedArray::Make(run_ends_array, values_array, length_));
+ *out = std::move(ree_array->data());
+ return Status::OK();
+}
+
+template <typename RunEndsType>
+Status RunEndEncodedBuilder::DoAppendRunEnd() {
+ return internal::checked_cast<typename CTypeTraits<RunEndsType>::BuilderType*>(
+ children_[0].get())
+ ->Append(static_cast<RunEndsType>(length_));
+}
+
+Status RunEndEncodedBuilder::FinishRun() {
+ if (length_ - run_start_ == 0) {
+ return Status::OK();
+ }
+ if (current_value_) {
+ RETURN_NOT_OK(value_builder().AppendScalar(*current_value_));
+ } else {
+ RETURN_NOT_OK(value_builder().AppendNull());
+ }
+ switch (type_->run_ends_type()->id()) {
+ case Type::INT16:
+ RETURN_NOT_OK(DoAppendRunEnd<int16_t>());
+ break;
+ case Type::INT32:
+ RETURN_NOT_OK(DoAppendRunEnd<int32_t>());
+ break;
+ case Type::INT64:
+ RETURN_NOT_OK(DoAppendRunEnd<int64_t>());
+ break;
+ default:
+ return Status::Invalid("Invalid type for run ends array: ", type_->run_ends_type());
+ }
+ current_value_.reset();
+ run_start_ = 0;
+ return Status::OK();
+}
+
+Status RunEndEncodedBuilder::ResizePhyiscal(int64_t capacity) {
+ RETURN_NOT_OK(value_builder().Resize(capacity));
+ return run_end_builder().Resize(capacity);
Review Comment:
Why resize both builders with the same capacity? This doesn't seem to make sense.
##########
cpp/src/parquet/arrow/path_internal.cc:
##########
@@ -801,6 +801,10 @@ class PathBuilder {
return Status::OK();
}
+ Status Visit(const ::arrow::RunEndEncodedArray& array) {
+ return Status::NotImplemented("Arrow REE array in Parquet");
Review Comment:
How about `NOT_IMPLEMENTED_VISIT` below?
##########
cpp/src/arrow/array/concatenate.cc:
##########
@@ -436,6 +438,23 @@ class ConcatenateImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
+ int64_t physical_length = 0;
+ for (auto input : in_) {
+ physical_length =
+ SafeSignedAdd(physical_length, ree_util::FindPhysicalOffset(ArraySpan(*input)));
+ }
+ ARROW_ASSIGN_OR_RAISE(auto builder, MakeBuilder(in_[0]->type, pool_));
+ RETURN_NOT_OK(internal::checked_cast<RunEndEncodedBuilder&>(*builder).ResizePhyiscal(
+ physical_length));
+ for (auto input : in_) {
Review Comment:
```suggestion
for (const auto& input : in_) {
```
##########
cpp/src/arrow/ipc/writer.cc:
##########
@@ -526,6 +526,10 @@ class RecordBatchSerializer {
return VisitType(*array.indices());
}
+ Status Visit(const RunEndEncodedArray& type) {
+ return Status::NotImplemented("run-end encoded array in IPC");
+ }
Review Comment:
I hope there will be a followup PR for this?
##########
cpp/src/arrow/array/concatenate.cc:
##########
@@ -30,6 +30,7 @@
#include "arrow/array/data.h"
#include "arrow/array/util.h"
#include "arrow/buffer.h"
+#include "arrow/builder.h"
Review Comment:
Just include the relevant builder header?
##########
cpp/src/arrow/array/builder_encoded.h:
##########
@@ -0,0 +1,85 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <limits>
+#include <memory>
+#include <utility>
+#include <vector>
+
+#include "arrow/array.h"
+#include "arrow/array/builder_base.h"
+
+namespace arrow {
+
+/// \addtogroup encoded-builders
+///
+/// @{
+
+// ----------------------------------------------------------------------
+// RunEndEncoded builder
+
+class ARROW_EXPORT RunEndEncodedBuilder : public ArrayBuilder {
+ public:
+ RunEndEncodedBuilder(MemoryPool* pool,
+ const std::shared_ptr<ArrayBuilder>& run_end_builder,
+ const std::shared_ptr<ArrayBuilder>& value_builder,
+ std::shared_ptr<DataType> type);
+
+ Status FinishInternal(std::shared_ptr<ArrayData>* out) final;
+ /// \cond FALSE
+ using ArrayBuilder::Finish;
+ /// \endcond
+
+ Status Resize(int64_t capacity) final;
+ void Reset() final;
+
+ Status Finish(std::shared_ptr<RunEndEncodedArray>* out) { return FinishTyped(out); }
+ Status AppendNull() final;
+ Status AppendNulls(int64_t length) final;
+ Status AppendEmptyValue() final;
+ Status AppendEmptyValues(int64_t length) final;
+ Status AppendScalar(const Scalar& scalar, int64_t n_repeats) final;
+ Status AppendScalars(const ScalarVector& scalars) final;
+ Status AppendArraySlice(const ArraySpan& array, int64_t offset, int64_t length) final;
+ std::shared_ptr<DataType> type() const final;
+
+ /// \brief Allocate enough memory for a given number of runs. Like Resize on non-REE
+ /// builders, it does not account for variable size data.
Review Comment:
You should perhaps take a look at the Reserve methods exposed by e.g. List or Binary builders.
##########
cpp/src/arrow/array/concatenate.cc:
##########
@@ -436,6 +438,23 @@ class ConcatenateImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
+ int64_t physical_length = 0;
+ for (auto input : in_) {
+ physical_length =
+ SafeSignedAdd(physical_length, ree_util::FindPhysicalOffset(ArraySpan(*input)));
Review Comment:
Are we sure this cannot overflow? If it can, we should detect overflow instead of letting it wrap around.
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -413,6 +414,20 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ Status Visit(const RunEndEncodedType& type) {
+ switch (type.run_ends_type()->id()) {
+ case Type::INT16:
+ return ValidateRunEndEncoded<int16_t>(type);
+ case Type::INT32:
+ return ValidateRunEndEncoded<int32_t>(type);
+ case Type::INT64:
+ return ValidateRunEndEncoded<int64_t>(type);
+ default:
+ return Status::Invalid("Run ends type must be int16, int32 or int64, but got: ",
+ type.run_ends_type());
Review Comment:
Hmm, does it print out the `shared_ptr<DataType>`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1090863816
##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +195,7 @@ int GetNumBuffers(const DataType& type) {
case Type::NA:
case Type::STRUCT:
case Type::FIXED_SIZE_LIST:
+ case Type::RUN_END_ENCODED:
return 1;
Review Comment:
I think REEs make thinking about nullability so different than normal that a dummy buffer won't really make things much easier. That's my impression without having much knowledge about all the codepaths that arrays go through.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1091314042
##########
cpp/src/arrow/scalar.cc:
##########
@@ -1149,7 +1149,7 @@ Result<std::shared_ptr<Scalar>> Scalar::CastTo(std::shared_ptr<DataType> to) con
if (is_valid) {
out->is_valid = true;
ToTypeVisitor unpack_to_type{*this, to, out.get()};
- RETURN_NOT_OK(VisitTypeInline(*to, &unpack_to_type));
+ RETURN_NOT_OK(VisitScalarTypeInline(*to, &unpack_to_type));
Review Comment:
And I totally agree. Not having the Scalar is not something we can really get away with.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1093933196
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
+ if (full_validation && run_ends_data.length != 0) {
+ const int64_t run_ends_length = ree_util::RunEndsArray(span).length;
+ int64_t last_run_end = 0;
+ int64_t physical_offset = 0;
+ int64_t physical_end = 0;
+ for (int64_t index = 0; index < run_ends_length; index++) {
+ int64_t run_end = run_ends[index];
+ if (run_end < 1) {
+ return Status::Invalid(
+ "Run ends array invalid: All run ends must be a positive integer but run "
+ "end ",
+ index, " is ", run_end);
+ }
+ if (run_end <= last_run_end) {
+ return Status::Invalid(
+ "Run ends array invalid: Each run end must be greater than the prevous "
+ "one, but run end ",
+ index, " is ", run_end, " and run end ", index - 1, " is ", last_run_end);
+ }
+ if (run_end > data.offset && last_run_end <= data.offset) {
+ physical_offset = index;
+ }
+ if (run_end >= data.offset + data.length &&
+ last_run_end < data.offset + data.length) {
+ physical_end = index + 1;
+ }
Review Comment:
I'm completely removing them as they are being calculated only to be compared on the asserts that I'm removing below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1092491250
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
+ return Status::Invalid("Parent type says this array encodes ", *type.encoded_type(),
+ " values, but values array has type ", *values_data.type);
+ }
+ const Status run_ends_valid = RecurseInto(run_ends_data);
+ if (!run_ends_valid.ok()) {
+ return Status::Invalid("Run ends array invalid: ", run_ends_valid.ToString());
+ }
+ const Status values_valid = RecurseInto(values_data);
+ if (!values_valid.ok()) {
+ return Status::Invalid("Values array invalid: ", values_valid.ToString());
+ }
+ if (data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for REE array, but was ",
+ data.null_count);
+ }
+ if (run_ends_data.null_count != 0) {
+ return Status::Invalid("Null count must be 0 for run ends array, but was ",
+ run_ends_data.null_count);
+ }
+ if (!run_ends_data.buffers[1]->is_cpu()) {
+ return Status::NotImplemented("Validating non-CPU run ends buffers");
+ }
+ ArraySpan span(data);
+ const RunEndsType* run_ends = ree_util::RunEnds<RunEndsType>(span);
+ if (run_ends_data.length == 0) {
+ if (data.length == 0) {
+ return Status::OK();
+ } else {
+ return Status::Invalid("REE array has non-zero length ", data.length,
+ ", but run ends array has zero length");
+ }
+ }
+ if (run_ends[run_ends_data.length - 1] < data.offset + data.length) {
+ return Status::Invalid(
+ "Last run in run ends array ends at ", run_ends[run_ends_data.length - 1],
+ " but this array requires at least ", data.offset + data.length, " (offset ",
+ data.offset, ", length ", data.length, ")");
+ }
+ if (full_validation && run_ends_data.length != 0) {
+ const int64_t run_ends_length = ree_util::RunEndsArray(span).length;
Review Comment:
`RunEnd<T>` returns a `cont T *`, `RunEndsArray(span)` is the `span.child_data[0]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1091674353
##########
cpp/src/arrow/array/data.cc:
##########
@@ -195,6 +195,7 @@ int GetNumBuffers(const DataType& type) {
case Type::NA:
case Type::STRUCT:
case Type::FIXED_SIZE_LIST:
+ case Type::RUN_END_ENCODED:
return 1;
Review Comment:
We shouldn't deviate from established practice on other datatypes.
* `IsValid` on `DictionaryArray` only queries the indices, not the dictionary values
* `IsValid` on `UnionArray` already returns false, even if the corresponding child entry is null
Also you'll notice that `IsValid` and `IsNull` are non-virtual methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] felipecrv commented on a diff in pull request #33641: GH-32104: [C++] Add support for Run-End encoded data to Arrow
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on code in PR #33641:
URL: https://github.com/apache/arrow/pull/33641#discussion_r1091318594
##########
cpp/src/arrow/array/validate.cc:
##########
@@ -622,6 +637,106 @@ struct ValidateArrayImpl {
return Status::OK();
}
+ template <typename RunEndsType>
+ Status ValidateRunEndEncoded(const RunEndEncodedType& type) {
+ // overflow was already checked at this point
+ if (data.offset + data.length > std::numeric_limits<RunEndsType>::max()) {
+ return Status::Invalid(
+ "Offset + length of an REE array must fit in a value of the run ends type ",
+ *type.run_ends_type(), ", but offset + length was ", data.offset + data.length,
+ " while the allowed maximum is ", std::numeric_limits<RunEndsType>::max());
+ }
+ if (!data.child_data[0]) {
+ return Status::Invalid("Run ends array is null pointer");
+ }
+ if (!data.child_data[1]) {
+ return Status::Invalid("Values array is null pointer");
+ }
+ const ArrayData& run_ends_data = *data.child_data[0];
+ const ArrayData& values_data = *data.child_data[1];
+ if (*run_ends_data.type != *type.run_ends_type()) {
+ return Status::Invalid("Run ends array of ", type, " must be ",
+ *type.run_ends_type(), ", but is ", *run_ends_data.type);
+ }
+ if (values_data.type != type.encoded_type()) {
Review Comment:
Fixing...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org