You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2020/06/21 18:41:24 UTC
[GitHub] [arrow] kiszk opened a new pull request #7507: [WIP] ARROW-8797: [C++] Create test to receive RecordBatch for different endian
kiszk opened a new pull request #7507:
URL: https://github.com/apache/arrow/pull/7507
This PR creates a test to receive RecordBatch for different endian (e.g. receive RecordBatch with big-endian schema on little-endian platform).
This PR changes
- Introduce Endianness enum class to represent endianness
- Add a new flag `endianness` to `arrow::schema` to represent endianness of Array in `RecordBatch`.
- Add `MakeIntRecordBatchNonnativeEndian` to generate a RecordBatch using non-native endian representation.
- Create a test to receive RecordBatch for different endian (Not enabled since this fails now).
If these approaches are acceptable, this PR will add additional tests for other types. Later, another PR will address passing this test.
For creating this PR, @kou helps me greatly (prototype of arrow::schema and teaching me about RecordBatch).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467942227
##########
File path: dev/archery/tests/generate_files_for_endian_test.sh
##########
@@ -0,0 +1,26 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+: ${ARROW_DIR:=/arrow}
+
+json_file=`archery integration --with-cpp=1 | grep "Testing file" | tail -1 | awk '{ print $3 }'`
Review comment:
Thank you for your good suggestion. I will try it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467891282
##########
File path: cpp/src/arrow/util/bit_util.h
##########
@@ -322,6 +322,31 @@ static inline uint16_t ByteSwap(uint16_t value) {
return static_cast<uint16_t>(ByteSwap(static_cast<int16_t>(value)));
}
static inline uint8_t ByteSwap(uint8_t value) { return value; }
+static inline int8_t ByteSwap(int8_t value) { return value; }
+static inline double ByteSwap(double value) {
+#if defined(__GNUC__)
+ // avoid dereferencing type-punned pointer will break strict-aliasing rules
+ int8_t* in_ptr = reinterpret_cast<int8_t*>(&value);
+ auto swapped = ARROW_BYTE_SWAP64(*reinterpret_cast<uint64_t*>(in_ptr));
+ int8_t* out_ptr = reinterpret_cast<int8_t*>(&swapped);
+ return *reinterpret_cast<double*>(out_ptr);
+#else
+ auto swapped = ARROW_BYTE_SWAP64(*reinterpret_cast<uint64_t*>(&value));
+ return *reinterpret_cast<double*>(&swapped);
+#endif
+}
+static inline float ByteSwap(float value) {
Review comment:
Same here.
##########
File path: cpp/src/arrow/util/bit_util.h
##########
@@ -322,6 +322,31 @@ static inline uint16_t ByteSwap(uint16_t value) {
return static_cast<uint16_t>(ByteSwap(static_cast<int16_t>(value)));
}
static inline uint8_t ByteSwap(uint8_t value) { return value; }
+static inline int8_t ByteSwap(int8_t value) { return value; }
+static inline double ByteSwap(double value) {
+#if defined(__GNUC__)
+ // avoid dereferencing type-punned pointer will break strict-aliasing rules
Review comment:
Instead, you should use the "Safe" functions from `util/ubsan.h` on all compilers. For example:
```c++
static inline double ByteSwap(double value) {
const uint64_t swapped = ARROW_BYTE_SWAP64(SafeCopy<uint64_t>(value));
return SafeCopy<double>(swapped);
}
```
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,177 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
Review comment:
Rather than patch this instance temporarily, it would seem cleaner to create a new `ArrayDataEndianSwapper` for this child data.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,177 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto buffer = const_cast<VALUE_TYPE*>(
+ reinterpret_cast<const VALUE_TYPE*>(data_->buffers[index]->data()));
+ // offset has one more element rather than data->length
+ int64_t length = length_ + 1;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ buffer[i] = BitUtil::FromBigEndian(buffer[i]);
+#else
+ buffer[i] = BitUtil::FromLittleEndian(buffer[i]);
+#endif
+ }
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ Status Visit(const T&) {
+ using value_type = typename T::c_type;
+ auto buffer = const_cast<value_type*>(
+ reinterpret_cast<const value_type*>(data_->buffers[1]->data()));
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ buffer[i] = BitUtil::FromBigEndian(buffer[i]);
+#else
+ buffer[i] = BitUtil::FromLittleEndian(buffer[i]);
+#endif
+ }
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto buffer = const_cast<uint64_t*>(
+ reinterpret_cast<const uint64_t*>(data_->buffers[1]->data()));
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(buffer[idx]);
+ buffer[idx] = BitUtil::FromBigEndian(buffer[idx + 1]);
+ buffer[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(buffer[idx]);
+ buffer[idx] = BitUtil::FromLittleEndian(buffer[idx + 1]);
+ buffer[idx + 1] = tmp;
+#endif
+ }
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ auto buffer = const_cast<uint32_t*>(
+ reinterpret_cast<const uint32_t*>(data_->buffers[1]->data()));
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ buffer[idx] = BitUtil::FromBigEndian(buffer[idx]);
+ buffer[idx + 1] = BitUtil::FromBigEndian(buffer[idx + 1]);
+#else
+ buffer[idx] = BitUtil::FromLittleEndian(buffer[idx]);
+ buffer[idx + 1] = BitUtil::FromLittleEndian(buffer[idx + 1]);
+#endif
+ }
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return Status::OK(); }
+ Status Visit(const Int8Type& type) { return Status::OK(); }
+ Status Visit(const UInt8Type& type) { return Status::OK(); }
+ Status Visit(const FixedSizeBinaryType& type) { return Status::OK(); }
+
+ Status Visit(const StringType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeStringType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+ Status Visit(const BinaryType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeBinaryType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+ Status Visit(const FixedSizeListType& type) {
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status Visit(const MapType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status Visit(const StructType& type) {
+ RETURN_NOT_OK(SwapChildren(type.fields()));
Review comment:
`SwapChildren` can be called in all cases instead of calling it specifically for some types.
##########
File path: cpp/src/arrow/type.h
##########
@@ -1582,13 +1583,26 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness {
+ LITTLE = 0,
+ BIG = 1,
+#if ARROW_LITTLE_ENDIAN
+ NATIVE = LITTLE
+#else
+ NATIVE = BIG
+#endif
+};
+
/// \class Schema
/// \brief Sequence of arrow::Field objects describing the columns of a record
/// batch or table data structure
class ARROW_EXPORT Schema : public detail::Fingerprintable,
public util::EqualityComparable<Schema>,
public util::ToStringOstreamable<Schema> {
public:
+ explicit Schema(std::vector<std::shared_ptr<Field>> fields, Endianness endianness,
+ std::shared_ptr<const KeyValueMetadata> metadata = NULLPTR);
Review comment:
I don't know if we want to allow non-native endianness in memory. @wesm What do you think?
##########
File path: cpp/src/arrow/type.h
##########
@@ -29,6 +29,7 @@
#include "arrow/result.h"
#include "arrow/type_fwd.h" // IWYU pragma: export
+#include "arrow/util/bit_util.h"
Review comment:
Is there a way to avoid including `bit_util.h` (which is a non-trivial header) here?
##########
File path: dev/archery/tests/generate_files_for_endian_test.sh
##########
@@ -0,0 +1,26 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+: ${ARROW_DIR:=/arrow}
+
+json_file=`archery integration --with-cpp=1 | grep "Testing file" | tail -1 | awk '{ print $3 }'`
Review comment:
It's not obvious to me what this script is doing precisely. In any case, I don't think we should rely on precise human-readable output of `archery integration`.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,177 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto buffer = const_cast<VALUE_TYPE*>(
Review comment:
This is definitely not safe at all. Another approach should be found IMHO. Worst case we reallocate, which I agree isn't desirable...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r576983710
##########
File path: ci/scripts/integration_arrow.sh
##########
@@ -24,9 +24,16 @@ source_dir=${1}/cpp
build_dir=${2}/cpp
gold_dir_0_14_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.14.1
gold_dir_0_17_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.17.1
+gold_dir_1_0_0_le=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-littleendian
+gold_dir_1_0_0_be=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-bigendian
pip install -e $arrow_dir/dev/archery
archery integration --with-all --run-flight \
--gold-dirs=$gold_dir_0_14_1 \
--gold-dirs=$gold_dir_0_17_1 \
+
+# TODO: support other languages
+archery integration --with-cpp=1 --run-flight \
Review comment:
You didn't answer my question above :-)
Also, instead of running `archery integration` a second time, you should add the new "gold dirs" to the command above, and add selective skips like in https://github.com/apache/arrow/blob/master/dev/archery/archery/integration/runner.py#L129
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r576992533
##########
File path: cpp/src/arrow/array/array_test.cc
##########
@@ -2598,4 +2599,325 @@ TEST(TestRechunkArraysConsistently, Plain) {
}
}
+// ----------------------------------------------------------------------
+// Test SwapEndianArrayData
+
+/// \brief Indicate if fields are equals.
+///
+/// \param[in] target ArrayData to be converted and tested
+/// \param[in] expected result ArrayData
+void AssertArrayDataEqualsWithSwapEndian(const std::shared_ptr<ArrayData>& target,
+ const std::shared_ptr<ArrayData>& expected) {
+ auto swap_array = MakeArray(*::arrow::internal::SwapEndianArrayData(target));
+ auto expected_array = MakeArray(expected);
+ ASSERT_ARRAYS_EQUAL(*swap_array, *expected_array);
+ ASSERT_OK(swap_array->ValidateFull());
+}
+
+TEST(TestSwapEndianArrayData, PrimitiveType) {
+ auto null_buffer = Buffer::FromString("\xff");
+ auto data_int_buffer = Buffer::FromString("01234567");
+
+ auto data = ArrayData::Make(null(), 0, {nullptr}, 0);
+ auto expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(boolean(), 8, {null_buffer, data_int_buffer}, 0);
+ expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(int8(), 8, {null_buffer, data_int_buffer}, 0);
+ expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(uint16(), 4, {null_buffer, data_int_buffer}, 0);
+ auto data_int16_buffer = Buffer::FromString("10325476");
+ expected_data = ArrayData::Make(uint16(), 4, {null_buffer, data_int16_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(int32(), 2, {null_buffer, data_int_buffer}, 0);
+ auto data_int32_buffer = Buffer::FromString("32107654");
+ expected_data = ArrayData::Make(int32(), 2, {null_buffer, data_int32_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(uint64(), 1, {null_buffer, data_int_buffer}, 0);
+ auto data_int64_buffer = Buffer::FromString("76543210");
+ expected_data = ArrayData::Make(uint64(), 1, {null_buffer, data_int64_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_16byte_buffer = Buffer::FromString("0123456789abcdef");
+ data = ArrayData::Make(decimal128(38, 10), 1, {null_buffer, data_16byte_buffer});
+ auto data_decimal128_buffer = Buffer::FromString("fedcba9876543210");
+ expected_data =
+ ArrayData::Make(decimal128(38, 10), 1, {null_buffer, data_decimal128_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_32byte_buffer = Buffer::FromString("0123456789abcdef123456789ABCDEF0");
+ data = ArrayData::Make(decimal256(76, 20), 1, {null_buffer, data_32byte_buffer});
+ auto data_decimal256_buffer = Buffer::FromString("0FEDCBA987654321fedcba9876543210");
+ expected_data =
+ ArrayData::Make(decimal256(76, 20), 1, {null_buffer, data_decimal256_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_float_buffer = Buffer::FromString("01200560");
+ data = ArrayData::Make(float32(), 2, {null_buffer, data_float_buffer}, 0);
+ auto data_float32_buffer = Buffer::FromString("02100650");
+ expected_data = ArrayData::Make(float32(), 2, {null_buffer, data_float32_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(float64(), 1, {null_buffer, data_float_buffer});
+ auto data_float64_buffer = Buffer::FromString("06500210");
+ expected_data = ArrayData::Make(float64(), 1, {null_buffer, data_float64_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ // With offset > 0
+ data =
+ ArrayData::Make(int64(), 1, {null_buffer, data_int_buffer}, kUnknownNullCount, 1);
+ ASSERT_RAISES(Invalid, ::arrow::internal::SwapEndianArrayData(data));
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffers(const std::shared_ptr<ArrayData>& data,
+ const int32_t buffer_index,
+ const std::vector<uint8_t>& buffer_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ test_data->buffers[buffer_index] =
+ std::make_shared<Buffer>(buffer_data.data(), buffer_data.size());
+ return test_data;
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffersInChild(const std::shared_ptr<ArrayData>& data,
+ const int32_t child_index,
+ const std::vector<uint8_t>& child_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ // assume updating only buffer[1] in child_data
+ auto child_array_data =
+ std::make_shared<ArrayData>(*test_data->child_data[child_index]);
+ child_array_data->buffers[1] =
+ std::make_shared<Buffer>(child_data.data(), child_data.size());
+ test_data->child_data[child_index] = child_array_data;
+ return test_data;
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffersInDictionary(
+ const std::shared_ptr<ArrayData>& data, const int32_t buffer_index,
+ const std::vector<uint8_t>& buffer_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ auto dict_array_data = std::make_shared<ArrayData>(*test_data->dictionary);
+ dict_array_data->buffers[buffer_index] =
+ std::make_shared<Buffer>(buffer_data.data(), buffer_data.size());
+ test_data->dictionary = dict_array_data;
+ return test_data;
+}
+
+TEST(TestSwapEndianArrayData, NonPrimitiveType) {
+ auto array = ArrayFromJSON(binary(), R"(["0123", null, "45"])");
+ const std::vector<uint8_t> offset1 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ auto expected_data = array->data();
+ auto test_data = ReplaceBuffers(expected_data, 1, offset1);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(large_binary(), R"(["01234", null, "567"])");
+ const std::vector<uint8_t> offset2 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5,
+ 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 8};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0,
+ 5, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset2);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(fixed_size_binary(3), R"(["012", null, "345"])");
+ expected_data = array->data();
+ AssertArrayDataEqualsWithSwapEndian(expected_data, expected_data);
+
+ array = ArrayFromJSON(utf8(), R"(["ABCD", null, "EF"])");
+ const std::vector<uint8_t> offset4 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset4);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(large_utf8(), R"(["ABCDE", null, "FGH"])");
+ const std::vector<uint8_t> offset5 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5,
+ 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 8};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0,
+ 5, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset5);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto type = std::make_shared<ListType>(int32());
+ array = ArrayFromJSON(type, "[[0, 1, 2, 3], null, [4, 5]]");
+ const std::vector<uint8_t> offset6 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data6 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 5};
+#else
+ {0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 5, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset6);
+ test_data = ReplaceBuffersInChild(test_data, 0, data6);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto type7 = std::make_shared<LargeListType>(int64());
+ array = ArrayFromJSON(type7, "[[0, 1, 2], null, [3]]");
+ const std::vector<uint8_t> offset7 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3,
+ 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0,
+ 3, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data7 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
+ 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 3};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
+ 2, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset7);
+ test_data = ReplaceBuffersInChild(test_data, 0, data7);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto type8 = std::make_shared<FixedSizeListType>(int32(), 2);
+ array = ArrayFromJSON(type8, "[[0, 1], null, [2, 3]]");
+ expected_data = array->data();
+ const std::vector<uint8_t> data8 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 3};
+#else
+ {0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0};
+#endif
+ test_data = ReplaceBuffersInChild(expected_data, 0, data8);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto type9 = dictionary(int32(), int16());
+ auto dict = ArrayFromJSON(int16(), "[4, 5, 6, 7]");
+ DictionaryArray array9(type9, ArrayFromJSON(int32(), "[0, 2, 3]"), dict);
+ expected_data = array9.data();
+ const std::vector<uint8_t> data9a =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 3};
+#else
+ {0, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data9b =
+#if ARROW_LITTLE_ENDIAN
+ {0, 4, 0, 5, 0, 6, 0, 7};
+#else
+ {4, 0, 5, 0, 6, 0, 7, 0};
+#endif
+ test_data = ReplaceBuffers(expected_data, 1, data9a);
+ test_data = ReplaceBuffersInDictionary(test_data, 1, data9b);
+ // dictionary must be explicitly swapped
+ test_data->dictionary = *::arrow::internal::SwapEndianArrayData(test_data->dictionary);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(struct_({field("a", int32()), field("b", utf8())}),
+ R"([{"a": 4, "b": null}, {"a": null, "b": "foo"}])");
+ expected_data = array->data();
+ const std::vector<uint8_t> data10a =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 4, 0, 0, 0, 0};
+#else
+ {4, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data10b =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0};
+#endif
+ test_data = ReplaceBuffersInChild(expected_data, 0, data10a);
+ test_data = ReplaceBuffersInChild(test_data, 1, data10b);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto expected_i8 = ArrayFromJSON(int8(), "[127, null, null, null, null]");
+ auto expected_str = ArrayFromJSON(utf8(), R"([null, "abcd", null, null, ""])");
+ auto expected_i32 = ArrayFromJSON(int32(), "[null, null, 1, 2, null]");
+ std::vector<uint8_t> expected_types_vector;
+ expected_types_vector.push_back(Type::INT8);
+ expected_types_vector.insert(expected_types_vector.end(), 2, Type::STRING);
+ expected_types_vector.insert(expected_types_vector.end(), 2, Type::INT32);
+ std::shared_ptr<Array> expected_types;
+ ArrayFromVector<Int8Type, uint8_t>(expected_types_vector, &expected_types);
+ auto arr11 = SparseUnionArray::Make(
+ *expected_types, {expected_i8, expected_str, expected_i32}, {"i8", "str", "i32"},
+ {Type::INT8, Type::STRING, Type::INT32});
+ expected_data = (*arr11)->data();
+ const std::vector<uint8_t> data11a =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 4};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data11b =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 0};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ test_data = ReplaceBuffersInChild(expected_data, 1, data11a);
+ test_data = ReplaceBuffersInChild(test_data, 2, data11b);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ expected_i8 = ArrayFromJSON(int8(), "[33, 10, -10]");
+ expected_str = ArrayFromJSON(utf8(), R"(["abc", "", "def"])");
+ expected_i32 = ArrayFromJSON(int32(), "[1, -259, 2]");
+ auto expected_offsets = ArrayFromJSON(int32(), "[0, 0, 0, 1, 1, 1, 2, 2, 2]");
+ auto arr12 = DenseUnionArray::Make(
+ *expected_types, *expected_offsets, {expected_i8, expected_str, expected_i32},
+ {"i8", "str", "i32"}, {Type::INT8, Type::STRING, Type::INT32});
+ expected_data = (*arr12)->data();
+ const std::vector<uint8_t> data12a =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,
+ 0, 1, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 2};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0,
+ 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data12b =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 3, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 3, 0, 0, 0, 3, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data12c =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 1, 255, 255, 254, 253, 0, 0, 0, 2};
+#else
+ {1, 0, 0, 0, 253, 254, 255, 255, 2, 0, 0, 0};
+#endif
+ test_data = ReplaceBuffers(expected_data, 2, data12a);
+ test_data = ReplaceBuffersInChild(test_data, 1, data12b);
+ test_data = ReplaceBuffersInChild(test_data, 2, data12c);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+}
+
Review comment:
Is it possible to add a test for extension types? The `SmallInt` type in `arrow/testing/extension_type.h` can help you.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-779998989
Thank you very much for the update @kiszk . I think we're nearing completion :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-780605746
@pitrou I appreciate a lot of your valuable feedbacks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r567381814
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
Review comment:
@pitrou Baed on [this suggestion](https://github.com/apache/arrow/pull/7507#discussion_r513711536), I created a new instance for `ArrayData` and related structures.
Is it fine with creating only a new `ArrayData` and reusing the existing buffer? cc @wesm
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-740138806
I am still working for this on my end since I have failures during addressing [this comment](https://github.com/apache/arrow/pull/7507#discussion_r513711536). Tomorrow, I will push the working version that has addressed other comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-667471857
It is ready for review now.
@wesm @pitrou @kou Would it be possible to review it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r577000124
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,11 +75,222 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(const std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {
+ const std::shared_ptr<DataType>& type = data->type;
+ std::vector<std::shared_ptr<Buffer>> buffers(data->buffers.size(), nullptr);
+ std::vector<std::shared_ptr<ArrayData>> child_data(data->child_data.size(), nullptr);
+ out_ = ArrayData::Make(type, data->length, buffers, child_data, data->null_count, 0);
+ }
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(const std::vector<std::shared_ptr<Field>>& child_fields) {
+ for (size_t i = 0; i < child_fields.size(); i++) {
+ ARROW_ASSIGN_OR_RAISE(out_->child_data[i],
+ internal::SwapEndianArrayData(data_->child_data[i]));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(
+ const std::shared_ptr<Buffer>& in_buffer) {
+ if (sizeof(T) == 1) {
+ // if data size is 1, element is not swapped. We can use the original buffer
+ return in_buffer;
+ }
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ int64_t length = in_buffer->size() / sizeof(T);
+ for (int64_t i = 0; i < length; i++) {
+ out_data[i] = BitUtil::ByteSwap(in_data[i]);
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffsets(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ out_->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // Except union, offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE(out_->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index]));
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE(out_->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1]));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ length = data_->buffers[1]->size() / (sizeof(uint64_t) * 2);
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ out_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ length = data_->buffers[1]->size() / (sizeof(uint64_t) * 4);
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ out_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE(out_->buffers[1], ByteSwapBuffer<uint32_t>(data_->buffers[1]));
+ return Status::OK();
+ }
+
+ Status ReuseDataBuffer() {
+ out_->buffers[1] = data_->buffers[1];
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return ReuseDataBuffer(); }
+ Status Visit(const Int8Type& type) { return ReuseDataBuffer(); }
+ Status Visit(const UInt8Type& type) { return ReuseDataBuffer(); }
+ Status Visit(const FixedSizeBinaryType& type) { return ReuseDataBuffer(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const UnionType& type) {
+ out_->buffers[1] = data_->buffers[1];
+ if (type.mode() == UnionMode::DENSE) {
+ RETURN_NOT_OK(SwapOffsets<int32_t>(2));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<BinaryType, T>::value || std::is_same<StringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapOffsets<int32_t>(1));
+ out_->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<LargeBinaryType, T>::value ||
+ std::is_same<LargeStringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapOffsets<int64_t>(1));
+ out_->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapOffsets<int32_t>(1));
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapOffsets<int64_t>(1));
+ return Status::OK();
+ }
+
+ Status Visit(const DictionaryType& type) {
+ RETURN_NOT_OK(SwapType(*type.index_type()));
+ // dictionary was already swapped in ReadDictionary() in ipc/reader.cc
+ out_->dictionary = data_->dictionary;
+ return Status::OK();
+ }
+
+ Status Visit(const ExtensionType& type) {
+ RETURN_NOT_OK(SwapType(*type.storage_type()));
+ // dictionary was already swapped in ReadDictionary() in ipc/reader.cc
+ out_->dictionary = data_->dictionary;
+ return Status::OK();
+ }
+
+ const std::shared_ptr<ArrayData>& data_;
+ int64_t length_;
+ std::shared_ptr<ArrayData> out_;
+};
+
+} // namespace
+
+namespace internal {
+
+Result<std::shared_ptr<ArrayData>> SwapEndianArrayData(
+ const std::shared_ptr<ArrayData>& data) {
+ if (data->offset != 0) {
+ return Status::Invalid("Unsupported data format: data.offset != 0");
+ }
+ const std::shared_ptr<DataType>& type = data->type;
+ ArrayDataEndianSwapper swapper_visitor(data, data->length);
+ DCHECK_OK(VisitTypeInline(*type, &swapper_visitor));
+ DCHECK_OK(swapper_visitor.SwapChildren((*type).fields()));
Review comment:
You should use `RETURN_NOT_OK` to return errors instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r537780408
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,204 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ArrayDataEndianSwapper swapper_child_visitor(data_->child_data[i],
+ data_->child_data[i]->length);
+ RETURN_NOT_OK(VisitTypeInline(*child_field.get()->type(), &swapper_child_visitor));
+ RETURN_NOT_OK(
+ swapper_child_visitor.SwapChildren((*child_field.get()->type()).fields()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto data = reinterpret_cast<const VALUE_TYPE*>(data_->buffers[index]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer,
+ AllocateBuffer(data_->buffers[index]->size() + 1));
+ auto new_data = reinterpret_cast<VALUE_TYPE*>(new_buffer->mutable_data());
+ // offset has one more element rather than data->length
+ int64_t length = length_ + 1;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[index] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ Status Visit(const T&) {
+ using value_type = typename T::c_type;
+ auto data = reinterpret_cast<const value_type*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<value_type*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ auto data = reinterpret_cast<const uint32_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint32_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ new_data[idx] = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromBigEndian(data[idx + 1]);
+#else
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromLittleEndian(data[idx + 1]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return Status::OK(); }
+ Status Visit(const Int8Type& type) { return Status::OK(); }
+ Status Visit(const UInt8Type& type) { return Status::OK(); }
+ Status Visit(const FixedSizeBinaryType& type) { return Status::OK(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const SparseUnionType& type) { return Status::OK(); }
+
+ Status Visit(const StringType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeStringType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+ Status Visit(const BinaryType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeBinaryType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const MapType& type) {
Review comment:
@wesm would it be possible to give me some comments?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7507: ARROW-8797: [C++] [WIP] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-647171266
https://issues.apache.org/jira/browse/ARROW-8797
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-657310102
Thank you for your clarification. I thought that I will check in binary files and write C++ files to validate the result to be read.
Now, I realized the integration test `archery integration ...` will kick `arrow-json-integration-test --integration --mode VALIDATE --json ... --arrow ...`. I will stick with ``archery integration ...`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-780578677
Thank you very much @kiszk ! I'll merge once CI is green.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467931144
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,177 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto buffer = const_cast<VALUE_TYPE*>(
Review comment:
I agree with safety. Since some of Arrow source files use `const_cast`, I also used it.
This is the tradeoff between effectiveness and safety.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r465839089
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -449,7 +449,7 @@ Result<std::shared_ptr<RecordBatch>> LoadRecordBatchSubset(
const flatbuf::RecordBatch* metadata, const std::shared_ptr<Schema>& schema,
const std::vector<bool>& inclusion_mask, const DictionaryMemo* dictionary_memo,
const IpcReadOptions& options, MetadataVersion metadata_version,
- Compression::type compression, io::RandomAccessFile* file) {
+ Compression::type compression, io::RandomAccessFile* file, bool swap_endian = false) {
Review comment:
I realized that we can drop `swap_endian` and `swap_endian_`. Then, I added `IpcReadOptions::use_native_endian` for a programmer to control whether the endian of data elements must be changed to native endian or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r563899594
##########
File path: cpp/src/arrow/array/util.h
##########
@@ -56,6 +56,14 @@ Result<std::shared_ptr<Array>> MakeArrayFromScalar(
namespace internal {
+/// \brief Swap endian of each element in a generic ArrayData
+/// \param[in] data the array contents
+/// \param[in] type the array type
+/// \return the resulting Array instance whose elements were swapped
+ARROW_EXPORT
+Result<std::shared_ptr<ArrayData>> SwapEndianArrayData(
+ std::shared_ptr<ArrayData>& data, const std::shared_ptr<DataType>& type);
+
/// Given a number of ArrayVectors, treat each ArrayVector as the
Review comment:
Thank you for your comments. I will address it.
Regarding the test, there are only integration tests as discussed at https://github.com/apache/arrow/pull/7507#issuecomment-657310102. I will add tests for `SwapEndianArrayData`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk edited a comment on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk edited a comment on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-667471857
It is ready for review now.
@wesm @pitrou @kou Would it be possible to review it if you have time?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467933180
##########
File path: cpp/src/arrow/type.h
##########
@@ -29,6 +29,7 @@
#include "arrow/result.h"
#include "arrow/type_fwd.h" // IWYU pragma: export
+#include "arrow/util/bit_util.h"
Review comment:
I see. Should I create a separate PR for refactoring `util/endian.h`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467911819
##########
File path: dev/archery/tests/generate_files_for_endian_test.sh
##########
@@ -0,0 +1,26 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+: ${ARROW_DIR:=/arrow}
+
+json_file=`archery integration --with-cpp=1 | grep "Testing file" | tail -1 | awk '{ print $3 }'`
Review comment:
Lines 21 and 22 try to get the directory where json files are generated. As far as I know, there is no way to get the directory name except stdout of `archery integration`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r521499370
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -107,6 +108,23 @@ Status InvalidMessageType(MessageType expected, MessageType actual) {
// ----------------------------------------------------------------------
// Record batch read path
+/// \brief Structure to keep common arguments to be passed
+struct IpcReadContext {
+ IpcReadContext(DictionaryMemo* memo, const IpcReadOptions& option, DictionaryKind* kind,
+ bool swap)
+ : dictionary_memo(memo), options(option), kind(kind), swap_endian(swap) {}
+
+ DictionaryMemo* dictionary_memo;
+
+ const IpcReadOptions& options;
+
+ DictionaryKind* kind;
Review comment:
This style, which uses a mutable pointer, follows the original code. Here is an example. My guess is that this code needs to have a special state `uninitialized`.
https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/reader.cc#L716-L726
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467934368
##########
File path: cpp/src/arrow/type.h
##########
@@ -29,6 +29,7 @@
#include "arrow/result.h"
#include "arrow/type_fwd.h" // IWYU pragma: export
+#include "arrow/util/bit_util.h"
Review comment:
No, it's minor enough that it can be done here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-717382815
@kiszk I'm on vacation at the moment, but I can take a look in 2 weeks, if you don't mind. I believe @wesm is quite busy at the moment...
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] [WIP] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r443912778
##########
File path: cpp/src/arrow/ipc/read_write_test.cc
##########
@@ -409,6 +414,32 @@ class IpcTestFixture : public io::MemoryMapFixture, public ExtensionTypesMixin {
CheckRoundtrip(*batch, options, IpcReadOptions::Defaults(), buffer_size);
}
+ void CheckRoundTrip(const RecordBatch& src_batch,
+ const RecordBatch& expected_batch,
+ IpcWriteOptions options = IpcWriteOptions::Defaults(),
+ IpcReadOptions read_options = IpcReadOptions::Defaults(),
+ int64_t buffer_size = 1 << 20) {
+ std::stringstream ss;
+ ss << "test-write-row-batch-" << g_file_number++;
+ ASSERT_OK_AND_ASSIGN(mmap_,
+ io::MemoryMapFixture::InitMemoryMap(buffer_size, ss.str()));
+
+ DictionaryMemo dictionary_memo;
+
+ std::shared_ptr<Schema> schema_result;
+ DoSchemaRoundTrip(*src_batch.schema(), &dictionary_memo, &schema_result);
+ ASSERT_TRUE(expected_batch.schema()->Equals(*schema_result));
Review comment:
Thank you for your comment. You are right. We should keep the original schema until `ReadRecordBatch`.
Then, if `LoadRecordBatchSubset` will automatically convert the endian based on the schema's endianness, `LoadRecordBatchSubset` will update the endian flag in the schema.
I will move this assertion after `Do*RoundTrip`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-663947048
After merging this PR https://github.com/apache/arrow-testing/pull/41, I will update the `integration_arrow.sh`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-739990481
ping
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467915870
##########
File path: cpp/src/arrow/type.h
##########
@@ -29,6 +29,7 @@
#include "arrow/result.h"
#include "arrow/type_fwd.h" // IWYU pragma: export
+#include "arrow/util/bit_util.h"
Review comment:
Just do you want not to include `bit_util.h` in `type.h`. There are possible two ways.
One way is to include one of the following header files that explicitly include `bit_util.h`.
- buffer_builder.h
- type_traits.h
- visitor_inline.h
Or, try to move the definition of `class Endianness`. On the other hand, the header file that will newly define `class Endianness` requires including `bit_util.h`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r468065835
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,177 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto buffer = const_cast<VALUE_TYPE*>(
Review comment:
I will address this issue tomorrow.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r465839089
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -449,7 +449,7 @@ Result<std::shared_ptr<RecordBatch>> LoadRecordBatchSubset(
const flatbuf::RecordBatch* metadata, const std::shared_ptr<Schema>& schema,
const std::vector<bool>& inclusion_mask, const DictionaryMemo* dictionary_memo,
const IpcReadOptions& options, MetadataVersion metadata_version,
- Compression::type compression, io::RandomAccessFile* file) {
+ Compression::type compression, io::RandomAccessFile* file, bool swap_endian = false) {
Review comment:
I realized that we can drop `swap_endian` and `swap_endian_`. Then, I added `IpcReadOptions::use_native_endian` for a programmer to control whether the endian of data elements must be changed to native endian or not.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-778937798
@pitrou Sorry for being late to address your comments. I addressed all of your comments. Now, it is ready for your review again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-748917409
@kiszk I'm in vacation, I can review this in two weeks. Would it be ok?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r465270135
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,186 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Status SwapOffset(const T&, int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ using value_type = typename T::c_type;
+ auto buffer = const_cast<value_type*>(
+ reinterpret_cast<const value_type*>(data_->buffers[index]->data()));
+ // offset has one more element rather than data->length
+ int64_t length = length_ + 1;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ buffer[i] = BitUtil::FromBigEndian(buffer[i]);
+#else
+ buffer[i] = BitUtil::FromLittleEndian(buffer[i]);
+#endif
+ }
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) {
+ Int32Type i32;
+ RETURN_NOT_OK(SwapOffset(i32, index));
+ return Status::OK();
+ }
+
+ Status SwapLargeOffset() {
+ Int64Type i64;
+ RETURN_NOT_OK(SwapOffset(i64, 1));
+ return Status::OK();
+ }
Review comment:
Thanks for the good suggestion. Done.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r568628575
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
Review comment:
I see. Thank you for your clarification.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r576995343
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,11 +75,222 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(const std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {
+ const std::shared_ptr<DataType>& type = data->type;
+ std::vector<std::shared_ptr<Buffer>> buffers(data->buffers.size(), nullptr);
+ std::vector<std::shared_ptr<ArrayData>> child_data(data->child_data.size(), nullptr);
+ out_ = ArrayData::Make(type, data->length, buffers, child_data, data->null_count, 0);
Review comment:
Now that I'm thinking of it, you can probably write `out_ = data->Copy();` instead.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r476686943
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,177 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto buffer = const_cast<VALUE_TYPE*>(
Review comment:
Sorry for my bad. I updated `SwapOffset`. The current failures in R are not related to this update.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r468100444
##########
File path: cpp/src/arrow/type.h
##########
@@ -29,6 +29,7 @@
#include "arrow/result.h"
#include "arrow/type_fwd.h" // IWYU pragma: export
+#include "arrow/util/bit_util.h"
Review comment:
Newly added `util/endian.h` by moving some definitions from `bit_util.h`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r472905861
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,177 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto buffer = const_cast<VALUE_TYPE*>(
Review comment:
@pitrou Since I was swamped with something in my company, I was late to address this. Now, I updated the code to reallocate the buffer. Could you please look at this again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r476481191
##########
File path: cpp/src/arrow/type.h
##########
@@ -1582,13 +1583,26 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness {
+ LITTLE = 0,
+ BIG = 1,
+#if ARROW_LITTLE_ENDIAN
+ NATIVE = LITTLE
+#else
+ NATIVE = BIG
+#endif
+};
+
/// \class Schema
/// \brief Sequence of arrow::Field objects describing the columns of a record
/// batch or table data structure
class ARROW_EXPORT Schema : public detail::Fingerprintable,
public util::EqualityComparable<Schema>,
public util::ToStringOstreamable<Schema> {
public:
+ explicit Schema(std::vector<std::shared_ptr<Field>> fields, Endianness endianness,
+ std::shared_ptr<const KeyValueMetadata> metadata = NULLPTR);
Review comment:
@kou Yes, that seems to make sense.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467953221
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,177 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto buffer = const_cast<VALUE_TYPE*>(
Review comment:
In the general case, we should certainly reallocate the buffer. I'm not sure if there are cases where we can avoid it, but our current APIs don't make it easy (a Buffer is a bit too abstracted for that).
In `gpu/cuda_memory.cc`, `io/file.cc` and `io/memory.cc`, the pointer is not dereferenced (the memory is not written to). Here, we are writing into the memory, so can't ignore `const`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-657261835
@wesm Can I ask a question? I am writing a test case after creating a point-of-truth JSON file and generating two arrow binary files for BE and LE. Is this my approach what you suggested? I am writing a function `MakePrimitiveBatch()` by importing values from the point-of-truth JSON file.
https://gist.github.com/kiszk/bccb2640a68b706049c1631ba9514eae
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7507: ARROW-8797: [C++] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-652142541
@kiszk I would suggest creating a LE and BE example corpus in apache/arrow-testing. You can use the integration test command line tools to create point-of-truth JSON files and then use the JSON_TO_ARROW mode of the integration test executable to create IPC binary files that can be checked in
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
wesm commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r513711536
##########
File path: cpp/src/arrow/array/util.h
##########
@@ -37,6 +37,11 @@ namespace arrow {
ARROW_EXPORT
std::shared_ptr<Array> MakeArray(const std::shared_ptr<ArrayData>& data);
+/// \brief Swap endian of each element in a generic ArrayData
+/// \param[in] data the array contents to be swapped
+ARROW_EXPORT
+void SwapEndianArrayData(std::shared_ptr<ArrayData>& data);
Review comment:
Since this is allocating memory anyway, it might be better for this to return a new ArrayData and leave the existing one unmodified, unless you are trying to optimize memory use. In either case, a mutable reference should not be passed -- `ArrayData*` for in-place modification (not recommended, since it will only work if the buffers are mutable, which is not guaranteed) or `const ArrayData&` for copy
Also, it may be better for this to be in an `internal::` namespace for now
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,204 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ArrayDataEndianSwapper swapper_child_visitor(data_->child_data[i],
+ data_->child_data[i]->length);
+ RETURN_NOT_OK(VisitTypeInline(*child_field.get()->type(), &swapper_child_visitor));
+ RETURN_NOT_OK(
+ swapper_child_visitor.SwapChildren((*child_field.get()->type()).fields()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto data = reinterpret_cast<const VALUE_TYPE*>(data_->buffers[index]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer,
+ AllocateBuffer(data_->buffers[index]->size() + 1));
+ auto new_data = reinterpret_cast<VALUE_TYPE*>(new_buffer->mutable_data());
+ // offset has one more element rather than data->length
+ int64_t length = length_ + 1;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[index] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ Status Visit(const T&) {
+ using value_type = typename T::c_type;
+ auto data = reinterpret_cast<const value_type*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<value_type*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ auto data = reinterpret_cast<const uint32_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint32_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ new_data[idx] = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromBigEndian(data[idx + 1]);
+#else
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromLittleEndian(data[idx + 1]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return Status::OK(); }
+ Status Visit(const Int8Type& type) { return Status::OK(); }
+ Status Visit(const UInt8Type& type) { return Status::OK(); }
+ Status Visit(const FixedSizeBinaryType& type) { return Status::OK(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const SparseUnionType& type) { return Status::OK(); }
+
+ Status Visit(const StringType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeStringType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+ Status Visit(const BinaryType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeBinaryType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const MapType& type) {
Review comment:
This case is covered by `ListType`
##########
File path: cpp/src/arrow/type.h
##########
@@ -1604,13 +1605,26 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness {
+ LITTLE = 0,
+ BIG = 1,
+#if ARROW_LITTLE_ENDIAN
+ NATIVE = LITTLE
+#else
+ NATIVE = BIG
+#endif
+};
Review comment:
Having endianness at the Schema level definitely has me scratching my head, since if you have only an `ArrayData` object, you have no way to know its endianness. In NumPy at least, which allows you to have non-native-endian data in-memory, the endianness is handled at the type level, so you could have a structured dtype in NumPy that has a mix of little and big-endian data.
Of course, in the Arrow protocol it's all or nothing -- either all of the data has to be little or big-endian in a protocol message.
One alternative to what's been done here is to have endianness at the type level and have an option in the IpcWriteOptions to force either native-endianness or little/big endian, which would cause any ArrayData with different endianness to be byte-swapped.
##########
File path: cpp/src/arrow/ipc/options.h
##########
@@ -93,6 +93,12 @@ struct ARROW_EXPORT IpcReadOptions {
/// like decompression
bool use_threads = true;
+ /// \brief Convert endian of data element to platform-native endianness
+ /// if the endianness of the received schema is not equal to
+ /// platform-native endianness. This is effective at RecordBatchFileReader.Open(),
+ /// RecordBatchStreamReader(), or StreamDecoder().
+ bool ensure_native_endian = true;
Review comment:
See comments below about this
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -963,6 +1003,12 @@ class RecordBatchFileReaderImpl : public RecordBatchFileReader {
// Get the schema and record any observed dictionaries
RETURN_NOT_OK(UnpackSchemaMessage(footer_->schema(), options, &dictionary_memo_,
&schema_, &out_schema_, &field_inclusion_mask_));
+ swap_endian_ = options.ensure_native_endian && !out_schema_->IsNativeEndianness();
+ if (swap_endian_) {
+ // create a new schema with native endianness before swapping endian in ArrayData
+ schema_ = schema_->WithNativeEndianness();
+ out_schema_ = out_schema_->WithNativeEndianness();
+ }
Review comment:
this is basically the same as above, this suggests that something needs to be refactored
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,204 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ArrayDataEndianSwapper swapper_child_visitor(data_->child_data[i],
+ data_->child_data[i]->length);
+ RETURN_NOT_OK(VisitTypeInline(*child_field.get()->type(), &swapper_child_visitor));
+ RETURN_NOT_OK(
+ swapper_child_visitor.SwapChildren((*child_field.get()->type()).fields()));
+ i++;
Review comment:
Can you not just call `SwapEndianArrayData` here?
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,204 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ArrayDataEndianSwapper swapper_child_visitor(data_->child_data[i],
+ data_->child_data[i]->length);
+ RETURN_NOT_OK(VisitTypeInline(*child_field.get()->type(), &swapper_child_visitor));
+ RETURN_NOT_OK(
+ swapper_child_visitor.SwapChildren((*child_field.get()->type()).fields()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto data = reinterpret_cast<const VALUE_TYPE*>(data_->buffers[index]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer,
+ AllocateBuffer(data_->buffers[index]->size() + 1));
+ auto new_data = reinterpret_cast<VALUE_TYPE*>(new_buffer->mutable_data());
+ // offset has one more element rather than data->length
+ int64_t length = length_ + 1;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[index] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ Status Visit(const T&) {
+ using value_type = typename T::c_type;
+ auto data = reinterpret_cast<const value_type*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<value_type*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ auto data = reinterpret_cast<const uint32_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint32_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ new_data[idx] = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromBigEndian(data[idx + 1]);
+#else
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromLittleEndian(data[idx + 1]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return Status::OK(); }
+ Status Visit(const Int8Type& type) { return Status::OK(); }
+ Status Visit(const UInt8Type& type) { return Status::OK(); }
+ Status Visit(const FixedSizeBinaryType& type) { return Status::OK(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const SparseUnionType& type) { return Status::OK(); }
+
+ Status Visit(const StringType& type) {
Review comment:
this case is covered by BinaryType (have to use `enable_if_binary`)
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,204 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ArrayDataEndianSwapper swapper_child_visitor(data_->child_data[i],
+ data_->child_data[i]->length);
+ RETURN_NOT_OK(VisitTypeInline(*child_field.get()->type(), &swapper_child_visitor));
+ RETURN_NOT_OK(
+ swapper_child_visitor.SwapChildren((*child_field.get()->type()).fields()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto data = reinterpret_cast<const VALUE_TYPE*>(data_->buffers[index]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer,
+ AllocateBuffer(data_->buffers[index]->size() + 1));
+ auto new_data = reinterpret_cast<VALUE_TYPE*>(new_buffer->mutable_data());
+ // offset has one more element rather than data->length
+ int64_t length = length_ + 1;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[index] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ Status Visit(const T&) {
+ using value_type = typename T::c_type;
+ auto data = reinterpret_cast<const value_type*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<value_type*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
Review comment:
There is code duplication here, could a function be factored out like:
```c++
ARROW_ASSIGN_OR_RAISE(out_buffers[i], ByteSwapBuffer<T>(*in_buffers[i], length, memory_pool));
```
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,204 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ArrayDataEndianSwapper swapper_child_visitor(data_->child_data[i],
+ data_->child_data[i]->length);
+ RETURN_NOT_OK(VisitTypeInline(*child_field.get()->type(), &swapper_child_visitor));
+ RETURN_NOT_OK(
+ swapper_child_visitor.SwapChildren((*child_field.get()->type()).fields()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto data = reinterpret_cast<const VALUE_TYPE*>(data_->buffers[index]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer,
+ AllocateBuffer(data_->buffers[index]->size() + 1));
+ auto new_data = reinterpret_cast<VALUE_TYPE*>(new_buffer->mutable_data());
+ // offset has one more element rather than data->length
+ int64_t length = length_ + 1;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[index] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ Status Visit(const T&) {
+ using value_type = typename T::c_type;
+ auto data = reinterpret_cast<const value_type*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<value_type*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ auto data = reinterpret_cast<const uint32_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint32_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ new_data[idx] = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromBigEndian(data[idx + 1]);
+#else
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromLittleEndian(data[idx + 1]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return Status::OK(); }
+ Status Visit(const Int8Type& type) { return Status::OK(); }
+ Status Visit(const UInt8Type& type) { return Status::OK(); }
+ Status Visit(const FixedSizeBinaryType& type) { return Status::OK(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const SparseUnionType& type) { return Status::OK(); }
+
+ Status Visit(const StringType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeStringType& type) {
Review comment:
Covered by LargeBinaryType
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -84,6 +283,12 @@ std::shared_ptr<Array> MakeArray(const std::shared_ptr<ArrayData>& data) {
return out;
}
+void SwapEndianArrayData(std::shared_ptr<ArrayData>& data) {
+ internal::ArrayDataEndianSwapper swapper_visitor(data, data->length);
+ DCHECK_OK(VisitTypeInline(*data->type, &swapper_visitor));
+ DCHECK_OK(swapper_visitor.SwapChildren((*data->type).fields()));
Review comment:
Possibly better to inline the SwapChildren implementation here
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,204 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ArrayDataEndianSwapper swapper_child_visitor(data_->child_data[i],
+ data_->child_data[i]->length);
+ RETURN_NOT_OK(VisitTypeInline(*child_field.get()->type(), &swapper_child_visitor));
+ RETURN_NOT_OK(
+ swapper_child_visitor.SwapChildren((*child_field.get()->type()).fields()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto data = reinterpret_cast<const VALUE_TYPE*>(data_->buffers[index]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer,
+ AllocateBuffer(data_->buffers[index]->size() + 1));
+ auto new_data = reinterpret_cast<VALUE_TYPE*>(new_buffer->mutable_data());
+ // offset has one more element rather than data->length
+ int64_t length = length_ + 1;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[index] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ Status Visit(const T&) {
+ using value_type = typename T::c_type;
+ auto data = reinterpret_cast<const value_type*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<value_type*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ auto data = reinterpret_cast<const uint32_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint32_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ new_data[idx] = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromBigEndian(data[idx + 1]);
+#else
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromLittleEndian(data[idx + 1]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
Review comment:
Wouldn't this be per above `ByteSwapBuffer<uint32_t>(*data_->buffers[1], length_ * 2, ...)`?
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -664,14 +690,15 @@ Result<std::shared_ptr<RecordBatch>> ReadRecordBatch(
std::shared_ptr<Schema> out_schema;
// Empty means do not use
std::vector<bool> inclusion_mask;
- RETURN_NOT_OK(GetInclusionMaskAndOutSchema(schema, options.included_fields,
+ DictionaryKind kind;
Review comment:
kind is never used? See above comment re: this parameter
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -481,24 +500,30 @@ Result<std::shared_ptr<RecordBatch>> LoadRecordBatchSubset(
filtered_columns = std::move(columns);
}
if (compression != Compression::UNCOMPRESSED) {
- RETURN_NOT_OK(DecompressBuffers(compression, options, &filtered_columns));
+ RETURN_NOT_OK(DecompressBuffers(compression, context.options, &filtered_columns));
}
+ // swap endian in a set of ArrayData if necessary (swap_endian == true)
+ if (context.swap_endian) {
+ for (int i = 0; i < static_cast<int>(filtered_columns.size()); ++i) {
+ SwapEndianArrayData(filtered_columns[i]);
+ }
+ }
return RecordBatch::Make(filtered_schema, metadata->length(),
std::move(filtered_columns));
}
Result<std::shared_ptr<RecordBatch>> LoadRecordBatch(
const flatbuf::RecordBatch* metadata, const std::shared_ptr<Schema>& schema,
- const std::vector<bool>& inclusion_mask, const DictionaryMemo* dictionary_memo,
- const IpcReadOptions& options, MetadataVersion metadata_version,
- Compression::type compression, io::RandomAccessFile* file) {
+ const std::vector<bool>& inclusion_mask, const IpcReadContext& context,
+ MetadataVersion metadata_version, Compression::type compression,
+ io::RandomAccessFile* file) {
if (inclusion_mask.size() > 0) {
- return LoadRecordBatchSubset(metadata, schema, &inclusion_mask, dictionary_memo,
- options, metadata_version, compression, file);
- } else {
- return LoadRecordBatchSubset(metadata, schema, nullptr, dictionary_memo, options,
+ return LoadRecordBatchSubset(metadata, schema, &inclusion_mask, context,
metadata_version, compression, file);
+ } else {
+ return LoadRecordBatchSubset(metadata, schema, nullptr, context, metadata_version,
Review comment:
nit, add `/*param_name=*/` for this nullptr
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -699,42 +726,45 @@ Status ReadDictionary(const Buffer& metadata, DictionaryMemo* dictionary_memo,
// Look up the dictionary value type, which must have been added to the
// DictionaryMemo already prior to invoking this function
- ARROW_ASSIGN_OR_RAISE(auto value_type, dictionary_memo->GetDictionaryType(id));
+ ARROW_ASSIGN_OR_RAISE(auto value_type, context.dictionary_memo->GetDictionaryType(id));
// Load the dictionary data from the dictionary batch
ArrayLoader loader(batch_meta, internal::GetMetadataVersion(message->version()),
- options, file);
- const auto dict_data = std::make_shared<ArrayData>();
+ context.options, file);
+ auto dict_data = std::make_shared<ArrayData>();
const Field dummy_field("", value_type);
RETURN_NOT_OK(loader.Load(&dummy_field, dict_data.get()));
if (compression != Compression::UNCOMPRESSED) {
ArrayDataVector dict_fields{dict_data};
- RETURN_NOT_OK(DecompressBuffers(compression, options, &dict_fields));
+ RETURN_NOT_OK(DecompressBuffers(compression, context.options, &dict_fields));
+ }
+
+ // swap endian in dict_data if necessary (swap_endian == true)
+ if (context.swap_endian) {
+ SwapEndianArrayData(dict_data);
}
if (dictionary_batch->isDelta()) {
- if (kind != nullptr) {
- *kind = DictionaryKind::Delta;
+ if (context.kind != nullptr) {
+ *context.kind = DictionaryKind::Delta;
}
- return dictionary_memo->AddDictionaryDelta(id, dict_data);
+ return context.dictionary_memo->AddDictionaryDelta(id, dict_data);
}
ARROW_ASSIGN_OR_RAISE(bool inserted,
- dictionary_memo->AddOrReplaceDictionary(id, dict_data));
- if (kind != nullptr) {
- *kind = inserted ? DictionaryKind::New : DictionaryKind::Replacement;
+ context.dictionary_memo->AddOrReplaceDictionary(id, dict_data));
+ if (context.kind != nullptr) {
+ *context.kind = inserted ? DictionaryKind::New : DictionaryKind::Replacement;
}
Review comment:
I think it's better to leave the DictionaryKind as an out param
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -107,6 +108,23 @@ Status InvalidMessageType(MessageType expected, MessageType actual) {
// ----------------------------------------------------------------------
// Record batch read path
+/// \brief Structure to keep common arguments to be passed
+struct IpcReadContext {
+ IpcReadContext(DictionaryMemo* memo, const IpcReadOptions& option, DictionaryKind* kind,
+ bool swap)
+ : dictionary_memo(memo), options(option), kind(kind), swap_endian(swap) {}
+
+ DictionaryMemo* dictionary_memo;
+
+ const IpcReadOptions& options;
+
+ DictionaryKind* kind;
Review comment:
Having a mutable pointer here for an enum seems odd, can you explain?
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -699,42 +726,45 @@ Status ReadDictionary(const Buffer& metadata, DictionaryMemo* dictionary_memo,
// Look up the dictionary value type, which must have been added to the
// DictionaryMemo already prior to invoking this function
- ARROW_ASSIGN_OR_RAISE(auto value_type, dictionary_memo->GetDictionaryType(id));
+ ARROW_ASSIGN_OR_RAISE(auto value_type, context.dictionary_memo->GetDictionaryType(id));
// Load the dictionary data from the dictionary batch
ArrayLoader loader(batch_meta, internal::GetMetadataVersion(message->version()),
- options, file);
- const auto dict_data = std::make_shared<ArrayData>();
+ context.options, file);
+ auto dict_data = std::make_shared<ArrayData>();
const Field dummy_field("", value_type);
RETURN_NOT_OK(loader.Load(&dummy_field, dict_data.get()));
if (compression != Compression::UNCOMPRESSED) {
ArrayDataVector dict_fields{dict_data};
- RETURN_NOT_OK(DecompressBuffers(compression, options, &dict_fields));
+ RETURN_NOT_OK(DecompressBuffers(compression, context.options, &dict_fields));
+ }
+
+ // swap endian in dict_data if necessary (swap_endian == true)
+ if (context.swap_endian) {
+ SwapEndianArrayData(dict_data);
Review comment:
Shouldn't this be able to fail (e.g. for a failed allocation)? I think it would be better to do
```
ARROW_ASSIGN_OR_RAISE(dict_data, SwapEndianness(*dict_data, ...));
```
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -753,8 +783,15 @@ class RecordBatchStreamReaderImpl : public RecordBatchStreamReader {
return Status::Invalid("Tried reading schema message, was null or length 0");
}
- return UnpackSchemaMessage(*message, options, &dictionary_memo_, &schema_,
- &out_schema_, &field_inclusion_mask_);
+ RETURN_NOT_OK(UnpackSchemaMessage(*message, options, &dictionary_memo_, &schema_,
+ &out_schema_, &field_inclusion_mask_));
+ swap_endian_ = options.ensure_native_endian && !out_schema_->IsNativeEndianness();
+ if (swap_endian_) {
+ // create a new schema with native endianness before swapping endian in ArrayData
+ schema_ = schema_->WithNativeEndianness();
+ out_schema_ = out_schema_->WithNativeEndianness();
+ }
Review comment:
This code block occurs multiple times, is there a way to avoid that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] [WIP] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-647170774
Comments are appreciated. In particular, `arrow::schema` and generation of RecordBatch using non-native endian representation
cc @pitrou @kou
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r576999927
##########
File path: cpp/src/arrow/array/data.cc
##########
@@ -130,6 +130,33 @@ int64_t ArrayData::GetNullCount() const {
return precomputed;
}
+bool ArrayData::Equals(const ArrayData& other) const {
Review comment:
Thank you for your comment. I realized that this method is not used now. It was implemented for my old versions. Actually, what I committed is your suggestion :)
I will drop it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-780387899
@pitrou I addressed your comments. Could you take a look again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r521763175
##########
File path: cpp/src/arrow/type.h
##########
@@ -1604,13 +1605,26 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness {
+ LITTLE = 0,
+ BIG = 1,
+#if ARROW_LITTLE_ENDIAN
+ NATIVE = LITTLE
+#else
+ NATIVE = BIG
+#endif
+};
Review comment:
I see.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r577081155
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -108,6 +109,20 @@ Status InvalidMessageType(MessageType expected, MessageType actual) {
// ----------------------------------------------------------------------
// Record batch read path
+/// \brief Structure to keep common arguments to be passed
+struct IpcReadContext {
+ IpcReadContext(DictionaryMemo* memo, const IpcReadOptions& option, bool swap)
+ : dictionary_memo(memo), options(option), swap_endian(swap) {}
+
+ DictionaryMemo* dictionary_memo;
+
+ const IpcReadOptions& options;
+
+ /// \brief LoadRecordBatch() or LoadRecordBatchSubset() swaps endianness of elements
+ /// if this flag is true
+ const bool swap_endian;
+};
Review comment:
If they're not used most of the time, then no, no need for it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r521501051
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -699,42 +726,45 @@ Status ReadDictionary(const Buffer& metadata, DictionaryMemo* dictionary_memo,
// Look up the dictionary value type, which must have been added to the
// DictionaryMemo already prior to invoking this function
- ARROW_ASSIGN_OR_RAISE(auto value_type, dictionary_memo->GetDictionaryType(id));
+ ARROW_ASSIGN_OR_RAISE(auto value_type, context.dictionary_memo->GetDictionaryType(id));
// Load the dictionary data from the dictionary batch
ArrayLoader loader(batch_meta, internal::GetMetadataVersion(message->version()),
- options, file);
- const auto dict_data = std::make_shared<ArrayData>();
+ context.options, file);
+ auto dict_data = std::make_shared<ArrayData>();
const Field dummy_field("", value_type);
RETURN_NOT_OK(loader.Load(&dummy_field, dict_data.get()));
if (compression != Compression::UNCOMPRESSED) {
ArrayDataVector dict_fields{dict_data};
- RETURN_NOT_OK(DecompressBuffers(compression, options, &dict_fields));
+ RETURN_NOT_OK(DecompressBuffers(compression, context.options, &dict_fields));
+ }
+
+ // swap endian in dict_data if necessary (swap_endian == true)
+ if (context.swap_endian) {
+ SwapEndianArrayData(dict_data);
}
if (dictionary_batch->isDelta()) {
- if (kind != nullptr) {
- *kind = DictionaryKind::Delta;
+ if (context.kind != nullptr) {
+ *context.kind = DictionaryKind::Delta;
}
- return dictionary_memo->AddDictionaryDelta(id, dict_data);
+ return context.dictionary_memo->AddDictionaryDelta(id, dict_data);
}
ARROW_ASSIGN_OR_RAISE(bool inserted,
- dictionary_memo->AddOrReplaceDictionary(id, dict_data));
- if (kind != nullptr) {
- *kind = inserted ? DictionaryKind::New : DictionaryKind::Replacement;
+ context.dictionary_memo->AddOrReplaceDictionary(id, dict_data));
+ if (context.kind != nullptr) {
+ *context.kind = inserted ? DictionaryKind::New : DictionaryKind::Replacement;
}
Review comment:
I see.
@kou Is it ok to exclude `param` from `IpcReadContext`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-740923764
Addressed comments except the following two items
- be addressing https://github.com/apache/arrow/pull/7507#discussion_r513711536
- postpone https://github.com/apache/arrow/pull/7507#discussion_r513725869 to another PR
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-652482871
@wesm Thank you for your suggestion. I will pursue the approach that you suggested. I will check the integration test command line tool and the integration test with the JSON_TO_ARROW mode.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r463912700
##########
File path: ci/scripts/integration_arrow.sh
##########
@@ -24,9 +24,16 @@ source_dir=${1}/cpp
build_dir=${2}/cpp
gold_dir_0_14_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.14.1
gold_dir_0_17_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.17.1
+gold_dir_1_0_0_le=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-littleendian
+gold_dir_1_0_0_be=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-bigendian
pip install -e $arrow_dir/dev/archery
archery integration --with-all --run-flight \
--gold-dirs=$gold_dir_0_14_1 \
--gold-dirs=$gold_dir_0_17_1 \
+
+# TODO: support other languages
+archery integration --with-cpp=1 --run-flight \
Review comment:
Since other languages (e.g. Java) cause failure of integration tests, future separate PR will fix failures in other languages.
This PR focuses on cpp integration test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r576989799
##########
File path: cpp/src/arrow/array/array_test.cc
##########
@@ -2598,4 +2599,325 @@ TEST(TestRechunkArraysConsistently, Plain) {
}
}
+// ----------------------------------------------------------------------
+// Test SwapEndianArrayData
+
+/// \brief Indicate if fields are equals.
+///
+/// \param[in] target ArrayData to be converted and tested
+/// \param[in] expected result ArrayData
+void AssertArrayDataEqualsWithSwapEndian(const std::shared_ptr<ArrayData>& target,
+ const std::shared_ptr<ArrayData>& expected) {
+ auto swap_array = MakeArray(*::arrow::internal::SwapEndianArrayData(target));
+ auto expected_array = MakeArray(expected);
+ ASSERT_ARRAYS_EQUAL(*swap_array, *expected_array);
+ ASSERT_OK(swap_array->ValidateFull());
+}
+
+TEST(TestSwapEndianArrayData, PrimitiveType) {
+ auto null_buffer = Buffer::FromString("\xff");
+ auto data_int_buffer = Buffer::FromString("01234567");
+
+ auto data = ArrayData::Make(null(), 0, {nullptr}, 0);
+ auto expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(boolean(), 8, {null_buffer, data_int_buffer}, 0);
+ expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(int8(), 8, {null_buffer, data_int_buffer}, 0);
+ expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(uint16(), 4, {null_buffer, data_int_buffer}, 0);
+ auto data_int16_buffer = Buffer::FromString("10325476");
+ expected_data = ArrayData::Make(uint16(), 4, {null_buffer, data_int16_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(int32(), 2, {null_buffer, data_int_buffer}, 0);
+ auto data_int32_buffer = Buffer::FromString("32107654");
+ expected_data = ArrayData::Make(int32(), 2, {null_buffer, data_int32_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(uint64(), 1, {null_buffer, data_int_buffer}, 0);
+ auto data_int64_buffer = Buffer::FromString("76543210");
+ expected_data = ArrayData::Make(uint64(), 1, {null_buffer, data_int64_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_16byte_buffer = Buffer::FromString("0123456789abcdef");
+ data = ArrayData::Make(decimal128(38, 10), 1, {null_buffer, data_16byte_buffer});
+ auto data_decimal128_buffer = Buffer::FromString("fedcba9876543210");
+ expected_data =
+ ArrayData::Make(decimal128(38, 10), 1, {null_buffer, data_decimal128_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_32byte_buffer = Buffer::FromString("0123456789abcdef123456789ABCDEF0");
+ data = ArrayData::Make(decimal256(76, 20), 1, {null_buffer, data_32byte_buffer});
+ auto data_decimal256_buffer = Buffer::FromString("0FEDCBA987654321fedcba9876543210");
+ expected_data =
+ ArrayData::Make(decimal256(76, 20), 1, {null_buffer, data_decimal256_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_float_buffer = Buffer::FromString("01200560");
+ data = ArrayData::Make(float32(), 2, {null_buffer, data_float_buffer}, 0);
+ auto data_float32_buffer = Buffer::FromString("02100650");
+ expected_data = ArrayData::Make(float32(), 2, {null_buffer, data_float32_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(float64(), 1, {null_buffer, data_float_buffer});
+ auto data_float64_buffer = Buffer::FromString("06500210");
+ expected_data = ArrayData::Make(float64(), 1, {null_buffer, data_float64_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ // With offset > 0
+ data =
+ ArrayData::Make(int64(), 1, {null_buffer, data_int_buffer}, kUnknownNullCount, 1);
+ ASSERT_RAISES(Invalid, ::arrow::internal::SwapEndianArrayData(data));
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffers(const std::shared_ptr<ArrayData>& data,
+ const int32_t buffer_index,
+ const std::vector<uint8_t>& buffer_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ test_data->buffers[buffer_index] =
+ std::make_shared<Buffer>(buffer_data.data(), buffer_data.size());
+ return test_data;
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffersInChild(const std::shared_ptr<ArrayData>& data,
+ const int32_t child_index,
+ const std::vector<uint8_t>& child_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ // assume updating only buffer[1] in child_data
+ auto child_array_data =
+ std::make_shared<ArrayData>(*test_data->child_data[child_index]);
+ child_array_data->buffers[1] =
+ std::make_shared<Buffer>(child_data.data(), child_data.size());
+ test_data->child_data[child_index] = child_array_data;
+ return test_data;
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffersInDictionary(
+ const std::shared_ptr<ArrayData>& data, const int32_t buffer_index,
+ const std::vector<uint8_t>& buffer_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ auto dict_array_data = std::make_shared<ArrayData>(*test_data->dictionary);
+ dict_array_data->buffers[buffer_index] =
+ std::make_shared<Buffer>(buffer_data.data(), buffer_data.size());
+ test_data->dictionary = dict_array_data;
+ return test_data;
+}
+
+TEST(TestSwapEndianArrayData, NonPrimitiveType) {
+ auto array = ArrayFromJSON(binary(), R"(["0123", null, "45"])");
+ const std::vector<uint8_t> offset1 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ auto expected_data = array->data();
+ auto test_data = ReplaceBuffers(expected_data, 1, offset1);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(large_binary(), R"(["01234", null, "567"])");
+ const std::vector<uint8_t> offset2 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5,
+ 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 8};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0,
+ 5, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset2);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(fixed_size_binary(3), R"(["012", null, "345"])");
+ expected_data = array->data();
+ AssertArrayDataEqualsWithSwapEndian(expected_data, expected_data);
+
+ array = ArrayFromJSON(utf8(), R"(["ABCD", null, "EF"])");
+ const std::vector<uint8_t> offset4 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset4);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(large_utf8(), R"(["ABCDE", null, "FGH"])");
+ const std::vector<uint8_t> offset5 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5,
+ 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 8};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0,
+ 5, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset5);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto type = std::make_shared<ListType>(int32());
+ array = ArrayFromJSON(type, "[[0, 1, 2, 3], null, [4, 5]]");
+ const std::vector<uint8_t> offset6 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data6 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 5};
+#else
+ {0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 5, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset6);
+ test_data = ReplaceBuffersInChild(test_data, 0, data6);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto type7 = std::make_shared<LargeListType>(int64());
+ array = ArrayFromJSON(type7, "[[0, 1, 2], null, [3]]");
+ const std::vector<uint8_t> offset7 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3,
+ 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0,
+ 3, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data7 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
+ 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 3};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
+ 2, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset7);
+ test_data = ReplaceBuffersInChild(test_data, 0, data7);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto type8 = std::make_shared<FixedSizeListType>(int32(), 2);
+ array = ArrayFromJSON(type8, "[[0, 1], null, [2, 3]]");
+ expected_data = array->data();
+ const std::vector<uint8_t> data8 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 3};
+#else
+ {0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0};
+#endif
+ test_data = ReplaceBuffersInChild(expected_data, 0, data8);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto type9 = dictionary(int32(), int16());
Review comment:
Nit, but instead of having a long single test named `NonPrimitiveType`, you could have several of them: `DictionaryType`, `StructType`, etc. It makes test failures easier to investigate too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467934939
##########
File path: dev/archery/tests/generate_files_for_endian_test.sh
##########
@@ -0,0 +1,26 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+: ${ARROW_DIR:=/arrow}
+
+json_file=`archery integration --with-cpp=1 | grep "Testing file" | tail -1 | awk '{ print $3 }'`
Review comment:
`archery integration --tempdir` should allow you to choose the directory.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r476447442
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,177 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto buffer = const_cast<VALUE_TYPE*>(
Review comment:
@pitrou How about this change?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk edited a comment on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk edited a comment on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-660690067
This commit can succeed to run an integration test with primitive types like this.
```
(on BE platform) > ../../../build/debug/arrow-json-integration-test --integration --json int32_primitive.json --arrow int32_primitive_le.arrow --mode=VALIDATE
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r576999769
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,11 +75,222 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(const std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {
+ const std::shared_ptr<DataType>& type = data->type;
+ std::vector<std::shared_ptr<Buffer>> buffers(data->buffers.size(), nullptr);
+ std::vector<std::shared_ptr<ArrayData>> child_data(data->child_data.size(), nullptr);
+ out_ = ArrayData::Make(type, data->length, buffers, child_data, data->null_count, 0);
+ }
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(const std::vector<std::shared_ptr<Field>>& child_fields) {
+ for (size_t i = 0; i < child_fields.size(); i++) {
+ ARROW_ASSIGN_OR_RAISE(out_->child_data[i],
+ internal::SwapEndianArrayData(data_->child_data[i]));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(
+ const std::shared_ptr<Buffer>& in_buffer) {
+ if (sizeof(T) == 1) {
+ // if data size is 1, element is not swapped. We can use the original buffer
+ return in_buffer;
+ }
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ int64_t length = in_buffer->size() / sizeof(T);
+ for (int64_t i = 0; i < length; i++) {
+ out_data[i] = BitUtil::ByteSwap(in_data[i]);
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffsets(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ out_->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // Except union, offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE(out_->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index]));
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE(out_->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1]));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ length = data_->buffers[1]->size() / (sizeof(uint64_t) * 2);
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ out_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ length = data_->buffers[1]->size() / (sizeof(uint64_t) * 4);
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ out_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE(out_->buffers[1], ByteSwapBuffer<uint32_t>(data_->buffers[1]));
+ return Status::OK();
+ }
+
+ Status ReuseDataBuffer() {
+ out_->buffers[1] = data_->buffers[1];
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return ReuseDataBuffer(); }
+ Status Visit(const Int8Type& type) { return ReuseDataBuffer(); }
+ Status Visit(const UInt8Type& type) { return ReuseDataBuffer(); }
+ Status Visit(const FixedSizeBinaryType& type) { return ReuseDataBuffer(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const UnionType& type) {
+ out_->buffers[1] = data_->buffers[1];
+ if (type.mode() == UnionMode::DENSE) {
+ RETURN_NOT_OK(SwapOffsets<int32_t>(2));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<BinaryType, T>::value || std::is_same<StringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapOffsets<int32_t>(1));
+ out_->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<LargeBinaryType, T>::value ||
+ std::is_same<LargeStringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapOffsets<int64_t>(1));
+ out_->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapOffsets<int32_t>(1));
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapOffsets<int64_t>(1));
+ return Status::OK();
+ }
+
+ Status Visit(const DictionaryType& type) {
+ RETURN_NOT_OK(SwapType(*type.index_type()));
+ // dictionary was already swapped in ReadDictionary() in ipc/reader.cc
+ out_->dictionary = data_->dictionary;
+ return Status::OK();
+ }
+
+ Status Visit(const ExtensionType& type) {
+ RETURN_NOT_OK(SwapType(*type.storage_type()));
+ // dictionary was already swapped in ReadDictionary() in ipc/reader.cc
+ out_->dictionary = data_->dictionary;
Review comment:
This doesn't seem necessary.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r576995848
##########
File path: cpp/src/arrow/array/array_test.cc
##########
@@ -2598,4 +2599,325 @@ TEST(TestRechunkArraysConsistently, Plain) {
}
}
+// ----------------------------------------------------------------------
+// Test SwapEndianArrayData
+
+/// \brief Indicate if fields are equals.
+///
+/// \param[in] target ArrayData to be converted and tested
+/// \param[in] expected result ArrayData
+void AssertArrayDataEqualsWithSwapEndian(const std::shared_ptr<ArrayData>& target,
+ const std::shared_ptr<ArrayData>& expected) {
+ auto swap_array = MakeArray(*::arrow::internal::SwapEndianArrayData(target));
+ auto expected_array = MakeArray(expected);
+ ASSERT_ARRAYS_EQUAL(*swap_array, *expected_array);
+ ASSERT_OK(swap_array->ValidateFull());
+}
+
+TEST(TestSwapEndianArrayData, PrimitiveType) {
+ auto null_buffer = Buffer::FromString("\xff");
+ auto data_int_buffer = Buffer::FromString("01234567");
+
+ auto data = ArrayData::Make(null(), 0, {nullptr}, 0);
+ auto expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(boolean(), 8, {null_buffer, data_int_buffer}, 0);
+ expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(int8(), 8, {null_buffer, data_int_buffer}, 0);
+ expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(uint16(), 4, {null_buffer, data_int_buffer}, 0);
+ auto data_int16_buffer = Buffer::FromString("10325476");
+ expected_data = ArrayData::Make(uint16(), 4, {null_buffer, data_int16_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(int32(), 2, {null_buffer, data_int_buffer}, 0);
+ auto data_int32_buffer = Buffer::FromString("32107654");
+ expected_data = ArrayData::Make(int32(), 2, {null_buffer, data_int32_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(uint64(), 1, {null_buffer, data_int_buffer}, 0);
+ auto data_int64_buffer = Buffer::FromString("76543210");
+ expected_data = ArrayData::Make(uint64(), 1, {null_buffer, data_int64_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_16byte_buffer = Buffer::FromString("0123456789abcdef");
+ data = ArrayData::Make(decimal128(38, 10), 1, {null_buffer, data_16byte_buffer});
+ auto data_decimal128_buffer = Buffer::FromString("fedcba9876543210");
+ expected_data =
+ ArrayData::Make(decimal128(38, 10), 1, {null_buffer, data_decimal128_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_32byte_buffer = Buffer::FromString("0123456789abcdef123456789ABCDEF0");
+ data = ArrayData::Make(decimal256(76, 20), 1, {null_buffer, data_32byte_buffer});
+ auto data_decimal256_buffer = Buffer::FromString("0FEDCBA987654321fedcba9876543210");
+ expected_data =
+ ArrayData::Make(decimal256(76, 20), 1, {null_buffer, data_decimal256_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_float_buffer = Buffer::FromString("01200560");
+ data = ArrayData::Make(float32(), 2, {null_buffer, data_float_buffer}, 0);
+ auto data_float32_buffer = Buffer::FromString("02100650");
+ expected_data = ArrayData::Make(float32(), 2, {null_buffer, data_float32_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(float64(), 1, {null_buffer, data_float_buffer});
+ auto data_float64_buffer = Buffer::FromString("06500210");
+ expected_data = ArrayData::Make(float64(), 1, {null_buffer, data_float64_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ // With offset > 0
+ data =
+ ArrayData::Make(int64(), 1, {null_buffer, data_int_buffer}, kUnknownNullCount, 1);
+ ASSERT_RAISES(Invalid, ::arrow::internal::SwapEndianArrayData(data));
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffers(const std::shared_ptr<ArrayData>& data,
+ const int32_t buffer_index,
+ const std::vector<uint8_t>& buffer_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ test_data->buffers[buffer_index] =
+ std::make_shared<Buffer>(buffer_data.data(), buffer_data.size());
+ return test_data;
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffersInChild(const std::shared_ptr<ArrayData>& data,
+ const int32_t child_index,
+ const std::vector<uint8_t>& child_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ // assume updating only buffer[1] in child_data
+ auto child_array_data =
+ std::make_shared<ArrayData>(*test_data->child_data[child_index]);
+ child_array_data->buffers[1] =
+ std::make_shared<Buffer>(child_data.data(), child_data.size());
+ test_data->child_data[child_index] = child_array_data;
+ return test_data;
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffersInDictionary(
+ const std::shared_ptr<ArrayData>& data, const int32_t buffer_index,
+ const std::vector<uint8_t>& buffer_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ auto dict_array_data = std::make_shared<ArrayData>(*test_data->dictionary);
+ dict_array_data->buffers[buffer_index] =
+ std::make_shared<Buffer>(buffer_data.data(), buffer_data.size());
+ test_data->dictionary = dict_array_data;
+ return test_data;
+}
+
+TEST(TestSwapEndianArrayData, NonPrimitiveType) {
+ auto array = ArrayFromJSON(binary(), R"(["0123", null, "45"])");
+ const std::vector<uint8_t> offset1 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ auto expected_data = array->data();
+ auto test_data = ReplaceBuffers(expected_data, 1, offset1);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(large_binary(), R"(["01234", null, "567"])");
+ const std::vector<uint8_t> offset2 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5,
+ 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 8};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0,
+ 5, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset2);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(fixed_size_binary(3), R"(["012", null, "345"])");
+ expected_data = array->data();
+ AssertArrayDataEqualsWithSwapEndian(expected_data, expected_data);
+
+ array = ArrayFromJSON(utf8(), R"(["ABCD", null, "EF"])");
+ const std::vector<uint8_t> offset4 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset4);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(large_utf8(), R"(["ABCDE", null, "FGH"])");
+ const std::vector<uint8_t> offset5 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5,
+ 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 8};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0,
+ 5, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset5);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto type = std::make_shared<ListType>(int32());
+ array = ArrayFromJSON(type, "[[0, 1, 2, 3], null, [4, 5]]");
+ const std::vector<uint8_t> offset6 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data6 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 5};
+#else
+ {0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0, 4, 0, 0, 0, 5, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset6);
+ test_data = ReplaceBuffersInChild(test_data, 0, data6);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto type7 = std::make_shared<LargeListType>(int64());
+ array = ArrayFromJSON(type7, "[[0, 1, 2], null, [3]]");
+ const std::vector<uint8_t> offset7 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3,
+ 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0, 4};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0,
+ 3, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data7 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1,
+ 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0, 3};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
+ 2, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset7);
+ test_data = ReplaceBuffersInChild(test_data, 0, data7);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto type8 = std::make_shared<FixedSizeListType>(int32(), 2);
+ array = ArrayFromJSON(type8, "[[0, 1], null, [2, 3]]");
+ expected_data = array->data();
+ const std::vector<uint8_t> data8 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 3};
+#else
+ {0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0};
+#endif
+ test_data = ReplaceBuffersInChild(expected_data, 0, data8);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto type9 = dictionary(int32(), int16());
+ auto dict = ArrayFromJSON(int16(), "[4, 5, 6, 7]");
+ DictionaryArray array9(type9, ArrayFromJSON(int32(), "[0, 2, 3]"), dict);
+ expected_data = array9.data();
+ const std::vector<uint8_t> data9a =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 2, 0, 0, 0, 3};
+#else
+ {0, 0, 0, 0, 2, 0, 0, 0, 3, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data9b =
+#if ARROW_LITTLE_ENDIAN
+ {0, 4, 0, 5, 0, 6, 0, 7};
+#else
+ {4, 0, 5, 0, 6, 0, 7, 0};
+#endif
+ test_data = ReplaceBuffers(expected_data, 1, data9a);
+ test_data = ReplaceBuffersInDictionary(test_data, 1, data9b);
+ // dictionary must be explicitly swapped
+ test_data->dictionary = *::arrow::internal::SwapEndianArrayData(test_data->dictionary);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(struct_({field("a", int32()), field("b", utf8())}),
+ R"([{"a": 4, "b": null}, {"a": null, "b": "foo"}])");
+ expected_data = array->data();
+ const std::vector<uint8_t> data10a =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 4, 0, 0, 0, 0};
+#else
+ {4, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data10b =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 3};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0};
+#endif
+ test_data = ReplaceBuffersInChild(expected_data, 0, data10a);
+ test_data = ReplaceBuffersInChild(test_data, 1, data10b);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ auto expected_i8 = ArrayFromJSON(int8(), "[127, null, null, null, null]");
+ auto expected_str = ArrayFromJSON(utf8(), R"([null, "abcd", null, null, ""])");
+ auto expected_i32 = ArrayFromJSON(int32(), "[null, null, 1, 2, null]");
+ std::vector<uint8_t> expected_types_vector;
+ expected_types_vector.push_back(Type::INT8);
+ expected_types_vector.insert(expected_types_vector.end(), 2, Type::STRING);
+ expected_types_vector.insert(expected_types_vector.end(), 2, Type::INT32);
+ std::shared_ptr<Array> expected_types;
+ ArrayFromVector<Int8Type, uint8_t>(expected_types_vector, &expected_types);
+ auto arr11 = SparseUnionArray::Make(
+ *expected_types, {expected_i8, expected_str, expected_i32}, {"i8", "str", "i32"},
+ {Type::INT8, Type::STRING, Type::INT32});
+ expected_data = (*arr11)->data();
+ const std::vector<uint8_t> data11a =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 4};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data11b =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 0};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ test_data = ReplaceBuffersInChild(expected_data, 1, data11a);
+ test_data = ReplaceBuffersInChild(test_data, 2, data11b);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ expected_i8 = ArrayFromJSON(int8(), "[33, 10, -10]");
+ expected_str = ArrayFromJSON(utf8(), R"(["abc", "", "def"])");
+ expected_i32 = ArrayFromJSON(int32(), "[1, -259, 2]");
+ auto expected_offsets = ArrayFromJSON(int32(), "[0, 0, 0, 1, 1, 1, 2, 2, 2]");
+ auto arr12 = DenseUnionArray::Make(
+ *expected_types, *expected_offsets, {expected_i8, expected_str, expected_i32},
+ {"i8", "str", "i32"}, {Type::INT8, Type::STRING, Type::INT32});
+ expected_data = (*arr12)->data();
+ const std::vector<uint8_t> data12a =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0,
+ 0, 1, 0, 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 2};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0,
+ 0, 0, 1, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0, 2, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data12b =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 3, 0, 0, 0, 3, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 3, 0, 0, 0, 3, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ const std::vector<uint8_t> data12c =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 1, 255, 255, 254, 253, 0, 0, 0, 2};
+#else
+ {1, 0, 0, 0, 253, 254, 255, 255, 2, 0, 0, 0};
+#endif
+ test_data = ReplaceBuffers(expected_data, 2, data12a);
+ test_data = ReplaceBuffersInChild(test_data, 1, data12b);
+ test_data = ReplaceBuffersInChild(test_data, 2, data12c);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+}
+
Review comment:
Sure, I will add it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r465269578
##########
File path: cpp/src/arrow/type.h
##########
@@ -1582,13 +1583,23 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness { LITTLE, BIG };
+#if ARROW_LITTLE_ENDIAN
+#define NATIVE_ENDIANNESS Endianness::LITTLE
+#else
+#define NATIVE_ENDIANNESS Endianness::BIG
Review comment:
sure, done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] [WIP] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r445289548
##########
File path: cpp/src/arrow/ipc/read_write_test.cc
##########
@@ -427,6 +469,14 @@ class TestIpcRoundTrip : public ::testing::TestWithParam<MakeRecordBatch*>,
void TearDown() { IpcTestFixture::TearDown(); }
};
+class TestIpcDifferentEndian
+ : public ::testing::TestWithParam<std::tuple<MakeRecordBatch*, MakeRecordBatch*>>,
+ public IpcTestFixture {
+ public:
+ void SetUp() { IpcTestFixture::SetUp(); }
+ void TearDown() { IpcTestFixture::TearDown(); }
+};
Review comment:
Good point. In my current thought, another version of [ArrayLoader](https://github.com/apache/arrow/blob/master/cpp/src/arrow/ipc/reader.cc#L107) (`LoadType` method), which is called from `LoadRecordBatchSubset`, will be implemented. One existing version is the non-endian conversion. The other is the endian conversion version. At this level, we know the type of each element. Thus, I imagine it is easy to swap endianness. What do you think?
To add a new API `Result<std::shared_ptr<RecordBatch>> RecordBatch::EnsureEndianness(Endianness)` is good idea to avoid unnecessary conversion in the above scenario. Technically, this could work correctly.
I have two questions.
1. I imagine that this API is explicitly called by a user. If so, is this change acceptable by a user? Of course, most of the scenarios (communicate between the same endianness) work well without calling this new API.
2. This new API needs to traverse all of the elements again if we need the data conversion. It may lead to additional overhead.
What do you think? cc @pitrou
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467906940
##########
File path: cpp/src/arrow/type.h
##########
@@ -1582,13 +1583,26 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness {
+ LITTLE = 0,
+ BIG = 1,
+#if ARROW_LITTLE_ENDIAN
+ NATIVE = LITTLE
+#else
+ NATIVE = BIG
+#endif
+};
+
/// \class Schema
/// \brief Sequence of arrow::Field objects describing the columns of a record
/// batch or table data structure
class ARROW_EXPORT Schema : public detail::Fingerprintable,
public util::EqualityComparable<Schema>,
public util::ToStringOstreamable<Schema> {
public:
+ explicit Schema(std::vector<std::shared_ptr<Field>> fields, Endianness endianness,
+ std::shared_ptr<const KeyValueMetadata> metadata = NULLPTR);
Review comment:
cc @kou
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r567382132
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
Review comment:
As I wrote below, it would be fine if we can reuse the existing buffer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-717603836
I will try to review before the end of the week, @emkornfield may be able to look also
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-660690067
This commit can succeed to run an integration test with primitive types like this.
```
../../../build/debug/arrow-json-integration-test --integration --json int32_primitive.json --arrow int32_primitive_le.arrow --mode=VALIDATE
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-740009830
@kiszk Did you forget to push your changes?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-671751774
@kou I addressed your two comments.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r532819514
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,204 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ArrayDataEndianSwapper swapper_child_visitor(data_->child_data[i],
+ data_->child_data[i]->length);
+ RETURN_NOT_OK(VisitTypeInline(*child_field.get()->type(), &swapper_child_visitor));
+ RETURN_NOT_OK(
+ swapper_child_visitor.SwapChildren((*child_field.get()->type()).fields()));
+ i++;
Review comment:
I need to add an additional argument to `SwapEndianArrayData` since `data->child_data[i]` is passed to `ArrayDataEndianSwapper`. I will do that.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r568628575
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
Review comment:
I see. Thank you for your clarification.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467936469
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,177 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto buffer = const_cast<VALUE_TYPE*>(
Review comment:
As far as I can tell, we only use `const_cast` when invoking third-party (usually C) libraries that didn't bother being `const`-friendly.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
wesm commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r521655583
##########
File path: cpp/src/arrow/type.h
##########
@@ -1604,13 +1605,26 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness {
+ LITTLE = 0,
+ BIG = 1,
+#if ARROW_LITTLE_ENDIAN
+ NATIVE = LITTLE
+#else
+ NATIVE = BIG
+#endif
+};
Review comment:
I will defer to @pitrou or @emkornfield on what to do -- I don't have a lot of bandwidth this month to help with writing PRs
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r465339816
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -449,7 +449,7 @@ Result<std::shared_ptr<RecordBatch>> LoadRecordBatchSubset(
const flatbuf::RecordBatch* metadata, const std::shared_ptr<Schema>& schema,
const std::vector<bool>& inclusion_mask, const DictionaryMemo* dictionary_memo,
const IpcReadOptions& options, MetadataVersion metadata_version,
- Compression::type compression, io::RandomAccessFile* file) {
+ Compression::type compression, io::RandomAccessFile* file, bool swap_endian = false) {
Review comment:
`force_native_endian` naming is OK to me. (It should be `false` by default.)
`use_native_endian` may also be good because we already have `use_threads`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #7507: ARROW-8797: [C++] [WIP] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r443832986
##########
File path: cpp/src/arrow/ipc/read_write_test.cc
##########
@@ -409,6 +414,32 @@ class IpcTestFixture : public io::MemoryMapFixture, public ExtensionTypesMixin {
CheckRoundtrip(*batch, options, IpcReadOptions::Defaults(), buffer_size);
}
+ void CheckRoundTrip(const RecordBatch& src_batch,
+ const RecordBatch& expected_batch,
+ IpcWriteOptions options = IpcWriteOptions::Defaults(),
+ IpcReadOptions read_options = IpcReadOptions::Defaults(),
+ int64_t buffer_size = 1 << 20) {
+ std::stringstream ss;
+ ss << "test-write-row-batch-" << g_file_number++;
+ ASSERT_OK_AND_ASSIGN(mmap_,
+ io::MemoryMapFixture::InitMemoryMap(buffer_size, ss.str()));
+
+ DictionaryMemo dictionary_memo;
+
+ std::shared_ptr<Schema> schema_result;
+ DoSchemaRoundTrip(*src_batch.schema(), &dictionary_memo, &schema_result);
+ ASSERT_TRUE(expected_batch.schema()->Equals(*schema_result));
Review comment:
Should we always convert schema's endianness to native endian?
I think that we should keep the original schema's endianness for serialization.
Did we discuss this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-775141706
Thank you. Ping you later.
I have to add test cases for review while I addressed all comments in the source files.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r522218040
##########
File path: cpp/src/arrow/type.h
##########
@@ -1604,13 +1605,26 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness {
+ LITTLE = 0,
+ BIG = 1,
+#if ARROW_LITTLE_ENDIAN
+ NATIVE = LITTLE
+#else
+ NATIVE = BIG
+#endif
+};
Review comment:
@pitrou I see. It will be another set of PRs. This PR will not address to add an endian field to `DataType`.
After the set of PRs will be available, another PR for this feature will support endian fields in `DataType`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou closed pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou closed pull request #7507:
URL: https://github.com/apache/arrow/pull/7507
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r575853573
##########
File path: ci/scripts/integration_arrow.sh
##########
@@ -24,9 +24,16 @@ source_dir=${1}/cpp
build_dir=${2}/cpp
gold_dir_0_14_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.14.1
gold_dir_0_17_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.17.1
+gold_dir_1_0_0_le=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-littleendian
+gold_dir_1_0_0_be=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-bigendian
Review comment:
If it takes too long, we can drop it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-774403485
> You mean even the little-endian cases fail on Java?
Yes, now failures may occur on Java. I will work for this kind on Java after getting the consensus in ML.
(Strangely, I cannot put the comment to [this comment](https://github.com/apache/arrow/pull/7507#discussion_r562063359))
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r476484988
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,177 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto buffer = const_cast<VALUE_TYPE*>(
Review comment:
Well, I still see a `const_cast` in `SwapOffset`. Otherwise the approach looks fine :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r465269890
##########
File path: cpp/src/arrow/type.h
##########
@@ -1600,6 +1611,12 @@ class ARROW_EXPORT Schema : public detail::Fingerprintable,
bool Equals(const Schema& other, bool check_metadata = false) const;
bool Equals(const std::shared_ptr<Schema>& other, bool check_metadata = false) const;
+ /// \brief Set platform-native endianness in the schema
+ void setNativeEndianness();
Review comment:
I see. Done using `WithNativeEndianness()`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-725211378
Thank you, I will address style issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r521499870
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -664,14 +690,15 @@ Result<std::shared_ptr<RecordBatch>> ReadRecordBatch(
std::shared_ptr<Schema> out_schema;
// Empty means do not use
std::vector<bool> inclusion_mask;
- RETURN_NOT_OK(GetInclusionMaskAndOutSchema(schema, options.included_fields,
+ DictionaryKind kind;
Review comment:
Yes, it is not used, This is a dummy variable to allocate an IpcReadContext().
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #7507: ARROW-8797: [C++] [WIP] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r445184425
##########
File path: cpp/src/arrow/ipc/read_write_test.cc
##########
@@ -427,6 +469,14 @@ class TestIpcRoundTrip : public ::testing::TestWithParam<MakeRecordBatch*>,
void TearDown() { IpcTestFixture::TearDown(); }
};
+class TestIpcDifferentEndian
+ : public ::testing::TestWithParam<std::tuple<MakeRecordBatch*, MakeRecordBatch*>>,
+ public IpcTestFixture {
+ public:
+ void SetUp() { IpcTestFixture::SetUp(); }
+ void TearDown() { IpcTestFixture::TearDown(); }
+};
Review comment:
I'm not sure how difficult to implement this idea...
How about enhancing existing `IpcTestFixture` instead of adding a new fixture?
```diff
diff --git a/cpp/src/arrow/ipc/read_write_test.cc b/cpp/src/arrow/ipc/read_write_test.cc
index 923670834..1e03472b8 100644
--- a/cpp/src/arrow/ipc/read_write_test.cc
+++ b/cpp/src/arrow/ipc/read_write_test.cc
@@ -397,6 +397,9 @@ class IpcTestFixture : public io::MemoryMapFixture, public ExtensionTypesMixin {
ASSERT_OK_AND_ASSIGN(result, DoLargeRoundTrip(batch, /*zero_data=*/true));
CheckReadResult(*result, batch);
+
+ ASSERT_OK_AND_ASSIGN(result, DoEndiannessRoundTrip(batch));
+ CheckReadResult(*result, batch);
}
void CheckRoundtrip(const std::shared_ptr<Array>& array,
IpcWriteOptions options = IpcWriteOptions::Defaults(),
```
`DoEndiannessRoundTrip()` will:
* create a nonnative endianness record batch by swapping the given record batch's endianness
* We need to implement this endianness conversion feature but how difficult...?
* run `DoStandardRoundTrip()` against the nonnative endianness record batch
BTW, should we always convert endianness in IPC? If users just relay data (A (little, generate data) -> B (big, not touch data) -> C (little, process data)), it'll be inefficient.
Or should we convert endianness explicitly out of IPC by providing `Result<std::shared_ptr<RecordBatch>> RecordBatch::EnsureEndianness(Endianness)` or something?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r465476936
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -449,7 +449,7 @@ Result<std::shared_ptr<RecordBatch>> LoadRecordBatchSubset(
const flatbuf::RecordBatch* metadata, const std::shared_ptr<Schema>& schema,
const std::vector<bool>& inclusion_mask, const DictionaryMemo* dictionary_memo,
const IpcReadOptions& options, MetadataVersion metadata_version,
- Compression::type compression, io::RandomAccessFile* file) {
+ Compression::type compression, io::RandomAccessFile* file, bool swap_endian = false) {
Review comment:
Can we remove `swap_endian` and `swap_endian_` by using `IpcReadOptions::use_native_endian`?
Should we keep `swap_endian` and `swap_endian_`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r468193502
##########
File path: cpp/src/arrow/ipc/options.h
##########
@@ -94,6 +94,11 @@ struct ARROW_EXPORT IpcReadOptions {
/// like decompression
bool use_threads = true;
+ /// \brief Convert endian of data element to platform-native endianness
+ /// if the endianness of the received schema is not equal to
+ /// platform-native endianness
+ bool use_native_endian = true;
Review comment:
Sorry. I suggested the `use_native_endian` name but it's not a good name...
It looks like that "if use_native_endian is false, the read data uses non native endian".
How about renaming it to `ensure_native_endian` or something?
##########
File path: cpp/src/arrow/type.h
##########
@@ -1582,13 +1583,26 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness {
+ LITTLE = 0,
+ BIG = 1,
+#if ARROW_LITTLE_ENDIAN
+ NATIVE = LITTLE
+#else
+ NATIVE = BIG
+#endif
+};
+
/// \class Schema
/// \brief Sequence of arrow::Field objects describing the columns of a record
/// batch or table data structure
class ARROW_EXPORT Schema : public detail::Fingerprintable,
public util::EqualityComparable<Schema>,
public util::ToStringOstreamable<Schema> {
public:
+ explicit Schema(std::vector<std::shared_ptr<Field>> fields, Endianness endianness,
+ std::shared_ptr<const KeyValueMetadata> metadata = NULLPTR);
Review comment:
Here are use cases I imagine:
* Router: It doesn't process any Arrow data and just passes the received Arrow data to another system. In this case, it's efficient that we allow non-native endianness in memory.
* Saver: It doesn't process any Arrow data and just saves the received Arrow data to filesystem. For example, it saves the received Arrow data to S3 as is.
Does it make sense?
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -449,7 +449,7 @@ Result<std::shared_ptr<RecordBatch>> LoadRecordBatchSubset(
const flatbuf::RecordBatch* metadata, const std::shared_ptr<Schema>& schema,
const std::vector<bool>& inclusion_mask, const DictionaryMemo* dictionary_memo,
const IpcReadOptions& options, MetadataVersion metadata_version,
- Compression::type compression, io::RandomAccessFile* file) {
+ Compression::type compression, io::RandomAccessFile* file, bool swap_endian = false) {
Review comment:
I see.
How about adding a new simple internal struct (`struct IpcReadContext`?) for organizing options (`IpcReadOptions`, `DictionaryMemo` and `swap_endian` for now)?
I think that many arguments API isn't good to maintain.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r465505197
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -449,7 +449,7 @@ Result<std::shared_ptr<RecordBatch>> LoadRecordBatchSubset(
const flatbuf::RecordBatch* metadata, const std::shared_ptr<Schema>& schema,
const std::vector<bool>& inclusion_mask, const DictionaryMemo* dictionary_memo,
const IpcReadOptions& options, MetadataVersion metadata_version,
- Compression::type compression, io::RandomAccessFile* file) {
+ Compression::type compression, io::RandomAccessFile* file, bool swap_endian = false) {
Review comment:
I think we need to keep `swap_endian` and `swap_endian_`.
IMHO, this is because `IpcReadOptions::use_native_endian` is a flag set by a user to determine whether the endianness of the schema will be changed or not if the endianness of the schema is not equal to the platform-native endianness.
`swap_endian_` is to show we need to actually swap endianness of each element.
For example, there is a case that `IpcReadOptions::use_native_endian=true` and `swap_endian_=false` if x86 platform gets the schema with the endianness = LITTLE.
Or, can we use `IpcReadOptions::use_native_endian` as a working area to show we need to actually swap endianness of each element?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r532786362
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -107,6 +108,23 @@ Status InvalidMessageType(MessageType expected, MessageType actual) {
// ----------------------------------------------------------------------
// Record batch read path
+/// \brief Structure to keep common arguments to be passed
+struct IpcReadContext {
+ IpcReadContext(DictionaryMemo* memo, const IpcReadOptions& option, DictionaryKind* kind,
+ bool swap)
+ : dictionary_memo(memo), options(option), kind(kind), swap_endian(swap) {}
+
+ DictionaryMemo* dictionary_memo;
+
+ const IpcReadOptions& options;
+
+ DictionaryKind* kind;
Review comment:
Based on [this suggestion](https://github.com/apache/arrow/pull/7507#discussion_r513782671), I dropped `kind` from `IpcReadContext`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r522162456
##########
File path: cpp/src/arrow/type.h
##########
@@ -1604,13 +1605,26 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness {
+ LITTLE = 0,
+ BIG = 1,
+#if ARROW_LITTLE_ENDIAN
+ NATIVE = LITTLE
+#else
+ NATIVE = BIG
+#endif
+};
Review comment:
@kiszk Indeed, I think we'll have to bit the bullet and add an endian field to `DataType`. Be sure to update other dependent methods (fingerprint computation, equality, tostring...) and add tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r521123817
##########
File path: cpp/src/arrow/type.h
##########
@@ -1604,13 +1605,26 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness {
+ LITTLE = 0,
+ BIG = 1,
+#if ARROW_LITTLE_ENDIAN
+ NATIVE = LITTLE
+#else
+ NATIVE = BIG
+#endif
+};
Review comment:
@wesm Good point. To have an endianness flag in the Schema causes an inefficiency that passes this property as an argument.
Do you want me to add an endian flag to `DataType` at https://github.com/apache/arrow/blob/master/cpp/src/arrow/type.h#L176 in this PR?
cc @pitrou
For my reference, there is an API in Numpy at https://numpy.org/doc/stable/reference/generated/numpy.dtype.byteorder.html.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7507: ARROW-8797: [C++] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-657343188
Yes you need to use the `--gold-dirs` argument
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r563899594
##########
File path: cpp/src/arrow/array/util.h
##########
@@ -56,6 +56,14 @@ Result<std::shared_ptr<Array>> MakeArrayFromScalar(
namespace internal {
+/// \brief Swap endian of each element in a generic ArrayData
+/// \param[in] data the array contents
+/// \param[in] type the array type
+/// \return the resulting Array instance whose elements were swapped
+ARROW_EXPORT
+Result<std::shared_ptr<ArrayData>> SwapEndianArrayData(
+ std::shared_ptr<ArrayData>& data, const std::shared_ptr<DataType>& type);
+
/// Given a number of ArrayVectors, treat each ArrayVector as the
Review comment:
Thank you for your comments. I will address it.
Regarding the test, there are only integration tests as discussed at https://github.com/apache/arrow/pull/7507#issuecomment-657310102. I will add tests for `SwapEndianArrayData`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r576988002
##########
File path: cpp/src/arrow/array/array_test.cc
##########
@@ -2598,4 +2599,325 @@ TEST(TestRechunkArraysConsistently, Plain) {
}
}
+// ----------------------------------------------------------------------
+// Test SwapEndianArrayData
+
+/// \brief Indicate if fields are equals.
+///
+/// \param[in] target ArrayData to be converted and tested
+/// \param[in] expected result ArrayData
+void AssertArrayDataEqualsWithSwapEndian(const std::shared_ptr<ArrayData>& target,
+ const std::shared_ptr<ArrayData>& expected) {
+ auto swap_array = MakeArray(*::arrow::internal::SwapEndianArrayData(target));
+ auto expected_array = MakeArray(expected);
+ ASSERT_ARRAYS_EQUAL(*swap_array, *expected_array);
+ ASSERT_OK(swap_array->ValidateFull());
+}
+
+TEST(TestSwapEndianArrayData, PrimitiveType) {
+ auto null_buffer = Buffer::FromString("\xff");
+ auto data_int_buffer = Buffer::FromString("01234567");
+
+ auto data = ArrayData::Make(null(), 0, {nullptr}, 0);
+ auto expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(boolean(), 8, {null_buffer, data_int_buffer}, 0);
+ expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(int8(), 8, {null_buffer, data_int_buffer}, 0);
+ expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(uint16(), 4, {null_buffer, data_int_buffer}, 0);
+ auto data_int16_buffer = Buffer::FromString("10325476");
+ expected_data = ArrayData::Make(uint16(), 4, {null_buffer, data_int16_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(int32(), 2, {null_buffer, data_int_buffer}, 0);
+ auto data_int32_buffer = Buffer::FromString("32107654");
+ expected_data = ArrayData::Make(int32(), 2, {null_buffer, data_int32_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(uint64(), 1, {null_buffer, data_int_buffer}, 0);
+ auto data_int64_buffer = Buffer::FromString("76543210");
+ expected_data = ArrayData::Make(uint64(), 1, {null_buffer, data_int64_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_16byte_buffer = Buffer::FromString("0123456789abcdef");
+ data = ArrayData::Make(decimal128(38, 10), 1, {null_buffer, data_16byte_buffer});
+ auto data_decimal128_buffer = Buffer::FromString("fedcba9876543210");
+ expected_data =
+ ArrayData::Make(decimal128(38, 10), 1, {null_buffer, data_decimal128_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_32byte_buffer = Buffer::FromString("0123456789abcdef123456789ABCDEF0");
+ data = ArrayData::Make(decimal256(76, 20), 1, {null_buffer, data_32byte_buffer});
+ auto data_decimal256_buffer = Buffer::FromString("0FEDCBA987654321fedcba9876543210");
+ expected_data =
+ ArrayData::Make(decimal256(76, 20), 1, {null_buffer, data_decimal256_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_float_buffer = Buffer::FromString("01200560");
+ data = ArrayData::Make(float32(), 2, {null_buffer, data_float_buffer}, 0);
+ auto data_float32_buffer = Buffer::FromString("02100650");
+ expected_data = ArrayData::Make(float32(), 2, {null_buffer, data_float32_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(float64(), 1, {null_buffer, data_float_buffer});
+ auto data_float64_buffer = Buffer::FromString("06500210");
+ expected_data = ArrayData::Make(float64(), 1, {null_buffer, data_float64_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ // With offset > 0
+ data =
+ ArrayData::Make(int64(), 1, {null_buffer, data_int_buffer}, kUnknownNullCount, 1);
+ ASSERT_RAISES(Invalid, ::arrow::internal::SwapEndianArrayData(data));
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffers(const std::shared_ptr<ArrayData>& data,
+ const int32_t buffer_index,
+ const std::vector<uint8_t>& buffer_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ test_data->buffers[buffer_index] =
+ std::make_shared<Buffer>(buffer_data.data(), buffer_data.size());
+ return test_data;
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffersInChild(const std::shared_ptr<ArrayData>& data,
+ const int32_t child_index,
+ const std::vector<uint8_t>& child_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ // assume updating only buffer[1] in child_data
+ auto child_array_data =
+ std::make_shared<ArrayData>(*test_data->child_data[child_index]);
+ child_array_data->buffers[1] =
+ std::make_shared<Buffer>(child_data.data(), child_data.size());
+ test_data->child_data[child_index] = child_array_data;
+ return test_data;
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffersInDictionary(
+ const std::shared_ptr<ArrayData>& data, const int32_t buffer_index,
+ const std::vector<uint8_t>& buffer_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ auto dict_array_data = std::make_shared<ArrayData>(*test_data->dictionary);
+ dict_array_data->buffers[buffer_index] =
+ std::make_shared<Buffer>(buffer_data.data(), buffer_data.size());
+ test_data->dictionary = dict_array_data;
+ return test_data;
+}
+
+TEST(TestSwapEndianArrayData, NonPrimitiveType) {
+ auto array = ArrayFromJSON(binary(), R"(["0123", null, "45"])");
+ const std::vector<uint8_t> offset1 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ auto expected_data = array->data();
+ auto test_data = ReplaceBuffers(expected_data, 1, offset1);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(large_binary(), R"(["01234", null, "567"])");
+ const std::vector<uint8_t> offset2 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5,
+ 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 8};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0,
+ 5, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset2);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(fixed_size_binary(3), R"(["012", null, "345"])");
+ expected_data = array->data();
+ AssertArrayDataEqualsWithSwapEndian(expected_data, expected_data);
+
+ array = ArrayFromJSON(utf8(), R"(["ABCD", null, "EF"])");
+ const std::vector<uint8_t> offset4 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset4);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(large_utf8(), R"(["ABCDE", null, "FGH"])");
+ const std::vector<uint8_t> offset5 =
Review comment:
Same as `offset2`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk edited a comment on pull request #7507: ARROW-8797: [C++] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
kiszk edited a comment on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-657310102
@wesm Thank you for your clarification. I thought that I will check in binary files and write C++ files to validate the result to be read.
Now, I realized the integration test `archery integration ...` will kick `arrow-json-integration-test --integration --mode VALIDATE --json ... --arrow ...`. I will stick with ``archery integration ...`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-717381980
@pitrou @wesm Could you please review this to support ipc between different endian clients?
[The document](https://github.com/apache/arrow/pull/8374) has been merged.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-748572198
@wesm @pitrou Sorry for being late. Could you please review this?
I addressed all of the comments except [this](https://github.com/apache/arrow/pull/7507#discussion_r513725869) that will be addressed by another PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r465252291
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -449,7 +449,7 @@ Result<std::shared_ptr<RecordBatch>> LoadRecordBatchSubset(
const flatbuf::RecordBatch* metadata, const std::shared_ptr<Schema>& schema,
const std::vector<bool>& inclusion_mask, const DictionaryMemo* dictionary_memo,
const IpcReadOptions& options, MetadataVersion metadata_version,
- Compression::type compression, io::RandomAccessFile* file) {
+ Compression::type compression, io::RandomAccessFile* file, bool swap_endian = false) {
Review comment:
From a technical perspective, we can naively put `native_endian` into `IpcReadOptions` in `reader.cc`.
From a semantics perspective, now, `native_endian` is to show the state on whether the endianness in the schema is equal to the endianness of the CPU. Since values in `IpcReadOptions` can be set by users, its flag should be like `force_naive_endian` to ask to change the endianness of the schema.
If we put `force_naive_endian` in `IpcReadOptions`, we need to keep `native_endian` somewhere.
What do you think?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467936947
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,177 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto buffer = const_cast<VALUE_TYPE*>(
Review comment:
In this case anyway, we don't know what kind of buffer we are dealing with (it could be a private memory buffer, it could be a memory-mapped file, it could be shared with another library...).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-775137661
@kiszk Feel free to ping when this is ready for review again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
wesm commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r513785728
##########
File path: cpp/src/arrow/array/util.h
##########
@@ -37,6 +37,11 @@ namespace arrow {
ARROW_EXPORT
std::shared_ptr<Array> MakeArray(const std::shared_ptr<ArrayData>& data);
+/// \brief Swap endian of each element in a generic ArrayData
+/// \param[in] data the array contents to be swapped
+ARROW_EXPORT
+void SwapEndianArrayData(std::shared_ptr<ArrayData>& data);
Review comment:
Also since this allocates memory it needs to be able to fail
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r576987861
##########
File path: cpp/src/arrow/array/array_test.cc
##########
@@ -2598,4 +2599,325 @@ TEST(TestRechunkArraysConsistently, Plain) {
}
}
+// ----------------------------------------------------------------------
+// Test SwapEndianArrayData
+
+/// \brief Indicate if fields are equals.
+///
+/// \param[in] target ArrayData to be converted and tested
+/// \param[in] expected result ArrayData
+void AssertArrayDataEqualsWithSwapEndian(const std::shared_ptr<ArrayData>& target,
+ const std::shared_ptr<ArrayData>& expected) {
+ auto swap_array = MakeArray(*::arrow::internal::SwapEndianArrayData(target));
+ auto expected_array = MakeArray(expected);
+ ASSERT_ARRAYS_EQUAL(*swap_array, *expected_array);
+ ASSERT_OK(swap_array->ValidateFull());
+}
+
+TEST(TestSwapEndianArrayData, PrimitiveType) {
+ auto null_buffer = Buffer::FromString("\xff");
+ auto data_int_buffer = Buffer::FromString("01234567");
+
+ auto data = ArrayData::Make(null(), 0, {nullptr}, 0);
+ auto expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(boolean(), 8, {null_buffer, data_int_buffer}, 0);
+ expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(int8(), 8, {null_buffer, data_int_buffer}, 0);
+ expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(uint16(), 4, {null_buffer, data_int_buffer}, 0);
+ auto data_int16_buffer = Buffer::FromString("10325476");
+ expected_data = ArrayData::Make(uint16(), 4, {null_buffer, data_int16_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(int32(), 2, {null_buffer, data_int_buffer}, 0);
+ auto data_int32_buffer = Buffer::FromString("32107654");
+ expected_data = ArrayData::Make(int32(), 2, {null_buffer, data_int32_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(uint64(), 1, {null_buffer, data_int_buffer}, 0);
+ auto data_int64_buffer = Buffer::FromString("76543210");
+ expected_data = ArrayData::Make(uint64(), 1, {null_buffer, data_int64_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_16byte_buffer = Buffer::FromString("0123456789abcdef");
+ data = ArrayData::Make(decimal128(38, 10), 1, {null_buffer, data_16byte_buffer});
+ auto data_decimal128_buffer = Buffer::FromString("fedcba9876543210");
+ expected_data =
+ ArrayData::Make(decimal128(38, 10), 1, {null_buffer, data_decimal128_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_32byte_buffer = Buffer::FromString("0123456789abcdef123456789ABCDEF0");
+ data = ArrayData::Make(decimal256(76, 20), 1, {null_buffer, data_32byte_buffer});
+ auto data_decimal256_buffer = Buffer::FromString("0FEDCBA987654321fedcba9876543210");
+ expected_data =
+ ArrayData::Make(decimal256(76, 20), 1, {null_buffer, data_decimal256_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_float_buffer = Buffer::FromString("01200560");
+ data = ArrayData::Make(float32(), 2, {null_buffer, data_float_buffer}, 0);
+ auto data_float32_buffer = Buffer::FromString("02100650");
+ expected_data = ArrayData::Make(float32(), 2, {null_buffer, data_float32_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(float64(), 1, {null_buffer, data_float_buffer});
+ auto data_float64_buffer = Buffer::FromString("06500210");
+ expected_data = ArrayData::Make(float64(), 1, {null_buffer, data_float64_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ // With offset > 0
+ data =
+ ArrayData::Make(int64(), 1, {null_buffer, data_int_buffer}, kUnknownNullCount, 1);
+ ASSERT_RAISES(Invalid, ::arrow::internal::SwapEndianArrayData(data));
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffers(const std::shared_ptr<ArrayData>& data,
+ const int32_t buffer_index,
+ const std::vector<uint8_t>& buffer_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ test_data->buffers[buffer_index] =
+ std::make_shared<Buffer>(buffer_data.data(), buffer_data.size());
+ return test_data;
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffersInChild(const std::shared_ptr<ArrayData>& data,
+ const int32_t child_index,
+ const std::vector<uint8_t>& child_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ // assume updating only buffer[1] in child_data
+ auto child_array_data =
+ std::make_shared<ArrayData>(*test_data->child_data[child_index]);
+ child_array_data->buffers[1] =
+ std::make_shared<Buffer>(child_data.data(), child_data.size());
+ test_data->child_data[child_index] = child_array_data;
+ return test_data;
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffersInDictionary(
+ const std::shared_ptr<ArrayData>& data, const int32_t buffer_index,
+ const std::vector<uint8_t>& buffer_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
+ auto dict_array_data = std::make_shared<ArrayData>(*test_data->dictionary);
+ dict_array_data->buffers[buffer_index] =
+ std::make_shared<Buffer>(buffer_data.data(), buffer_data.size());
+ test_data->dictionary = dict_array_data;
+ return test_data;
+}
+
+TEST(TestSwapEndianArrayData, NonPrimitiveType) {
+ auto array = ArrayFromJSON(binary(), R"(["0123", null, "45"])");
+ const std::vector<uint8_t> offset1 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6};
+#else
+ {0, 0, 0, 0, 4, 0, 0, 0, 4, 0, 0, 0, 6, 0, 0, 0};
+#endif
+ auto expected_data = array->data();
+ auto test_data = ReplaceBuffers(expected_data, 1, offset1);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(large_binary(), R"(["01234", null, "567"])");
+ const std::vector<uint8_t> offset2 =
+#if ARROW_LITTLE_ENDIAN
+ {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 5,
+ 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0, 8};
+#else
+ {0, 0, 0, 0, 0, 0, 0, 0, 5, 0, 0, 0, 0, 0, 0, 0,
+ 5, 0, 0, 0, 0, 0, 0, 0, 8, 0, 0, 0, 0, 0, 0, 0};
+#endif
+ expected_data = array->data();
+ test_data = ReplaceBuffers(expected_data, 1, offset2);
+ AssertArrayDataEqualsWithSwapEndian(test_data, expected_data);
+
+ array = ArrayFromJSON(fixed_size_binary(3), R"(["012", null, "345"])");
+ expected_data = array->data();
+ AssertArrayDataEqualsWithSwapEndian(expected_data, expected_data);
+
+ array = ArrayFromJSON(utf8(), R"(["ABCD", null, "EF"])");
+ const std::vector<uint8_t> offset4 =
Review comment:
It seems this is the same as `offset1`, no need to declare it again :-)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r567756770
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
Review comment:
You can reuse existing buffers as long as you don't modify the contents.
When you want to modify the contents of a buffer (for example to byte-swap it), you must do a copy.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
Review comment:
You can reuse existing buffers as long as you don't modify their contents.
When you want to modify the contents of a buffer (for example to byte-swap it), you must do a copy.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r464651784
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,186 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Status SwapOffset(const T&, int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ using value_type = typename T::c_type;
+ auto buffer = const_cast<value_type*>(
+ reinterpret_cast<const value_type*>(data_->buffers[index]->data()));
+ // offset has one more element rather than data->length
+ int64_t length = length_ + 1;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ buffer[i] = BitUtil::FromBigEndian(buffer[i]);
+#else
+ buffer[i] = BitUtil::FromLittleEndian(buffer[i]);
+#endif
+ }
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) {
+ Int32Type i32;
+ RETURN_NOT_OK(SwapOffset(i32, index));
+ return Status::OK();
+ }
+
+ Status SwapLargeOffset() {
+ Int64Type i64;
+ RETURN_NOT_OK(SwapOffset(i64, 1));
+ return Status::OK();
+ }
Review comment:
Can we simplify them like the following?
```diff
diff --git a/cpp/src/arrow/array/util.cc b/cpp/src/arrow/array/util.cc
index e2f88016f..6f6d4bc78 100644
--- a/cpp/src/arrow/array/util.cc
+++ b/cpp/src/arrow/array/util.cc
@@ -95,14 +95,13 @@ class ArrayDataEndianSwapper {
return Status::OK();
}
- template <typename T>
- Status SwapOffset(const T&, int index) {
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
if (data_->buffers[index] == nullptr) {
return Status::OK();
}
- using value_type = typename T::c_type;
- auto buffer = const_cast<value_type*>(
- reinterpret_cast<const value_type*>(data_->buffers[index]->data()));
+ auto buffer = const_cast<VALUE_TYPE*>(
+ reinterpret_cast<const VALUE_TYPE*>(data_->buffers[index]->data()));
// offset has one more element rather than data->length
int64_t length = length_ + 1;
for (int64_t i = 0; i < length; i++) {
@@ -116,15 +115,11 @@ class ArrayDataEndianSwapper {
}
Status SwapSmallOffset(int index = 1) {
- Int32Type i32;
- RETURN_NOT_OK(SwapOffset(i32, index));
- return Status::OK();
+ return SwapOffset<int32_t>(index);
}
Status SwapLargeOffset() {
- Int64Type i64;
- RETURN_NOT_OK(SwapOffset(i64, 1));
- return Status::OK();
+ return SwapOffset<int64_t>(1);
}
template <typename T>
```
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -449,7 +449,7 @@ Result<std::shared_ptr<RecordBatch>> LoadRecordBatchSubset(
const flatbuf::RecordBatch* metadata, const std::shared_ptr<Schema>& schema,
const std::vector<bool>& inclusion_mask, const DictionaryMemo* dictionary_memo,
const IpcReadOptions& options, MetadataVersion metadata_version,
- Compression::type compression, io::RandomAccessFile* file) {
+ Compression::type compression, io::RandomAccessFile* file, bool swap_endian = false) {
Review comment:
I'm not sure `swap_endian` is useful.
`native_endian` may be more useful.
##########
File path: cpp/src/arrow/type.h
##########
@@ -1582,13 +1583,23 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness { LITTLE, BIG };
+#if ARROW_LITTLE_ENDIAN
+#define NATIVE_ENDIANNESS Endianness::LITTLE
+#else
+#define NATIVE_ENDIANNESS Endianness::BIG
Review comment:
Could you hide this from public API?
It doesn't have `ARROW_` prefix.
##########
File path: cpp/src/arrow/type.h
##########
@@ -1600,6 +1611,12 @@ class ARROW_EXPORT Schema : public detail::Fingerprintable,
bool Equals(const Schema& other, bool check_metadata = false) const;
bool Equals(const std::shared_ptr<Schema>& other, bool check_metadata = false) const;
+ /// \brief Set platform-native endianness in the schema
+ void setNativeEndianness();
Review comment:
`Schema` is read-only.
How about adding `withNativeEndianness()` or `withEndianness()` like `withMetadata()` instead?
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -449,7 +449,7 @@ Result<std::shared_ptr<RecordBatch>> LoadRecordBatchSubset(
const flatbuf::RecordBatch* metadata, const std::shared_ptr<Schema>& schema,
const std::vector<bool>& inclusion_mask, const DictionaryMemo* dictionary_memo,
const IpcReadOptions& options, MetadataVersion metadata_version,
- Compression::type compression, io::RandomAccessFile* file) {
+ Compression::type compression, io::RandomAccessFile* file, bool swap_endian = false) {
Review comment:
Can we put the `swap_endian` to `IpcReadOptions` from function argument?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r521553432
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -699,42 +726,45 @@ Status ReadDictionary(const Buffer& metadata, DictionaryMemo* dictionary_memo,
// Look up the dictionary value type, which must have been added to the
// DictionaryMemo already prior to invoking this function
- ARROW_ASSIGN_OR_RAISE(auto value_type, dictionary_memo->GetDictionaryType(id));
+ ARROW_ASSIGN_OR_RAISE(auto value_type, context.dictionary_memo->GetDictionaryType(id));
// Load the dictionary data from the dictionary batch
ArrayLoader loader(batch_meta, internal::GetMetadataVersion(message->version()),
- options, file);
- const auto dict_data = std::make_shared<ArrayData>();
+ context.options, file);
+ auto dict_data = std::make_shared<ArrayData>();
const Field dummy_field("", value_type);
RETURN_NOT_OK(loader.Load(&dummy_field, dict_data.get()));
if (compression != Compression::UNCOMPRESSED) {
ArrayDataVector dict_fields{dict_data};
- RETURN_NOT_OK(DecompressBuffers(compression, options, &dict_fields));
+ RETURN_NOT_OK(DecompressBuffers(compression, context.options, &dict_fields));
+ }
+
+ // swap endian in dict_data if necessary (swap_endian == true)
+ if (context.swap_endian) {
+ SwapEndianArrayData(dict_data);
}
if (dictionary_batch->isDelta()) {
- if (kind != nullptr) {
- *kind = DictionaryKind::Delta;
+ if (context.kind != nullptr) {
+ *context.kind = DictionaryKind::Delta;
}
- return dictionary_memo->AddDictionaryDelta(id, dict_data);
+ return context.dictionary_memo->AddDictionaryDelta(id, dict_data);
}
ARROW_ASSIGN_OR_RAISE(bool inserted,
- dictionary_memo->AddOrReplaceDictionary(id, dict_data));
- if (kind != nullptr) {
- *kind = inserted ? DictionaryKind::New : DictionaryKind::Replacement;
+ context.dictionary_memo->AddOrReplaceDictionary(id, dict_data));
+ if (context.kind != nullptr) {
+ *context.kind = inserted ? DictionaryKind::New : DictionaryKind::Replacement;
}
Review comment:
nvm, after your suggestion, I added `param` into `IpcReadContext`. This is because `param` has recently added.
I will keep the DictionaryKind as an out param
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-717385965
@pitou Thank you for letting us the status. It is fine with me. Enjoy your vacation 👍
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r521518379
##########
File path: cpp/src/arrow/type.h
##########
@@ -1604,13 +1605,26 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness {
+ LITTLE = 0,
+ BIG = 1,
+#if ARROW_LITTLE_ENDIAN
+ NATIVE = LITTLE
+#else
+ NATIVE = BIG
+#endif
+};
Review comment:
@wesm Or, would it be possible to create a small PR to add an endian field to `DataType`? Then, I will continue to work for this PR after merging your PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] [WIP] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-648320579
Are there any comments about this approach for preparing test cases between different endians? cc @pitrou @wesm
If not, I will prepare other tests (but disabled now) with this approach.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r577078249
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -108,6 +109,20 @@ Status InvalidMessageType(MessageType expected, MessageType actual) {
// ----------------------------------------------------------------------
// Record batch read path
+/// \brief Structure to keep common arguments to be passed
+struct IpcReadContext {
+ IpcReadContext(DictionaryMemo* memo, const IpcReadOptions& option, bool swap)
+ : dictionary_memo(memo), options(option), swap_endian(swap) {}
+
+ DictionaryMemo* dictionary_memo;
+
+ const IpcReadOptions& options;
+
+ /// \brief LoadRecordBatch() or LoadRecordBatchSubset() swaps endianness of elements
+ /// if this flag is true
+ const bool swap_endian;
+};
Review comment:
@pitrou Most use cases of `IpcReadContext`, `ReadRecordBatchInternal()` do not receieve `MetadataVersion` and `Compression::type`.
If we pass default values for these two variables at the constructor, it would be useful to add them. So, we do not need to pass values to `IpcReadContent ipc(...)` before calling ReadRecordBatchInternal().
Do you want to add them while we use default values at the constructor? I am neutral on this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-780603113
The CI failure is unrelated (see https://issues.apache.org/jira/browse/ARROW-11668).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7507: ARROW-8797: [C++] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
wesm commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-657282736
I was suggesting to generate BE and LE files to be put in this directory
https://github.com/apache/arrow-testing/tree/master/data/arrow-ipc-stream/integration
Then you can specify the respective directory of e.g. BE files to be read on an LE platform
I think it's better to stick with the integration test tools for this rather than writing new C++ code
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r532814153
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,204 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ArrayDataEndianSwapper swapper_child_visitor(data_->child_data[i],
+ data_->child_data[i]->length);
+ RETURN_NOT_OK(VisitTypeInline(*child_field.get()->type(), &swapper_child_visitor));
+ RETURN_NOT_OK(
+ swapper_child_visitor.SwapChildren((*child_field.get()->type()).fields()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto data = reinterpret_cast<const VALUE_TYPE*>(data_->buffers[index]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer,
+ AllocateBuffer(data_->buffers[index]->size() + 1));
+ auto new_data = reinterpret_cast<VALUE_TYPE*>(new_buffer->mutable_data());
+ // offset has one more element rather than data->length
+ int64_t length = length_ + 1;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[index] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ Status Visit(const T&) {
+ using value_type = typename T::c_type;
+ auto data = reinterpret_cast<const value_type*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<value_type*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ auto data = reinterpret_cast<const uint32_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint32_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ new_data[idx] = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromBigEndian(data[idx + 1]);
+#else
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromLittleEndian(data[idx + 1]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return Status::OK(); }
+ Status Visit(const Int8Type& type) { return Status::OK(); }
+ Status Visit(const UInt8Type& type) { return Status::OK(); }
+ Status Visit(const FixedSizeBinaryType& type) { return Status::OK(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const SparseUnionType& type) { return Status::OK(); }
+
+ Status Visit(const StringType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeStringType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+ Status Visit(const BinaryType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeBinaryType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const MapType& type) {
Review comment:
@wesm Could you please elaborate on this comment? I cannot cover `MapType` by `ListType` even I looked at `type_trais.h`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r577001270
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,11 +75,222 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(const std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {
+ const std::shared_ptr<DataType>& type = data->type;
+ std::vector<std::shared_ptr<Buffer>> buffers(data->buffers.size(), nullptr);
+ std::vector<std::shared_ptr<ArrayData>> child_data(data->child_data.size(), nullptr);
+ out_ = ArrayData::Make(type, data->length, buffers, child_data, data->null_count, 0);
+ }
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(const std::vector<std::shared_ptr<Field>>& child_fields) {
+ for (size_t i = 0; i < child_fields.size(); i++) {
+ ARROW_ASSIGN_OR_RAISE(out_->child_data[i],
+ internal::SwapEndianArrayData(data_->child_data[i]));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(
+ const std::shared_ptr<Buffer>& in_buffer) {
+ if (sizeof(T) == 1) {
+ // if data size is 1, element is not swapped. We can use the original buffer
+ return in_buffer;
+ }
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ int64_t length = in_buffer->size() / sizeof(T);
+ for (int64_t i = 0; i < length; i++) {
+ out_data[i] = BitUtil::ByteSwap(in_data[i]);
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffsets(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ out_->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // Except union, offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE(out_->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index]));
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE(out_->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1]));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ length = data_->buffers[1]->size() / (sizeof(uint64_t) * 2);
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ out_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ length = data_->buffers[1]->size() / (sizeof(uint64_t) * 4);
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ out_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE(out_->buffers[1], ByteSwapBuffer<uint32_t>(data_->buffers[1]));
+ return Status::OK();
+ }
+
+ Status ReuseDataBuffer() {
+ out_->buffers[1] = data_->buffers[1];
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return ReuseDataBuffer(); }
+ Status Visit(const Int8Type& type) { return ReuseDataBuffer(); }
+ Status Visit(const UInt8Type& type) { return ReuseDataBuffer(); }
+ Status Visit(const FixedSizeBinaryType& type) { return ReuseDataBuffer(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const UnionType& type) {
+ out_->buffers[1] = data_->buffers[1];
+ if (type.mode() == UnionMode::DENSE) {
+ RETURN_NOT_OK(SwapOffsets<int32_t>(2));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<BinaryType, T>::value || std::is_same<StringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapOffsets<int32_t>(1));
+ out_->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<LargeBinaryType, T>::value ||
+ std::is_same<LargeStringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapOffsets<int64_t>(1));
+ out_->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapOffsets<int32_t>(1));
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapOffsets<int64_t>(1));
+ return Status::OK();
+ }
+
+ Status Visit(const DictionaryType& type) {
+ RETURN_NOT_OK(SwapType(*type.index_type()));
+ // dictionary was already swapped in ReadDictionary() in ipc/reader.cc
+ out_->dictionary = data_->dictionary;
+ return Status::OK();
+ }
+
+ Status Visit(const ExtensionType& type) {
+ RETURN_NOT_OK(SwapType(*type.storage_type()));
+ // dictionary was already swapped in ReadDictionary() in ipc/reader.cc
+ out_->dictionary = data_->dictionary;
+ return Status::OK();
+ }
+
+ const std::shared_ptr<ArrayData>& data_;
+ int64_t length_;
+ std::shared_ptr<ArrayData> out_;
+};
+
+} // namespace
+
+namespace internal {
+
+Result<std::shared_ptr<ArrayData>> SwapEndianArrayData(
+ const std::shared_ptr<ArrayData>& data) {
+ if (data->offset != 0) {
+ return Status::Invalid("Unsupported data format: data.offset != 0");
+ }
+ const std::shared_ptr<DataType>& type = data->type;
+ ArrayDataEndianSwapper swapper_visitor(data, data->length);
+ DCHECK_OK(VisitTypeInline(*type, &swapper_visitor));
+ DCHECK_OK(swapper_visitor.SwapChildren((*type).fields()));
Review comment:
Also, why don't you call `SwapType` directly?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r577002938
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -108,6 +109,20 @@ Status InvalidMessageType(MessageType expected, MessageType actual) {
// ----------------------------------------------------------------------
// Record batch read path
+/// \brief Structure to keep common arguments to be passed
+struct IpcReadContext {
+ IpcReadContext(DictionaryMemo* memo, const IpcReadOptions& option, bool swap)
+ : dictionary_memo(memo), options(option), swap_endian(swap) {}
+
+ DictionaryMemo* dictionary_memo;
+
+ const IpcReadOptions& options;
+
+ /// \brief LoadRecordBatch() or LoadRecordBatchSubset() swaps endianness of elements
+ /// if this flag is true
+ const bool swap_endian;
+};
Review comment:
Would it be useful to add `MetadataVersion` and `Compression::type` here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r576994839
##########
File path: cpp/src/arrow/array/data.cc
##########
@@ -130,6 +130,33 @@ int64_t ArrayData::GetNullCount() const {
return precomputed;
}
+bool ArrayData::Equals(const ArrayData& other) const {
Review comment:
I don't think this is right. Equality is more complicated than that.
Instead of adding this, I would suggest you simply use `MakeArray` in tests and then call `AssertArraysEqual`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-755108332
@pitrou gentle ping
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7507: [WIP] ARROW-8797: [C++] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-647166030
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
Could you open an issue for this pull request on JIRA?
https://issues.apache.org/jira/browse/ARROW
Then could you also rename pull request title in the following format?
ARROW-${JIRA_ID}: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - How to contribute patches](https://arrow.apache.org/docs/developers/contributing.html#how-to-contribute-patches)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r571378590
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
Review comment:
null bitmap is handled at [here](https://github.com/apache/arrow/pull/7507/files#diff-c3b0484ad8586ff46fa035d446a7d1c3a30cd35d13cd05678c99814938e07d5bR287).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-740006741
I'll take a look at this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r465423744
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -449,7 +449,7 @@ Result<std::shared_ptr<RecordBatch>> LoadRecordBatchSubset(
const flatbuf::RecordBatch* metadata, const std::shared_ptr<Schema>& schema,
const std::vector<bool>& inclusion_mask, const DictionaryMemo* dictionary_memo,
const IpcReadOptions& options, MetadataVersion metadata_version,
- Compression::type compression, io::RandomAccessFile* file) {
+ Compression::type compression, io::RandomAccessFile* file, bool swap_endian = false) {
Review comment:
I see. Thus, we will have `use_native_endian` in `IpcReadOptions`. In addition, keep `swap_endian` and `swap_endian_` as now.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] [WIP] Create test to receive RecordBatch for different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r443306549
##########
File path: cpp/src/arrow/ipc/read_write_test.cc
##########
@@ -1289,6 +1338,10 @@ INSTANTIATE_TEST_SUITE_P(StreamDecoderSmallChunksRoundTripTests,
INSTANTIATE_TEST_SUITE_P(StreamDecoderLargeChunksRoundTripTests,
TestStreamDecoderLargeChunks, BATCH_CASES());
+// TODO(kiszk): Enable this test since it is not successful now
Review comment:
Now, I intetionally comment out this test since this type of communication is not supported.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467917391
##########
File path: cpp/src/arrow/type.h
##########
@@ -29,6 +29,7 @@
#include "arrow/result.h"
#include "arrow/type_fwd.h" // IWYU pragma: export
+#include "arrow/util/bit_util.h"
Review comment:
If we just need endian-related definitions (such as `ARROW_LITTLE_ENDIAN` and `ARROW_BIG_ENDIAN`), then they can be moved to a separate `util/endian.h`, IMHO.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r576985802
##########
File path: cpp/src/arrow/array/array_test.cc
##########
@@ -2598,4 +2599,325 @@ TEST(TestRechunkArraysConsistently, Plain) {
}
}
+// ----------------------------------------------------------------------
+// Test SwapEndianArrayData
+
+/// \brief Indicate if fields are equals.
+///
+/// \param[in] target ArrayData to be converted and tested
+/// \param[in] expected result ArrayData
+void AssertArrayDataEqualsWithSwapEndian(const std::shared_ptr<ArrayData>& target,
+ const std::shared_ptr<ArrayData>& expected) {
+ auto swap_array = MakeArray(*::arrow::internal::SwapEndianArrayData(target));
+ auto expected_array = MakeArray(expected);
+ ASSERT_ARRAYS_EQUAL(*swap_array, *expected_array);
+ ASSERT_OK(swap_array->ValidateFull());
+}
+
+TEST(TestSwapEndianArrayData, PrimitiveType) {
+ auto null_buffer = Buffer::FromString("\xff");
+ auto data_int_buffer = Buffer::FromString("01234567");
+
+ auto data = ArrayData::Make(null(), 0, {nullptr}, 0);
+ auto expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(boolean(), 8, {null_buffer, data_int_buffer}, 0);
+ expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(int8(), 8, {null_buffer, data_int_buffer}, 0);
+ expected_data = data;
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(uint16(), 4, {null_buffer, data_int_buffer}, 0);
+ auto data_int16_buffer = Buffer::FromString("10325476");
+ expected_data = ArrayData::Make(uint16(), 4, {null_buffer, data_int16_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(int32(), 2, {null_buffer, data_int_buffer}, 0);
+ auto data_int32_buffer = Buffer::FromString("32107654");
+ expected_data = ArrayData::Make(int32(), 2, {null_buffer, data_int32_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(uint64(), 1, {null_buffer, data_int_buffer}, 0);
+ auto data_int64_buffer = Buffer::FromString("76543210");
+ expected_data = ArrayData::Make(uint64(), 1, {null_buffer, data_int64_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_16byte_buffer = Buffer::FromString("0123456789abcdef");
+ data = ArrayData::Make(decimal128(38, 10), 1, {null_buffer, data_16byte_buffer});
+ auto data_decimal128_buffer = Buffer::FromString("fedcba9876543210");
+ expected_data =
+ ArrayData::Make(decimal128(38, 10), 1, {null_buffer, data_decimal128_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_32byte_buffer = Buffer::FromString("0123456789abcdef123456789ABCDEF0");
+ data = ArrayData::Make(decimal256(76, 20), 1, {null_buffer, data_32byte_buffer});
+ auto data_decimal256_buffer = Buffer::FromString("0FEDCBA987654321fedcba9876543210");
+ expected_data =
+ ArrayData::Make(decimal256(76, 20), 1, {null_buffer, data_decimal256_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ auto data_float_buffer = Buffer::FromString("01200560");
+ data = ArrayData::Make(float32(), 2, {null_buffer, data_float_buffer}, 0);
+ auto data_float32_buffer = Buffer::FromString("02100650");
+ expected_data = ArrayData::Make(float32(), 2, {null_buffer, data_float32_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ data = ArrayData::Make(float64(), 1, {null_buffer, data_float_buffer});
+ auto data_float64_buffer = Buffer::FromString("06500210");
+ expected_data = ArrayData::Make(float64(), 1, {null_buffer, data_float64_buffer}, 0);
+ AssertArrayDataEqualsWithSwapEndian(data, expected_data);
+
+ // With offset > 0
+ data =
+ ArrayData::Make(int64(), 1, {null_buffer, data_int_buffer}, kUnknownNullCount, 1);
+ ASSERT_RAISES(Invalid, ::arrow::internal::SwapEndianArrayData(data));
+}
+
+std::shared_ptr<ArrayData> ReplaceBuffers(const std::shared_ptr<ArrayData>& data,
+ const int32_t buffer_index,
+ const std::vector<uint8_t>& buffer_data) {
+ const auto test_data = std::make_shared<ArrayData>(*data);
Review comment:
Better: `auto test_data = data->Copy();`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r576988294
##########
File path: ci/scripts/integration_arrow.sh
##########
@@ -24,9 +24,16 @@ source_dir=${1}/cpp
build_dir=${2}/cpp
gold_dir_0_14_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.14.1
gold_dir_0_17_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.17.1
+gold_dir_1_0_0_le=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-littleendian
+gold_dir_1_0_0_be=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-bigendian
pip install -e $arrow_dir/dev/archery
archery integration --with-all --run-flight \
--gold-dirs=$gold_dir_0_14_1 \
--gold-dirs=$gold_dir_0_17_1 \
+
+# TODO: support other languages
+archery integration --with-cpp=1 --run-flight \
Review comment:
Oh, sorry, I should mention you [here](https://github.com/apache/arrow/pull/7507#issuecomment-774403485)
The answer is yes. It fails on Java at least.
Sure, I will update `runner.py` instead of `integration_arrow.sh`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r577078249
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -108,6 +109,20 @@ Status InvalidMessageType(MessageType expected, MessageType actual) {
// ----------------------------------------------------------------------
// Record batch read path
+/// \brief Structure to keep common arguments to be passed
+struct IpcReadContext {
+ IpcReadContext(DictionaryMemo* memo, const IpcReadOptions& option, bool swap)
+ : dictionary_memo(memo), options(option), swap_endian(swap) {}
+
+ DictionaryMemo* dictionary_memo;
+
+ const IpcReadOptions& options;
+
+ /// \brief LoadRecordBatch() or LoadRecordBatchSubset() swaps endianness of elements
+ /// if this flag is true
+ const bool swap_endian;
+};
Review comment:
Most use cases of `IpcReadContext`, `ReadRecordBatchInternal()` do not receieve `MetadataVersion` and `Compression::type`.
If we pass default values for these two variables at the constructor, it would be useful to add them. So, we do not need to pass values to `IpcReadContent ipc(...)` before calling ReadRecordBatchInternal().
Do you want to add them while we use default values at the constructor? I am neutral on this.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r577000812
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,11 +75,222 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(const std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {
+ const std::shared_ptr<DataType>& type = data->type;
+ std::vector<std::shared_ptr<Buffer>> buffers(data->buffers.size(), nullptr);
+ std::vector<std::shared_ptr<ArrayData>> child_data(data->child_data.size(), nullptr);
+ out_ = ArrayData::Make(type, data->length, buffers, child_data, data->null_count, 0);
+ }
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(const std::vector<std::shared_ptr<Field>>& child_fields) {
Review comment:
I think you can write `const FieldVector& child_fields`
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r465505197
##########
File path: cpp/src/arrow/ipc/reader.cc
##########
@@ -449,7 +449,7 @@ Result<std::shared_ptr<RecordBatch>> LoadRecordBatchSubset(
const flatbuf::RecordBatch* metadata, const std::shared_ptr<Schema>& schema,
const std::vector<bool>& inclusion_mask, const DictionaryMemo* dictionary_memo,
const IpcReadOptions& options, MetadataVersion metadata_version,
- Compression::type compression, io::RandomAccessFile* file) {
+ Compression::type compression, io::RandomAccessFile* file, bool swap_endian = false) {
Review comment:
I think we need to keep `swap_endian` and `swap_endian_`.
IMHO, this is because `IpcReadOptions::use_native_endian` is a flag set by a user to determine whether the endianness of the schema will be changed or not if the endianness of the schema is not equal to the platform-native endianness.
`swap_endian_` is to show we need to actually swap endianness of each element.
For example, there is a case that `IpcReadOptions::use_native_endian=true` and `swap_endian_=false` if x86 platform gets the schema with the endianness = LITTLE.
Or, can we use `IpcReadOptions::use_native_endian` as a working area o show we need to actually swap endianness of each element?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r562063757
##########
File path: cpp/src/arrow/array/util.h
##########
@@ -56,6 +56,14 @@ Result<std::shared_ptr<Array>> MakeArrayFromScalar(
namespace internal {
+/// \brief Swap endian of each element in a generic ArrayData
+/// \param[in] data the array contents
+/// \param[in] type the array type
+/// \return the resulting Array instance whose elements were swapped
+ARROW_EXPORT
+Result<std::shared_ptr<ArrayData>> SwapEndianArrayData(
+ std::shared_ptr<ArrayData>& data, const std::shared_ptr<DataType>& type);
Review comment:
The first parameter should be `const` as well, it seems?
##########
File path: cpp/src/arrow/array/util.h
##########
@@ -56,6 +56,14 @@ Result<std::shared_ptr<Array>> MakeArrayFromScalar(
namespace internal {
+/// \brief Swap endian of each element in a generic ArrayData
+/// \param[in] data the array contents
+/// \param[in] type the array type
Review comment:
The type is already on the data, is it necessary to pass it separately?
##########
File path: dev/archery/tests/generate_files_for_endian_test.sh
##########
@@ -0,0 +1,28 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+: ${ARROW_DIR:=/arrow}
Review comment:
Can you add a comment explaining how to use this script?
##########
File path: cpp/src/arrow/type.h
##########
@@ -1641,6 +1655,17 @@ class ARROW_EXPORT Schema : public detail::Fingerprintable,
bool Equals(const Schema& other, bool check_metadata = false) const;
bool Equals(const std::shared_ptr<Schema>& other, bool check_metadata = false) const;
+ /// \brief Replace endianness with platform-native endianness in the schema
+ ///
+ /// \return new Schema
+ std::shared_ptr<Schema> WithNativeEndianness() const;
+
+ /// \brief Return endianness in the schema
+ Endianness endianness() const;
+
+ /// \brief Indicate if endianness is equal to platform-native endianness
+ bool IsNativeEndianness() const;
Review comment:
I would call it `is_native_endian()`.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
Review comment:
And where is the null bitmap handled? Did I miss something?
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
Review comment:
Either `const&` or no reference at all.
##########
File path: ci/scripts/integration_arrow.sh
##########
@@ -24,9 +24,16 @@ source_dir=${1}/cpp
build_dir=${2}/cpp
gold_dir_0_14_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.14.1
gold_dir_0_17_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.17.1
+gold_dir_1_0_0_le=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-littleendian
+gold_dir_1_0_0_be=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-bigendian
Review comment:
Sidenote: I see that "generated_large_batch" was added to these directories. Is it necessary?
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
Review comment:
By the way, what happens if `data_.offset` is non-zero? Is it supported? If not, should probably return an error.
##########
File path: cpp/src/arrow/ipc/reader.h
##########
@@ -169,6 +172,9 @@ class ARROW_EXPORT RecordBatchFileReader {
/// \brief Return current read statistics
virtual ReadStats stats() const = 0;
+
+ protected:
+ bool swap_endian_;
Review comment:
Same here.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
Review comment:
`SwapOffsets` would be better.
##########
File path: cpp/src/arrow/ipc/reader.h
##########
@@ -96,6 +96,9 @@ class ARROW_EXPORT RecordBatchStreamReader : public RecordBatchReader {
/// \brief Return current read statistics
virtual ReadStats stats() const = 0;
+
+ protected:
+ bool swap_endian_;
Review comment:
It doesn't seem necessary to put this in `RecordBatchStreamReader`, you can move it to the implementation class.
##########
File path: cpp/src/arrow/type.h
##########
@@ -1623,13 +1624,26 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness {
+ LITTLE = 0,
+ BIG = 1,
+#if ARROW_LITTLE_ENDIAN
+ NATIVE = LITTLE
+#else
+ NATIVE = BIG
Review comment:
Nit, but to avoid issues with third-party macros, I would use `Little`, `Big`, `Native`.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return CopyDataBuffer(); }
+ Status Visit(const Int8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const UInt8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeBinaryType& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const UnionType& type) {
+ (*out_)->buffers[1] = data_->buffers[1];
+ if (type.mode() == UnionMode::DENSE) {
+ RETURN_NOT_OK(SwapSmallOffset(2));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<BinaryType, T>::value || std::is_same<StringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ (*out_)->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<LargeBinaryType, T>::value ||
+ std::is_same<LargeStringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ (*out_)->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const MapType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const DictionaryType& type) {
+ RETURN_NOT_OK(SwapType(*type.index_type()));
+ (*out_)->dictionary = data_->dictionary;
+ return Status::OK();
+ }
+
+ Status Visit(const ExtensionType& type) {
+ RETURN_NOT_OK(SwapType(*type.storage_type()));
+ (*out_)->dictionary = data_->dictionary;
+ return Status::OK();
+ }
+
+ std::shared_ptr<ArrayData>& data_;
+ int64_t length_;
+ std::shared_ptr<ArrayData>* out_;
+};
+
+Result<std::shared_ptr<ArrayData>> SwapEndianArrayData(
+ std::shared_ptr<ArrayData>& data, const std::shared_ptr<DataType>& type) {
+ std::vector<std::shared_ptr<Buffer>> buffers(data->buffers.size(), nullptr);
+ std::vector<std::shared_ptr<ArrayData>> child_data(data->child_data.size(), nullptr);
+ std::shared_ptr<ArrayData> out =
+ ArrayData::Make(type, data->length, buffers, child_data, data->null_count, 0);
Review comment:
I think it would make the code more regular to create the buffers and child_data inside `ArrayDataEndianSwapper` (since you also need to do it for child fields recursively)
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
Review comment:
Instead of taking the result `ArrayData` as an argument, you can probably create it in this class and return it to the caller.
##########
File path: cpp/src/arrow/array/util.h
##########
@@ -56,6 +56,14 @@ Result<std::shared_ptr<Array>> MakeArrayFromScalar(
namespace internal {
+/// \brief Swap endian of each element in a generic ArrayData
+/// \param[in] data the array contents
+/// \param[in] type the array type
+/// \return the resulting Array instance whose elements were swapped
+ARROW_EXPORT
+Result<std::shared_ptr<ArrayData>> SwapEndianArrayData(
+ std::shared_ptr<ArrayData>& data, const std::shared_ptr<DataType>& type);
+
/// Given a number of ArrayVectors, treat each ArrayVector as the
Review comment:
Hmm... did I miss the tests for `SwapEndianArrayData` or didn't you add any?
I would expect some simple basic tests, at least to exercise nulls, numbers, structs, lists, decimals...
##########
File path: cpp/src/arrow/type.cc
##########
@@ -1352,6 +1369,11 @@ bool Schema::Equals(const Schema& other, bool check_metadata) const {
return true;
}
+ // checks endianness equality
+ if (endianness() != other.endianness()) {
+ return false;
+ }
Review comment:
Can you add equality tests to `type_test.cc`?
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
Review comment:
One other thing: if `sizeof(T) == 1`, then we can simply reuse the existing buffer.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
Review comment:
Hmm... are we sure `length + extra_size` and `in_buffer->size()` are consistent?
I would expect either:
* byte-swap `length` entries
* `memset()` the remaining bytes to 0
... or to allocate exactly the required size (i.e. `length * sizeof(T)`)
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
Review comment:
We should also `memset` the excess bytes to 0. We probably want to factor the buffer allocation out.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
Review comment:
Since this is a non-exposed class, can you put it in the anonymous namespace?
(you can do the same for `ArrayDataWrapper` at the same time :-))
##########
File path: ci/scripts/integration_arrow.sh
##########
@@ -24,9 +24,16 @@ source_dir=${1}/cpp
build_dir=${2}/cpp
gold_dir_0_14_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.14.1
gold_dir_0_17_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.17.1
+gold_dir_1_0_0_le=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-littleendian
+gold_dir_1_0_0_be=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-bigendian
pip install -e $arrow_dir/dev/archery
archery integration --with-all --run-flight \
--gold-dirs=$gold_dir_0_14_1 \
--gold-dirs=$gold_dir_0_17_1 \
+
+# TODO: support other languages
+archery integration --with-cpp=1 --run-flight \
Review comment:
You mean even the little-endian cases fail on Java?
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
Review comment:
Pass a const reference here to avoid copying the vector.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
Review comment:
I'm frankly not sure those two mini-methods are useful?
##########
File path: cpp/src/arrow/type.cc
##########
@@ -1352,6 +1369,11 @@ bool Schema::Equals(const Schema& other, bool check_metadata) const {
return true;
}
+ // checks endianness equality
+ if (endianness() != other.endianness()) {
+ return false;
+ }
Review comment:
Also, you'll need to fix fingerprint computation.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,204 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ArrayDataEndianSwapper swapper_child_visitor(data_->child_data[i],
+ data_->child_data[i]->length);
+ RETURN_NOT_OK(VisitTypeInline(*child_field.get()->type(), &swapper_child_visitor));
+ RETURN_NOT_OK(
+ swapper_child_visitor.SwapChildren((*child_field.get()->type()).fields()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto data = reinterpret_cast<const VALUE_TYPE*>(data_->buffers[index]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer,
+ AllocateBuffer(data_->buffers[index]->size() + 1));
+ auto new_data = reinterpret_cast<VALUE_TYPE*>(new_buffer->mutable_data());
+ // offset has one more element rather than data->length
+ int64_t length = length_ + 1;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[index] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ Status Visit(const T&) {
+ using value_type = typename T::c_type;
+ auto data = reinterpret_cast<const value_type*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<value_type*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ auto data = reinterpret_cast<const uint32_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint32_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ new_data[idx] = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromBigEndian(data[idx + 1]);
+#else
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromLittleEndian(data[idx + 1]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return Status::OK(); }
+ Status Visit(const Int8Type& type) { return Status::OK(); }
+ Status Visit(const UInt8Type& type) { return Status::OK(); }
+ Status Visit(const FixedSizeBinaryType& type) { return Status::OK(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const SparseUnionType& type) { return Status::OK(); }
+
+ Status Visit(const StringType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeStringType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+ Status Visit(const BinaryType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeBinaryType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const MapType& type) {
Review comment:
`MapType` is a subclass of `ListType`, so you don't need a specific visitor since it's the same code :-)
##########
File path: cpp/src/arrow/type.h
##########
@@ -1641,6 +1655,17 @@ class ARROW_EXPORT Schema : public detail::Fingerprintable,
bool Equals(const Schema& other, bool check_metadata = false) const;
bool Equals(const std::shared_ptr<Schema>& other, bool check_metadata = false) const;
+ /// \brief Replace endianness with platform-native endianness in the schema
+ ///
+ /// \return new Schema
+ std::shared_ptr<Schema> WithNativeEndianness() const;
Review comment:
Wouldn't `WithEndianness(Endianness) const` be more generally useful?
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return CopyDataBuffer(); }
+ Status Visit(const Int8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const UInt8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeBinaryType& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const UnionType& type) {
+ (*out_)->buffers[1] = data_->buffers[1];
+ if (type.mode() == UnionMode::DENSE) {
+ RETURN_NOT_OK(SwapSmallOffset(2));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<BinaryType, T>::value || std::is_same<StringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
Review comment:
You can use `SwapOffsets<T::offset_type>` and merge with the below function.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
Review comment:
Why is a copy required? You can just reuse the buffer as-is.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
Review comment:
Why not use `BitUtil::ByteSwap`?
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return CopyDataBuffer(); }
+ Status Visit(const Int8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const UInt8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeBinaryType& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const UnionType& type) {
+ (*out_)->buffers[1] = data_->buffers[1];
+ if (type.mode() == UnionMode::DENSE) {
+ RETURN_NOT_OK(SwapSmallOffset(2));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<BinaryType, T>::value || std::is_same<StringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ (*out_)->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<LargeBinaryType, T>::value ||
+ std::is_same<LargeStringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ (*out_)->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
Review comment:
Same here (`SwapOffsets<T::offset_type>`)
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return CopyDataBuffer(); }
+ Status Visit(const Int8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const UInt8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeBinaryType& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const UnionType& type) {
+ (*out_)->buffers[1] = data_->buffers[1];
+ if (type.mode() == UnionMode::DENSE) {
+ RETURN_NOT_OK(SwapSmallOffset(2));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<BinaryType, T>::value || std::is_same<StringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ (*out_)->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<LargeBinaryType, T>::value ||
+ std::is_same<LargeStringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ (*out_)->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const MapType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const DictionaryType& type) {
+ RETURN_NOT_OK(SwapType(*type.index_type()));
+ (*out_)->dictionary = data_->dictionary;
+ return Status::OK();
+ }
+
+ Status Visit(const ExtensionType& type) {
+ RETURN_NOT_OK(SwapType(*type.storage_type()));
+ (*out_)->dictionary = data_->dictionary;
+ return Status::OK();
+ }
+
+ std::shared_ptr<ArrayData>& data_;
Review comment:
const reference here
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
Review comment:
Why `extra_size`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r576998571
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,11 +75,222 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(const std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {
+ const std::shared_ptr<DataType>& type = data->type;
+ std::vector<std::shared_ptr<Buffer>> buffers(data->buffers.size(), nullptr);
+ std::vector<std::shared_ptr<ArrayData>> child_data(data->child_data.size(), nullptr);
+ out_ = ArrayData::Make(type, data->length, buffers, child_data, data->null_count, 0);
+ }
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(const std::vector<std::shared_ptr<Field>>& child_fields) {
+ for (size_t i = 0; i < child_fields.size(); i++) {
+ ARROW_ASSIGN_OR_RAISE(out_->child_data[i],
+ internal::SwapEndianArrayData(data_->child_data[i]));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(
+ const std::shared_ptr<Buffer>& in_buffer) {
+ if (sizeof(T) == 1) {
+ // if data size is 1, element is not swapped. We can use the original buffer
+ return in_buffer;
+ }
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ int64_t length = in_buffer->size() / sizeof(T);
+ for (int64_t i = 0; i < length; i++) {
+ out_data[i] = BitUtil::ByteSwap(in_data[i]);
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffsets(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ out_->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // Except union, offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE(out_->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index]));
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE(out_->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1]));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ length = data_->buffers[1]->size() / (sizeof(uint64_t) * 2);
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ out_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ length = data_->buffers[1]->size() / (sizeof(uint64_t) * 4);
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ out_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE(out_->buffers[1], ByteSwapBuffer<uint32_t>(data_->buffers[1]));
+ return Status::OK();
+ }
+
+ Status ReuseDataBuffer() {
+ out_->buffers[1] = data_->buffers[1];
Review comment:
Isn't this automatically the case? (since `out_` is initialized from `data_`)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
pitrou commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r562063757
##########
File path: cpp/src/arrow/array/util.h
##########
@@ -56,6 +56,14 @@ Result<std::shared_ptr<Array>> MakeArrayFromScalar(
namespace internal {
+/// \brief Swap endian of each element in a generic ArrayData
+/// \param[in] data the array contents
+/// \param[in] type the array type
+/// \return the resulting Array instance whose elements were swapped
+ARROW_EXPORT
+Result<std::shared_ptr<ArrayData>> SwapEndianArrayData(
+ std::shared_ptr<ArrayData>& data, const std::shared_ptr<DataType>& type);
Review comment:
The first parameter should be `const` as well, it seems?
##########
File path: cpp/src/arrow/array/util.h
##########
@@ -56,6 +56,14 @@ Result<std::shared_ptr<Array>> MakeArrayFromScalar(
namespace internal {
+/// \brief Swap endian of each element in a generic ArrayData
+/// \param[in] data the array contents
+/// \param[in] type the array type
Review comment:
The type is already on the data, is it necessary to pass it separately?
##########
File path: dev/archery/tests/generate_files_for_endian_test.sh
##########
@@ -0,0 +1,28 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+: ${ARROW_DIR:=/arrow}
Review comment:
Can you add a comment explaining how to use this script?
##########
File path: cpp/src/arrow/type.h
##########
@@ -1641,6 +1655,17 @@ class ARROW_EXPORT Schema : public detail::Fingerprintable,
bool Equals(const Schema& other, bool check_metadata = false) const;
bool Equals(const std::shared_ptr<Schema>& other, bool check_metadata = false) const;
+ /// \brief Replace endianness with platform-native endianness in the schema
+ ///
+ /// \return new Schema
+ std::shared_ptr<Schema> WithNativeEndianness() const;
+
+ /// \brief Return endianness in the schema
+ Endianness endianness() const;
+
+ /// \brief Indicate if endianness is equal to platform-native endianness
+ bool IsNativeEndianness() const;
Review comment:
I would call it `is_native_endian()`.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
Review comment:
And where is the null bitmap handled? Did I miss something?
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
Review comment:
Either `const&` or no reference at all.
##########
File path: ci/scripts/integration_arrow.sh
##########
@@ -24,9 +24,16 @@ source_dir=${1}/cpp
build_dir=${2}/cpp
gold_dir_0_14_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.14.1
gold_dir_0_17_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.17.1
+gold_dir_1_0_0_le=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-littleendian
+gold_dir_1_0_0_be=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-bigendian
Review comment:
Sidenote: I see that "generated_large_batch" was added to these directories. Is it necessary?
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
Review comment:
By the way, what happens if `data_.offset` is non-zero? Is it supported? If not, should probably return an error.
##########
File path: cpp/src/arrow/ipc/reader.h
##########
@@ -169,6 +172,9 @@ class ARROW_EXPORT RecordBatchFileReader {
/// \brief Return current read statistics
virtual ReadStats stats() const = 0;
+
+ protected:
+ bool swap_endian_;
Review comment:
Same here.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
Review comment:
`SwapOffsets` would be better.
##########
File path: cpp/src/arrow/ipc/reader.h
##########
@@ -96,6 +96,9 @@ class ARROW_EXPORT RecordBatchStreamReader : public RecordBatchReader {
/// \brief Return current read statistics
virtual ReadStats stats() const = 0;
+
+ protected:
+ bool swap_endian_;
Review comment:
It doesn't seem necessary to put this in `RecordBatchStreamReader`, you can move it to the implementation class.
##########
File path: cpp/src/arrow/type.h
##########
@@ -1623,13 +1624,26 @@ class ARROW_EXPORT FieldRef {
// ----------------------------------------------------------------------
// Schema
+enum class Endianness {
+ LITTLE = 0,
+ BIG = 1,
+#if ARROW_LITTLE_ENDIAN
+ NATIVE = LITTLE
+#else
+ NATIVE = BIG
Review comment:
Nit, but to avoid issues with third-party macros, I would use `Little`, `Big`, `Native`.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return CopyDataBuffer(); }
+ Status Visit(const Int8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const UInt8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeBinaryType& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const UnionType& type) {
+ (*out_)->buffers[1] = data_->buffers[1];
+ if (type.mode() == UnionMode::DENSE) {
+ RETURN_NOT_OK(SwapSmallOffset(2));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<BinaryType, T>::value || std::is_same<StringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ (*out_)->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<LargeBinaryType, T>::value ||
+ std::is_same<LargeStringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ (*out_)->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const MapType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const DictionaryType& type) {
+ RETURN_NOT_OK(SwapType(*type.index_type()));
+ (*out_)->dictionary = data_->dictionary;
+ return Status::OK();
+ }
+
+ Status Visit(const ExtensionType& type) {
+ RETURN_NOT_OK(SwapType(*type.storage_type()));
+ (*out_)->dictionary = data_->dictionary;
+ return Status::OK();
+ }
+
+ std::shared_ptr<ArrayData>& data_;
+ int64_t length_;
+ std::shared_ptr<ArrayData>* out_;
+};
+
+Result<std::shared_ptr<ArrayData>> SwapEndianArrayData(
+ std::shared_ptr<ArrayData>& data, const std::shared_ptr<DataType>& type) {
+ std::vector<std::shared_ptr<Buffer>> buffers(data->buffers.size(), nullptr);
+ std::vector<std::shared_ptr<ArrayData>> child_data(data->child_data.size(), nullptr);
+ std::shared_ptr<ArrayData> out =
+ ArrayData::Make(type, data->length, buffers, child_data, data->null_count, 0);
Review comment:
I think it would make the code more regular to create the buffers and child_data inside `ArrayDataEndianSwapper` (since you also need to do it for child fields recursively)
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
Review comment:
Instead of taking the result `ArrayData` as an argument, you can probably create it in this class and return it to the caller.
##########
File path: cpp/src/arrow/array/util.h
##########
@@ -56,6 +56,14 @@ Result<std::shared_ptr<Array>> MakeArrayFromScalar(
namespace internal {
+/// \brief Swap endian of each element in a generic ArrayData
+/// \param[in] data the array contents
+/// \param[in] type the array type
+/// \return the resulting Array instance whose elements were swapped
+ARROW_EXPORT
+Result<std::shared_ptr<ArrayData>> SwapEndianArrayData(
+ std::shared_ptr<ArrayData>& data, const std::shared_ptr<DataType>& type);
+
/// Given a number of ArrayVectors, treat each ArrayVector as the
Review comment:
Hmm... did I miss the tests for `SwapEndianArrayData` or didn't you add any?
I would expect some simple basic tests, at least to exercise nulls, numbers, structs, lists, decimals...
##########
File path: cpp/src/arrow/type.cc
##########
@@ -1352,6 +1369,11 @@ bool Schema::Equals(const Schema& other, bool check_metadata) const {
return true;
}
+ // checks endianness equality
+ if (endianness() != other.endianness()) {
+ return false;
+ }
Review comment:
Can you add equality tests to `type_test.cc`?
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
Review comment:
One other thing: if `sizeof(T) == 1`, then we can simply reuse the existing buffer.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
Review comment:
Hmm... are we sure `length + extra_size` and `in_buffer->size()` are consistent?
I would expect either:
* byte-swap `length` entries
* `memset()` the remaining bytes to 0
... or to allocate exactly the required size (i.e. `length * sizeof(T)`)
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
Review comment:
We should also `memset` the excess bytes to 0. We probably want to factor the buffer allocation out.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
Review comment:
Since this is a non-exposed class, can you put it in the anonymous namespace?
(you can do the same for `ArrayDataWrapper` at the same time :-))
##########
File path: ci/scripts/integration_arrow.sh
##########
@@ -24,9 +24,16 @@ source_dir=${1}/cpp
build_dir=${2}/cpp
gold_dir_0_14_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.14.1
gold_dir_0_17_1=$arrow_dir/testing/data/arrow-ipc-stream/integration/0.17.1
+gold_dir_1_0_0_le=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-littleendian
+gold_dir_1_0_0_be=$arrow_dir/testing/data/arrow-ipc-stream/integration/1.0.0-bigendian
pip install -e $arrow_dir/dev/archery
archery integration --with-all --run-flight \
--gold-dirs=$gold_dir_0_14_1 \
--gold-dirs=$gold_dir_0_17_1 \
+
+# TODO: support other languages
+archery integration --with-cpp=1 --run-flight \
Review comment:
You mean even the little-endian cases fail on Java?
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
Review comment:
Pass a const reference here to avoid copying the vector.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
Review comment:
I'm frankly not sure those two mini-methods are useful?
##########
File path: cpp/src/arrow/type.cc
##########
@@ -1352,6 +1369,11 @@ bool Schema::Equals(const Schema& other, bool check_metadata) const {
return true;
}
+ // checks endianness equality
+ if (endianness() != other.endianness()) {
+ return false;
+ }
Review comment:
Also, you'll need to fix fingerprint computation.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,204 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ArrayDataEndianSwapper swapper_child_visitor(data_->child_data[i],
+ data_->child_data[i]->length);
+ RETURN_NOT_OK(VisitTypeInline(*child_field.get()->type(), &swapper_child_visitor));
+ RETURN_NOT_OK(
+ swapper_child_visitor.SwapChildren((*child_field.get()->type()).fields()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto data = reinterpret_cast<const VALUE_TYPE*>(data_->buffers[index]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer,
+ AllocateBuffer(data_->buffers[index]->size() + 1));
+ auto new_data = reinterpret_cast<VALUE_TYPE*>(new_buffer->mutable_data());
+ // offset has one more element rather than data->length
+ int64_t length = length_ + 1;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[index] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ Status Visit(const T&) {
+ using value_type = typename T::c_type;
+ auto data = reinterpret_cast<const value_type*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<value_type*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+#if ARROW_LITTLE_ENDIAN
+ new_data[i] = BitUtil::FromBigEndian(data[i]);
+#else
+ new_data[i] = BitUtil::FromLittleEndian(data[i]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ auto data = reinterpret_cast<const uint32_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint32_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ new_data[idx] = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromBigEndian(data[idx + 1]);
+#else
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx + 1] = BitUtil::FromLittleEndian(data[idx + 1]);
+#endif
+ }
+ data_->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return Status::OK(); }
+ Status Visit(const Int8Type& type) { return Status::OK(); }
+ Status Visit(const UInt8Type& type) { return Status::OK(); }
+ Status Visit(const FixedSizeBinaryType& type) { return Status::OK(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const SparseUnionType& type) { return Status::OK(); }
+
+ Status Visit(const StringType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeStringType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+ Status Visit(const BinaryType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeBinaryType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const MapType& type) {
Review comment:
`MapType` is a subclass of `ListType`, so you don't need a specific visitor since it's the same code :-)
##########
File path: cpp/src/arrow/type.h
##########
@@ -1641,6 +1655,17 @@ class ARROW_EXPORT Schema : public detail::Fingerprintable,
bool Equals(const Schema& other, bool check_metadata = false) const;
bool Equals(const std::shared_ptr<Schema>& other, bool check_metadata = false) const;
+ /// \brief Replace endianness with platform-native endianness in the schema
+ ///
+ /// \return new Schema
+ std::shared_ptr<Schema> WithNativeEndianness() const;
Review comment:
Wouldn't `WithEndianness(Endianness) const` be more generally useful?
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return CopyDataBuffer(); }
+ Status Visit(const Int8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const UInt8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeBinaryType& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const UnionType& type) {
+ (*out_)->buffers[1] = data_->buffers[1];
+ if (type.mode() == UnionMode::DENSE) {
+ RETURN_NOT_OK(SwapSmallOffset(2));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<BinaryType, T>::value || std::is_same<StringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
Review comment:
You can use `SwapOffsets<T::offset_type>` and merge with the below function.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
Review comment:
Why is a copy required? You can just reuse the buffer as-is.
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
Review comment:
Why not use `BitUtil::ByteSwap`?
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return CopyDataBuffer(); }
+ Status Visit(const Int8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const UInt8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeBinaryType& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const UnionType& type) {
+ (*out_)->buffers[1] = data_->buffers[1];
+ if (type.mode() == UnionMode::DENSE) {
+ RETURN_NOT_OK(SwapSmallOffset(2));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<BinaryType, T>::value || std::is_same<StringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ (*out_)->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<LargeBinaryType, T>::value ||
+ std::is_same<LargeStringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ (*out_)->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
Review comment:
Same here (`SwapOffsets<T::offset_type>`)
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
+ auto in_data = reinterpret_cast<const T*>(in_buffer->data());
+ ARROW_ASSIGN_OR_RAISE(auto out_buffer, AllocateBuffer(in_buffer->size()));
+ auto out_data = reinterpret_cast<T*>(out_buffer->mutable_data());
+ for (int64_t i = 0; i < length + extra_size; i++) {
+#if ARROW_LITTLE_ENDIAN
+ out_data[i] = BitUtil::FromBigEndian(in_data[i]);
+#else
+ out_data[i] = BitUtil::FromLittleEndian(in_data[i]);
+#endif
+ }
+ return std::move(out_buffer);
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr || data_->buffers[index]->size() == 0) {
+ (*out_)->buffers[index] = data_->buffers[index];
+ return Status::OK();
+ }
+ // offset has one more element rather than data->length
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[index],
+ ByteSwapBuffer<VALUE_TYPE>(data_->buffers[index], length_, 1));
+ return Status::OK();
+ }
+
+ Status SwapSmallOffset(int index = 1) { return SwapOffset<int32_t>(index); }
+
+ Status SwapLargeOffset() { return SwapOffset<int64_t>(1); }
+
+ template <typename T>
+ enable_if_t<std::is_base_of<FixedWidthType, T>::value &&
+ !std::is_base_of<FixedSizeBinaryType, T>::value &&
+ !std::is_base_of<DictionaryType, T>::value,
+ Status>
+ Visit(const T& type) {
+ using value_type = typename T::c_type;
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<value_type>(data_->buffers[1], length_, 0));
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal128Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp;
+ auto idx = i * 2;
+#if ARROW_LITTLE_ENDIAN
+ tmp = BitUtil::FromBigEndian(data[idx]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#else
+ tmp = BitUtil::FromLittleEndian(data[idx]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 1]);
+ new_data[idx + 1] = tmp;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const Decimal256Type& type) {
+ auto data = reinterpret_cast<const uint64_t*>(data_->buffers[1]->data());
+ ARROW_ASSIGN_OR_RAISE(auto new_buffer, AllocateBuffer(data_->buffers[1]->size()));
+ auto new_data = reinterpret_cast<uint64_t*>(new_buffer->mutable_data());
+ int64_t length = length_;
+ for (int64_t i = 0; i < length; i++) {
+ uint64_t tmp0, tmp1, tmp2;
+ auto idx = i * 4;
+#if ARROW_LITTLE_ENDIAN
+ tmp0 = BitUtil::FromBigEndian(data[idx]);
+ tmp1 = BitUtil::FromBigEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromBigEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromBigEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#else
+ tmp0 = BitUtil::FromLittleEndian(data[idx]);
+ tmp1 = BitUtil::FromLittleEndian(data[idx + 1]);
+ tmp2 = BitUtil::FromLittleEndian(data[idx + 2]);
+ new_data[idx] = BitUtil::FromLittleEndian(data[idx + 3]);
+ new_data[idx + 1] = tmp2;
+ new_data[idx + 2] = tmp1;
+ new_data[idx + 3] = tmp0;
+#endif
+ }
+ (*out_)->buffers[1] = std::move(new_buffer);
+ return Status::OK();
+ }
+
+ Status Visit(const DayTimeIntervalType& type) {
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ ByteSwapBuffer<uint32_t>(data_->buffers[1], length_ * 2, 0));
+ return Status::OK();
+ }
+
+ Status CopyDataBuffer() {
+ if (data_->buffers[1]->data() == nullptr) {
+ return Status::OK();
+ }
+ ARROW_ASSIGN_OR_RAISE((*out_)->buffers[1],
+ data_->buffers[1]->CopySlice(0, data_->buffers[1]->size()));
+ return Status::OK();
+ }
+
+ Status Visit(const NullType& type) { return Status::OK(); }
+ Status Visit(const BooleanType& type) { return CopyDataBuffer(); }
+ Status Visit(const Int8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const UInt8Type& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeBinaryType& type) { return CopyDataBuffer(); }
+ Status Visit(const FixedSizeListType& type) { return Status::OK(); }
+ Status Visit(const StructType& type) { return Status::OK(); }
+ Status Visit(const UnionType& type) {
+ (*out_)->buffers[1] = data_->buffers[1];
+ if (type.mode() == UnionMode::DENSE) {
+ RETURN_NOT_OK(SwapSmallOffset(2));
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<BinaryType, T>::value || std::is_same<StringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ (*out_)->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ template <typename T>
+ enable_if_t<std::is_same<LargeBinaryType, T>::value ||
+ std::is_same<LargeStringType, T>::value,
+ Status>
+ Visit(const T& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ (*out_)->buffers[2] = data_->buffers[2];
+ return Status::OK();
+ }
+
+ Status Visit(const ListType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+ Status Visit(const LargeListType& type) {
+ RETURN_NOT_OK(SwapLargeOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const MapType& type) {
+ RETURN_NOT_OK(SwapSmallOffset());
+ return Status::OK();
+ }
+
+ Status Visit(const DictionaryType& type) {
+ RETURN_NOT_OK(SwapType(*type.index_type()));
+ (*out_)->dictionary = data_->dictionary;
+ return Status::OK();
+ }
+
+ Status Visit(const ExtensionType& type) {
+ RETURN_NOT_OK(SwapType(*type.storage_type()));
+ (*out_)->dictionary = data_->dictionary;
+ return Status::OK();
+ }
+
+ std::shared_ptr<ArrayData>& data_;
Review comment:
const reference here
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +75,220 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length,
+ std::shared_ptr<ArrayData>* out)
+ : data_(data), length_(length), out_(out) {}
+
+ Status SwapType(const DataType& type) {
+ RETURN_NOT_OK(VisitTypeInline(type, this));
+ RETURN_NOT_OK(SwapChildren(type.fields()));
+ return Status::OK();
+ }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ ARROW_ASSIGN_OR_RAISE(
+ (*out_)->child_data[i],
+ SwapEndianArrayData(data_->child_data[i], child_field.get()->type()));
+ i++;
+ }
+ return Status::OK();
+ }
+
+ template <typename T>
+ Result<std::shared_ptr<Buffer>> ByteSwapBuffer(std::shared_ptr<Buffer>& in_buffer,
+ int64_t length, int64_t extra_size) {
Review comment:
Why `extra_size`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r467950372
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -74,6 +74,177 @@ class ArrayDataWrapper {
std::shared_ptr<Array>* out_;
};
+class ArrayDataEndianSwapper {
+ public:
+ ArrayDataEndianSwapper(std::shared_ptr<ArrayData>& data, int64_t length)
+ : data_(data), length_(length) {}
+
+ Status SwapType(const DataType& type) { return VisitTypeInline(type, this); }
+
+ Status SwapChildren(std::vector<std::shared_ptr<Field>> child_fields) {
+ int i = 0;
+ for (const auto& child_field : child_fields) {
+ auto orig_data = data_;
+ auto orig_length = length_;
+ data_ = data_->child_data[i++];
+ length_ = data_->length;
+ RETURN_NOT_OK(SwapType(*child_field.get()->type()));
+ length_ = orig_length;
+ data_ = orig_data;
+ }
+ return Status::OK();
+ }
+
+ template <typename VALUE_TYPE>
+ Status SwapOffset(int index) {
+ if (data_->buffers[index] == nullptr) {
+ return Status::OK();
+ }
+ auto buffer = const_cast<VALUE_TYPE*>(
Review comment:
`gpu/cuda_memory.cc`, `io/file.cc`, and `io/memory.cc` are what I saw.
If we should avoid the usage of `const_cast`, I agree. Should we reallocate the buffer?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on a change in pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on a change in pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#discussion_r538208202
##########
File path: cpp/src/arrow/array/util.cc
##########
@@ -84,6 +283,12 @@ std::shared_ptr<Array> MakeArray(const std::shared_ptr<ArrayData>& data) {
return out;
}
+void SwapEndianArrayData(std::shared_ptr<ArrayData>& data) {
+ internal::ArrayDataEndianSwapper swapper_visitor(data, data->length);
+ DCHECK_OK(VisitTypeInline(*data->type, &swapper_visitor));
+ DCHECK_OK(swapper_visitor.SwapChildren((*data->type).fields()));
Review comment:
@wesm I just keep this code as-is. This is because `SwapChildren()` is also called from`SwapType()`. Any comments are appreciated.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kiszk commented on pull request #7507: ARROW-8797: [C++] Read RecordBatch in a different endian
Posted by GitBox <gi...@apache.org>.
kiszk commented on pull request #7507:
URL: https://github.com/apache/arrow/pull/7507#issuecomment-749068143
@pitrou Sure, it is fine with me. Happy holiday and new year!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org