You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "mapleFU (via GitHub)" <gi...@apache.org> on 2025/05/09 10:37:45 UTC
[PR] [C++][Parquet] Parquet Variant decoding tools [arrow]
mapleFU opened a new pull request, #46372:
URL: https://github.com/apache/arrow/pull/46372
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
Please remove this line and the above text before creating your pull request.
### Rationale for this change
### What changes are included in this PR?
### Are these changes tested?
### Are there any user-facing changes?
**This PR includes breaking changes to public APIs.** (If there are any breaking changes to public APIs, please explain which changes are breaking. If not, you can remove this.)
**This PR contains a "Critical Fix".** (If the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld), please provide explanation. If not, you can remove this.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2093320904
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+std::string variantBasicTypeToString(VariantBasicType type);
Review Comment:
It's a good idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2102015282
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+#include "parquet/platform.h"
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+PARQUET_EXPORT std::string VariantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ TimestampMicros = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampMicrosNtz = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeMicrosNtz = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampNanosTz = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNanosNtz = 19, // Differentiating from TimestampNtz (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+PARQUET_EXPORT std::string VariantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ Object,
+ Array,
+ Null,
+ Boolean,
+ Int8,
+ Int16,
+ Int32,
+ Int64,
+ String,
+ Double,
+ Decimal4,
+ Decimal8,
+ Decimal16,
+ Date,
+ TimestampMicrosTz,
+ TimestampMicrosNtz,
+ Float,
+ Binary,
+ Time,
+ TimestampNanosTz,
+ TimestampNanosNtz,
+ Uuid
+};
+
+PARQUET_EXPORT std::string VariantTypeToString(VariantType type);
+
+class PARQUET_EXPORT VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary_size).
+ std::string_view GetMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sorted_and_unique(), the metadata key is not guaranteed to be unique, so we use a
Review Comment:
I don't know, but I think whether the writer promises that it doesn't produce value like this, whether the reader should pay the effort. Otherwise it's a bug
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2917016088
> @pitrou SAX style parser means consume the whole token and parsing it to a variant object.
It does not. See https://rapidjson.org/md_doc_sax.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2085849335
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,177 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+enum class VariantPrimitiveType {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNTZ = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNTZ = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTZ = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNTZNanos = 19, // Differentiating from TimestampNTZ (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ OBJECT,
+ ARRAY,
+ VARIANT_NULL,
+ BOOLEAN,
+ BYTE,
+ SHORT,
+ INT,
+ LONG,
+ STRING,
+ DOUBLE,
+ DECIMAL4,
+ DECIMAL8,
+ DECIMAL16,
+ DATE,
+ TIMESTAMP_TZ,
+ TIMESTAMP_NTZ,
+ FLOAT,
+ BINARY,
+ TIME,
+ TIMESTAMP_NANOS_TZ,
+ TIMESTAMP_NANOS_NTZ,
+ UUID
+};
+
+class VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ int8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ std::string_view getMetadataKey(int32_t variantId) const;
Review Comment:
Should we also implement query whether a key exists?
##########
cpp/src/parquet/variant_test.cpp:
##########
@@ -0,0 +1,75 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <gtest/gtest.h>
+
+#include <string>
+
+#include "parquet/exception.h"
+#include "parquet/test_util.h"
+#include "parquet/variant.h"
+
+#include <arrow/filesystem/localfs.h>
+#include <arrow/testing/gtest_util.h>
+
+namespace parquet::variant {
+
+TEST(ParquetVariant, MetadataBase) {
+ std::string dir_string(parquet::test::get_variant_dir());
+ auto file_system = std::make_shared<::arrow::fs::LocalFileSystem>();
+ std::vector<std::string> primitive_metadatas = {
+ // "primitive_null.metadata",
Review Comment:
This is because: https://github.com/apache/parquet-testing/issues/81
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #46372: GH-46371: [C++][Parquet] Parquet Variant decoding tools
URL: https://github.com/apache/arrow/pull/46372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2093316223
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+std::string variantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNTZ = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNTZ = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTZ = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNTZNanos = 19, // Differentiating from TimestampNTZ (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+std::string variantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ OBJECT,
+ ARRAY,
+ VARIANT_NULL,
+ BOOLEAN,
+ INT8,
+ INT16,
+ INT32,
+ INT64,
+ STRING,
+ DOUBLE,
+ DECIMAL4,
+ DECIMAL8,
+ DECIMAL16,
+ DATE,
+ TIMESTAMP_TZ,
+ TIMESTAMP_NTZ,
+ FLOAT,
+ BINARY,
+ TIME,
+ TIMESTAMP_NANOS_TZ,
+ TIMESTAMP_NANOS_NTZ,
+ UUID
+};
+
+std::string variantTypeToString(VariantType type);
+
+class VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary size).
+ std::string_view getMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sortedStrings(), the metadata key is not guaranteed to be unique, so we use a
+ /// vector to store all the metadata ids.
+ ::arrow::internal::SmallVector<uint32_t, 1> getMetadataId(std::string_view key) const;
+
+ bool sortedStrings() const;
+ uint8_t offsetSize() const;
+ uint32_t dictionarySize() const;
+
+ private:
+ static uint32_t loadDictionarySize(std::string_view metadata, uint8_t offset_size);
+
+ private:
+ static constexpr uint8_t VERSION_MASK = 0b1111;
+ static constexpr uint8_t SORTED_STRING_MASK = 0b10000;
+ static constexpr size_t HEADER_SIZE_BYTES = 1;
+ static constexpr size_t MINIMAL_OFFSET_SIZE_BYTES = 1;
+ static constexpr size_t MAXIMUM_OFFSET_SIZE_BYTES = 4;
+ // mask is applied after shift, it's like 0b11000000 before shift.
+ static constexpr uint8_t OFFSET_SIZE_MASK = 0b11;
+ static constexpr uint8_t OFFSET_SIZE_BIT_SHIFT = 6;
+ static constexpr uint8_t SUPPORTED_VERSION = 1;
+
+ private:
+ std::string_view metadata_;
+ uint32_t dictionary_size_{0};
+};
+
+template <typename DecimalType>
+struct DecimalValue {
+ uint8_t scale;
+ DecimalType value;
+};
+
+struct VariantValue {
+ VariantMetadata metadata;
Review Comment:
Why not? They logically shares same `VariantMetadata`. And `VariantMetadata` is so lightweight ( about 24bytes and not allocated on heap, and we can make it cheaper if using ByteArray rather than string view, which could be 16bytes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "emkornfield (via GitHub)" <gi...@apache.org>.
emkornfield commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2124420321
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,315 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+#include "parquet/platform.h"
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+PARQUET_EXPORT std::string VariantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ TimestampMicros = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampMicrosNtz = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeMicrosNtz = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampNanosTz = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNanosNtz = 19, // Differentiating from TimestampNtz (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+PARQUET_EXPORT std::string VariantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ Object,
+ Array,
+ Null,
+ Boolean,
+ Int8,
+ Int16,
+ Int32,
+ Int64,
+ String,
+ Double,
+ Decimal4,
+ Decimal8,
+ Decimal16,
+ Date,
+ TimestampMicrosTz,
+ TimestampMicrosNtz,
+ Float,
+ Binary,
+ Time,
+ TimestampNanosTz,
+ TimestampNanosNtz,
+ Uuid
+};
+
+PARQUET_EXPORT std::string VariantTypeToString(VariantType type);
+
+class PARQUET_EXPORT VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary_size).
+ std::string_view GetMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sorted_and_unique(), the metadata key is not guaranteed to be unique, so we use a
+ /// vector to store all the metadata ids.
+ ::arrow::internal::SmallVector<uint32_t, 1> GetMetadataId(std::string_view key) const;
+
+ bool sorted_and_unique() const;
+ uint8_t offset_size() const;
+ uint32_t dictionary_size() const;
+
+ /// Metadata for primitive types and any nested types
+ /// without key dictionary.
+ static constexpr char kEmptyMetadataChars[] = {0x1, 0x0, 0x0};
+ static constexpr std::string_view kEmptyMetadataStringView{kEmptyMetadataChars,
+ sizeof(kEmptyMetadataChars)};
+
+ private:
+ static uint32_t loadDictionarySize(std::string_view metadata, uint8_t offset_size);
+
+ private:
+ static constexpr uint8_t kVersionMask = 0b1111;
+ static constexpr uint8_t kSortedStringMask = 0b10000;
+ static constexpr size_t kHeaderSizeBytes = 1;
+ static constexpr size_t kMinimalOffsetSizeBytes = 1;
+ static constexpr size_t kMaximumOffsetSizeBytes = 4;
+ // mask is applied after shift, it's like 0b11000000 before shift.
+ static constexpr uint8_t kOffsetSizeMask = 0b11;
+ static constexpr uint8_t kOffsetSizeBitShift = 6;
+ static constexpr uint8_t kSupportedVersion = 1;
+
+ private:
+ std::string_view metadata_;
+ uint32_t dictionary_size_{0};
+};
+
+template <typename DecimalType>
+struct PARQUET_EXPORT DecimalValue {
+ uint8_t scale;
+ DecimalType value;
+};
+
+class PARQUET_EXPORT VariantValue {
+ public:
+ VariantValue(std::string_view metadata, std::string_view value);
+
+ VariantBasicType getBasicType() const;
Review Comment:
nit: style in this class looks off. either `GetBasicType` or `get_basic_type`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2866056854
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - Contributing Overview](https://arrow.apache.org/docs/developers/overview.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2879251495
@emkornfield @wgtmac @pitrou @zeroshade
This patch add some basic variant decoding tools. Some thoughts:
How would the interface for visiting variant like? The simplist way is cast `<metadata, value>` pairs to a `ptr<::arrow::Scalar>`, but this is too slow and needs to read whole data. We can also wraps a `std::variant`, but I think it's also slow and needs to dynamic dispatch the visitor. Here I just add visitor for every type. And currently, `getInt64` would only supports read int64. Any idea is welcome
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2093316223
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+std::string variantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNTZ = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNTZ = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTZ = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNTZNanos = 19, // Differentiating from TimestampNTZ (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+std::string variantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ OBJECT,
+ ARRAY,
+ VARIANT_NULL,
+ BOOLEAN,
+ INT8,
+ INT16,
+ INT32,
+ INT64,
+ STRING,
+ DOUBLE,
+ DECIMAL4,
+ DECIMAL8,
+ DECIMAL16,
+ DATE,
+ TIMESTAMP_TZ,
+ TIMESTAMP_NTZ,
+ FLOAT,
+ BINARY,
+ TIME,
+ TIMESTAMP_NANOS_TZ,
+ TIMESTAMP_NANOS_NTZ,
+ UUID
+};
+
+std::string variantTypeToString(VariantType type);
+
+class VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary size).
+ std::string_view getMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sortedStrings(), the metadata key is not guaranteed to be unique, so we use a
+ /// vector to store all the metadata ids.
+ ::arrow::internal::SmallVector<uint32_t, 1> getMetadataId(std::string_view key) const;
+
+ bool sortedStrings() const;
+ uint8_t offsetSize() const;
+ uint32_t dictionarySize() const;
+
+ private:
+ static uint32_t loadDictionarySize(std::string_view metadata, uint8_t offset_size);
+
+ private:
+ static constexpr uint8_t VERSION_MASK = 0b1111;
+ static constexpr uint8_t SORTED_STRING_MASK = 0b10000;
+ static constexpr size_t HEADER_SIZE_BYTES = 1;
+ static constexpr size_t MINIMAL_OFFSET_SIZE_BYTES = 1;
+ static constexpr size_t MAXIMUM_OFFSET_SIZE_BYTES = 4;
+ // mask is applied after shift, it's like 0b11000000 before shift.
+ static constexpr uint8_t OFFSET_SIZE_MASK = 0b11;
+ static constexpr uint8_t OFFSET_SIZE_BIT_SHIFT = 6;
+ static constexpr uint8_t SUPPORTED_VERSION = 1;
+
+ private:
+ std::string_view metadata_;
+ uint32_t dictionary_size_{0};
+};
+
+template <typename DecimalType>
+struct DecimalValue {
+ uint8_t scale;
+ DecimalType value;
+};
+
+struct VariantValue {
+ VariantMetadata metadata;
Review Comment:
Why not? They logically shares same `VariantMetadata`. And `VariantMetadata` is so heavyweight ( about 24bytes and not allocated on heap)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2890349188
> Is it possible to manually add unit tests covering scenarios such as null values, UUID, etc.?
@xxubai I've added some hand-written binary fmt for testing..
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2866212036
:warning: GitHub issue #46371 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2093319616
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+std::string variantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNTZ = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNTZ = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTZ = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNTZNanos = 19, // Differentiating from TimestampNTZ (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+std::string variantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ OBJECT,
+ ARRAY,
+ VARIANT_NULL,
+ BOOLEAN,
+ INT8,
+ INT16,
+ INT32,
+ INT64,
+ STRING,
+ DOUBLE,
+ DECIMAL4,
+ DECIMAL8,
+ DECIMAL16,
+ DATE,
+ TIMESTAMP_TZ,
+ TIMESTAMP_NTZ,
+ FLOAT,
+ BINARY,
+ TIME,
+ TIMESTAMP_NANOS_TZ,
+ TIMESTAMP_NANOS_NTZ,
+ UUID
+};
+
+std::string variantTypeToString(VariantType type);
+
+class VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary size).
+ std::string_view getMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sortedStrings(), the metadata key is not guaranteed to be unique, so we use a
+ /// vector to store all the metadata ids.
+ ::arrow::internal::SmallVector<uint32_t, 1> getMetadataId(std::string_view key) const;
+
+ bool sortedStrings() const;
+ uint8_t offsetSize() const;
+ uint32_t dictionarySize() const;
+
+ private:
+ static uint32_t loadDictionarySize(std::string_view metadata, uint8_t offset_size);
+
+ private:
+ static constexpr uint8_t VERSION_MASK = 0b1111;
Review Comment:
Actually they're private? I can move if neccessary but it's private class member ( both way is ok to me)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2088188833
##########
cpp/src/parquet/variant.cc:
##########
@@ -0,0 +1,618 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/variant.h"
+
+#include <cstdint>
+#include <iostream>
+#include <string_view>
+
+#include "arrow/util/endian.h"
+#include "parquet/exception.h"
+
+namespace parquet::variant {
+
+VariantMetadata::VariantMetadata(std::string_view metadata) : metadata_(metadata) {
+ if (metadata.size() < 2) {
+ throw ParquetException("Invalid Variant metadata: too short: " +
+ std::to_string(metadata.size()));
+ }
+}
+
+int8_t VariantMetadata::version() const {
+ return static_cast<int8_t>(metadata_[0]) & 0x0F;
+}
+
+bool VariantMetadata::sortedStrings() const { return (metadata_[0] & 0b10000) != 0; }
+
+uint8_t VariantMetadata::offsetSize() const { return ((metadata_[0] >> 6) & 0x3) + 1; }
+
+uint32_t VariantMetadata::dictionarySize() const {
+ uint8_t length = offsetSize();
+ if (length > 4) {
+ throw ParquetException("Invalid offset size: " + std::to_string(length));
+ }
+ if (length + 1 > metadata_.size()) {
+ throw ParquetException("Invalid Variant metadata: too short for dictionary size");
+ }
+ uint32_t dict_size = 0;
+ memcpy(&dict_size, metadata_.data() + 1, length);
+ dict_size = arrow::bit_util::FromLittleEndian(dict_size);
+ return dict_size;
+}
+
+std::string_view VariantMetadata::getMetadataKey(int32_t variantId) const {
+ uint32_t offset_size = offsetSize();
+ uint32_t dict_size = dictionarySize();
+
+ if (variantId < 0 || variantId >= static_cast<int32_t>(dict_size)) {
+ throw ParquetException("Invalid Variant metadata: variantId out of range");
+ }
+
+ if ((dict_size + 1) * offset_size > metadata_.size()) {
+ throw ParquetException("Invalid Variant metadata: offset out of range");
+ }
+
+ size_t offset_start_pos = 1 + offset_size + (variantId * offset_size);
+
+ uint32_t variant_offset = 0;
+ uint32_t variant_next_offset = 0;
+ memcpy(&variant_offset, metadata_.data() + offset_start_pos, offset_size);
+ variant_offset = arrow::bit_util::FromLittleEndian(variant_offset);
+ memcpy(&variant_next_offset, metadata_.data() + offset_start_pos + offset_size,
+ offset_size);
+ variant_next_offset = arrow::bit_util::FromLittleEndian(variant_next_offset);
+
+ uint32_t key_size = variant_next_offset - variant_offset;
+
+ size_t string_start = 1 + offset_size * (dict_size + 2) + variant_offset;
+ if (string_start + key_size > metadata_.size()) {
+ throw ParquetException("Invalid Variant metadata: string data out of range");
+ }
+ return {metadata_.data() + string_start, key_size};
+}
+
+VariantBasicType VariantValue::getBasicType() const {
+ if (value.empty()) {
+ throw ParquetException("Empty variant value");
+ }
+ return static_cast<VariantBasicType>(value[0] & BASIC_TYPE_MASK);
+}
+
+VariantType VariantValue::getType() const {
+ VariantBasicType basic_type = getBasicType();
+ // std::cout << "Variant first byte:" << static_cast<int>(value[0] >> 2) << ", "
+ // << static_cast<int>(value[0] && BASIC_TYPE_MASK) << '\n';
+ switch (basic_type) {
+ case VariantBasicType::Primitive: {
+ auto primitive_type = static_cast<VariantPrimitiveType>(value[0] >> 2);
+ switch (primitive_type) {
+ case VariantPrimitiveType::NullType:
+ return VariantType::VARIANT_NULL;
+ case VariantPrimitiveType::BooleanTrue:
+ case VariantPrimitiveType::BooleanFalse:
+ return VariantType::BOOLEAN;
+ case VariantPrimitiveType::Int8:
+ return VariantType::BYTE;
+ case VariantPrimitiveType::Int16:
+ return VariantType::SHORT;
+ case VariantPrimitiveType::Int32:
+ return VariantType::INT;
+ case VariantPrimitiveType::Int64:
+ return VariantType::LONG;
+ case VariantPrimitiveType::Double:
+ return VariantType::DOUBLE;
+ case VariantPrimitiveType::Decimal4:
+ return VariantType::DECIMAL4;
+ case VariantPrimitiveType::Decimal8:
+ return VariantType::DECIMAL8;
+ case VariantPrimitiveType::Decimal16:
+ return VariantType::DECIMAL16;
+ case VariantPrimitiveType::Date:
+ return VariantType::DATE;
+ case VariantPrimitiveType::Timestamp:
+ return VariantType::TIMESTAMP_TZ;
+ case VariantPrimitiveType::TimestampNTZ:
+ return VariantType::TIMESTAMP_NTZ;
+ case VariantPrimitiveType::Float:
+ return VariantType::FLOAT;
+ case VariantPrimitiveType::Binary:
+ return VariantType::BINARY;
+ case VariantPrimitiveType::String:
+ return VariantType::STRING;
+ case VariantPrimitiveType::TimeNTZ:
+ return VariantType::TIME;
+ case VariantPrimitiveType::TimestampTZ:
+ return VariantType::TIMESTAMP_NANOS_TZ;
+ case VariantPrimitiveType::TimestampNTZNanos:
+ return VariantType::TIMESTAMP_NANOS_NTZ;
+ case VariantPrimitiveType::Uuid:
+ return VariantType::UUID;
+ default:
+ throw ParquetException("Unknown primitive type: " +
+ std::to_string(static_cast<int>(primitive_type)));
+ }
+ }
+ case VariantBasicType::ShortString:
+ return VariantType::STRING;
+ case VariantBasicType::Object:
+ return VariantType::OBJECT;
+ case VariantBasicType::Array:
+ return VariantType::ARRAY;
+ default:
+ throw ParquetException("Unknown basic type: " +
+ std::to_string(static_cast<int>(basic_type)));
+ }
+}
+
+std::string VariantValue::typeDebugString() const {
+ VariantType type = getType();
+ switch (type) {
+ case VariantType::OBJECT:
+ return "OBJECT";
+ case VariantType::ARRAY:
+ return "ARRAY";
+ case VariantType::VARIANT_NULL:
+ return "NULL";
+ case VariantType::BOOLEAN:
+ return "BOOLEAN";
+ case VariantType::BYTE:
+ return "BYTE";
+ case VariantType::SHORT:
+ return "SHORT";
+ case VariantType::INT:
+ return "INT";
+ case VariantType::LONG:
+ return "LONG";
+ case VariantType::STRING:
+ return "STRING";
+ case VariantType::DOUBLE:
+ return "DOUBLE";
+ case VariantType::DECIMAL4:
+ return "DECIMAL4";
+ case VariantType::DECIMAL8:
+ return "DECIMAL8";
+ case VariantType::DECIMAL16:
+ return "DECIMAL16";
+ case VariantType::DATE:
+ return "DATE";
+ case VariantType::TIMESTAMP_TZ:
+ return "TIMESTAMP_TZ";
+ case VariantType::TIMESTAMP_NTZ:
+ return "TIMESTAMP_NTZ";
+ case VariantType::FLOAT:
+ return "FLOAT";
+ case VariantType::BINARY:
+ return "BINARY";
+ case VariantType::TIME:
+ return "TIME";
+ case VariantType::TIMESTAMP_NANOS_TZ:
+ return "TIMESTAMP_NANOS_TZ";
+ case VariantType::TIMESTAMP_NANOS_NTZ:
+ return "TIMESTAMP_NANOS_NTZ";
+ case VariantType::UUID:
+ return "UUID";
+ default:
+ return "UNKNOWN";
+ }
+}
+
+bool VariantValue::getBool() const {
+ if (getBasicType() != VariantBasicType::Primitive) {
+ throw ParquetException("Not a primitive type");
+ }
+
+ int8_t primitive_type = static_cast<int8_t>(value[0]) >> 2;
+ if (primitive_type == static_cast<int8_t>(VariantPrimitiveType::BooleanTrue)) {
+ return true;
+ }
+ if (primitive_type == static_cast<int8_t>(VariantPrimitiveType::BooleanFalse)) {
+ return false;
+ }
+
+ throw ParquetException("Not a variant primitive boolean type with primitive type: " +
+ std::to_string(primitive_type));
+}
+
+template <typename PrimitiveType>
+PrimitiveType VariantValue::getPrimitiveVariantType(VariantPrimitiveType type) const {
+ if (getBasicType() != VariantBasicType::Primitive) {
+ throw ParquetException("Not a primitive type");
+ }
+
+ auto primitive_type = static_cast<VariantPrimitiveType>(value[0] >> 2);
+ if (primitive_type != VariantPrimitiveType::Int8) {
+ throw ParquetException("Not an correspond type");
+ }
+
+ if (value.size() < 1 + sizeof(PrimitiveType)) {
+ throw ParquetException("Invalid value: too short");
+ }
+
+ PrimitiveType decimal_value{};
+ memcpy(&decimal_value, value.data() + 1, sizeof(PrimitiveType));
+ return decimal_value;
+}
+
+int8_t VariantValue::getInt8() const {
+ return getPrimitiveVariantType<int8_t>(VariantPrimitiveType::Int8);
+}
+
+int16_t VariantValue::getInt16() const {
+ return getPrimitiveVariantType<int8_t>(VariantPrimitiveType::Int16);
+}
+
+int32_t VariantValue::getInt32() const {
+ return getPrimitiveVariantType<int8_t>(VariantPrimitiveType::Int32);
+}
+
+int64_t VariantValue::getInt64() const {
+ return getPrimitiveVariantType<int8_t>(VariantPrimitiveType::Int64);
+}
+
+float VariantValue::getFloat() const {
+ return getPrimitiveVariantType<float>(VariantPrimitiveType::Float);
+}
+
+double VariantValue::getDouble() const {
+ return getPrimitiveVariantType<float>(VariantPrimitiveType::Double);
+}
+
+std::string_view VariantValue::getPrimitiveBinaryType(VariantPrimitiveType type) const {
+ VariantBasicType basic_type = getBasicType();
+ if (basic_type != VariantBasicType::Primitive) {
+ throw ParquetException("Not a primitive type");
+ }
+ auto primitive_type = static_cast<VariantPrimitiveType>(value[0] >> 2);
+ if (primitive_type != VariantPrimitiveType::String) {
+ throw ParquetException("Not a string type");
+ }
+
+ if (value.size() < 5) {
+ throw ParquetException("Invalid string value: too short");
+ }
+
+ uint32_t length;
+ memcpy(&length, value.data() + 1, sizeof(uint32_t));
+ length = arrow::bit_util::FromLittleEndian(length);
+
+ if (value.size() < length + 5) {
+ throw ParquetException("Invalid string value: too short for specified length");
+ }
+
+ return {value.data() + 5, length};
+}
+
+std::string_view VariantValue::getString() const {
+ VariantBasicType basic_type = getBasicType();
+
+ if (basic_type == VariantBasicType::ShortString) {
+ uint8_t length = (value[0] >> 2) & MAX_SHORT_STR_SIZE_MASK;
+ if (value.size() < length + 1) {
+ throw ParquetException("Invalid short string: too short");
+ }
+ return {value.data() + 1, length};
+ }
+ if (basic_type == VariantBasicType::Primitive) {
+ // TODO(mwish): Should we validate utf8 here?
+ return getPrimitiveBinaryType(VariantPrimitiveType::String);
+ }
+
+ throw ParquetException("Not a primitive or short string type calls getString");
+}
+
+std::string_view VariantValue::getBinary() const {
+ return getPrimitiveBinaryType(VariantPrimitiveType::Binary);
+}
+
+template <typename DecimalType>
+DecimalValue<DecimalType> VariantValue::getPrimitiveDecimalType(
+ VariantPrimitiveType type) const {
+ using DecimalValueType = typename DecimalType::ValueType;
+ if (getBasicType() != VariantBasicType::Primitive) {
+ throw ParquetException("Not a primitive type");
+ }
+
+ auto primitive_type = static_cast<VariantPrimitiveType>(value[0] >> 2);
+ if (primitive_type != type) {
+ throw ParquetException("Not a decimal type");
+ }
+
+ if (value.size() < 2 + sizeof(DecimalValueType)) {
+ throw ParquetException("Invalid decimal value: too short");
+ }
+
+ uint8_t scale = value[1];
+ DecimalValueType decimal_value;
+ memcpy(&decimal_value, value.data() + 2, sizeof(DecimalValueType));
+ decimal_value = arrow::bit_util::FromLittleEndian(decimal_value);
+
+ return {scale, DecimalType(decimal_value)};
+}
+
+DecimalValue<::arrow::Decimal32> VariantValue::getDecimal4() const {
+ return getPrimitiveDecimalType<::arrow::Decimal32>(VariantPrimitiveType::Decimal4);
+}
+
+DecimalValue<::arrow::Decimal64> VariantValue::getDecimal8() const {
+ return getPrimitiveDecimalType<::arrow::Decimal64>(VariantPrimitiveType::Decimal8);
+}
+
+DecimalValue<::arrow::Decimal128> VariantValue::getDecimal16() const {
+ if (getBasicType() != VariantBasicType::Primitive) {
+ throw ParquetException("Not a primitive type");
+ }
+
+ auto primitive_type = static_cast<VariantPrimitiveType>(value[0] >> 2);
+ if (primitive_type != VariantPrimitiveType::Decimal16) {
+ throw ParquetException("Not a decimal16 type");
+ }
+
+ if (value.size() < 2 + sizeof(int64_t) * 2) {
+ throw ParquetException("Invalid decimal16 value: too short");
+ }
+
+ uint8_t scale = value[1];
+
+ // TODO(mwish): Do we have better way for this?
+ std::array<int64_t, 2> low_high_bits;
+ memcpy(&low_high_bits[0], value.data() + 2, sizeof(int64_t));
+ memcpy(&low_high_bits[1], value.data() + 10, sizeof(int64_t));
+ ::arrow::bit_util::little_endian::ToNative(low_high_bits);
+ return {scale, ::arrow::Decimal128(low_high_bits[1], low_high_bits[0])};
+}
+
+int64_t VariantValue::timeNTZ() const {
+ return getPrimitiveVariantType<int64_t>(VariantPrimitiveType::TimeNTZ);
+}
+
+int64_t VariantValue::getTimestamp() const {
+ return getPrimitiveVariantType<int64_t>(VariantPrimitiveType::Timestamp);
+}
+
+int64_t VariantValue::getTimestampNTZ() const {
+ return getPrimitiveVariantType<int64_t>(VariantPrimitiveType::TimestampNTZ);
+}
+
+const uint8_t* VariantValue::getUuid() const {
+ throw ParquetException("VariantValue::getUuid Not implemented");
+}
+
+std::string VariantValue::ObjectInfo::toDebugString() const {
+ std::stringstream ss;
+ ss << "ObjectInfo{"
+ << "num_elements=" << num_elements
+ << ", id_size=" << static_cast<int>(id_size)
+ << ", offset_size=" << static_cast<int>(offset_size)
+ << ", id_start_offset=" << id_start_offset
+ << ", offset_start_offset=" << offset_start_offset
+ << ", data_start_offset=" << data_start_offset
+ << "}";
+ return ss.str();
+}
+
+
+VariantValue::ObjectInfo VariantValue::getObjectInfo() const {
+ if (getBasicType() != VariantBasicType::Object) {
+ throw ParquetException("Not an object type");
+ }
+ uint8_t value_header = value[0] >> 2;
+ uint8_t field_offset_size = (value_header & 0b11) + 1;
+ uint8_t field_id_size = ((value_header >> 2) & 0b11) + 1;
+ bool is_large = ((value_header >> 4) & 0b1);
+ uint8_t num_elements_size = is_large ? 4 : 1;
+ if (value.size() < 1 + num_elements_size) {
+ throw ParquetException("Invalid object value: too short: " +
+ std::to_string(value.size()) + " for at least " +
+ std::to_string(1 + num_elements_size));
+ }
+ // parse num_elements
+ uint32_t num_elements = 0;
+ {
+ memcpy(&num_elements, value.data() + 1, num_elements_size);
+ num_elements = arrow::bit_util::FromLittleEndian(num_elements);
+ }
+ ObjectInfo info{};
+ info.num_elements = num_elements;
+ info.id_size = field_id_size;
+ info.offset_size = field_offset_size;
+ info.id_start_offset = 1 + num_elements_size;
+ info.offset_start_offset = info.id_start_offset + num_elements * field_id_size;
+ info.data_start_offset = info.offset_start_offset + (num_elements + 1) * field_offset_size;
+ // Check the boundary with the final offset
+ if (info.data_start_offset > value.size()) {
+ throw ParquetException("Invalid object value: data_start_offset=" +
+ std::to_string(info.data_start_offset) +
+ ", value_size=" + std::to_string(value.size()));
+ }
+ {
+ uint32_t final_offset = 0;
Review Comment:
Should we delay this check to another function?
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,228 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+std::string variantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNTZ = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNTZ = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTZ = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNTZNanos = 19, // Differentiating from TimestampNTZ (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+std::string variantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ OBJECT,
+ ARRAY,
+ VARIANT_NULL,
+ BOOLEAN,
+ BYTE,
+ SHORT,
+ INT,
+ LONG,
+ STRING,
+ DOUBLE,
+ DECIMAL4,
+ DECIMAL8,
+ DECIMAL16,
+ DATE,
+ TIMESTAMP_TZ,
+ TIMESTAMP_NTZ,
+ FLOAT,
+ BINARY,
+ TIME,
+ TIMESTAMP_NANOS_TZ,
+ TIMESTAMP_NANOS_NTZ,
+ UUID
+};
+
+std::string variantTypeToString(VariantType type);
+
+class VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ int8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ std::string_view getMetadataKey(int32_t variantId) const;
+
+ private:
+ bool sortedStrings() const;
+ uint8_t offsetSize() const;
+ uint32_t dictionarySize() const;
+
+ private:
+ std::string_view metadata_;
+};
+
+template <typename DecimalType>
+struct DecimalValue {
+ uint8_t scale;
+ DecimalType value;
+};
+
+struct VariantValue {
+ VariantMetadata metadata;
+ std::string_view value;
+
+ VariantBasicType getBasicType() const;
+ VariantType getType() const;
+ std::string typeDebugString() const;
+
+ /// \defgroup ValueAccessors
+ /// @{
+
+ // Note: Null doesn't need visitor.
+ bool getBool() const;
+ int8_t getInt8() const;
+ int16_t getInt16() const;
+ int32_t getInt32() const;
+ int64_t getInt64() const;
Review Comment:
Currently, `getInt64` only supports read from int64, which is too strict for integer. I think we can also uses some way to allow getInt64 to get some "smaller types" like int32, int16, int8.
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,228 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+std::string variantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNTZ = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNTZ = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTZ = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNTZNanos = 19, // Differentiating from TimestampNTZ (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+std::string variantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ OBJECT,
+ ARRAY,
+ VARIANT_NULL,
+ BOOLEAN,
+ BYTE,
+ SHORT,
+ INT,
+ LONG,
+ STRING,
+ DOUBLE,
+ DECIMAL4,
+ DECIMAL8,
+ DECIMAL16,
+ DATE,
+ TIMESTAMP_TZ,
+ TIMESTAMP_NTZ,
+ FLOAT,
+ BINARY,
+ TIME,
+ TIMESTAMP_NANOS_TZ,
+ TIMESTAMP_NANOS_NTZ,
+ UUID
+};
+
+std::string variantTypeToString(VariantType type);
+
+class VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ int8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ std::string_view getMetadataKey(int32_t variantId) const;
+
+ private:
+ bool sortedStrings() const;
+ uint8_t offsetSize() const;
+ uint32_t dictionarySize() const;
+
+ private:
+ std::string_view metadata_;
+};
+
+template <typename DecimalType>
+struct DecimalValue {
+ uint8_t scale;
+ DecimalType value;
+};
+
+struct VariantValue {
+ VariantMetadata metadata;
+ std::string_view value;
+
+ VariantBasicType getBasicType() const;
+ VariantType getType() const;
+ std::string typeDebugString() const;
+
+ /// \defgroup ValueAccessors
+ /// @{
+
+ // Note: Null doesn't need visitor.
+ bool getBool() const;
+ int8_t getInt8() const;
+ int16_t getInt16() const;
+ int32_t getInt32() const;
+ int64_t getInt64() const;
+ /// Include short_string optimization and primitive string type
+ std::string_view getString() const;
Review Comment:
Currently I didn't check utf-8 here.
##########
cpp/src/parquet/variant.cc:
##########
@@ -0,0 +1,744 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/variant.h"
+
+#include <cstdint>
+#include <iostream>
+#include <string_view>
+
+#include "arrow/util/endian.h"
+#include "parquet/exception.h"
+
+namespace parquet::variant {
+
+std::string variantBasicTypeToString(VariantBasicType type) {
+ switch (type) {
+ case VariantBasicType::Primitive:
+ return "Primitive";
+ case VariantBasicType::ShortString:
+ return "ShortString";
+ case VariantBasicType::Object:
+ return "Object";
+ case VariantBasicType::Array:
+ return "Array";
+ default:
+ return "Unknown";
+ }
+}
+
+std::string variantPrimitiveTypeToString(VariantPrimitiveType type) {
+ switch (type) {
+ case VariantPrimitiveType::NullType:
+ return "NullType";
+ case VariantPrimitiveType::BooleanTrue:
+ return "BooleanTrue";
+ case VariantPrimitiveType::BooleanFalse:
+ return "BooleanFalse";
+ case VariantPrimitiveType::Int8:
+ return "Int8";
+ case VariantPrimitiveType::Int16:
+ return "Int16";
+ case VariantPrimitiveType::Int32:
+ return "Int32";
+ case VariantPrimitiveType::Int64:
+ return "Int64";
+ case VariantPrimitiveType::Double:
+ return "Double";
+ case VariantPrimitiveType::Decimal4:
+ return "Decimal4";
+ case VariantPrimitiveType::Decimal8:
+ return "Decimal8";
+ case VariantPrimitiveType::Decimal16:
+ return "Decimal16";
+ case VariantPrimitiveType::Date:
+ return "Date";
+ case VariantPrimitiveType::Timestamp:
+ return "Timestamp";
+ case VariantPrimitiveType::TimestampNTZ:
+ return "TimestampNTZ";
+ case VariantPrimitiveType::Float:
+ return "Float";
+ case VariantPrimitiveType::Binary:
+ return "Binary";
+ case VariantPrimitiveType::String:
+ return "String";
+ case VariantPrimitiveType::TimeNTZ:
+ return "TimeNTZ";
+ case VariantPrimitiveType::TimestampTZ:
+ return "TimestampTZ";
+ case VariantPrimitiveType::TimestampNTZNanos:
+ return "TimestampNTZNanos";
+ case VariantPrimitiveType::Uuid:
+ return "Uuid";
+ default:
+ return "Unknown";
+ }
+}
+
+std::string variantTypeToString(VariantType type) {
+ switch (type) {
+ case VariantType::OBJECT:
+ return "OBJECT";
+ case VariantType::ARRAY:
+ return "ARRAY";
+ case VariantType::VARIANT_NULL:
+ return "NULL";
+ case VariantType::BOOLEAN:
+ return "BOOLEAN";
+ case VariantType::BYTE:
+ return "BYTE";
+ case VariantType::SHORT:
+ return "SHORT";
+ case VariantType::INT:
+ return "INT";
+ case VariantType::LONG:
+ return "LONG";
+ case VariantType::STRING:
+ return "STRING";
+ case VariantType::DOUBLE:
+ return "DOUBLE";
+ case VariantType::DECIMAL4:
+ return "DECIMAL4";
+ case VariantType::DECIMAL8:
+ return "DECIMAL8";
+ case VariantType::DECIMAL16:
+ return "DECIMAL16";
+ case VariantType::DATE:
+ return "DATE";
+ case VariantType::TIMESTAMP_TZ:
+ return "TIMESTAMP_TZ";
+ case VariantType::TIMESTAMP_NTZ:
+ return "TIMESTAMP_NTZ";
+ case VariantType::FLOAT:
+ return "FLOAT";
+ case VariantType::BINARY:
+ return "BINARY";
+ case VariantType::TIME:
+ return "TIME";
+ case VariantType::TIMESTAMP_NANOS_TZ:
+ return "TIMESTAMP_NANOS_TZ";
+ case VariantType::TIMESTAMP_NANOS_NTZ:
+ return "TIMESTAMP_NANOS_NTZ";
+ case VariantType::UUID:
+ return "UUID";
+ default:
+ return "UNKNOWN";
+ }
+}
+
+VariantMetadata::VariantMetadata(std::string_view metadata) : metadata_(metadata) {
+ if (metadata.size() < 2) {
+ throw ParquetException("Invalid Variant metadata: too short: " +
+ std::to_string(metadata.size()));
+ }
+}
+
+int8_t VariantMetadata::version() const {
+ return static_cast<int8_t>(metadata_[0]) & 0x0F;
+}
+
+bool VariantMetadata::sortedStrings() const { return (metadata_[0] & 0b10000) != 0; }
+
+uint8_t VariantMetadata::offsetSize() const { return ((metadata_[0] >> 6) & 0x3) + 1; }
+
+uint32_t VariantMetadata::dictionarySize() const {
+ uint8_t length = offsetSize();
+ if (length > 4) {
+ throw ParquetException("Invalid offset size: " + std::to_string(length));
+ }
+ if (length + 1 > metadata_.size()) {
+ throw ParquetException("Invalid Variant metadata: too short for dictionary size");
+ }
+ uint32_t dict_size = 0;
+ memcpy(&dict_size, metadata_.data() + 1, length);
+ dict_size = arrow::bit_util::FromLittleEndian(dict_size);
+ return dict_size;
+}
+
+std::string_view VariantMetadata::getMetadataKey(int32_t variantId) const {
+ uint32_t offset_size = offsetSize();
+ uint32_t dict_size = dictionarySize();
+
+ if (variantId < 0 || variantId >= static_cast<int32_t>(dict_size)) {
+ throw ParquetException("Invalid Variant metadata: variantId out of range");
+ }
+
+ if ((dict_size + 1) * offset_size > metadata_.size()) {
+ throw ParquetException("Invalid Variant metadata: offset out of range");
+ }
+
+ size_t offset_start_pos = 1 + offset_size + (variantId * offset_size);
+
+ uint32_t variant_offset = 0;
+ uint32_t variant_next_offset = 0;
+ memcpy(&variant_offset, metadata_.data() + offset_start_pos, offset_size);
+ variant_offset = arrow::bit_util::FromLittleEndian(variant_offset);
+ memcpy(&variant_next_offset, metadata_.data() + offset_start_pos + offset_size,
+ offset_size);
+ variant_next_offset = arrow::bit_util::FromLittleEndian(variant_next_offset);
+
+ uint32_t key_size = variant_next_offset - variant_offset;
+
+ size_t string_start = 1 + offset_size * (dict_size + 2) + variant_offset;
+ if (string_start + key_size > metadata_.size()) {
+ throw ParquetException("Invalid Variant metadata: string data out of range");
+ }
+ return {metadata_.data() + string_start, key_size};
+}
+
+VariantBasicType VariantValue::getBasicType() const {
+ if (value.empty()) {
+ throw ParquetException("Empty variant value");
+ }
+ return static_cast<VariantBasicType>(value[0] & BASIC_TYPE_MASK);
+}
+
+VariantType VariantValue::getType() const {
+ VariantBasicType basic_type = getBasicType();
+ switch (basic_type) {
+ case VariantBasicType::Primitive: {
+ auto primitive_type = static_cast<VariantPrimitiveType>(value[0] >> 2);
+ switch (primitive_type) {
+ case VariantPrimitiveType::NullType:
+ return VariantType::VARIANT_NULL;
+ case VariantPrimitiveType::BooleanTrue:
+ case VariantPrimitiveType::BooleanFalse:
+ return VariantType::BOOLEAN;
+ case VariantPrimitiveType::Int8:
+ return VariantType::BYTE;
+ case VariantPrimitiveType::Int16:
+ return VariantType::SHORT;
+ case VariantPrimitiveType::Int32:
+ return VariantType::INT;
+ case VariantPrimitiveType::Int64:
+ return VariantType::LONG;
+ case VariantPrimitiveType::Double:
+ return VariantType::DOUBLE;
+ case VariantPrimitiveType::Decimal4:
+ return VariantType::DECIMAL4;
+ case VariantPrimitiveType::Decimal8:
+ return VariantType::DECIMAL8;
+ case VariantPrimitiveType::Decimal16:
+ return VariantType::DECIMAL16;
+ case VariantPrimitiveType::Date:
+ return VariantType::DATE;
+ case VariantPrimitiveType::Timestamp:
+ return VariantType::TIMESTAMP_TZ;
+ case VariantPrimitiveType::TimestampNTZ:
+ return VariantType::TIMESTAMP_NTZ;
+ case VariantPrimitiveType::Float:
+ return VariantType::FLOAT;
+ case VariantPrimitiveType::Binary:
+ return VariantType::BINARY;
+ case VariantPrimitiveType::String:
+ return VariantType::STRING;
+ case VariantPrimitiveType::TimeNTZ:
+ return VariantType::TIME;
+ case VariantPrimitiveType::TimestampTZ:
+ return VariantType::TIMESTAMP_NANOS_TZ;
+ case VariantPrimitiveType::TimestampNTZNanos:
+ return VariantType::TIMESTAMP_NANOS_NTZ;
+ case VariantPrimitiveType::Uuid:
+ return VariantType::UUID;
+ default:
+ throw ParquetException("Unknown primitive type: " +
+ std::to_string(static_cast<int>(primitive_type)));
+ }
+ }
+ case VariantBasicType::ShortString:
+ return VariantType::STRING;
+ case VariantBasicType::Object:
+ return VariantType::OBJECT;
+ case VariantBasicType::Array:
+ return VariantType::ARRAY;
+ default:
+ throw ParquetException("Unknown basic type: " +
+ std::to_string(static_cast<int>(basic_type)));
+ }
+}
+
+std::string VariantValue::typeDebugString() const {
+ VariantType type = getType();
+ switch (type) {
+ case VariantType::OBJECT:
+ return "OBJECT";
+ case VariantType::ARRAY:
+ return "ARRAY";
+ case VariantType::VARIANT_NULL:
+ return "NULL";
+ case VariantType::BOOLEAN:
+ return "BOOLEAN";
+ case VariantType::BYTE:
+ return "BYTE";
+ case VariantType::SHORT:
+ return "SHORT";
+ case VariantType::INT:
+ return "INT";
+ case VariantType::LONG:
+ return "LONG";
+ case VariantType::STRING:
+ return "STRING";
+ case VariantType::DOUBLE:
+ return "DOUBLE";
+ case VariantType::DECIMAL4:
+ return "DECIMAL4";
+ case VariantType::DECIMAL8:
+ return "DECIMAL8";
+ case VariantType::DECIMAL16:
+ return "DECIMAL16";
+ case VariantType::DATE:
+ return "DATE";
+ case VariantType::TIMESTAMP_TZ:
+ return "TIMESTAMP_TZ";
+ case VariantType::TIMESTAMP_NTZ:
+ return "TIMESTAMP_NTZ";
+ case VariantType::FLOAT:
+ return "FLOAT";
+ case VariantType::BINARY:
+ return "BINARY";
+ case VariantType::TIME:
+ return "TIME";
+ case VariantType::TIMESTAMP_NANOS_TZ:
+ return "TIMESTAMP_NANOS_TZ";
+ case VariantType::TIMESTAMP_NANOS_NTZ:
+ return "TIMESTAMP_NANOS_NTZ";
+ case VariantType::UUID:
+ return "UUID";
+ default:
+ return "UNKNOWN";
+ }
+}
+
+bool VariantValue::getBool() const {
+ if (getBasicType() != VariantBasicType::Primitive) {
+ throw ParquetException("Expected primitive type, but got: " +
+ variantBasicTypeToString(getBasicType()));
+ }
+
+ int8_t primitive_type = static_cast<int8_t>(value[0]) >> 2;
+ if (primitive_type == static_cast<int8_t>(VariantPrimitiveType::BooleanTrue)) {
+ return true;
+ }
+ if (primitive_type == static_cast<int8_t>(VariantPrimitiveType::BooleanFalse)) {
+ return false;
+ }
+
+ throw ParquetException("Not a variant primitive boolean type with primitive type: " +
+ std::to_string(primitive_type));
+}
+
+void VariantValue::checkBasicType(VariantBasicType type) const {
+ if (getBasicType() != type) {
+ throw ParquetException("Expected basic type: " + variantBasicTypeToString(type) +
+ ", but got: " + variantBasicTypeToString(getBasicType()));
+ }
+}
+
+void VariantValue::checkPrimitiveType(VariantPrimitiveType type,
+ size_t size_required) const {
+ checkBasicType(VariantBasicType::Primitive);
+
+ auto primitive_type = static_cast<VariantPrimitiveType>(value[0] >> 2);
+ if (primitive_type != type) {
+ throw ParquetException(
+ "Expected primitive type: " + variantPrimitiveTypeToString(type) +
+ ", but got: " + variantPrimitiveTypeToString(primitive_type));
+ }
+
+ if (value.size() < size_required) {
+ throw ParquetException("Invalid value: too short, expected at least " +
+ std::to_string(size_required) + " bytes for type " +
+ variantPrimitiveTypeToString(type) +
+ ", but got: " + std::to_string(value.size()) + " bytes");
+ }
+}
+
+template <typename PrimitiveType>
+PrimitiveType VariantValue::getPrimitiveType(VariantPrimitiveType type) const {
+ checkPrimitiveType(type, sizeof(PrimitiveType) + 1);
+
+ PrimitiveType primitive_value{};
+ memcpy(&primitive_value, value.data() + 1, sizeof(PrimitiveType));
+ // Here we should cast from Little endian.
+ primitive_value = ::arrow::bit_util::FromLittleEndian(primitive_value);
+ return primitive_value;
+}
+
+int8_t VariantValue::getInt8() const {
+ return getPrimitiveType<int8_t>(VariantPrimitiveType::Int8);
+}
+
+int16_t VariantValue::getInt16() const {
+ return getPrimitiveType<int16_t>(VariantPrimitiveType::Int16);
+}
+
+int32_t VariantValue::getInt32() const {
+ return getPrimitiveType<int32_t>(VariantPrimitiveType::Int32);
+}
+
+int64_t VariantValue::getInt64() const {
+ return getPrimitiveType<int64_t>(VariantPrimitiveType::Int64);
+}
+
+float VariantValue::getFloat() const {
+ return getPrimitiveType<float>(VariantPrimitiveType::Float);
+}
+
+double VariantValue::getDouble() const {
+ return getPrimitiveType<double>(VariantPrimitiveType::Double);
+}
+
+std::string_view VariantValue::getPrimitiveBinaryType(VariantPrimitiveType type) const {
+ checkPrimitiveType(type, /*size_required=*/5);
+
+ uint32_t length;
+ memcpy(&length, value.data() + 1, sizeof(uint32_t));
+ length = arrow::bit_util::FromLittleEndian(length);
+
+ if (value.size() < length + 5) {
+ throw ParquetException("Invalid string value: too short for specified length");
+ }
+
+ return {value.data() + 5, length};
+}
+
+std::string_view VariantValue::getString() const {
+ VariantBasicType basic_type = getBasicType();
+
+ if (basic_type == VariantBasicType::ShortString) {
+ uint8_t length = (value[0] >> 2) & MAX_SHORT_STR_SIZE_MASK;
+ if (value.size() < length + 1) {
+ throw ParquetException("Invalid short string: too short");
+ }
+ return {value.data() + 1, length};
+ }
+ if (basic_type == VariantBasicType::Primitive) {
+ // TODO(mwish): Should we validate utf8 here?
+ return getPrimitiveBinaryType(VariantPrimitiveType::String);
+ }
+
+ throw ParquetException("Not a primitive or short string type calls getString");
+}
+
+std::string_view VariantValue::getBinary() const {
+ return getPrimitiveBinaryType(VariantPrimitiveType::Binary);
+}
+
+template <typename DecimalType>
+DecimalValue<DecimalType> VariantValue::getPrimitiveDecimalType(
+ VariantPrimitiveType type) const {
+ using DecimalValueType = typename DecimalType::ValueType;
+ checkPrimitiveType(type, sizeof(DecimalValueType) + 2);
+
+ uint8_t scale = value[1];
+ DecimalValueType decimal_value;
+ memcpy(&decimal_value, value.data() + 2, sizeof(DecimalValueType));
+ decimal_value = arrow::bit_util::FromLittleEndian(decimal_value);
+
+ return {scale, DecimalType(decimal_value)};
+}
+
+DecimalValue<::arrow::Decimal32> VariantValue::getDecimal4() const {
+ return getPrimitiveDecimalType<::arrow::Decimal32>(VariantPrimitiveType::Decimal4);
+}
+
+DecimalValue<::arrow::Decimal64> VariantValue::getDecimal8() const {
+ return getPrimitiveDecimalType<::arrow::Decimal64>(VariantPrimitiveType::Decimal8);
+}
+
+DecimalValue<::arrow::Decimal128> VariantValue::getDecimal16() const {
+ checkPrimitiveType(VariantPrimitiveType::Decimal16,
+ /*size_required=*/sizeof(int64_t) * 2 + 2);
+
+ uint8_t scale = value[1];
+
+ // TODO(mwish): Do we have better way for this?
+ std::array<int64_t, 2> low_high_bits;
+ memcpy(&low_high_bits[0], value.data() + 2, sizeof(int64_t));
+ memcpy(&low_high_bits[1], value.data() + 10, sizeof(int64_t));
+ ::arrow::bit_util::little_endian::ToNative(low_high_bits);
+ return {scale, ::arrow::Decimal128(low_high_bits[1], low_high_bits[0])};
+}
+
+int32_t VariantValue::getDate() const {
+ return getPrimitiveType<int32_t>(VariantPrimitiveType::Date);
+}
+
+int64_t VariantValue::getTimeNTZ() const {
+ return getPrimitiveType<int64_t>(VariantPrimitiveType::TimeNTZ);
+}
+
+int64_t VariantValue::getTimestamp() const {
+ return getPrimitiveType<int64_t>(VariantPrimitiveType::Timestamp);
+}
+
+int64_t VariantValue::getTimestampNTZ() const {
+ return getPrimitiveType<int64_t>(VariantPrimitiveType::TimestampNTZ);
+}
+
+std::array<uint8_t, 16> VariantValue::getUuid() const {
+ checkPrimitiveType(VariantPrimitiveType::Uuid, /*size_required=*/17);
+ std::array<uint8_t, 16> uuid_value;
+ memcpy(uuid_value.data(), value.data() + 1, sizeof(uuid_value));
+#if ARROW_LITTLE_ENDIAN
+ std::array<uint8_t, 16> uuid_value_le;
+ ::arrow::bit_util::ByteSwap(uuid_value_le.data(), uuid_value.data(), uuid_value.size());
+ return uuid_value_le;
+#else
+ return uuid_value;
+#endif
+}
+
+std::string VariantValue::ObjectInfo::toDebugString() const {
+ std::stringstream ss;
+ ss << "ObjectInfo{"
+ << "num_elements=" << num_elements
+ << ", id_size=" << static_cast<int>(id_size)
+ << ", offset_size=" << static_cast<int>(offset_size)
+ << ", id_start_offset=" << id_start_offset
+ << ", offset_start_offset=" << offset_start_offset
+ << ", data_start_offset=" << data_start_offset
+ << "}";
+ return ss.str();
+}
+
+
+VariantValue::ObjectInfo VariantValue::getObjectInfo() const {
+ checkBasicType(VariantBasicType::Object);
+ uint8_t value_header = value[0] >> 2;
+ uint8_t field_offset_size = (value_header & 0b11) + 1;
+ uint8_t field_id_size = ((value_header >> 2) & 0b11) + 1;
+ bool is_large = ((value_header >> 4) & 0b1);
+ uint8_t num_elements_size = is_large ? 4 : 1;
+ if (value.size() < 1 + num_elements_size) {
+ throw ParquetException("Invalid object value: too short: " +
+ std::to_string(value.size()) + " for at least " +
+ std::to_string(1 + num_elements_size));
+ }
+ // parse num_elements
+ uint32_t num_elements = 0;
+ {
+ memcpy(&num_elements, value.data() + 1, num_elements_size);
+ num_elements = arrow::bit_util::FromLittleEndian(num_elements);
+ }
+ ObjectInfo info{};
+ info.num_elements = num_elements;
+ info.id_size = field_id_size;
+ info.offset_size = field_offset_size;
+ info.id_start_offset = 1 + num_elements_size;
+ info.offset_start_offset = info.id_start_offset + num_elements * field_id_size;
+ info.data_start_offset = info.offset_start_offset + (num_elements + 1) * field_offset_size;
+ // Check the boundary with the final offset
+ if (info.data_start_offset > value.size()) {
+ throw ParquetException("Invalid object value: data_start_offset=" +
+ std::to_string(info.data_start_offset) +
+ ", value_size=" + std::to_string(value.size()));
+ }
+ {
+ uint32_t final_offset = 0;
+ memcpy(&final_offset,
+ value.data() + info.offset_start_offset + num_elements * field_offset_size,
+ field_offset_size);
+ // It could be less than value size since it could be a sub-object.
+ if (final_offset + info.data_start_offset > value.size()) {
+ throw ParquetException("Invalid object value: final_offset=" +
+ std::to_string(final_offset) +
+ ", data_start_offset=" + std::to_string(info.data_start_offset) +
+ ", value_size=" + std::to_string(value.size()));
+ }
+ }
+ return info;
+}
+
+std::optional<VariantValue> VariantValue::getObjectValueByKey(
+ std::string_view key) const {
+ ObjectInfo info = getObjectInfo();
+
+ return getObjectValueByKey(key, info);
+}
+
+std::optional<VariantValue> VariantValue::getObjectValueByKey(
+ std::string_view key, const VariantValue::ObjectInfo& info) const {
+ // TODO(mwish): Currently we just linear search here. The best way here is:
Review Comment:
Currently I just do linear search, this could be optimized later, but I don't want to finish it in this pr (will create an github issue when everyone agree with this interface)
##########
cpp/src/parquet/variant.cc:
##########
@@ -0,0 +1,744 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/variant.h"
+
+#include <cstdint>
+#include <iostream>
+#include <string_view>
+
+#include "arrow/util/endian.h"
+#include "parquet/exception.h"
+
+namespace parquet::variant {
+
+std::string variantBasicTypeToString(VariantBasicType type) {
+ switch (type) {
+ case VariantBasicType::Primitive:
+ return "Primitive";
+ case VariantBasicType::ShortString:
+ return "ShortString";
+ case VariantBasicType::Object:
+ return "Object";
+ case VariantBasicType::Array:
+ return "Array";
+ default:
+ return "Unknown";
+ }
+}
+
+std::string variantPrimitiveTypeToString(VariantPrimitiveType type) {
+ switch (type) {
+ case VariantPrimitiveType::NullType:
+ return "NullType";
+ case VariantPrimitiveType::BooleanTrue:
+ return "BooleanTrue";
+ case VariantPrimitiveType::BooleanFalse:
+ return "BooleanFalse";
+ case VariantPrimitiveType::Int8:
+ return "Int8";
+ case VariantPrimitiveType::Int16:
+ return "Int16";
+ case VariantPrimitiveType::Int32:
+ return "Int32";
+ case VariantPrimitiveType::Int64:
+ return "Int64";
+ case VariantPrimitiveType::Double:
+ return "Double";
+ case VariantPrimitiveType::Decimal4:
+ return "Decimal4";
+ case VariantPrimitiveType::Decimal8:
+ return "Decimal8";
+ case VariantPrimitiveType::Decimal16:
+ return "Decimal16";
+ case VariantPrimitiveType::Date:
+ return "Date";
+ case VariantPrimitiveType::Timestamp:
+ return "Timestamp";
+ case VariantPrimitiveType::TimestampNTZ:
+ return "TimestampNTZ";
+ case VariantPrimitiveType::Float:
+ return "Float";
+ case VariantPrimitiveType::Binary:
+ return "Binary";
+ case VariantPrimitiveType::String:
+ return "String";
+ case VariantPrimitiveType::TimeNTZ:
+ return "TimeNTZ";
+ case VariantPrimitiveType::TimestampTZ:
+ return "TimestampTZ";
+ case VariantPrimitiveType::TimestampNTZNanos:
+ return "TimestampNTZNanos";
+ case VariantPrimitiveType::Uuid:
+ return "Uuid";
+ default:
+ return "Unknown";
+ }
+}
+
+std::string variantTypeToString(VariantType type) {
+ switch (type) {
+ case VariantType::OBJECT:
+ return "OBJECT";
+ case VariantType::ARRAY:
+ return "ARRAY";
+ case VariantType::VARIANT_NULL:
+ return "NULL";
+ case VariantType::BOOLEAN:
+ return "BOOLEAN";
+ case VariantType::BYTE:
+ return "BYTE";
+ case VariantType::SHORT:
+ return "SHORT";
+ case VariantType::INT:
+ return "INT";
+ case VariantType::LONG:
+ return "LONG";
+ case VariantType::STRING:
+ return "STRING";
+ case VariantType::DOUBLE:
+ return "DOUBLE";
+ case VariantType::DECIMAL4:
+ return "DECIMAL4";
+ case VariantType::DECIMAL8:
+ return "DECIMAL8";
+ case VariantType::DECIMAL16:
+ return "DECIMAL16";
+ case VariantType::DATE:
+ return "DATE";
+ case VariantType::TIMESTAMP_TZ:
+ return "TIMESTAMP_TZ";
+ case VariantType::TIMESTAMP_NTZ:
+ return "TIMESTAMP_NTZ";
+ case VariantType::FLOAT:
+ return "FLOAT";
+ case VariantType::BINARY:
+ return "BINARY";
+ case VariantType::TIME:
+ return "TIME";
+ case VariantType::TIMESTAMP_NANOS_TZ:
+ return "TIMESTAMP_NANOS_TZ";
+ case VariantType::TIMESTAMP_NANOS_NTZ:
+ return "TIMESTAMP_NANOS_NTZ";
+ case VariantType::UUID:
+ return "UUID";
+ default:
+ return "UNKNOWN";
+ }
+}
+
+VariantMetadata::VariantMetadata(std::string_view metadata) : metadata_(metadata) {
+ if (metadata.size() < 2) {
+ throw ParquetException("Invalid Variant metadata: too short: " +
+ std::to_string(metadata.size()));
+ }
+}
+
+int8_t VariantMetadata::version() const {
+ return static_cast<int8_t>(metadata_[0]) & 0x0F;
+}
+
+bool VariantMetadata::sortedStrings() const { return (metadata_[0] & 0b10000) != 0; }
+
+uint8_t VariantMetadata::offsetSize() const { return ((metadata_[0] >> 6) & 0x3) + 1; }
+
+uint32_t VariantMetadata::dictionarySize() const {
+ uint8_t length = offsetSize();
+ if (length > 4) {
+ throw ParquetException("Invalid offset size: " + std::to_string(length));
+ }
+ if (length + 1 > metadata_.size()) {
+ throw ParquetException("Invalid Variant metadata: too short for dictionary size");
+ }
+ uint32_t dict_size = 0;
+ memcpy(&dict_size, metadata_.data() + 1, length);
+ dict_size = arrow::bit_util::FromLittleEndian(dict_size);
+ return dict_size;
+}
+
+std::string_view VariantMetadata::getMetadataKey(int32_t variantId) const {
+ uint32_t offset_size = offsetSize();
+ uint32_t dict_size = dictionarySize();
+
+ if (variantId < 0 || variantId >= static_cast<int32_t>(dict_size)) {
+ throw ParquetException("Invalid Variant metadata: variantId out of range");
+ }
+
+ if ((dict_size + 1) * offset_size > metadata_.size()) {
+ throw ParquetException("Invalid Variant metadata: offset out of range");
+ }
+
+ size_t offset_start_pos = 1 + offset_size + (variantId * offset_size);
+
+ uint32_t variant_offset = 0;
+ uint32_t variant_next_offset = 0;
+ memcpy(&variant_offset, metadata_.data() + offset_start_pos, offset_size);
+ variant_offset = arrow::bit_util::FromLittleEndian(variant_offset);
+ memcpy(&variant_next_offset, metadata_.data() + offset_start_pos + offset_size,
+ offset_size);
+ variant_next_offset = arrow::bit_util::FromLittleEndian(variant_next_offset);
+
+ uint32_t key_size = variant_next_offset - variant_offset;
+
+ size_t string_start = 1 + offset_size * (dict_size + 2) + variant_offset;
+ if (string_start + key_size > metadata_.size()) {
+ throw ParquetException("Invalid Variant metadata: string data out of range");
+ }
+ return {metadata_.data() + string_start, key_size};
+}
+
+VariantBasicType VariantValue::getBasicType() const {
+ if (value.empty()) {
+ throw ParquetException("Empty variant value");
+ }
+ return static_cast<VariantBasicType>(value[0] & BASIC_TYPE_MASK);
+}
+
+VariantType VariantValue::getType() const {
+ VariantBasicType basic_type = getBasicType();
+ switch (basic_type) {
+ case VariantBasicType::Primitive: {
+ auto primitive_type = static_cast<VariantPrimitiveType>(value[0] >> 2);
+ switch (primitive_type) {
+ case VariantPrimitiveType::NullType:
+ return VariantType::VARIANT_NULL;
+ case VariantPrimitiveType::BooleanTrue:
+ case VariantPrimitiveType::BooleanFalse:
+ return VariantType::BOOLEAN;
+ case VariantPrimitiveType::Int8:
+ return VariantType::BYTE;
+ case VariantPrimitiveType::Int16:
+ return VariantType::SHORT;
+ case VariantPrimitiveType::Int32:
+ return VariantType::INT;
+ case VariantPrimitiveType::Int64:
+ return VariantType::LONG;
+ case VariantPrimitiveType::Double:
+ return VariantType::DOUBLE;
+ case VariantPrimitiveType::Decimal4:
+ return VariantType::DECIMAL4;
+ case VariantPrimitiveType::Decimal8:
+ return VariantType::DECIMAL8;
+ case VariantPrimitiveType::Decimal16:
+ return VariantType::DECIMAL16;
+ case VariantPrimitiveType::Date:
+ return VariantType::DATE;
+ case VariantPrimitiveType::Timestamp:
+ return VariantType::TIMESTAMP_TZ;
+ case VariantPrimitiveType::TimestampNTZ:
+ return VariantType::TIMESTAMP_NTZ;
+ case VariantPrimitiveType::Float:
+ return VariantType::FLOAT;
+ case VariantPrimitiveType::Binary:
+ return VariantType::BINARY;
+ case VariantPrimitiveType::String:
+ return VariantType::STRING;
+ case VariantPrimitiveType::TimeNTZ:
+ return VariantType::TIME;
+ case VariantPrimitiveType::TimestampTZ:
+ return VariantType::TIMESTAMP_NANOS_TZ;
+ case VariantPrimitiveType::TimestampNTZNanos:
+ return VariantType::TIMESTAMP_NANOS_NTZ;
+ case VariantPrimitiveType::Uuid:
+ return VariantType::UUID;
+ default:
+ throw ParquetException("Unknown primitive type: " +
+ std::to_string(static_cast<int>(primitive_type)));
+ }
+ }
+ case VariantBasicType::ShortString:
+ return VariantType::STRING;
+ case VariantBasicType::Object:
+ return VariantType::OBJECT;
+ case VariantBasicType::Array:
+ return VariantType::ARRAY;
+ default:
+ throw ParquetException("Unknown basic type: " +
+ std::to_string(static_cast<int>(basic_type)));
+ }
+}
+
+std::string VariantValue::typeDebugString() const {
+ VariantType type = getType();
+ switch (type) {
+ case VariantType::OBJECT:
+ return "OBJECT";
+ case VariantType::ARRAY:
+ return "ARRAY";
+ case VariantType::VARIANT_NULL:
+ return "NULL";
+ case VariantType::BOOLEAN:
+ return "BOOLEAN";
+ case VariantType::BYTE:
+ return "BYTE";
+ case VariantType::SHORT:
+ return "SHORT";
+ case VariantType::INT:
+ return "INT";
+ case VariantType::LONG:
+ return "LONG";
+ case VariantType::STRING:
+ return "STRING";
+ case VariantType::DOUBLE:
+ return "DOUBLE";
+ case VariantType::DECIMAL4:
+ return "DECIMAL4";
+ case VariantType::DECIMAL8:
+ return "DECIMAL8";
+ case VariantType::DECIMAL16:
+ return "DECIMAL16";
+ case VariantType::DATE:
+ return "DATE";
+ case VariantType::TIMESTAMP_TZ:
+ return "TIMESTAMP_TZ";
+ case VariantType::TIMESTAMP_NTZ:
+ return "TIMESTAMP_NTZ";
+ case VariantType::FLOAT:
+ return "FLOAT";
+ case VariantType::BINARY:
+ return "BINARY";
+ case VariantType::TIME:
+ return "TIME";
+ case VariantType::TIMESTAMP_NANOS_TZ:
+ return "TIMESTAMP_NANOS_TZ";
+ case VariantType::TIMESTAMP_NANOS_NTZ:
+ return "TIMESTAMP_NANOS_NTZ";
+ case VariantType::UUID:
+ return "UUID";
+ default:
+ return "UNKNOWN";
+ }
+}
+
+bool VariantValue::getBool() const {
+ if (getBasicType() != VariantBasicType::Primitive) {
+ throw ParquetException("Expected primitive type, but got: " +
+ variantBasicTypeToString(getBasicType()));
+ }
+
+ int8_t primitive_type = static_cast<int8_t>(value[0]) >> 2;
+ if (primitive_type == static_cast<int8_t>(VariantPrimitiveType::BooleanTrue)) {
+ return true;
+ }
+ if (primitive_type == static_cast<int8_t>(VariantPrimitiveType::BooleanFalse)) {
+ return false;
+ }
+
+ throw ParquetException("Not a variant primitive boolean type with primitive type: " +
+ std::to_string(primitive_type));
+}
+
+void VariantValue::checkBasicType(VariantBasicType type) const {
+ if (getBasicType() != type) {
+ throw ParquetException("Expected basic type: " + variantBasicTypeToString(type) +
+ ", but got: " + variantBasicTypeToString(getBasicType()));
+ }
+}
+
+void VariantValue::checkPrimitiveType(VariantPrimitiveType type,
+ size_t size_required) const {
+ checkBasicType(VariantBasicType::Primitive);
+
+ auto primitive_type = static_cast<VariantPrimitiveType>(value[0] >> 2);
+ if (primitive_type != type) {
+ throw ParquetException(
+ "Expected primitive type: " + variantPrimitiveTypeToString(type) +
+ ", but got: " + variantPrimitiveTypeToString(primitive_type));
+ }
+
+ if (value.size() < size_required) {
+ throw ParquetException("Invalid value: too short, expected at least " +
+ std::to_string(size_required) + " bytes for type " +
+ variantPrimitiveTypeToString(type) +
+ ", but got: " + std::to_string(value.size()) + " bytes");
+ }
+}
+
+template <typename PrimitiveType>
+PrimitiveType VariantValue::getPrimitiveType(VariantPrimitiveType type) const {
+ checkPrimitiveType(type, sizeof(PrimitiveType) + 1);
+
+ PrimitiveType primitive_value{};
+ memcpy(&primitive_value, value.data() + 1, sizeof(PrimitiveType));
+ // Here we should cast from Little endian.
+ primitive_value = ::arrow::bit_util::FromLittleEndian(primitive_value);
+ return primitive_value;
+}
+
+int8_t VariantValue::getInt8() const {
+ return getPrimitiveType<int8_t>(VariantPrimitiveType::Int8);
+}
+
+int16_t VariantValue::getInt16() const {
+ return getPrimitiveType<int16_t>(VariantPrimitiveType::Int16);
+}
+
+int32_t VariantValue::getInt32() const {
+ return getPrimitiveType<int32_t>(VariantPrimitiveType::Int32);
+}
+
+int64_t VariantValue::getInt64() const {
+ return getPrimitiveType<int64_t>(VariantPrimitiveType::Int64);
+}
+
+float VariantValue::getFloat() const {
+ return getPrimitiveType<float>(VariantPrimitiveType::Float);
+}
+
+double VariantValue::getDouble() const {
+ return getPrimitiveType<double>(VariantPrimitiveType::Double);
+}
+
+std::string_view VariantValue::getPrimitiveBinaryType(VariantPrimitiveType type) const {
+ checkPrimitiveType(type, /*size_required=*/5);
+
+ uint32_t length;
+ memcpy(&length, value.data() + 1, sizeof(uint32_t));
+ length = arrow::bit_util::FromLittleEndian(length);
+
+ if (value.size() < length + 5) {
+ throw ParquetException("Invalid string value: too short for specified length");
+ }
+
+ return {value.data() + 5, length};
+}
+
+std::string_view VariantValue::getString() const {
+ VariantBasicType basic_type = getBasicType();
+
+ if (basic_type == VariantBasicType::ShortString) {
+ uint8_t length = (value[0] >> 2) & MAX_SHORT_STR_SIZE_MASK;
+ if (value.size() < length + 1) {
+ throw ParquetException("Invalid short string: too short");
+ }
+ return {value.data() + 1, length};
+ }
+ if (basic_type == VariantBasicType::Primitive) {
+ // TODO(mwish): Should we validate utf8 here?
+ return getPrimitiveBinaryType(VariantPrimitiveType::String);
+ }
+
+ throw ParquetException("Not a primitive or short string type calls getString");
+}
+
+std::string_view VariantValue::getBinary() const {
+ return getPrimitiveBinaryType(VariantPrimitiveType::Binary);
+}
+
+template <typename DecimalType>
+DecimalValue<DecimalType> VariantValue::getPrimitiveDecimalType(
+ VariantPrimitiveType type) const {
+ using DecimalValueType = typename DecimalType::ValueType;
+ checkPrimitiveType(type, sizeof(DecimalValueType) + 2);
+
+ uint8_t scale = value[1];
+ DecimalValueType decimal_value;
+ memcpy(&decimal_value, value.data() + 2, sizeof(DecimalValueType));
+ decimal_value = arrow::bit_util::FromLittleEndian(decimal_value);
+
+ return {scale, DecimalType(decimal_value)};
+}
+
+DecimalValue<::arrow::Decimal32> VariantValue::getDecimal4() const {
+ return getPrimitiveDecimalType<::arrow::Decimal32>(VariantPrimitiveType::Decimal4);
+}
+
+DecimalValue<::arrow::Decimal64> VariantValue::getDecimal8() const {
+ return getPrimitiveDecimalType<::arrow::Decimal64>(VariantPrimitiveType::Decimal8);
+}
+
+DecimalValue<::arrow::Decimal128> VariantValue::getDecimal16() const {
+ checkPrimitiveType(VariantPrimitiveType::Decimal16,
+ /*size_required=*/sizeof(int64_t) * 2 + 2);
+
+ uint8_t scale = value[1];
+
+ // TODO(mwish): Do we have better way for this?
+ std::array<int64_t, 2> low_high_bits;
+ memcpy(&low_high_bits[0], value.data() + 2, sizeof(int64_t));
+ memcpy(&low_high_bits[1], value.data() + 10, sizeof(int64_t));
+ ::arrow::bit_util::little_endian::ToNative(low_high_bits);
+ return {scale, ::arrow::Decimal128(low_high_bits[1], low_high_bits[0])};
+}
+
+int32_t VariantValue::getDate() const {
+ return getPrimitiveType<int32_t>(VariantPrimitiveType::Date);
+}
+
+int64_t VariantValue::getTimeNTZ() const {
+ return getPrimitiveType<int64_t>(VariantPrimitiveType::TimeNTZ);
+}
+
+int64_t VariantValue::getTimestamp() const {
+ return getPrimitiveType<int64_t>(VariantPrimitiveType::Timestamp);
+}
+
+int64_t VariantValue::getTimestampNTZ() const {
+ return getPrimitiveType<int64_t>(VariantPrimitiveType::TimestampNTZ);
+}
+
+std::array<uint8_t, 16> VariantValue::getUuid() const {
+ checkPrimitiveType(VariantPrimitiveType::Uuid, /*size_required=*/17);
+ std::array<uint8_t, 16> uuid_value;
+ memcpy(uuid_value.data(), value.data() + 1, sizeof(uuid_value));
+#if ARROW_LITTLE_ENDIAN
+ std::array<uint8_t, 16> uuid_value_le;
+ ::arrow::bit_util::ByteSwap(uuid_value_le.data(), uuid_value.data(), uuid_value.size());
+ return uuid_value_le;
+#else
+ return uuid_value;
+#endif
+}
+
+std::string VariantValue::ObjectInfo::toDebugString() const {
+ std::stringstream ss;
+ ss << "ObjectInfo{"
+ << "num_elements=" << num_elements
+ << ", id_size=" << static_cast<int>(id_size)
+ << ", offset_size=" << static_cast<int>(offset_size)
+ << ", id_start_offset=" << id_start_offset
+ << ", offset_start_offset=" << offset_start_offset
+ << ", data_start_offset=" << data_start_offset
+ << "}";
+ return ss.str();
+}
+
+
+VariantValue::ObjectInfo VariantValue::getObjectInfo() const {
+ checkBasicType(VariantBasicType::Object);
+ uint8_t value_header = value[0] >> 2;
+ uint8_t field_offset_size = (value_header & 0b11) + 1;
+ uint8_t field_id_size = ((value_header >> 2) & 0b11) + 1;
+ bool is_large = ((value_header >> 4) & 0b1);
+ uint8_t num_elements_size = is_large ? 4 : 1;
+ if (value.size() < 1 + num_elements_size) {
+ throw ParquetException("Invalid object value: too short: " +
+ std::to_string(value.size()) + " for at least " +
+ std::to_string(1 + num_elements_size));
+ }
+ // parse num_elements
+ uint32_t num_elements = 0;
+ {
+ memcpy(&num_elements, value.data() + 1, num_elements_size);
+ num_elements = arrow::bit_util::FromLittleEndian(num_elements);
+ }
+ ObjectInfo info{};
+ info.num_elements = num_elements;
+ info.id_size = field_id_size;
+ info.offset_size = field_offset_size;
+ info.id_start_offset = 1 + num_elements_size;
+ info.offset_start_offset = info.id_start_offset + num_elements * field_id_size;
+ info.data_start_offset = info.offset_start_offset + (num_elements + 1) * field_offset_size;
+ // Check the boundary with the final offset
+ if (info.data_start_offset > value.size()) {
+ throw ParquetException("Invalid object value: data_start_offset=" +
+ std::to_string(info.data_start_offset) +
+ ", value_size=" + std::to_string(value.size()));
+ }
+ {
+ uint32_t final_offset = 0;
+ memcpy(&final_offset,
+ value.data() + info.offset_start_offset + num_elements * field_offset_size,
+ field_offset_size);
+ // It could be less than value size since it could be a sub-object.
+ if (final_offset + info.data_start_offset > value.size()) {
+ throw ParquetException("Invalid object value: final_offset=" +
+ std::to_string(final_offset) +
+ ", data_start_offset=" + std::to_string(info.data_start_offset) +
+ ", value_size=" + std::to_string(value.size()));
+ }
+ }
+ return info;
+}
+
+std::optional<VariantValue> VariantValue::getObjectValueByKey(
+ std::string_view key) const {
+ ObjectInfo info = getObjectInfo();
+
+ return getObjectValueByKey(key, info);
+}
+
+std::optional<VariantValue> VariantValue::getObjectValueByKey(
+ std::string_view key, const VariantValue::ObjectInfo& info) const {
+ // TODO(mwish): Currently we just linear search here. The best way here is:
+ // 1. check the num_elements
+ // 2.1. If the element number is less than 8(or other magic number), we can keep
+ // current method.
+ // 2.2. If the element number is larger than 8, and metadata.sorted_strings is true,
+ // we can first apply binary search on the metadata, and then binary search the
+ // field id.
+
+ for (uint32_t i = 0; i < info.num_elements; ++i) {
+ std::string_view field_key;
+ std::optional<VariantValue> field_value = getObjectFieldByFieldId(i, &field_key);
+
+ if (field_key == key) {
+ return field_value;
+ }
+ }
+
+ return std::nullopt;
+}
+
+VariantValue VariantValue::getObjectFieldByFieldId(uint32_t variantId,
+ std::string_view* key) const {
+ ObjectInfo info = getObjectInfo();
+
+ if (variantId >= info.num_elements) {
+ throw ParquetException("Field ID out of range: " + std::to_string(variantId) +
+ " >= " + std::to_string(info.num_elements));
+ }
+
+ // Read the field ID
+ uint32_t field_id = 0;
+ memcpy(&field_id, value.data() + info.id_start_offset + variantId * info.id_size,
+ info.id_size);
+ field_id = arrow::bit_util::FromLittleEndian(field_id);
+
+ // Get the key from metadata
+ *key = metadata.getMetadataKey(static_cast<int32_t>(field_id));
+
+ // Read the offset and next offset
+ uint32_t offset = 0, next_offset = 0;
+ memcpy(&offset, value.data() + info.offset_start_offset + variantId * info.offset_size,
+ info.offset_size);
+ memcpy(&next_offset,
+ value.data() + info.offset_start_offset + (variantId + 1) * info.offset_size,
+ info.offset_size);
+ offset = arrow::bit_util::FromLittleEndian(offset);
+
+ if (info.data_start_offset + offset > value.size()) {
+ throw ParquetException("Invalid object field offsets: data_start_offset=" +
+ std::to_string(info.data_start_offset) +
+ ", offset=" + std::to_string(offset) +
+ ", value_size=" + std::to_string(value.size()));
+ }
+
+ // Create a VariantValue for the field
+ VariantValue field_value{
+ .metadata = metadata,
+ .value = value.substr(info.data_start_offset + offset)};
+
+ return field_value;
+}
+
+VariantValue::ArrayInfo VariantValue::getArrayInfo() const {
+ checkBasicType(VariantBasicType::Array);
+ uint8_t value_header = value[0] >> 2;
+ uint8_t field_offset_size = (value_header & 0b11) + 1;
+ bool is_large = ((value_header >> 2) & 0b1);
+
+ // check the array header
+ uint8_t num_elements_size = is_large ? 4 : 1;
+ if (value.size() < 1 + num_elements_size) {
+ throw ParquetException(
+ "Invalid array value: too short: " + std::to_string(value.size()) +
+ " for at least " + std::to_string(1 + num_elements_size));
+ }
+
+ // parse num_elements
+ uint32_t num_elements = 0;
+ {
+ memcpy(&num_elements, value.data() + 1, num_elements_size);
+ num_elements = arrow::bit_util::FromLittleEndian(num_elements);
+ }
+
+ ArrayInfo info{};
+ info.num_elements = num_elements;
+ info.offset_size = field_offset_size;
+ info.offset_start_offset = 1 + num_elements_size;
+ info.data_start_offset =
+ info.offset_start_offset + (num_elements + 1) * field_offset_size;
+
+ // Boundary check
+ if (info.data_start_offset > value.size()) {
+ throw ParquetException("Invalid array value: data_start_offset=" +
+ std::to_string(info.data_start_offset) +
+ ", value_size=" + std::to_string(value.size()));
+ }
+
+ // Validate final offset is equal to the size of the value,
+ // it would work since even empty array would have an offset of 0.
+ {
+ uint32_t final_offset = 0;
+ memcpy(&final_offset,
+ value.data() + info.offset_start_offset + num_elements * field_offset_size,
+ field_offset_size);
+ final_offset = arrow::bit_util::FromLittleEndian(final_offset);
+
+ if (info.data_start_offset + final_offset > value.size()) {
+ throw ParquetException(
+ "Invalid array value: final_offset=" + std::to_string(final_offset) +
+ ", data_start_offset=" + std::to_string(info.data_start_offset) +
+ ", value_size=" + std::to_string(value.size()));
+ }
+ }
+
+ // checking the element is incremental.
+ // TODO(mwish): Remove this or encapsulate this range check to function
Review Comment:
I think should we use a extra function here like "Validate", or just checks them here?
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,228 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+std::string variantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNTZ = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNTZ = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTZ = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNTZNanos = 19, // Differentiating from TimestampNTZ (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+std::string variantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ OBJECT,
+ ARRAY,
+ VARIANT_NULL,
+ BOOLEAN,
+ BYTE,
+ SHORT,
+ INT,
+ LONG,
+ STRING,
+ DOUBLE,
+ DECIMAL4,
+ DECIMAL8,
+ DECIMAL16,
+ DATE,
+ TIMESTAMP_TZ,
+ TIMESTAMP_NTZ,
+ FLOAT,
+ BINARY,
+ TIME,
+ TIMESTAMP_NANOS_TZ,
+ TIMESTAMP_NANOS_NTZ,
+ UUID
+};
+
+std::string variantTypeToString(VariantType type);
+
+class VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ int8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ std::string_view getMetadataKey(int32_t variantId) const;
+
+ private:
+ bool sortedStrings() const;
+ uint8_t offsetSize() const;
+ uint32_t dictionarySize() const;
+
+ private:
+ std::string_view metadata_;
+};
+
+template <typename DecimalType>
+struct DecimalValue {
+ uint8_t scale;
+ DecimalType value;
+};
+
+struct VariantValue {
+ VariantMetadata metadata;
+ std::string_view value;
+
+ VariantBasicType getBasicType() const;
+ VariantType getType() const;
+ std::string typeDebugString() const;
+
+ /// \defgroup ValueAccessors
+ /// @{
+
+ // Note: Null doesn't need visitor.
+ bool getBool() const;
+ int8_t getInt8() const;
+ int16_t getInt16() const;
+ int32_t getInt32() const;
+ int64_t getInt64() const;
+ /// Include short_string optimization and primitive string type
+ std::string_view getString() const;
+ std::string_view getBinary() const;
+ float getFloat() const;
+ double getDouble() const;
Review Comment:
Currently, `getDouble` only supports read from `getFloat`, which is too strict for. Maybe we can also uses some way to allow `getDouble` get other types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2897505907
Thanks! The read/write implementation will not be too complex without shredding. However design the proper interface would be a little challenge here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "scovich (via GitHub)" <gi...@apache.org>.
scovich commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2100787349
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+std::string variantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNTZ = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNTZ = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTZ = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNTZNanos = 19, // Differentiating from TimestampNTZ (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+std::string variantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ OBJECT,
+ ARRAY,
+ VARIANT_NULL,
+ BOOLEAN,
+ INT8,
+ INT16,
+ INT32,
+ INT64,
+ STRING,
+ DOUBLE,
+ DECIMAL4,
+ DECIMAL8,
+ DECIMAL16,
+ DATE,
+ TIMESTAMP_TZ,
+ TIMESTAMP_NTZ,
+ FLOAT,
+ BINARY,
+ TIME,
+ TIMESTAMP_NANOS_TZ,
+ TIMESTAMP_NANOS_NTZ,
+ UUID
+};
+
+std::string variantTypeToString(VariantType type);
+
+class VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary size).
+ std::string_view getMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sortedStrings(), the metadata key is not guaranteed to be unique, so we use a
+ /// vector to store all the metadata ids.
+ ::arrow::internal::SmallVector<uint32_t, 1> getMetadataId(std::string_view key) const;
+
+ bool sortedStrings() const;
+ uint8_t offsetSize() const;
+ uint32_t dictionarySize() const;
+
+ private:
+ static uint32_t loadDictionarySize(std::string_view metadata, uint8_t offset_size);
+
+ private:
+ static constexpr uint8_t VERSION_MASK = 0b1111;
+ static constexpr uint8_t SORTED_STRING_MASK = 0b10000;
+ static constexpr size_t HEADER_SIZE_BYTES = 1;
+ static constexpr size_t MINIMAL_OFFSET_SIZE_BYTES = 1;
+ static constexpr size_t MAXIMUM_OFFSET_SIZE_BYTES = 4;
+ // mask is applied after shift, it's like 0b11000000 before shift.
+ static constexpr uint8_t OFFSET_SIZE_MASK = 0b11;
+ static constexpr uint8_t OFFSET_SIZE_BIT_SHIFT = 6;
+ static constexpr uint8_t SUPPORTED_VERSION = 1;
+
+ private:
+ std::string_view metadata_;
+ uint32_t dictionary_size_{0};
+};
+
+template <typename DecimalType>
+struct DecimalValue {
+ uint8_t scale;
+ DecimalType value;
+};
+
+struct VariantValue {
+ VariantMetadata metadata;
+ std::string_view value;
+
+ VariantBasicType getBasicType() const;
+ VariantType getType() const;
+ std::string typeDebugString() const;
+
+ /// \defgroup ValueAccessors
+ /// @{
+
+ // Note: Null doesn't need visitor.
+ bool getBool() const;
+ int8_t getInt8() const;
+ int16_t getInt16() const;
+ int32_t getInt32() const;
+ int64_t getInt64() const;
+ /// Include short_string optimization and primitive string type
+ std::string_view getString() const;
+ std::string_view getBinary() const;
+ float getFloat() const;
+ double getDouble() const;
+
+ DecimalValue<::arrow::Decimal32> getDecimal4() const;
+ DecimalValue<::arrow::Decimal64> getDecimal8() const;
+ DecimalValue<::arrow::Decimal128> getDecimal16() const;
+
+ int32_t getDate() const;
+ int64_t getTimeNTZ() const;
+ // timestamp with adjusted to UTC
+ int64_t getTimestamp() const;
+ int64_t getTimestampNTZ() const;
+ // 16 bytes UUID
+ std::array<uint8_t, 16> getUuid() const;
Review Comment:
Also, `string_view` would occupy (at least) the same 16 bytes as this array, with a more complex structure to access. And byte-swapping is a single machine instruction in most architectures so the conversion should be super cheap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2109621841
##########
cpp/src/parquet/variant.cc:
##########
@@ -300,8 +300,7 @@ VariantType VariantValue::getType() const {
VariantBasicType basic_type = getBasicType();
switch (basic_type) {
case VariantBasicType::Primitive: {
- auto primitive_type =
- static_cast<VariantPrimitiveType>(value_[0] >> kValueHeaderBitShift);
+ auto primitive_type = static_cast<VariantPrimitiveType>(valueHeader());
Review Comment:
I'm a bit confused. `valueHeader` seems to be just doing `>> kValueHeaderBitShift`, which means that the primitive type would be `valueHeader() & 0x3f`, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2109729997
##########
cpp/src/parquet/variant.cc:
##########
@@ -300,8 +300,7 @@ VariantType VariantValue::getType() const {
VariantBasicType basic_type = getBasicType();
switch (basic_type) {
case VariantBasicType::Primitive: {
- auto primitive_type =
- static_cast<VariantPrimitiveType>(value_[0] >> kValueHeaderBitShift);
+ auto primitive_type = static_cast<VariantPrimitiveType>(valueHeader());
Review Comment:
if it's not primitive type, `value_header` might be length for string, or be used as `field_offset_length` etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2105481208
##########
cpp/src/parquet/variant.cc:
##########
@@ -0,0 +1,779 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/variant.h"
+
+#include <cstdint>
+#include <iostream>
+#include <string_view>
+
+#include <arrow/util/endian.h>
+#include <arrow/util/logging.h>
+#include <arrow/util/unreachable.h>
+
+#include "parquet/exception.h"
+
+namespace parquet::variant {
+
+std::string VariantBasicTypeToString(VariantBasicType type) {
+ switch (type) {
+ case VariantBasicType::Primitive:
+ return "Primitive";
+ case VariantBasicType::ShortString:
+ return "ShortString";
+ case VariantBasicType::Object:
+ return "Object";
+ case VariantBasicType::Array:
+ return "Array";
+ default:
+ return "Unknown";
+ }
+}
+
+std::string VariantPrimitiveTypeToString(VariantPrimitiveType type) {
+ switch (type) {
+ case VariantPrimitiveType::NullType:
+ return "NullType";
+ case VariantPrimitiveType::BooleanTrue:
+ return "BooleanTrue";
+ case VariantPrimitiveType::BooleanFalse:
+ return "BooleanFalse";
+ case VariantPrimitiveType::Int8:
+ return "Int8";
+ case VariantPrimitiveType::Int16:
+ return "Int16";
+ case VariantPrimitiveType::Int32:
+ return "Int32";
+ case VariantPrimitiveType::Int64:
+ return "Int64";
+ case VariantPrimitiveType::Double:
+ return "Double";
+ case VariantPrimitiveType::Decimal4:
+ return "Decimal4";
+ case VariantPrimitiveType::Decimal8:
+ return "Decimal8";
+ case VariantPrimitiveType::Decimal16:
+ return "Decimal16";
+ case VariantPrimitiveType::Date:
+ return "Date";
+ case VariantPrimitiveType::TimestampMicros:
+ return "TimestampMicros";
+ case VariantPrimitiveType::TimestampMicrosNtz:
+ return "TimestampMicrosNtz";
+ case VariantPrimitiveType::Float:
+ return "Float";
+ case VariantPrimitiveType::Binary:
+ return "Binary";
+ case VariantPrimitiveType::String:
+ return "String";
+ case VariantPrimitiveType::TimeMicrosNtz:
+ return "TimeMicrosNtz";
+ case VariantPrimitiveType::TimestampNanosTz:
+ return "TimestampNanosTz";
+ case VariantPrimitiveType::TimestampNanosNtz:
+ return "TimestampNanosNtz";
+ case VariantPrimitiveType::Uuid:
+ return "Uuid";
+ default:
+ return "Unknown";
+ }
+}
+
+std::string VariantTypeToString(VariantType type) {
+ switch (type) {
+ case VariantType::Object:
+ return "Object";
+ case VariantType::Array:
+ return "Array";
+ case VariantType::Null:
+ return "Null";
+ case VariantType::Boolean:
+ return "Boolean";
+ case VariantType::Int8:
+ return "Int8";
+ case VariantType::Int16:
+ return "Int16";
+ case VariantType::Int32:
+ return "Int32";
+ case VariantType::Int64:
+ return "Int64";
+ case VariantType::String:
+ return "String";
+ case VariantType::Double:
+ return "Double";
+ case VariantType::Decimal4:
+ return "Decimal4";
+ case VariantType::Decimal8:
+ return "Decimal8";
+ case VariantType::Decimal16:
+ return "Decimal16";
+ case VariantType::Date:
+ return "Date";
+ case VariantType::TimestampMicrosTz:
+ return "TimestampMicrosTz";
+ case VariantType::TimestampMicrosNtz:
+ return "TimestampMicrosNtz";
+ case VariantType::Float:
+ return "Float";
+ case VariantType::Binary:
+ return "Binary";
+ case VariantType::Time:
+ return "Time";
+ case VariantType::TimestampNanosTz:
+ return "TimestampNanosTz";
+ case VariantType::TimestampNanosNtz:
+ return "TimestampNanosNtz";
+ case VariantType::Uuid:
+ return "Uuid";
+ default:
+ return "Unknown";
+ }
+}
+
+inline uint32_t readLittleEndianU32(const void* from, uint8_t size) {
+ ARROW_DCHECK_LE(size, 4);
+ ARROW_DCHECK_GE(size, 1);
+
+ uint32_t result = 0;
+ memcpy(&result, from, size);
+ return ::arrow::bit_util::FromLittleEndian(result);
+}
+
+VariantMetadata::VariantMetadata(std::string_view metadata) : metadata_(metadata) {
+ if (metadata.size() < kHeaderSizeBytes + kMinimalOffsetSizeBytes * 2) {
+ // Empty metadata is at least 3 bytes: version, dictionarySize and
+ // at least one offset.
+ throw ParquetException("Invalid Variant metadata: too short: size=" +
+ std::to_string(metadata.size()));
+ }
+ if (version() != kSupportedVersion) {
+ // Currently we only supports version 1.
+ throw ParquetException("Unsupported Variant metadata version: " +
+ std::to_string(version()));
+ }
+ uint8_t offset_sz = offset_size();
+ if (offset_sz < kMinimalOffsetSizeBytes || offset_sz > kMaximumOffsetSizeBytes) {
+ throw ParquetException("Invalid Variant metadata: invalid offset size: " +
+ std::to_string(offset_sz));
+ }
+ dictionary_size_ = loadDictionarySize(metadata, offset_sz);
+ if (kHeaderSizeBytes + (dictionary_size_ + 2) * offset_sz > metadata_.size()) {
+ throw ParquetException(
+ "Invalid Variant metadata: offset out of range: " +
+ std::to_string((dictionary_size_ + kHeaderSizeBytes) * offset_sz) + " > " +
+ std::to_string(metadata_.size()));
+ }
Review Comment:
In metadata I delayed the validating offsets to getter, it can also put here to validate
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2917006069
> Since there seems to be a controversy about a good API for this, how about we start with something lower-level, such as a SAX-like parser (i.e. event-driven)? Then we can build up a higher-level API on top of it once we know which kind of API would be efficient?
@pitrou SAX style parser means consume the whole token and parsing it to a variant object. It's ok when we'd like to parse the whole object 🤔, however, maybe it would be a bit expansive when we want to visit one or multiple columns? Currently I just implement a `Variant` object for this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-4622014721
Thank you for your contribution. Unfortunately, this pull request has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this PR will be closed in 14 days. Feel free to re-open this if it has been closed in error. If you do not have repository permissions to reopen the PR, please tag a maintainer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2917028244
Emm i mean, for writer it might be good, for reader, get a key not requires visit/parse the whole binary with `Key("...")` api?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2093316956
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+std::string variantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNTZ = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNTZ = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTZ = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNTZNanos = 19, // Differentiating from TimestampNTZ (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+std::string variantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ OBJECT,
+ ARRAY,
+ VARIANT_NULL,
+ BOOLEAN,
+ INT8,
+ INT16,
+ INT32,
+ INT64,
+ STRING,
+ DOUBLE,
+ DECIMAL4,
+ DECIMAL8,
+ DECIMAL16,
+ DATE,
+ TIMESTAMP_TZ,
+ TIMESTAMP_NTZ,
+ FLOAT,
+ BINARY,
+ TIME,
+ TIMESTAMP_NANOS_TZ,
+ TIMESTAMP_NANOS_NTZ,
+ UUID
+};
+
+std::string variantTypeToString(VariantType type);
+
+class VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary size).
+ std::string_view getMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sortedStrings(), the metadata key is not guaranteed to be unique, so we use a
+ /// vector to store all the metadata ids.
+ ::arrow::internal::SmallVector<uint32_t, 1> getMetadataId(std::string_view key) const;
+
+ bool sortedStrings() const;
+ uint8_t offsetSize() const;
+ uint32_t dictionarySize() const;
+
+ private:
+ static uint32_t loadDictionarySize(std::string_view metadata, uint8_t offset_size);
+
+ private:
+ static constexpr uint8_t VERSION_MASK = 0b1111;
+ static constexpr uint8_t SORTED_STRING_MASK = 0b10000;
+ static constexpr size_t HEADER_SIZE_BYTES = 1;
+ static constexpr size_t MINIMAL_OFFSET_SIZE_BYTES = 1;
+ static constexpr size_t MAXIMUM_OFFSET_SIZE_BYTES = 4;
+ // mask is applied after shift, it's like 0b11000000 before shift.
+ static constexpr uint8_t OFFSET_SIZE_MASK = 0b11;
+ static constexpr uint8_t OFFSET_SIZE_BIT_SHIFT = 6;
+ static constexpr uint8_t SUPPORTED_VERSION = 1;
+
+ private:
+ std::string_view metadata_;
+ uint32_t dictionary_size_{0};
+};
+
+template <typename DecimalType>
+struct DecimalValue {
+ uint8_t scale;
+ DecimalType value;
+};
+
+struct VariantValue {
+ VariantMetadata metadata;
+ std::string_view value;
+
+ VariantBasicType getBasicType() const;
+ VariantType getType() const;
+ std::string typeDebugString() const;
+
+ /// \defgroup ValueAccessors
+ /// @{
+
+ // Note: Null doesn't need visitor.
+ bool getBool() const;
Review Comment:
Fair enough. `std::variant` pays for a full deserialization.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2102015282
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+#include "parquet/platform.h"
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+PARQUET_EXPORT std::string VariantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ TimestampMicros = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampMicrosNtz = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeMicrosNtz = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampNanosTz = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNanosNtz = 19, // Differentiating from TimestampNtz (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+PARQUET_EXPORT std::string VariantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ Object,
+ Array,
+ Null,
+ Boolean,
+ Int8,
+ Int16,
+ Int32,
+ Int64,
+ String,
+ Double,
+ Decimal4,
+ Decimal8,
+ Decimal16,
+ Date,
+ TimestampMicrosTz,
+ TimestampMicrosNtz,
+ Float,
+ Binary,
+ Time,
+ TimestampNanosTz,
+ TimestampNanosNtz,
+ Uuid
+};
+
+PARQUET_EXPORT std::string VariantTypeToString(VariantType type);
+
+class PARQUET_EXPORT VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary_size).
+ std::string_view GetMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sorted_and_unique(), the metadata key is not guaranteed to be unique, so we use a
Review Comment:
I don't know, but I think whether the writer promises that it doesn't produce value like this, whether the reader should pay the effort. Otherwise it's a bug
For example, `{"a": {"a": 1}}` reads "a" in inner object, it has `field_id: 1` in field-id list. But the metadata returns 0 for "a", so it returns "not exists" for get "a". In this implementation it should get "field_ids: [0, 1]", and find "1" in it's field-id list.
I think most of the scenerio we don't need care about this, so uses `SmallVector` here. And if `sorted_and_unique`, it can have an optimization on find the keys
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2918581135
Thanks for the explanation @wgtmac . I retract my suggestion then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2103684369
##########
cpp/src/parquet/variant.cc:
##########
@@ -0,0 +1,771 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/variant.h"
+
+#include <cstdint>
+#include <iostream>
+#include <string_view>
+
+#include <arrow/util/endian.h>
+#include <arrow/util/logging.h>
+#include <arrow/util/unreachable.h>
+
+#include "parquet/exception.h"
+
+namespace parquet::variant {
+
+std::string VariantBasicTypeToString(VariantBasicType type) {
+ switch (type) {
+ case VariantBasicType::Primitive:
+ return "Primitive";
+ case VariantBasicType::ShortString:
+ return "ShortString";
+ case VariantBasicType::Object:
+ return "Object";
+ case VariantBasicType::Array:
+ return "Array";
+ default:
+ return "Unknown";
+ }
+}
+
+std::string VariantPrimitiveTypeToString(VariantPrimitiveType type) {
+ switch (type) {
+ case VariantPrimitiveType::NullType:
+ return "NullType";
+ case VariantPrimitiveType::BooleanTrue:
+ return "BooleanTrue";
+ case VariantPrimitiveType::BooleanFalse:
+ return "BooleanFalse";
+ case VariantPrimitiveType::Int8:
+ return "Int8";
+ case VariantPrimitiveType::Int16:
+ return "Int16";
+ case VariantPrimitiveType::Int32:
+ return "Int32";
+ case VariantPrimitiveType::Int64:
+ return "Int64";
+ case VariantPrimitiveType::Double:
+ return "Double";
+ case VariantPrimitiveType::Decimal4:
+ return "Decimal4";
+ case VariantPrimitiveType::Decimal8:
+ return "Decimal8";
+ case VariantPrimitiveType::Decimal16:
+ return "Decimal16";
+ case VariantPrimitiveType::Date:
+ return "Date";
+ case VariantPrimitiveType::TimestampMicros:
+ return "TimestampMicros";
+ case VariantPrimitiveType::TimestampMicrosNtz:
+ return "TimestampMicrosNtz";
+ case VariantPrimitiveType::Float:
+ return "Float";
+ case VariantPrimitiveType::Binary:
+ return "Binary";
+ case VariantPrimitiveType::String:
+ return "String";
+ case VariantPrimitiveType::TimeMicrosNtz:
+ return "TimeMicrosNtz";
+ case VariantPrimitiveType::TimestampNanosTz:
+ return "TimestampNanosTz";
+ case VariantPrimitiveType::TimestampNanosNtz:
+ return "TimestampNanosNtz";
+ case VariantPrimitiveType::Uuid:
+ return "Uuid";
+ default:
+ return "Unknown";
+ }
+}
+
+std::string VariantTypeToString(VariantType type) {
+ switch (type) {
+ case VariantType::Object:
+ return "Object";
+ case VariantType::Array:
+ return "Array";
+ case VariantType::Null:
+ return "Null";
+ case VariantType::Boolean:
+ return "Boolean";
+ case VariantType::Int8:
+ return "Int8";
+ case VariantType::Int16:
+ return "Int16";
+ case VariantType::Int32:
+ return "Int32";
+ case VariantType::Int64:
+ return "Int64";
+ case VariantType::String:
+ return "String";
+ case VariantType::Double:
+ return "Double";
+ case VariantType::Decimal4:
+ return "Decimal4";
+ case VariantType::Decimal8:
+ return "Decimal8";
+ case VariantType::Decimal16:
+ return "Decimal16";
+ case VariantType::Date:
+ return "Date";
+ case VariantType::TimestampMicrosTz:
+ return "TimestampMicrosTz";
+ case VariantType::TimestampMicrosNtz:
+ return "TimestampMicrosNtz";
+ case VariantType::Float:
+ return "Float";
+ case VariantType::Binary:
+ return "Binary";
+ case VariantType::Time:
+ return "Time";
+ case VariantType::TimestampNanosTz:
+ return "TimestampNanosTz";
+ case VariantType::TimestampNanosNtz:
+ return "TimestampNanosNtz";
+ case VariantType::Uuid:
+ return "Uuid";
+ default:
+ return "Unknown";
+ }
+}
+
+inline uint32_t readLittleEndianU32(const void* from, uint8_t size) {
+ ARROW_DCHECK_LE(size, 4);
+ ARROW_DCHECK_GE(size, 1);
+
+ uint32_t result = 0;
+ memcpy(&result, from, size);
+ return ::arrow::bit_util::FromLittleEndian(result);
+}
+
+VariantMetadata::VariantMetadata(std::string_view metadata) : metadata_(metadata) {
+ if (metadata.size() < kHeaderSizeBytes + kMinimalOffsetSizeBytes * 2) {
+ // Empty metadata is at least 3 bytes: version, dictionarySize and
+ // at least one offset.
+ throw ParquetException("Invalid Variant metadata: too short: size=" +
+ std::to_string(metadata.size()));
+ }
+ if (version() != kSupportedVersion) {
+ // Currently we only supports version 1.
+ throw ParquetException("Unsupported Variant metadata version: " +
+ std::to_string(version()));
+ }
+ uint8_t offset_sz = offset_size();
+ if (offset_sz < kMinimalOffsetSizeBytes || offset_sz > kMaximumOffsetSizeBytes) {
+ throw ParquetException("Invalid Variant metadata: invalid offset size: " +
+ std::to_string(offset_sz));
+ }
Review Comment:
offset_sz already checked in `readLittleEndianU32`?
```
ARROW_DCHECK_LE(size, 4);
ARROW_DCHECK_GE(size, 1);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2105481208
##########
cpp/src/parquet/variant.cc:
##########
@@ -0,0 +1,779 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/variant.h"
+
+#include <cstdint>
+#include <iostream>
+#include <string_view>
+
+#include <arrow/util/endian.h>
+#include <arrow/util/logging.h>
+#include <arrow/util/unreachable.h>
+
+#include "parquet/exception.h"
+
+namespace parquet::variant {
+
+std::string VariantBasicTypeToString(VariantBasicType type) {
+ switch (type) {
+ case VariantBasicType::Primitive:
+ return "Primitive";
+ case VariantBasicType::ShortString:
+ return "ShortString";
+ case VariantBasicType::Object:
+ return "Object";
+ case VariantBasicType::Array:
+ return "Array";
+ default:
+ return "Unknown";
+ }
+}
+
+std::string VariantPrimitiveTypeToString(VariantPrimitiveType type) {
+ switch (type) {
+ case VariantPrimitiveType::NullType:
+ return "NullType";
+ case VariantPrimitiveType::BooleanTrue:
+ return "BooleanTrue";
+ case VariantPrimitiveType::BooleanFalse:
+ return "BooleanFalse";
+ case VariantPrimitiveType::Int8:
+ return "Int8";
+ case VariantPrimitiveType::Int16:
+ return "Int16";
+ case VariantPrimitiveType::Int32:
+ return "Int32";
+ case VariantPrimitiveType::Int64:
+ return "Int64";
+ case VariantPrimitiveType::Double:
+ return "Double";
+ case VariantPrimitiveType::Decimal4:
+ return "Decimal4";
+ case VariantPrimitiveType::Decimal8:
+ return "Decimal8";
+ case VariantPrimitiveType::Decimal16:
+ return "Decimal16";
+ case VariantPrimitiveType::Date:
+ return "Date";
+ case VariantPrimitiveType::TimestampMicros:
+ return "TimestampMicros";
+ case VariantPrimitiveType::TimestampMicrosNtz:
+ return "TimestampMicrosNtz";
+ case VariantPrimitiveType::Float:
+ return "Float";
+ case VariantPrimitiveType::Binary:
+ return "Binary";
+ case VariantPrimitiveType::String:
+ return "String";
+ case VariantPrimitiveType::TimeMicrosNtz:
+ return "TimeMicrosNtz";
+ case VariantPrimitiveType::TimestampNanosTz:
+ return "TimestampNanosTz";
+ case VariantPrimitiveType::TimestampNanosNtz:
+ return "TimestampNanosNtz";
+ case VariantPrimitiveType::Uuid:
+ return "Uuid";
+ default:
+ return "Unknown";
+ }
+}
+
+std::string VariantTypeToString(VariantType type) {
+ switch (type) {
+ case VariantType::Object:
+ return "Object";
+ case VariantType::Array:
+ return "Array";
+ case VariantType::Null:
+ return "Null";
+ case VariantType::Boolean:
+ return "Boolean";
+ case VariantType::Int8:
+ return "Int8";
+ case VariantType::Int16:
+ return "Int16";
+ case VariantType::Int32:
+ return "Int32";
+ case VariantType::Int64:
+ return "Int64";
+ case VariantType::String:
+ return "String";
+ case VariantType::Double:
+ return "Double";
+ case VariantType::Decimal4:
+ return "Decimal4";
+ case VariantType::Decimal8:
+ return "Decimal8";
+ case VariantType::Decimal16:
+ return "Decimal16";
+ case VariantType::Date:
+ return "Date";
+ case VariantType::TimestampMicrosTz:
+ return "TimestampMicrosTz";
+ case VariantType::TimestampMicrosNtz:
+ return "TimestampMicrosNtz";
+ case VariantType::Float:
+ return "Float";
+ case VariantType::Binary:
+ return "Binary";
+ case VariantType::Time:
+ return "Time";
+ case VariantType::TimestampNanosTz:
+ return "TimestampNanosTz";
+ case VariantType::TimestampNanosNtz:
+ return "TimestampNanosNtz";
+ case VariantType::Uuid:
+ return "Uuid";
+ default:
+ return "Unknown";
+ }
+}
+
+inline uint32_t readLittleEndianU32(const void* from, uint8_t size) {
+ ARROW_DCHECK_LE(size, 4);
+ ARROW_DCHECK_GE(size, 1);
+
+ uint32_t result = 0;
+ memcpy(&result, from, size);
+ return ::arrow::bit_util::FromLittleEndian(result);
+}
+
+VariantMetadata::VariantMetadata(std::string_view metadata) : metadata_(metadata) {
+ if (metadata.size() < kHeaderSizeBytes + kMinimalOffsetSizeBytes * 2) {
+ // Empty metadata is at least 3 bytes: version, dictionarySize and
+ // at least one offset.
+ throw ParquetException("Invalid Variant metadata: too short: size=" +
+ std::to_string(metadata.size()));
+ }
+ if (version() != kSupportedVersion) {
+ // Currently we only supports version 1.
+ throw ParquetException("Unsupported Variant metadata version: " +
+ std::to_string(version()));
+ }
+ uint8_t offset_sz = offset_size();
+ if (offset_sz < kMinimalOffsetSizeBytes || offset_sz > kMaximumOffsetSizeBytes) {
+ throw ParquetException("Invalid Variant metadata: invalid offset size: " +
+ std::to_string(offset_sz));
+ }
+ dictionary_size_ = loadDictionarySize(metadata, offset_sz);
+ if (kHeaderSizeBytes + (dictionary_size_ + 2) * offset_sz > metadata_.size()) {
+ throw ParquetException(
+ "Invalid Variant metadata: offset out of range: " +
+ std::to_string((dictionary_size_ + kHeaderSizeBytes) * offset_sz) + " > " +
+ std::to_string(metadata_.size()));
+ }
Review Comment:
In metadata I delayed the validating offsets to getter, it can also put here to validate all offsets are monotonic and the final offset is equal to the metadata size boundary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2918052042
> How would you avoid parsing the whole binary?
The variant spec has provided sufficient metadata (dictionary of all keys and offset to any value) to jump into a key at arbitrary nesting level without decoding irrelevant binary data. It is already a parsed binary. This is something that XML or JSON texts cannot do.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2911515974
Since there seems to be a controversy about a good API for this, how about we start with something lower-level, such as a SAX-like parser (i.e. event-driven)? Then we can build up a higher-level API on top of it once we know which kind of API would be efficient?
An example of event-driven API is [in RapidJSON](https://rapidjson.org/md_doc_sax.html).
(of course, we should use more modern C++ features)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2101899093
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+#include "parquet/platform.h"
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+PARQUET_EXPORT std::string VariantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ TimestampMicros = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampMicrosNtz = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeMicrosNtz = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampNanosTz = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNanosNtz = 19, // Differentiating from TimestampNtz (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+PARQUET_EXPORT std::string VariantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ Object,
+ Array,
+ Null,
+ Boolean,
+ Int8,
+ Int16,
+ Int32,
+ Int64,
+ String,
+ Double,
+ Decimal4,
+ Decimal8,
+ Decimal16,
+ Date,
+ TimestampMicrosTz,
+ TimestampMicrosNtz,
+ Float,
+ Binary,
+ Time,
+ TimestampNanosTz,
+ TimestampNanosNtz,
+ Uuid
+};
+
+PARQUET_EXPORT std::string VariantTypeToString(VariantType type);
+
+class PARQUET_EXPORT VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary_size).
+ std::string_view GetMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sorted_and_unique(), the metadata key is not guaranteed to be unique, so we use a
Review Comment:
IMO. In Iceberg and Arrow implementation it returns first matched variant value in a object. But it's different in [parquet-java](https://github.com/apache/parquet-java/blob/1f1e07bbf750fba228851c2d63470c3da5726831/parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantObjectBuilder.java#L230-L242) which returns the latest pushed value.
This is a bit confusing for me — please correct me if I misunderstood.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2101774972
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+#include "parquet/platform.h"
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+PARQUET_EXPORT std::string VariantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ TimestampMicros = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampMicrosNtz = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeMicrosNtz = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampNanosTz = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNanosNtz = 19, // Differentiating from TimestampNtz (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+PARQUET_EXPORT std::string VariantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ Object,
+ Array,
+ Null,
+ Boolean,
+ Int8,
+ Int16,
+ Int32,
+ Int64,
+ String,
+ Double,
+ Decimal4,
+ Decimal8,
+ Decimal16,
+ Date,
+ TimestampMicrosTz,
+ TimestampMicrosNtz,
+ Float,
+ Binary,
+ Time,
+ TimestampNanosTz,
+ TimestampNanosNtz,
+ Uuid
+};
+
+PARQUET_EXPORT std::string VariantTypeToString(VariantType type);
+
+class PARQUET_EXPORT VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary_size).
+ std::string_view GetMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sorted_and_unique(), the metadata key is not guaranteed to be unique, so we use a
Review Comment:
Assume a `{"a": {"a" : 1 } }` here. And metadata is duplicate ( standard not require it's unique if !sorted )
The metadata is: `"a": 0, "a": 1`. Assume inner object gets "a", if only return field_id = 0, it cannot get any info from the inner object, which requires 1 as field_id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2102008095
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+#include "parquet/platform.h"
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+PARQUET_EXPORT std::string VariantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ TimestampMicros = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampMicrosNtz = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeMicrosNtz = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampNanosTz = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNanosNtz = 19, // Differentiating from TimestampNtz (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+PARQUET_EXPORT std::string VariantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ Object,
+ Array,
+ Null,
+ Boolean,
+ Int8,
+ Int16,
+ Int32,
+ Int64,
+ String,
+ Double,
+ Decimal4,
+ Decimal8,
+ Decimal16,
+ Date,
+ TimestampMicrosTz,
+ TimestampMicrosNtz,
+ Float,
+ Binary,
+ Time,
+ TimestampNanosTz,
+ TimestampNanosNtz,
+ Uuid
+};
+
+PARQUET_EXPORT std::string VariantTypeToString(VariantType type);
+
+class PARQUET_EXPORT VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary_size).
+ std::string_view GetMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sorted_and_unique(), the metadata key is not guaranteed to be unique, so we use a
Review Comment:
Okay. So for object variant, is it acceptable that different reader implementations return a different value if there are duplicate keys in the dictionary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2887148562
@wgtmac I'm think of keep some elements inside the `VariantValue`, the way below adds 24B or 28B overheads on `VariantValue`. It adds some cost when constructing the `VariantValue` but maybe more friendly for user.
```
VariantBasicType type_;
uint32_t num_elements_; // exists when type_ is array or object
uint32_t id_size_; // exists when type_ is object
uint32_t offset_size_; // exists when type_ is array or object
uint32_t id_start_offset_; // exists when type_ is object
uint32_t offset_start_offset_; // exists when type_ is array or object
uint32_t data_start_offset_; // exists when type_ is array or object
```
Another problem is the error style for decoding tools. If we just leave VariantValue/VariantMetadata a tool in parquet, and uses it in arrow compute or extension type, we also needs cast the handling value to status...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2109613192
##########
cpp/src/parquet/variant.cc:
##########
@@ -300,8 +300,7 @@ VariantType VariantValue::getType() const {
VariantBasicType basic_type = getBasicType();
switch (basic_type) {
case VariantBasicType::Primitive: {
- auto primitive_type =
- static_cast<VariantPrimitiveType>(value_[0] >> kValueHeaderBitShift);
+ auto primitive_type = static_cast<VariantPrimitiveType>(valueHeader());
Review Comment:
any reason not to push the `static_cast<VariantPrimitiveType>` call into the `valueHeader` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2917030686
How would you avoid parsing the whole binary?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2887163947
Also cc @pitrou for interfaces because you're interface expert here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2101769873
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+#include "parquet/platform.h"
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+PARQUET_EXPORT std::string VariantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ TimestampMicros = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampMicrosNtz = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeMicrosNtz = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampNanosTz = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNanosNtz = 19, // Differentiating from TimestampNtz (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+PARQUET_EXPORT std::string VariantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ Object,
+ Array,
+ Null,
+ Boolean,
+ Int8,
+ Int16,
+ Int32,
+ Int64,
+ String,
+ Double,
+ Decimal4,
+ Decimal8,
+ Decimal16,
+ Date,
+ TimestampMicrosTz,
+ TimestampMicrosNtz,
+ Float,
+ Binary,
+ Time,
+ TimestampNanosTz,
+ TimestampNanosNtz,
+ Uuid
+};
+
+PARQUET_EXPORT std::string VariantTypeToString(VariantType type);
+
+class PARQUET_EXPORT VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary_size).
+ std::string_view GetMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sorted_and_unique(), the metadata key is not guaranteed to be unique, so we use a
Review Comment:
I check the Iceberg implementation just return the first matched string index(https://github.com/apache/iceberg/blob/1911c94ea605a3d3f10a1994b046f00a5e9fdceb/api/src/main/java/org/apache/iceberg/variants/SerializedMetadata.java#L88-L102).
Kindly ask: Why we need to return a vector here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2101769873
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+#include "parquet/platform.h"
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+PARQUET_EXPORT std::string VariantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ TimestampMicros = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampMicrosNtz = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeMicrosNtz = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampNanosTz = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNanosNtz = 19, // Differentiating from TimestampNtz (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+PARQUET_EXPORT std::string VariantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ Object,
+ Array,
+ Null,
+ Boolean,
+ Int8,
+ Int16,
+ Int32,
+ Int64,
+ String,
+ Double,
+ Decimal4,
+ Decimal8,
+ Decimal16,
+ Date,
+ TimestampMicrosTz,
+ TimestampMicrosNtz,
+ Float,
+ Binary,
+ Time,
+ TimestampNanosTz,
+ TimestampNanosNtz,
+ Uuid
+};
+
+PARQUET_EXPORT std::string VariantTypeToString(VariantType type);
+
+class PARQUET_EXPORT VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary_size).
+ std::string_view GetMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sorted_and_unique(), the metadata key is not guaranteed to be unique, so we use a
Review Comment:
I check the implementation just return the first matched string index(https://github.com/apache/iceberg/blob/1911c94ea605a3d3f10a1994b046f00a5e9fdceb/api/src/main/java/org/apache/iceberg/variants/SerializedMetadata.java#L88-L102).
Kindly ask: Why we need to return a vector here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2103693613
##########
cpp/src/parquet/variant.cc:
##########
@@ -0,0 +1,771 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/variant.h"
+
+#include <cstdint>
+#include <iostream>
+#include <string_view>
+
+#include <arrow/util/endian.h>
+#include <arrow/util/logging.h>
+#include <arrow/util/unreachable.h>
+
+#include "parquet/exception.h"
+
+namespace parquet::variant {
+
+std::string VariantBasicTypeToString(VariantBasicType type) {
+ switch (type) {
+ case VariantBasicType::Primitive:
+ return "Primitive";
+ case VariantBasicType::ShortString:
+ return "ShortString";
+ case VariantBasicType::Object:
+ return "Object";
+ case VariantBasicType::Array:
+ return "Array";
+ default:
+ return "Unknown";
+ }
+}
+
+std::string VariantPrimitiveTypeToString(VariantPrimitiveType type) {
+ switch (type) {
+ case VariantPrimitiveType::NullType:
+ return "NullType";
+ case VariantPrimitiveType::BooleanTrue:
+ return "BooleanTrue";
+ case VariantPrimitiveType::BooleanFalse:
+ return "BooleanFalse";
+ case VariantPrimitiveType::Int8:
+ return "Int8";
+ case VariantPrimitiveType::Int16:
+ return "Int16";
+ case VariantPrimitiveType::Int32:
+ return "Int32";
+ case VariantPrimitiveType::Int64:
+ return "Int64";
+ case VariantPrimitiveType::Double:
+ return "Double";
+ case VariantPrimitiveType::Decimal4:
+ return "Decimal4";
+ case VariantPrimitiveType::Decimal8:
+ return "Decimal8";
+ case VariantPrimitiveType::Decimal16:
+ return "Decimal16";
+ case VariantPrimitiveType::Date:
+ return "Date";
+ case VariantPrimitiveType::TimestampMicros:
+ return "TimestampMicros";
+ case VariantPrimitiveType::TimestampMicrosNtz:
+ return "TimestampMicrosNtz";
+ case VariantPrimitiveType::Float:
+ return "Float";
+ case VariantPrimitiveType::Binary:
+ return "Binary";
+ case VariantPrimitiveType::String:
+ return "String";
+ case VariantPrimitiveType::TimeMicrosNtz:
+ return "TimeMicrosNtz";
+ case VariantPrimitiveType::TimestampNanosTz:
+ return "TimestampNanosTz";
+ case VariantPrimitiveType::TimestampNanosNtz:
+ return "TimestampNanosNtz";
+ case VariantPrimitiveType::Uuid:
+ return "Uuid";
+ default:
+ return "Unknown";
+ }
+}
+
+std::string VariantTypeToString(VariantType type) {
+ switch (type) {
+ case VariantType::Object:
+ return "Object";
+ case VariantType::Array:
+ return "Array";
+ case VariantType::Null:
+ return "Null";
+ case VariantType::Boolean:
+ return "Boolean";
+ case VariantType::Int8:
+ return "Int8";
+ case VariantType::Int16:
+ return "Int16";
+ case VariantType::Int32:
+ return "Int32";
+ case VariantType::Int64:
+ return "Int64";
+ case VariantType::String:
+ return "String";
+ case VariantType::Double:
+ return "Double";
+ case VariantType::Decimal4:
+ return "Decimal4";
+ case VariantType::Decimal8:
+ return "Decimal8";
+ case VariantType::Decimal16:
+ return "Decimal16";
+ case VariantType::Date:
+ return "Date";
+ case VariantType::TimestampMicrosTz:
+ return "TimestampMicrosTz";
+ case VariantType::TimestampMicrosNtz:
+ return "TimestampMicrosNtz";
+ case VariantType::Float:
+ return "Float";
+ case VariantType::Binary:
+ return "Binary";
+ case VariantType::Time:
+ return "Time";
+ case VariantType::TimestampNanosTz:
+ return "TimestampNanosTz";
+ case VariantType::TimestampNanosNtz:
+ return "TimestampNanosNtz";
+ case VariantType::Uuid:
+ return "Uuid";
+ default:
+ return "Unknown";
+ }
+}
+
+inline uint32_t readLittleEndianU32(const void* from, uint8_t size) {
+ ARROW_DCHECK_LE(size, 4);
+ ARROW_DCHECK_GE(size, 1);
+
+ uint32_t result = 0;
+ memcpy(&result, from, size);
+ return ::arrow::bit_util::FromLittleEndian(result);
+}
+
+VariantMetadata::VariantMetadata(std::string_view metadata) : metadata_(metadata) {
+ if (metadata.size() < kHeaderSizeBytes + kMinimalOffsetSizeBytes * 2) {
+ // Empty metadata is at least 3 bytes: version, dictionarySize and
+ // at least one offset.
+ throw ParquetException("Invalid Variant metadata: too short: size=" +
+ std::to_string(metadata.size()));
+ }
+ if (version() != kSupportedVersion) {
+ // Currently we only supports version 1.
+ throw ParquetException("Unsupported Variant metadata version: " +
+ std::to_string(version()));
+ }
+ uint8_t offset_sz = offset_size();
+ if (offset_sz < kMinimalOffsetSizeBytes || offset_sz > kMaximumOffsetSizeBytes) {
+ throw ParquetException("Invalid Variant metadata: invalid offset size: " +
+ std::to_string(offset_sz));
+ }
Review Comment:
DCHECK means debug check, it should be regarded as an requires assertion rather than runtime dynamically check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2093685041
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,297 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+#include "parquet/platform.h"
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+PARQUET_EXPORT std::string_view VariantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNtz = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNtz = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTz = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNtzNanos = 19, // Differentiating from TimestampNtz (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+PARQUET_EXPORT std::string_view VariantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ Object,
+ Array,
+ Null,
+ Boolean,
+ Int8,
+ Int16,
+ Int32,
+ Int64,
+ String,
+ Double,
+ Decimal4,
+ Decimal8,
+ Decimal16,
+ Date,
+ TimestampTz,
+ TimestampNtz,
+ Float,
+ Binary,
+ Time,
+ TimestampNanosTz,
+ TimestampNanosNtz,
+ Uuid
+};
+
+PARQUET_EXPORT std::string_view VariantTypeToString(VariantType type);
+
+class PARQUET_EXPORT VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary_size).
+ std::string_view GetMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sorted_and_unique(), the metadata key is not guaranteed to be unique, so we use a
+ /// vector to store all the metadata ids.
+ ::arrow::internal::SmallVector<uint32_t, 1> GetMetadataId(std::string_view key) const;
+
+ bool sorted_and_unique() const;
+ uint8_t offset_size() const;
+ uint32_t dictionary_size() const;
+
+ private:
+ static uint32_t loadDictionarySize(std::string_view metadata, uint8_t offset_size);
+
+ private:
+ static constexpr uint8_t kVersionMask = 0b1111;
+ static constexpr uint8_t kSortedStringMask = 0b10000;
+ static constexpr size_t kHeaderSizeBytes = 1;
+ static constexpr size_t kMinimalOffsetSizeBytes = 1;
+ static constexpr size_t kMaximumOffsetSizeBytes = 4;
+ // mask is applied after shift, it's like 0b11000000 before shift.
+ static constexpr uint8_t kOffsetSizeMask = 0b11;
+ static constexpr uint8_t kOffsetSizeBitShift = 6;
+ static constexpr uint8_t kSupportedVersion = 1;
+
+ private:
+ std::string_view metadata_;
+ uint32_t dictionary_size_{0};
Review Comment:
I wonder if it's worth to change this to below case, which could make this 24B -> 16B
```
const uint8_t* metadata_ptr_;
const uint32_t metadata_size_;
uint32_t dictionary_size_
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2917037940
Emm it's by the design of parquet variant format. In memory there is `VariantObject`, `VariantObject::Get(key)` would search key in metadata, find `field_id`, and search for `field_id` in object variant. These operations are operated directed in variant binary format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2086632875
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNTZ = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNTZ = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTZ = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNTZNanos = 19, // Differentiating from TimestampNTZ (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ OBJECT,
+ ARRAY,
+ VARIANT_NULL,
+ BOOLEAN,
+ BYTE,
+ SHORT,
+ INT,
+ LONG,
+ STRING,
+ DOUBLE,
+ DECIMAL4,
+ DECIMAL8,
+ DECIMAL16,
+ DATE,
+ TIMESTAMP_TZ,
+ TIMESTAMP_NTZ,
+ FLOAT,
+ BINARY,
+ TIME,
+ TIMESTAMP_NANOS_TZ,
+ TIMESTAMP_NANOS_NTZ,
+ UUID
+};
+
+class VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ int8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ std::string_view getMetadataKey(int32_t variantId) const;
+
+ private:
+ bool sortedStrings() const;
+ uint8_t offsetSize() const;
+ uint32_t dictionarySize() const;
+
+ private:
+ std::string_view metadata_;
+};
+
+template <typename DecimalType>
+struct DecimalValue {
+ uint8_t scale;
+ DecimalType value;
+};
+
+struct VariantValue {
+ VariantMetadata metadata;
+ std::string_view value;
+
+ VariantBasicType getBasicType() const;
+ VariantType getType() const;
+ std::string typeDebugString() const;
+
+ /// \defgroup ValueAccessors
+ /// @{
+
+ // Note: Null doesn't need visitor.
+ bool getBool() const;
+ int8_t getInt8() const;
+ int16_t getInt16() const;
+ int32_t getInt32() const;
+ int64_t getInt64() const;
+ /// Include short_string optimization and primitive string type
+ std::string_view getString() const;
+ std::string_view getBinary() const;
+ float getFloat() const;
+ double getDouble() const;
+
+ DecimalValue<::arrow::Decimal32> getDecimal4() const;
+ DecimalValue<::arrow::Decimal64> getDecimal8() const;
+ DecimalValue<::arrow::Decimal128> getDecimal16() const;
+
+ int64_t timeNTZ() const;
+ // timestamp with adjusted to UTC
+ int64_t getTimestamp() const;
+ int64_t getTimestampNTZ() const;
+ // 16 bytes UUID
+ const uint8_t* getUuid() const;
+
+ /// }@
+
+ struct ObjectInfo {
+ uint32_t num_elements;
+ uint32_t id_size;
+ uint32_t offset_size;
+ uint32_t id_start_offset;
+ uint32_t offset_start_offset;
+ uint32_t data_start_offset;
+
+ std::string toDebugString() const;
+ };
+ ObjectInfo getObjectInfo() const;
+ std::optional<VariantValue> getObjectValueByKey(std::string_view key) const;
+ std::optional<VariantValue> getObjectFieldByFieldId(uint32_t variantId,
+ std::string_view* key) const;
+
+ struct ArrayInfo {
+ uint32_t num_elements;
+ uint32_t offset_size;
+ uint32_t offset_start_offset;
+ uint32_t data_start_offset;
+ };
+ ArrayInfo getArrayInfo() const;
+ // Would throw ParquetException if index is out of range.
+ VariantValue getArrayValueByIndex(uint32_t index) const;
+
+ static constexpr uint8_t BASIC_TYPE_MASK = 0b00000011;
Review Comment:
This code needs to cleanup
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNTZ = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNTZ = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTZ = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNTZNanos = 19, // Differentiating from TimestampNTZ (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ OBJECT,
+ ARRAY,
+ VARIANT_NULL,
+ BOOLEAN,
+ BYTE,
+ SHORT,
+ INT,
+ LONG,
+ STRING,
+ DOUBLE,
+ DECIMAL4,
+ DECIMAL8,
+ DECIMAL16,
+ DATE,
+ TIMESTAMP_TZ,
+ TIMESTAMP_NTZ,
+ FLOAT,
+ BINARY,
+ TIME,
+ TIMESTAMP_NANOS_TZ,
+ TIMESTAMP_NANOS_NTZ,
+ UUID
+};
+
+class VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ int8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ std::string_view getMetadataKey(int32_t variantId) const;
+
+ private:
+ bool sortedStrings() const;
+ uint8_t offsetSize() const;
+ uint32_t dictionarySize() const;
+
+ private:
+ std::string_view metadata_;
+};
+
+template <typename DecimalType>
Review Comment:
This is because decimal data is with a byte's precision. So use an extra "scale" here
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNTZ = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNTZ = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTZ = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNTZNanos = 19, // Differentiating from TimestampNTZ (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ OBJECT,
+ ARRAY,
+ VARIANT_NULL,
+ BOOLEAN,
+ BYTE,
+ SHORT,
+ INT,
+ LONG,
+ STRING,
+ DOUBLE,
+ DECIMAL4,
+ DECIMAL8,
+ DECIMAL16,
+ DATE,
+ TIMESTAMP_TZ,
+ TIMESTAMP_NTZ,
+ FLOAT,
+ BINARY,
+ TIME,
+ TIMESTAMP_NANOS_TZ,
+ TIMESTAMP_NANOS_NTZ,
+ UUID
+};
+
+class VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ int8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ std::string_view getMetadataKey(int32_t variantId) const;
+
+ private:
+ bool sortedStrings() const;
+ uint8_t offsetSize() const;
+ uint32_t dictionarySize() const;
+
+ private:
+ std::string_view metadata_;
+};
+
+template <typename DecimalType>
+struct DecimalValue {
+ uint8_t scale;
+ DecimalType value;
+};
+
+struct VariantValue {
+ VariantMetadata metadata;
+ std::string_view value;
+
+ VariantBasicType getBasicType() const;
+ VariantType getType() const;
+ std::string typeDebugString() const;
+
+ /// \defgroup ValueAccessors
+ /// @{
+
+ // Note: Null doesn't need visitor.
Review Comment:
I don't know should we just return an arrow's `Scalar`, it would be easy to use but in-efficient.
##########
cpp/src/parquet/variant.cc:
##########
@@ -0,0 +1,592 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "parquet/variant.h"
+
+#include <cstdint>
+#include <iostream>
+#include <string_view>
+
+#include "arrow/util/endian.h"
+#include "parquet/exception.h"
+
+namespace parquet::variant {
+
+VariantMetadata::VariantMetadata(std::string_view metadata) : metadata_(metadata) {
+ if (metadata.size() < 2) {
+ throw ParquetException("Invalid Variant metadata: too short: " +
+ std::to_string(metadata.size()));
+ }
+}
+
+int8_t VariantMetadata::version() const {
+ return static_cast<int8_t>(metadata_[0]) & 0x0F;
+}
+
+bool VariantMetadata::sortedStrings() const { return (metadata_[0] & 0b10000) != 0; }
+
+uint8_t VariantMetadata::offsetSize() const { return ((metadata_[0] >> 6) & 0x3) + 1; }
+
+uint32_t VariantMetadata::dictionarySize() const {
+ uint8_t length = offsetSize();
+ if (length > 4) {
+ throw ParquetException("Invalid offset size: " + std::to_string(length));
+ }
+ if (length + 1 > metadata_.size()) {
+ throw ParquetException("Invalid Variant metadata: too short for dictionary size");
+ }
+ uint32_t dict_size = 0;
+ memcpy(&dict_size, metadata_.data() + 1, length);
+ dict_size = arrow::bit_util::FromLittleEndian(dict_size);
+ return dict_size;
+}
+
+std::string_view VariantMetadata::getMetadataKey(int32_t variantId) const {
+ uint32_t offset_size = offsetSize();
+ uint32_t dict_size = dictionarySize();
+
+ if (variantId < 0 || variantId >= static_cast<int32_t>(dict_size)) {
+ throw ParquetException("Invalid Variant metadata: variantId out of range");
+ }
+
+ if ((dict_size + 1) * offset_size > metadata_.size()) {
+ throw ParquetException("Invalid Variant metadata: offset out of range");
+ }
+
+ size_t offset_start_pos = 1 + offset_size + (variantId * offset_size);
+
+ uint32_t variant_offset = 0;
+ uint32_t variant_next_offset = 0;
+ memcpy(&variant_offset, metadata_.data() + offset_start_pos, offset_size);
+ variant_offset = arrow::bit_util::FromLittleEndian(variant_offset);
+ memcpy(&variant_next_offset, metadata_.data() + offset_start_pos + offset_size,
+ offset_size);
+ variant_next_offset = arrow::bit_util::FromLittleEndian(variant_next_offset);
+
+ uint32_t key_size = variant_next_offset - variant_offset;
+
+ size_t string_start = 1 + offset_size * (dict_size + 2) + variant_offset;
+ if (string_start + key_size > metadata_.size()) {
+ throw ParquetException("Invalid Variant metadata: string data out of range");
+ }
+ return std::string_view(metadata_.data() + string_start, key_size);
+}
+
+VariantBasicType VariantValue::getBasicType() const {
+ if (value.empty()) {
+ throw ParquetException("Empty variant value");
+ }
+ return static_cast<VariantBasicType>(value[0] & BASIC_TYPE_MASK);
+}
+
+VariantType VariantValue::getType() const {
+ VariantBasicType basic_type = getBasicType();
+ // std::cout << "Variant first byte:" << static_cast<int>(value[0] >> 2) << ", "
Review Comment:
todo: remove these debug messages...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2095298696
##########
cpp/src/parquet/variant_test.cc:
##########
@@ -27,6 +27,9 @@
#include <arrow/testing/gtest_util.h>
#include <arrow/util/base64.h>
+#include <boost/uuid/uuid.hpp>
Review Comment:
use boost just for testing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2101933563
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+#include "parquet/platform.h"
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+PARQUET_EXPORT std::string VariantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ TimestampMicros = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampMicrosNtz = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeMicrosNtz = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampNanosTz = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNanosNtz = 19, // Differentiating from TimestampNtz (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+PARQUET_EXPORT std::string VariantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ Object,
+ Array,
+ Null,
+ Boolean,
+ Int8,
+ Int16,
+ Int32,
+ Int64,
+ String,
+ Double,
+ Decimal4,
+ Decimal8,
+ Decimal16,
+ Date,
+ TimestampMicrosTz,
+ TimestampMicrosNtz,
+ Float,
+ Binary,
+ Time,
+ TimestampNanosTz,
+ TimestampNanosNtz,
+ Uuid
+};
+
+PARQUET_EXPORT std::string VariantTypeToString(VariantType type);
+
+class PARQUET_EXPORT VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary_size).
+ std::string_view GetMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sorted_and_unique(), the metadata key is not guaranteed to be unique, so we use a
Review Comment:
Refer to: https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96
> Keys may appear in nested objects, but cannot appear in the same object. So the first example, {"a": {"a": 1}} is allowed. The second example, {"a": 1, "a": 2} is not allowed.
This parquet-java test prevent from the key in same object. But
> If sorted_strings is set to 1, strings in the dictionary must be unique and sorted in lexicographic order. If the value is set to 0, readers may not make any assumptions about string order or uniqueness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "xxubai (via GitHub)" <gi...@apache.org>.
xxubai commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2101899093
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,307 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+#include "parquet/platform.h"
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+PARQUET_EXPORT std::string VariantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ TimestampMicros = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampMicrosNtz = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeMicrosNtz = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampNanosTz = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNanosNtz = 19, // Differentiating from TimestampNtz (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+PARQUET_EXPORT std::string VariantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ Object,
+ Array,
+ Null,
+ Boolean,
+ Int8,
+ Int16,
+ Int32,
+ Int64,
+ String,
+ Double,
+ Decimal4,
+ Decimal8,
+ Decimal16,
+ Date,
+ TimestampMicrosTz,
+ TimestampMicrosNtz,
+ Float,
+ Binary,
+ Time,
+ TimestampNanosTz,
+ TimestampNanosNtz,
+ Uuid
+};
+
+PARQUET_EXPORT std::string VariantTypeToString(VariantType type);
+
+class PARQUET_EXPORT VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary_size).
+ std::string_view GetMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sorted_and_unique(), the metadata key is not guaranteed to be unique, so we use a
Review Comment:
IMO. In Iceberg and Arrow implementation it returns first matched variant value in a object. But it's different in [parquet-java](https://github.com/apache/parquet-java/blob/1f1e07bbf750fba228851c2d63470c3da5726831/parquet-variant/src/test/java/org/apache/parquet/variant/TestVariantObjectBuilder.java#L230-L242) which returns the latest pushed value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #46372:
URL: https://github.com/apache/arrow/pull/46372#issuecomment-2897136400
Please let me find the time to digest the Variant spec and review this. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2093314603
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
+ /// A string with a length less than 64 bytes
+ ShortString = 1,
+ /// A collection of (string-key, variant-value) pairs
+ Object = 2,
+ /// An ordered sequence of variant values
+ Array = 3
+};
+
+std::string variantBasicTypeToString(VariantBasicType type);
+
+enum class VariantPrimitiveType : int8_t {
+ /// Equivalent Parquet Type: UNKNOWN
+ NullType = 0,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanTrue = 1,
+ /// Equivalent Parquet Type: BOOLEAN
+ BooleanFalse = 2,
+ /// Equivalent Parquet Type: INT(8, signed)
+ Int8 = 3,
+ /// Equivalent Parquet Type: INT(16, signed)
+ Int16 = 4,
+ /// Equivalent Parquet Type: INT(32, signed)
+ Int32 = 5,
+ /// Equivalent Parquet Type: INT(64, signed)
+ Int64 = 6,
+ /// Equivalent Parquet Type: DOUBLE
+ Double = 7,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal4 = 8,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal8 = 9,
+ /// Equivalent Parquet Type: DECIMAL(precision, scale)
+ Decimal16 = 10,
+ /// Equivalent Parquet Type: DATE
+ Date = 11,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, MICROS)
+ Timestamp = 12,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, MICROS)
+ TimestampNTZ = 13,
+ /// Equivalent Parquet Type: FLOAT
+ Float = 14,
+ /// Equivalent Parquet Type: BINARY
+ Binary = 15,
+ /// Equivalent Parquet Type: STRING
+ String = 16,
+ /// Equivalent Parquet Type: TIME(isAdjustedToUTC=false, MICROS)
+ TimeNTZ = 17,
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=true, NANOS)
+ TimestampTZ = 18, // Assuming TZ stands for TimeZone, and follows the document's
+ // 'timestamp with time zone'
+ /// Equivalent Parquet Type: TIMESTAMP(isAdjustedToUTC=false, NANOS)
+ TimestampNTZNanos = 19, // Differentiating from TimestampNTZ (MICROS)
+ /// Equivalent Parquet Type: UUID
+ Uuid = 20
+};
+
+std::string variantPrimitiveTypeToString(VariantPrimitiveType type);
+
+/// VariantType is from basic type and primitive type.
+enum class VariantType {
+ OBJECT,
+ ARRAY,
+ VARIANT_NULL,
+ BOOLEAN,
+ INT8,
+ INT16,
+ INT32,
+ INT64,
+ STRING,
+ DOUBLE,
+ DECIMAL4,
+ DECIMAL8,
+ DECIMAL16,
+ DATE,
+ TIMESTAMP_TZ,
+ TIMESTAMP_NTZ,
+ FLOAT,
+ BINARY,
+ TIME,
+ TIMESTAMP_NANOS_TZ,
+ TIMESTAMP_NANOS_NTZ,
+ UUID
+};
+
+std::string variantTypeToString(VariantType type);
+
+class VariantMetadata {
+ public:
+ explicit VariantMetadata(std::string_view metadata);
+ /// \brief Get the variant metadata version. Currently, always 1.
+ uint8_t version() const;
+ /// \brief Get the metadata key for a given variant field id.
+ /// \throw ParquetException if the variant_id is out of range(larger than
+ /// dictionary size).
+ std::string_view getMetadataKey(uint32_t variant_id) const;
+ /// \brief Get the metadata id for a given key.
+ /// From the discussion in ML:
+ /// https://lists.apache.org/thread/b68tjmrjmy64mbv9dknpmqs28vnzjj96 if
+ /// !sortedStrings(), the metadata key is not guaranteed to be unique, so we use a
+ /// vector to store all the metadata ids.
+ ::arrow::internal::SmallVector<uint32_t, 1> getMetadataId(std::string_view key) const;
+
+ bool sortedStrings() const;
+ uint8_t offsetSize() const;
+ uint32_t dictionarySize() const;
+
+ private:
+ static uint32_t loadDictionarySize(std::string_view metadata, uint8_t offset_size);
+
+ private:
+ static constexpr uint8_t VERSION_MASK = 0b1111;
+ static constexpr uint8_t SORTED_STRING_MASK = 0b10000;
+ static constexpr size_t HEADER_SIZE_BYTES = 1;
+ static constexpr size_t MINIMAL_OFFSET_SIZE_BYTES = 1;
+ static constexpr size_t MAXIMUM_OFFSET_SIZE_BYTES = 4;
+ // mask is applied after shift, it's like 0b11000000 before shift.
+ static constexpr uint8_t OFFSET_SIZE_MASK = 0b11;
+ static constexpr uint8_t OFFSET_SIZE_BIT_SHIFT = 6;
+ static constexpr uint8_t SUPPORTED_VERSION = 1;
+
+ private:
+ std::string_view metadata_;
+ uint32_t dictionary_size_{0};
+};
+
+template <typename DecimalType>
+struct DecimalValue {
+ uint8_t scale;
+ DecimalType value;
+};
+
+struct VariantValue {
+ VariantMetadata metadata;
+ std::string_view value;
+
+ VariantBasicType getBasicType() const;
+ VariantType getType() const;
+ std::string typeDebugString() const;
+
+ /// \defgroup ValueAccessors
+ /// @{
+
+ // Note: Null doesn't need visitor.
+ bool getBool() const;
Review Comment:
I don't think variant is a good idea, `VariantValue` is a "variant" type, casting it to std::variant and get from it is (1) not lazy, needs to deserialize whole structure (2) it decode twice
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-46371: [C++][Parquet] Parquet Variant decoding tools [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #46372:
URL: https://github.com/apache/arrow/pull/46372#discussion_r2093545719
##########
cpp/src/parquet/variant.h:
##########
@@ -0,0 +1,261 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <optional>
+#include <string_view>
+#include <vector>
+
+#include <arrow/util/decimal.h>
+#include <arrow/util/small_vector.h>
+
+namespace parquet::variant {
+
+// TODO(mwish): Should I use parquet::ByteArray rather than
+// std::string_view?
+
+enum class VariantBasicType {
+ /// One of the primitive types
+ Primitive = 0,
Review Comment:
Actually this codebase doesn't have this style? Like:
```c++
enum class ParquetDataPageVersion { V1, V2 };
/// Controls the level of size statistics that are written to the file.
enum class SizeStatisticsLevel : uint8_t {
// No size statistics are written.
None = 0,
// Only column chunk size statistics are written.
ColumnChunk,
// Both size statistics in the column chunk and page index are written.
PageAndColumnChunk
};
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org