You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "qzyu999 (via GitHub)" <gi...@apache.org> on 2026/06/08 04:54:13 UTC
[PR] Variant encoding [arrow]
qzyu999 opened a new pull request, #50122:
URL: https://github.com/apache/arrow/pull/50122
### Rationale for this change
This is part of the GH-45937 umbrella (Add variant support to C++ Parquet). It adds the encoding (writing) side of the Variant binary format, building on the decoder from GH-45946. The encoder is required for GH-45948 (variant shredding) and for any Parquet writer that needs to produce Variant columns.
As with the decoder, the implementation targets feature parity with the [arrow-go `parquet/variant.Builder`](https://github.com/apache/arrow-go/tree/main/parquet/variant), adapted to idiomatic C++ patterns. Divergences are deliberate and documented.
### What changes are included in this PR?
Adds `VariantBuilder` class in `variant_internal.h` / `variant_builder.cc` for encoding Variant binary values per the [Variant Encoding Spec](https://github.com/apache/parquet-format/blob/master/VariantEncoding.md).
**Builder API:**
- All 21 primitive types: `Null()`, `Bool()`, `Int()` (auto-sizes), `Int8/16/32/64()`, `Float()`, `Double()`, `Date()`, `TimestampMicros/NTZ()`, `TimestampNanos/NTZ()`, `TimeNTZ()`, `Decimal4/8/16()`, `String()` (auto short-string for ≤63 bytes), `Binary()`, `UUID()`
- Container construction: `Offset()` / `NextElement()` / `FinishArray()` for arrays, `NextField()` / `FinishObject()` for objects
- `Finish()` — produces encoded metadata + value buffers with sorted-flag detection
- `Reset()` — clears buffer for builder reuse; dictionary preserved across `Finish()` calls
- Constructor from existing `VariantMetadata` for shared-dictionary workflows
**Key design points:**
- Move-only (non-copyable, `noexcept` movable)
- `FinishObject()` sorts fields in-place by key — spec requires field IDs in lexicographic key order
- Strict duplicate key rejection (`Status::Invalid`) — spec says "An object may not contain duplicate keys"; configurable tolerance deferred to GH-45948 with TODO
- `FinishArray()` validates offsets are non-negative
- `Finish()` validates total dictionary size fits in 4-byte offsets
- Decimal scale validation (≤ 38) in encoder; decoder is lenient
- Go enforces a 128MB metadata limit (`metadataMaxSizeLimit`); C++ only enforces the spec's ~4GB 4-byte offset maximum
**TODOs for GH-45948 (shredding):**
```cpp
// TODO GH-45948: Add BuildWithoutMeta() — raw value bytes without metadata
// TODO GH-45948: Add UnsafeAppendEncoded() — append pre-encoded bytes
// TODO GH-45948: Add SetAllowDuplicates(bool) — last-value-wins semantics
```
### Are these changes tested?
Yes. 238 total tests pass with `BUILD_WARNING_LEVEL=CHECKIN` (73 encoder + 165 decoder):
- Primitive round-trips (14 tests including short/long string boundary at 63/64 bytes)
- Int auto-sizing boundaries: Int8→Int16→Int32→Int64 transitions (8 tests)
- Direct int type methods: `Int8/16/32/64` without auto-sizing (4 tests)
- Array round-trips: empty, simple, nested (3 tests)
- Object round-trips: empty, simple, nested, duplicate rejection, field sorting (5 tests)
- Builder features: reset, from-existing-metadata, sorted/unsorted flag (4 tests)
- Integration: complex nested object, large metadata (300 keys), offset-size computation, invalid start, negative offsets (5 tests)
- Special floats: NaN, ±Inf for float and double (6 tests)
- Large containers triggering `is_large` flag: 300-element array + 300-field object (2 tests)
- Decoder utility round-trips through builder output: FindObjectField, GetArrayElement, GetObjectFieldAt, ValueSize (4 tests)
- Builder reuse: dictionary preservation across multiple `Finish()` calls (2 tests)
- Pre-existing buffer: FinishObject/FinishArray with start > 0 (2 tests)
- Decimal scale validation: rejects scale > 38 (1 test)
### Are there any user-facing changes?
No breaking changes. This extends the public API added in GH-45946 with the `VariantBuilder` class in the same `arrow::extension::variant` namespace.
**AI Disclosure:** AI coding assistants were used during development for scaffolding, test generation, and review iteration. All code has been reviewed, debugged, and verified by the author who owns and understands the changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45947 : [C++][Parquet] Variant encoding [arrow]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #50122:
URL: https://github.com/apache/arrow/pull/50122#discussion_r3485323349
##########
cpp/src/arrow/extension/variant_builder_test.cc:
##########
@@ -0,0 +1,1180 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/extension/variant_internal.h"
+#include "arrow/extension/variant_test_util.h"
+
+#include <cstring>
+#include <limits>
+#include <string>
+#include <vector>
+
+#include "arrow/testing/gtest_util.h"
+
+namespace arrow::extension::variant_internal {
+
+// ===========================================================================
+// Helper: decode an EncodedVariant and return visitor events
+// ===========================================================================
+
+/// Encode with builder, decode, return events.
+/// Note: Uses .ValueOrDie() because ASSERT_OK_AND_ASSIGN cannot be used
+/// in a non-void function. Test-only; will crash with a descriptive message
+/// on failure rather than producing a clean test failure.
+std::vector<std::string> RoundTrip(VariantBuilder& builder) {
+ auto result = builder.Finish().ValueOrDie();
+ auto metadata =
+ DecodeMetadata(result.metadata.data(), static_cast<int64_t>(result.metadata.size()))
+ .ValueOrDie();
+ RecordingVisitor visitor;
+ DecodeVariantValue(metadata, result.value.data(),
+ static_cast<int64_t>(result.value.size()), &visitor)
+ .ok();
+ return visitor.events;
+}
+
+// ===========================================================================
+// Primitive round-trip tests
+// ===========================================================================
+
+class VariantBuilderPrimitiveTest : public ::testing::Test {};
+
+TEST_F(VariantBuilderPrimitiveTest, Null) {
+ VariantBuilder b;
+ ASSERT_OK(b.Null());
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Null");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, BoolTrue) {
+ VariantBuilder b;
+ ASSERT_OK(b.Bool(true));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Bool(true)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, BoolFalse) {
+ VariantBuilder b;
+ ASSERT_OK(b.Bool(false));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Bool(false)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, IntAutoSizesInt8) {
+ VariantBuilder b;
+ ASSERT_OK(b.Int(42));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Int8(42)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, IntAutoSizesInt16) {
+ VariantBuilder b;
+ ASSERT_OK(b.Int(300));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Int16(300)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, IntAutoSizesInt32) {
+ VariantBuilder b;
+ ASSERT_OK(b.Int(100000));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Int32(100000)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, IntAutoSizesInt64) {
+ VariantBuilder b;
+ ASSERT_OK(b.Int(5000000000LL));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Int64(5000000000)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, IntNegative) {
+ VariantBuilder b;
+ ASSERT_OK(b.Int(-42));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Int8(-42)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, ShortString) {
+ VariantBuilder b;
+ ASSERT_OK(b.String("hello"));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "String(\"hello\")");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, LongString) {
+ std::string long_str(100, 'x');
+ VariantBuilder b;
+ ASSERT_OK(b.String(long_str));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "String(\"" + long_str + "\")");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, ShortStringBoundary63) {
+ std::string str63(63, 'a');
+ VariantBuilder b;
+ ASSERT_OK(b.String(str63));
+ ASSERT_OK_AND_ASSIGN(auto result, b.Finish());
+ // Should use short string encoding: 1 byte header + 63 bytes
+ ASSERT_EQ(result.value.size(), 64);
+}
+
+TEST_F(VariantBuilderPrimitiveTest, LongStringBoundary64) {
+ std::string str64(64, 'a');
+ VariantBuilder b;
+ ASSERT_OK(b.String(str64));
+ ASSERT_OK_AND_ASSIGN(auto result, b.Finish());
+ // Should use long string encoding: 1 byte header + 4 byte length + 64 bytes
+ ASSERT_EQ(result.value.size(), 69);
+}
+
+TEST_F(VariantBuilderPrimitiveTest, Date) {
+ VariantBuilder b;
+ ASSERT_OK(b.Date(19000));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Date(19000)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, Double) {
+ VariantBuilder b;
+ ASSERT_OK(b.Double(3.14));
+ auto events = RoundTrip(b);
+ ASSERT_TRUE(events[0].find("Double(") == 0);
+}
+
+// ===========================================================================
+// Array round-trip tests
+// ===========================================================================
+
+class VariantBuilderArrayTest : public ::testing::Test {};
+
+TEST_F(VariantBuilderArrayTest, EmptyArray) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<int64_t> offsets;
+ ASSERT_OK(b.FinishArray(start, offsets));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events.size(), 2);
+ ASSERT_EQ(events[0], "StartArray(0)");
+ ASSERT_EQ(events[1], "EndArray");
+}
+
+TEST_F(VariantBuilderArrayTest, SimpleArray) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<int64_t> offsets;
+ offsets.push_back(b.NextElement(start));
+ ASSERT_OK(b.Int(1));
+ offsets.push_back(b.NextElement(start));
+ ASSERT_OK(b.Int(2));
+ offsets.push_back(b.NextElement(start));
+ ASSERT_OK(b.Int(3));
+ ASSERT_OK(b.FinishArray(start, offsets));
+
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events.size(), 5);
+ ASSERT_EQ(events[0], "StartArray(3)");
+ ASSERT_EQ(events[1], "Int8(1)");
+ ASSERT_EQ(events[2], "Int8(2)");
+ ASSERT_EQ(events[3], "Int8(3)");
+ ASSERT_EQ(events[4], "EndArray");
+}
+
+TEST_F(VariantBuilderArrayTest, NestedArray) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<int64_t> offsets;
+
+ // First element: nested array [10, 20]
+ offsets.push_back(b.NextElement(start));
+ auto inner_start = b.Offset();
+ std::vector<int64_t> inner_offsets;
+ inner_offsets.push_back(b.NextElement(inner_start));
+ ASSERT_OK(b.Int(10));
+ inner_offsets.push_back(b.NextElement(inner_start));
+ ASSERT_OK(b.Int(20));
+ ASSERT_OK(b.FinishArray(inner_start, inner_offsets));
+
+ // Second element: 30
+ offsets.push_back(b.NextElement(start));
+ ASSERT_OK(b.Int(30));
+
+ ASSERT_OK(b.FinishArray(start, offsets));
+
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "StartArray(2)");
+ ASSERT_EQ(events[1], "StartArray(2)");
+ ASSERT_EQ(events[2], "Int8(10)");
+ ASSERT_EQ(events[3], "Int8(20)");
+ ASSERT_EQ(events[4], "EndArray");
+ ASSERT_EQ(events[5], "Int8(30)");
+ ASSERT_EQ(events[6], "EndArray");
+}
+
+// ===========================================================================
+// Object round-trip tests
+// ===========================================================================
+
+class VariantBuilderObjectTest : public ::testing::Test {};
+
+TEST_F(VariantBuilderObjectTest, EmptyObject) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<VariantBuilder::FieldEntry> fields;
+ ASSERT_OK(b.FinishObject(start, fields));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events.size(), 2);
+ ASSERT_EQ(events[0], "StartObject(0)");
+ ASSERT_EQ(events[1], "EndObject");
+}
+
+TEST_F(VariantBuilderObjectTest, SimpleObject) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<VariantBuilder::FieldEntry> fields;
+ fields.push_back(b.NextField(start, "name"));
+ ASSERT_OK(b.String("Alice"));
+ fields.push_back(b.NextField(start, "age"));
+ ASSERT_OK(b.Int(30));
+ ASSERT_OK(b.FinishObject(start, fields));
+
+ auto events = RoundTrip(b);
+ // Fields sorted by key: "age" before "name"
+ ASSERT_EQ(events[0], "StartObject(2)");
+ ASSERT_EQ(events[1], "FieldName(\"age\")");
+ ASSERT_EQ(events[2], "Int8(30)");
+ ASSERT_EQ(events[3], "FieldName(\"name\")");
+ ASSERT_EQ(events[4], "String(\"Alice\")");
+ ASSERT_EQ(events[5], "EndObject");
+}
+
+TEST_F(VariantBuilderObjectTest, NestedObject) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<VariantBuilder::FieldEntry> fields;
+ fields.push_back(b.NextField(start, "inner"));
+ {
+ auto inner_start = b.Offset();
+ std::vector<VariantBuilder::FieldEntry> inner_fields;
+ inner_fields.push_back(b.NextField(inner_start, "key"));
+ ASSERT_OK(b.String("value"));
+ ASSERT_OK(b.FinishObject(inner_start, inner_fields));
+ }
+ ASSERT_OK(b.FinishObject(start, fields));
+
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "StartObject(1)");
+ ASSERT_EQ(events[1], "FieldName(\"inner\")");
+ ASSERT_EQ(events[2], "StartObject(1)");
+ ASSERT_EQ(events[3], "FieldName(\"key\")");
+ ASSERT_EQ(events[4], "String(\"value\")");
+ ASSERT_EQ(events[5], "EndObject");
+ ASSERT_EQ(events[6], "EndObject");
+}
+
+TEST_F(VariantBuilderObjectTest, DuplicateKeyError) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<VariantBuilder::FieldEntry> fields;
+ fields.push_back(b.NextField(start, "key"));
+ ASSERT_OK(b.Int(1));
+ fields.push_back(b.NextField(start, "key"));
+ ASSERT_OK(b.Int(2));
+ ASSERT_RAISES(Invalid, b.FinishObject(start, fields));
+}
+
+TEST_F(VariantBuilderObjectTest, FieldsSortedByKey) {
+ // Insert fields in reverse order; verify they come out sorted
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<VariantBuilder::FieldEntry> fields;
+ fields.push_back(b.NextField(start, "z_last"));
+ ASSERT_OK(b.Int(3));
+ fields.push_back(b.NextField(start, "a_first"));
+ ASSERT_OK(b.Int(1));
+ fields.push_back(b.NextField(start, "m_middle"));
+ ASSERT_OK(b.Int(2));
+ ASSERT_OK(b.FinishObject(start, fields));
+
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[1], "FieldName(\"a_first\")");
+ ASSERT_EQ(events[2], "Int8(1)");
+ ASSERT_EQ(events[3], "FieldName(\"m_middle\")");
+ ASSERT_EQ(events[4], "Int8(2)");
+ ASSERT_EQ(events[5], "FieldName(\"z_last\")");
+ ASSERT_EQ(events[6], "Int8(3)");
+}
+
+// ===========================================================================
+// Builder features
+// ===========================================================================
+
+class VariantBuilderFeatureTest : public ::testing::Test {};
+
+TEST_F(VariantBuilderFeatureTest, Reset) {
+ VariantBuilder b;
+ ASSERT_OK(b.Int(42));
+ auto events1 = RoundTrip(b);
+ ASSERT_EQ(events1[0], "Int8(42)");
+
+ b.Reset();
+ ASSERT_OK(b.String("hello"));
+ auto events2 = RoundTrip(b);
+ ASSERT_EQ(events2[0], "String(\"hello\")");
+}
+
+TEST_F(VariantBuilderFeatureTest, BuilderFromExistingMetadata) {
+ // First, build a variant to get metadata
+ VariantBuilder b1;
+ auto start = b1.Offset();
+ std::vector<VariantBuilder::FieldEntry> fields;
+ fields.push_back(b1.NextField(start, "name"));
+ ASSERT_OK(b1.String("Alice"));
+ ASSERT_OK(b1.FinishObject(start, fields));
+ ASSERT_OK_AND_ASSIGN(auto encoded1, b1.Finish());
+
+ // Decode the metadata
+ ASSERT_OK_AND_ASSIGN(auto meta,
+ DecodeMetadata(encoded1.metadata.data(),
+ static_cast<int64_t>(encoded1.metadata.size())));
+
+ // Build a new variant reusing the same metadata
Review Comment:
Thanks for the suggestion. After working through the refactoring, I realized this test isn't possible because of how the variant format works, the metadata dictionary contains only key names (string interning for object field names), not value types. The format is self-describing: each value carries its own type tag in its header byte.
A "type mismatch between metadata and data" is architecturally impossible because metadata doesn't encode types at all. The refactored `VariantMetadata` struct's docstring makes this explicit: "This is NOT a schema, it contains key names only, not value types."
What you can test (and what the validation tests cover) is structural invariant violations: field IDs that exceed the dictionary size, offsets that point out of bounds, truncated values, etc. These are exercised by the `ValidateVariant` tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45947 : [C++][Parquet] Variant encoding [arrow]
Posted by "Explooosion-code (via GitHub)" <gi...@apache.org>.
Explooosion-code commented on code in PR #50122:
URL: https://github.com/apache/arrow/pull/50122#discussion_r3664209445
##########
cpp/src/arrow/extension/variant.h:
##########
@@ -0,0 +1,814 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+/// \file variant.h
+/// \brief Public C++ API for Variant binary encoding/decoding.
+///
+/// Provides zero-copy view classes for reading variant values (VariantView,
+/// VariantObjectView, VariantArrayView) and a visitor interface for full
+/// tree traversal. Implements the Variant Encoding Spec:
+/// https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
+///
+/// Design principles:
+/// - Parse once, query many (view classes pre-parse headers at construction)
+/// - Zero-copy (string_view into source buffers, no heap allocation for reads)
+/// - Type safety at boundaries (views validate at construction, not on access)
+/// - O(log n) field lookup always (no threshold heuristics)
+
+#include <cstdint>
+#include <functional>
+#include <optional>
+#include <string>
+#include <string_view>
+#include <unordered_map>
+#include <vector>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow::extension::variant {
+
+// ---------------------------------------------------------------------------
+// Constants
+// ---------------------------------------------------------------------------
+
+/// Variant encoding spec version 1.
+constexpr uint8_t kVariantVersion = 1;
+
+/// Maximum nesting depth for recursive value decoding.
+/// Prevents stack overflow on deeply nested (possibly malicious) input.
+constexpr int32_t kMaxNestingDepth = 128;
+
+/// UUID values are always 16 bytes (128-bit, big-endian per RFC 4122).
+constexpr int32_t kUUIDByteLength = 16;
+
+/// Maximum length for short-string encoding (6 bits → 0..63).
+/// Strings longer than this use the long-string (4-byte length prefix) format.
+constexpr int32_t kMaxShortStringLength = 63;
+
+/// Maximum supported decimal scale per the variant encoding spec.
+constexpr uint8_t kMaxDecimalScale = 38;
+
+/// Container element count threshold for large encoding.
+/// Objects/arrays with more than 255 elements use 4-byte num_elements fields.
+constexpr int32_t kLargeContainerThreshold = 255;
+
+// ---------------------------------------------------------------------------
+// Enumerations
+// ---------------------------------------------------------------------------
+
+/// \brief Basic type codes from bits 0-1 of the value header byte.
+///
+/// See:
+/// https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types
+enum class BasicType : uint8_t {
+ kPrimitive = 0,
+ kShortString = 1,
+ kObject = 2,
+ kArray = 3,
+};
+
+/// \brief Primitive type codes from bits 2-7 when basic_type == kPrimitive.
+///
+/// See:
+/// https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types
+enum class PrimitiveType : uint8_t {
+ kNull = 0,
+ kTrue = 1,
+ kFalse = 2,
+ kInt8 = 3,
+ kInt16 = 4,
+ kInt32 = 5,
+ kInt64 = 6,
+ kDouble = 7,
+ kDecimal4 = 8,
+ kDecimal8 = 9,
+ kDecimal16 = 10,
+ kDate = 11,
+ kTimestampMicros = 12,
+ kTimestampMicrosNTZ = 13,
+ kFloat = 14,
+ kBinary = 15,
+ kString = 16,
+ kTimeNTZ = 17,
+ kTimestampNanos = 18,
+ kTimestampNanosNTZ = 19,
+ kUUID = 20,
+};
+
+// ---------------------------------------------------------------------------
+// Metadata
+// ---------------------------------------------------------------------------
+
+/// \brief Parsed variant metadata (string dictionary).
+///
+/// The metadata buffer contains a header byte followed by a dictionary of
+/// interned strings used as object field names. String views reference the
+/// raw buffer and are valid only as long as the underlying buffer is alive.
+///
+/// \note This is NOT a schema — it contains key names only, not value types.
+struct ARROW_EXPORT VariantMetadata {
+ /// Spec version (must be kVariantVersion).
+ uint8_t version = 0;
+
+ /// Whether the dictionary strings are sorted lexicographically.
+ bool is_sorted = false;
+
+ /// Number of bytes used for each offset (1, 2, 3, or 4).
+ int32_t offset_size = 0;
+
+ /// Dictionary of interned strings. Views into the raw metadata buffer.
+ std::vector<std::string_view> strings;
+};
+
+/// \brief Decode a variant metadata buffer.
+///
+/// Parses the header byte and string dictionary from the raw metadata
+/// buffer. The returned VariantMetadata contains string_views that
+/// reference the input buffer directly (zero-copy).
+///
+/// \param[in] data Pointer to the metadata buffer (must not be null)
+/// \param[in] length Length of the metadata buffer in bytes
+/// \return Parsed VariantMetadata on success, Status::Invalid on
+/// malformed input
+///
+/// \note The input buffer must outlive the returned VariantMetadata.
+ARROW_EXPORT Result<VariantMetadata> DecodeMetadata(const uint8_t* data, int64_t length);
+
+/// \brief Find the dictionary ID for a given key name.
+///
+/// Uses binary search if the metadata is sorted, otherwise linear scan.
+///
+/// \param[in] metadata Parsed metadata
+/// \param[in] key The key to search for
+/// \return The dictionary ID if found, or -1 if not present
+ARROW_EXPORT int32_t FindMetadataKey(const VariantMetadata& metadata,
+ std::string_view key);
+
+// ---------------------------------------------------------------------------
+// Header utilities
+// ---------------------------------------------------------------------------
+
+/// \brief Extract the basic type from a value header byte.
+inline BasicType GetBasicType(uint8_t header) {
+ return static_cast<BasicType>(header & 0x03);
+}
+
+/// \brief Extract the primitive type from a value header byte.
+/// Only valid when GetBasicType(header) == BasicType::kPrimitive.
+inline PrimitiveType GetPrimitiveType(uint8_t header) {
+ return static_cast<PrimitiveType>((header >> 2) & 0x3F);
+}
+
+/// \brief Get the byte size of a primitive value (excluding header).
+/// Returns -1 for variable-length types (Binary, String).
+ARROW_EXPORT int32_t PrimitiveValueSize(PrimitiveType primitive_type);
+
+/// \brief Compute the total byte size of a variant value (header + data).
+///
+/// Determines how many bytes a variant value occupies without full decoding.
+/// \param[in] data Pointer to the start of a variant value
+/// \param[in] length Maximum bytes available
+/// \return Total byte count, or Status::Invalid if truncated
+ARROW_EXPORT Result<int64_t> ValueSize(const uint8_t* data, int64_t length);
+
+/// \brief Recursively validate a variant value and all nested children.
+///
+/// Performs deep structural validation of the entire value tree:
+/// - All headers are well-formed
+/// - All field IDs reference valid dictionary entries
+/// - All offsets are within bounds and monotonically non-decreasing
+/// - All nested values are recursively valid
+/// - Nesting depth does not exceed kMaxNestingDepth
+///
+/// Use this for untrusted input where you want a single pass/fail before
+/// operating on the data. For trusted data (e.g., builder output), the
+/// per-access validation in view classes is sufficient.
+///
+/// \param[in] metadata Parsed metadata (for validating field ID references)
+/// \param[in] data Pointer to the variant value buffer
+/// \param[in] length Length of the variant value buffer in bytes
+/// \return Status::OK if valid, Status::Invalid with description of first error
+ARROW_EXPORT Status ValidateVariant(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length);
+
+// ---------------------------------------------------------------------------
+// Forward declarations
+// ---------------------------------------------------------------------------
+
+class VariantObjectView;
+class VariantArrayView;
+
+// ---------------------------------------------------------------------------
+// VariantView — non-owning view over a single variant value
+// ---------------------------------------------------------------------------
+
+/// \brief A non-owning, zero-copy view over a single variant value.
+///
+/// Construction validates the header byte is readable. Subsequent typed
+/// accessors validate type compatibility and return errors on mismatch.
+///
+/// Stack-allocated, ~32 bytes. No heap allocation.
+///
+/// \note The metadata and data buffers must outlive this view.
+class ARROW_EXPORT VariantView {
+ public:
+ /// \brief Construct a view over a variant value.
+ ///
+ /// Validates that the buffer has at least one byte and computes the
+ /// value's total size. Returns Invalid on empty/null buffers or if
+ /// the value is truncated.
+ ///
+ /// \param[in] metadata Parsed metadata (for resolving object field names)
+ /// \param[in] data Pointer to the value buffer
+ /// \param[in] length Length of the value buffer in bytes
+ static Result<VariantView> Make(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length);
+
+ /// \brief The basic type of this value.
+ BasicType type() const { return type_; }
+
+ /// \brief Whether this value is null (PrimitiveType::kNull).
+ bool is_null() const;
+
+ /// \brief Total byte size of this value (header + payload).
+ int64_t size_bytes() const { return size_; }
+
+ /// \brief Raw pointer to the value bytes.
+ const uint8_t* data() const { return data_; }
+
+ /// @name Primitive accessors
+ /// Each returns the value if the type matches, or Status::Invalid.
+ /// @{
+ Result<bool> as_bool() const;
+ Result<int8_t> as_int8() const;
+ Result<int16_t> as_int16() const;
+ Result<int32_t> as_int32() const;
+ Result<int64_t> as_int64() const;
+ Result<float> as_float() const;
+ Result<double> as_double() const;
+ Result<std::string_view> as_string() const;
+ Result<std::string_view> as_binary() const;
+ Result<int32_t> as_date() const;
+ Result<int64_t> as_timestamp_micros() const;
+ Result<int64_t> as_timestamp_micros_ntz() const;
+ Result<int64_t> as_timestamp_nanos() const;
+ Result<int64_t> as_timestamp_nanos_ntz() const;
+ Result<int64_t> as_time_ntz() const;
+ /// @}
+
+ /// @name Widening numeric accessors (Rust parity)
+ /// These coerce narrower integer types to a wider target type.
+ /// e.g., as_int64_coerced() succeeds on Int8, Int16, Int32, or Int64 values.
+ /// @{
+
+ /// \brief Read any integer variant as int64, widening if necessary.
+ /// Succeeds for Int8, Int16, Int32, and Int64 encoded values.
+ Result<int64_t> as_int64_coerced() const;
+
+ /// \brief Read any integer variant as int32, widening if necessary.
+ /// Succeeds for Int8 and Int16 and Int32 encoded values.
+ /// Returns Invalid for Int64 (would narrow).
+ Result<int32_t> as_int32_coerced() const;
+
+ /// \brief Read any numeric variant as double, coercing if necessary.
+ /// Succeeds for Int8, Int16, Int32, Int64, Float, and Double.
+ /// Note: Int64 -> double may lose precision for large values.
+ Result<double> as_double_coerced() const;
+ /// @}
+
+ /// \brief Access UUID value (16 bytes, big-endian).
+ /// Returns pointer to the 16 UUID bytes within the source buffer.
+ Result<const uint8_t*> as_uuid() const;
+
+ /// \brief Access Decimal4 (scale + 4 bytes unscaled value).
+ /// \param[out] scale Set to the decimal scale
+ /// \return Pointer to the 4 unscaled value bytes
+ Result<const uint8_t*> as_decimal4(int32_t* scale) const;
+
+ /// \brief Access Decimal8 (scale + 8 bytes unscaled value).
+ Result<const uint8_t*> as_decimal8(int32_t* scale) const;
+
+ /// \brief Access Decimal16 (scale + 16 bytes unscaled value).
+ Result<const uint8_t*> as_decimal16(int32_t* scale) const;
+ /// @}
+
+ /// @name Container accessors
+ /// @{
+
+ /// \brief Interpret this value as an object.
+ /// Returns Invalid if the basic type is not kObject.
+ Result<VariantObjectView> as_object() const;
+
+ /// \brief Interpret this value as an array.
+ /// Returns Invalid if the basic type is not kArray.
+ Result<VariantArrayView> as_array() const;
+ /// @}
+
+ /// @name Visitor traversal
+ /// @{
+
+ /// \brief Full recursive traversal via visitor pattern.
+ ///
+ /// Visits every node in the value tree, calling appropriate visitor
+ /// methods. For bulk processing of entire variant values (e.g., JSON
+ /// serialization, schema inference).
+ ///
+ /// \param[in] visitor Callback interface for decoded values
+ /// \return Status::OK on success, or first error from visitor/decode
+ Status Visit(class VariantVisitor* visitor) const;
+ /// @}
+
+ private:
+ VariantView(const VariantMetadata* metadata, const uint8_t* data, int64_t size,
+ BasicType type);
+
+ const VariantMetadata* metadata_;
+ const uint8_t* data_;
+ int64_t size_;
+ BasicType type_;
+};
+
+// ---------------------------------------------------------------------------
+// VariantObjectView — pre-parsed object for O(log n) field lookup
+// ---------------------------------------------------------------------------
+
+/// \brief A non-owning view over a variant object value with pre-parsed header.
+///
+/// Construction validates the object header structure (field counts, array
+/// bounds). After construction, field lookup is O(log n) via binary search
+/// with no redundant parsing.
+///
+/// Stack-allocated, ~72 bytes. No heap allocation.
+///
+/// \note The metadata and data buffers must outlive this view.
+class ARROW_EXPORT VariantObjectView {
+ public:
+ /// \brief Construct an object view, pre-parsing the header.
+ ///
+ /// Validates:
+ /// 1. Basic type is kObject
+ /// 2. num_fields is readable
+ /// 3. Field ID array fits in buffer
+ /// 4. Offset array fits in buffer
+ /// 5. Last offset (total data size) is within buffer
+ ///
+ /// After Make() succeeds, all field accessors are bounds-safe.
+ static Result<VariantObjectView> Make(const VariantMetadata& metadata,
+ const uint8_t* data, int64_t length);
+
+ /// \brief Number of fields in this object.
+ int32_t num_fields() const { return num_fields_; }
+
+ /// \brief Get the name of the i-th field (0-indexed).
+ /// \return The field name, or Invalid if index is out of range or
+ /// field ID references an invalid dictionary entry.
+ Result<std::string_view> field_name(int32_t index) const;
+
+ /// \brief Get the value of the i-th field (0-indexed).
+ /// \return A VariantView over the field's value.
+ Result<VariantView> field_value(int32_t index) const;
+
+ /// \brief Lookup a field by name using binary search.
+ ///
+ /// Per spec, field IDs are sorted by lexicographic order of their
+ /// corresponding key names. This enables O(log n) lookup for all
+ /// object sizes.
+ ///
+ /// \param[in] name The field name to search for
+ /// \return The field's value if found, or std::nullopt if not present.
+ ///
+ /// \note Returns std::nullopt for both "field not found" AND "malformed data"
+ /// (e.g., field ID exceeds dictionary, or field value bytes are truncated).
+ /// For untrusted data where error reporting is needed, use field_name(i)
+ /// and field_value(i) directly which return Result<T>.
+ std::optional<VariantView> get(std::string_view name) const;
+
+ /// \brief Check if a field exists by name.
+ bool contains(std::string_view name) const;
+
+ /// \brief Location of a field's value bytes within the object buffer.
+ struct FieldLocation {
+ int64_t offset; ///< Byte offset from object start to field value
+ int64_t size; ///< Byte size of the field value
+ };
+
+ /// \brief Locate a field's raw bytes by name without constructing a view.
+ ///
+ /// Useful for zero-copy extraction when you need the offset+size for
+ /// raw byte operations (e.g., shredding's UnsafeAppendEncoded).
+ ///
+ /// \param[in] name The field name to search for
+ /// \return The field location, or std::nullopt if not present.
+ std::optional<FieldLocation> locate(std::string_view name) const;
+
+ /// @name Iteration support
+ /// @{
+
+ /// \brief Iterator over (name, value) pairs.
+ ///
+ /// \warning Iteration uses ValueOrDie() internally. If the object contains
+ /// malformed field data (corrupt value bytes), dereferencing the
+ /// iterator will abort. Use field_name(i)/field_value(i) for
+ /// error-safe access on untrusted data.
+ class Iterator {
+ public:
+ using value_type = std::pair<std::string_view, VariantView>;
+
+ Iterator(const VariantObjectView* obj, int32_t index);
+
+ value_type operator*() const;
+ Iterator& operator++();
+ bool operator!=(const Iterator& other) const;
+
+ private:
+ const VariantObjectView* obj_;
+ int32_t index_;
+ };
+
+ Iterator begin() const;
+ Iterator end() const;
+ /// @}
+
+ private:
+ VariantObjectView(const VariantMetadata* metadata, const uint8_t* data, int64_t length,
+ int32_t num_fields, int8_t field_id_size, int8_t field_offset_size,
+ int64_t id_start, int64_t offset_start, int64_t data_start);
+
+ /// \brief Read the field ID at position i.
+ uint32_t field_id_at(int32_t i) const;
+
+ /// \brief Read the value offset at position i.
+ int64_t value_offset_at(int32_t i) const;
+
+ const VariantMetadata* metadata_;
+ const uint8_t* data_;
+ int64_t length_;
+ int32_t num_fields_;
+ int8_t field_id_size_;
+ int8_t field_offset_size_;
+ int64_t id_start_;
+ int64_t offset_start_;
+ int64_t data_start_;
+};
+
+// ---------------------------------------------------------------------------
+// VariantArrayView — pre-parsed array for O(1) element access
+// ---------------------------------------------------------------------------
+
+/// \brief A non-owning view over a variant array value with pre-parsed header.
+///
+/// Construction validates the array header structure. After construction,
+/// element access is O(1) via the pre-computed offset table.
+///
+/// Stack-allocated, ~56 bytes. No heap allocation.
+class ARROW_EXPORT VariantArrayView {
+ public:
+ /// \brief Construct an array view, pre-parsing the header.
+ ///
+ /// Validates:
+ /// 1. Basic type is kArray
+ /// 2. num_elements is readable
+ /// 3. Offset array fits in buffer
+ /// 4. Offsets are monotonically non-decreasing
+ /// 5. Last offset is within buffer
+ static Result<VariantArrayView> Make(const VariantMetadata& metadata,
+ const uint8_t* data, int64_t length);
+
+ /// \brief Number of elements in this array.
+ int32_t num_elements() const { return num_elements_; }
+
+ /// \brief Get the i-th element (0-indexed, O(1) access).
+ /// \return A VariantView over the element, or Invalid if index is out of range.
+ Result<VariantView> get(int32_t index) const;
+
+ /// @name Iteration support
+ /// @{
+
+ /// \warning Iteration uses ValueOrDie() internally. If the array contains
+ /// malformed element data, dereferencing the iterator will abort.
+ /// Use get(i) for error-safe access on untrusted data.
+ class Iterator {
+ public:
+ using value_type = VariantView;
+
+ Iterator(const VariantArrayView* arr, int32_t index);
+
+ value_type operator*() const;
+ Iterator& operator++();
+ bool operator!=(const Iterator& other) const;
+
+ private:
+ const VariantArrayView* arr_;
+ int32_t index_;
+ };
+
+ Iterator begin() const;
+ Iterator end() const;
+ /// @}
+
+ private:
+ VariantArrayView(const VariantMetadata* metadata, const uint8_t* data, int64_t length,
+ int32_t num_elements, int8_t offset_size, int64_t offset_start,
+ int64_t data_start);
+
+ /// \brief Read the element offset at position i.
+ int64_t element_offset_at(int32_t i) const;
+
+ const VariantMetadata* metadata_;
+ const uint8_t* data_;
+ int64_t length_;
+ int32_t num_elements_;
+ int8_t offset_size_;
+ int64_t offset_start_;
+ int64_t data_start_;
+};
+
+// ---------------------------------------------------------------------------
+// Visitor interface (for full tree traversal)
+// ---------------------------------------------------------------------------
+
+/// \brief Visitor interface for variant value traversal (SAX-style).
+///
+/// Implement this interface to receive callbacks during recursive variant
+/// value traversal. Use VariantView::Visit() to drive the traversal.
+///
+/// For point-queries (reading a specific field), use the view classes
+/// directly instead. The visitor is for bulk operations that need to
+/// process every node in the tree.
+///
+/// \note String values are raw bytes without UTF-8 validation.
+class ARROW_EXPORT VariantVisitor {
+ public:
+ virtual ~VariantVisitor() = default;
+
+ /// @name Primitive value callbacks
+ /// @{
+ virtual Status Null() = 0;
+ virtual Status Bool(bool value) = 0;
+ virtual Status Int8(int8_t value) = 0;
+ virtual Status Int16(int16_t value) = 0;
+ virtual Status Int32(int32_t value) = 0;
+ virtual Status Int64(int64_t value) = 0;
+ virtual Status Float(float value) = 0;
+ virtual Status Double(double value) = 0;
+ virtual Status Decimal4(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal8(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal16(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Date(int32_t days_since_epoch) = 0;
+ virtual Status TimestampMicros(int64_t micros_since_epoch) = 0;
+ virtual Status TimestampMicrosNTZ(int64_t micros_since_epoch) = 0;
+ virtual Status String(std::string_view value) = 0;
+ virtual Status Binary(std::string_view value) = 0;
+ virtual Status TimeNTZ(int64_t micros_since_midnight) = 0;
+ virtual Status TimestampNanos(int64_t nanos_since_epoch) = 0;
+ virtual Status TimestampNanosNTZ(int64_t nanos_since_epoch) = 0;
+ virtual Status UUID(const uint8_t* bytes) = 0;
+ /// @}
+
+ /// @name Container callbacks
+ /// @{
+ virtual Status StartObject(int32_t num_fields) = 0;
+ virtual Status FieldName(std::string_view name) = 0;
+ virtual Status EndObject() = 0;
+ virtual Status StartArray(int32_t num_elements) = 0;
+ virtual Status EndArray() = 0;
+ /// @}
+};
+
+// ---------------------------------------------------------------------------
+// VariantBuilder (encoder)
+// ---------------------------------------------------------------------------
+
+class ObjectScope;
+class ListScope;
+
+/// \brief Builder for constructing Variant binary values.
+///
+/// Provides both low-level (Offset/NextField/FinishObject) and high-level
+/// (StartObject/StartList returning RAII scopes) APIs for encoding.
+class ARROW_EXPORT VariantBuilder {
+ public:
+ VariantBuilder();
+ explicit VariantBuilder(const VariantMetadata& existing_metadata);
+ ~VariantBuilder() = default;
+
+ VariantBuilder(VariantBuilder&&) noexcept = default;
+ VariantBuilder& operator=(VariantBuilder&&) noexcept = default;
+ VariantBuilder(const VariantBuilder&) = delete;
+ VariantBuilder& operator=(const VariantBuilder&) = delete;
+
+ /// @name Primitive value setters
+ /// @{
+ Status Null();
+ Status Bool(bool value);
+ Status Int(int64_t value); ///< Auto-selects smallest int type
+ Status Int8(int8_t value);
+ Status Int16(int16_t value);
+ Status Int32(int32_t value);
+ Status Int64(int64_t value);
+ Status Float(float value);
+ Status Double(double value);
+ Status Decimal4(uint8_t scale, const uint8_t* value_bytes);
+ Status Decimal8(uint8_t scale, const uint8_t* value_bytes);
+ Status Decimal16(uint8_t scale, const uint8_t* value_bytes);
+ Status Date(int32_t days_since_epoch);
+ Status TimestampMicros(int64_t micros);
+ Status TimestampMicrosNTZ(int64_t micros);
+ Status TimeNTZ(int64_t micros);
+ Status TimestampNanos(int64_t nanos);
+ Status TimestampNanosNTZ(int64_t nanos);
+ Status String(std::string_view value); ///< Auto short-string for <=63 bytes
+ Status Binary(std::string_view value);
+ Status UUID(const uint8_t* bytes);
+ /// @}
+
+ /// @name Low-level container construction
+ /// @{
+ int64_t Offset() const;
+ int64_t NextElement(int64_t start) const;
+
+ struct FieldEntry {
+ std::string key;
+ uint32_t id;
+ int64_t offset;
+ };
+
+ FieldEntry NextField(int64_t start, std::string_view key);
+ Status FinishArray(int64_t start, const std::vector<int64_t>& offsets);
+ Status FinishObject(int64_t start, std::vector<FieldEntry>& fields);
+ /// @}
+
+ /// @name RAII container construction
+ /// @{
+
+ /// \brief Start building an object. Returns an RAII scope that auto-rolls
+ /// back if Finish() is not called (e.g., on exception/early return).
+ [[nodiscard]] ObjectScope StartObject();
+
+ /// \brief Start building a list/array. Returns an RAII scope.
+ [[nodiscard]] ListScope StartList();
+ /// @}
+
+ /// @name Output
+ /// @{
+ struct EncodedVariant {
+ std::vector<uint8_t> metadata;
+ std::vector<uint8_t> value;
+ };
+
+ Result<EncodedVariant> Finish();
+ void Reset();
+ /// @}
+
+ /// @name Internal (used by scopes and shredding)
+ /// @name Internal (used by scopes)
+ void Truncate(int64_t offset);
+ /// @}
+
+ private:
+ friend class ObjectScope;
+ friend class ListScope;
+
+ uint32_t AddKey(std::string_view key);
+
+ std::vector<uint8_t> buffer_;
+ /// \brief Dictionary mapping key names to their IDs.
+ /// Uses a custom transparent hasher to allow lookups with string_view
+ /// without constructing a std::string (C++17 heterogeneous lookup).
+ struct StringHash {
+ using is_transparent = void;
+ size_t operator()(std::string_view sv) const noexcept {
+ return std::hash<std::string_view>{}(sv);
+ }
+ };
+ struct StringEqual {
+ using is_transparent = void;
+ bool operator()(std::string_view a, std::string_view b) const noexcept {
+ return a == b;
+ }
+ };
+ std::unordered_map<std::string, uint32_t, StringHash, StringEqual> dict_;
+ std::vector<std::string> dict_keys_;
+ bool allow_duplicates_ = false;
+};
+
+// ---------------------------------------------------------------------------
+// ObjectScope — RAII scoped object builder
+// ---------------------------------------------------------------------------
+
+/// \brief RAII scope for building an object. Destructor rolls back if not committed.
+///
+/// Usage:
+/// auto obj = builder.StartObject();
+/// obj.Insert("name", "Alice");
+/// obj.Insert("age", 30);
+/// obj.Finish(); // commits
+/// // If Finish() not called, destructor truncates buffer to pre-scope state
+class ARROW_EXPORT ObjectScope {
+ public:
+ ~ObjectScope();
+
+ ObjectScope(const ObjectScope&) = delete;
+ ObjectScope& operator=(const ObjectScope&) = delete;
+ ObjectScope(ObjectScope&& other) noexcept;
+ ObjectScope& operator=(ObjectScope&&) = delete;
+
+ /// @name Insert fields (delegates to VariantBuilder primitives)
+ /// @{
+ Status Insert(std::string_view key, std::nullptr_t);
+ Status Insert(std::string_view key, bool value);
+ Status Insert(std::string_view key, int64_t value);
+ Status Insert(std::string_view key, double value);
+ Status Insert(std::string_view key, std::string_view value);
Review Comment:
Btw this also applies to the Array logic of `Append`
##########
cpp/src/arrow/extension/variant_builder.cc:
##########
Review Comment:
A general comment on this implementation. It lacks depth tracking for LIFO operations like `StartList`, `AppendObject` etc. For example there is no error (but generated variant is invalid) for
- Starting a Builder
- Starting an Object
- Finishing Builder
##########
cpp/src/arrow/extension/variant.h:
##########
@@ -0,0 +1,814 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+/// \file variant.h
+/// \brief Public C++ API for Variant binary encoding/decoding.
+///
+/// Provides zero-copy view classes for reading variant values (VariantView,
+/// VariantObjectView, VariantArrayView) and a visitor interface for full
+/// tree traversal. Implements the Variant Encoding Spec:
+/// https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
+///
+/// Design principles:
+/// - Parse once, query many (view classes pre-parse headers at construction)
+/// - Zero-copy (string_view into source buffers, no heap allocation for reads)
+/// - Type safety at boundaries (views validate at construction, not on access)
+/// - O(log n) field lookup always (no threshold heuristics)
+
+#include <cstdint>
+#include <functional>
+#include <optional>
+#include <string>
+#include <string_view>
+#include <unordered_map>
+#include <vector>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow::extension::variant {
+
+// ---------------------------------------------------------------------------
+// Constants
+// ---------------------------------------------------------------------------
+
+/// Variant encoding spec version 1.
+constexpr uint8_t kVariantVersion = 1;
+
+/// Maximum nesting depth for recursive value decoding.
+/// Prevents stack overflow on deeply nested (possibly malicious) input.
+constexpr int32_t kMaxNestingDepth = 128;
+
+/// UUID values are always 16 bytes (128-bit, big-endian per RFC 4122).
+constexpr int32_t kUUIDByteLength = 16;
+
+/// Maximum length for short-string encoding (6 bits → 0..63).
+/// Strings longer than this use the long-string (4-byte length prefix) format.
+constexpr int32_t kMaxShortStringLength = 63;
+
+/// Maximum supported decimal scale per the variant encoding spec.
+constexpr uint8_t kMaxDecimalScale = 38;
+
+/// Container element count threshold for large encoding.
+/// Objects/arrays with more than 255 elements use 4-byte num_elements fields.
+constexpr int32_t kLargeContainerThreshold = 255;
+
+// ---------------------------------------------------------------------------
+// Enumerations
+// ---------------------------------------------------------------------------
+
+/// \brief Basic type codes from bits 0-1 of the value header byte.
+///
+/// See:
+/// https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types
+enum class BasicType : uint8_t {
+ kPrimitive = 0,
+ kShortString = 1,
+ kObject = 2,
+ kArray = 3,
+};
+
+/// \brief Primitive type codes from bits 2-7 when basic_type == kPrimitive.
+///
+/// See:
+/// https://github.com/apache/parquet-format/blob/master/VariantEncoding.md#encoding-types
+enum class PrimitiveType : uint8_t {
+ kNull = 0,
+ kTrue = 1,
+ kFalse = 2,
+ kInt8 = 3,
+ kInt16 = 4,
+ kInt32 = 5,
+ kInt64 = 6,
+ kDouble = 7,
+ kDecimal4 = 8,
+ kDecimal8 = 9,
+ kDecimal16 = 10,
+ kDate = 11,
+ kTimestampMicros = 12,
+ kTimestampMicrosNTZ = 13,
+ kFloat = 14,
+ kBinary = 15,
+ kString = 16,
+ kTimeNTZ = 17,
+ kTimestampNanos = 18,
+ kTimestampNanosNTZ = 19,
+ kUUID = 20,
+};
+
+// ---------------------------------------------------------------------------
+// Metadata
+// ---------------------------------------------------------------------------
+
+/// \brief Parsed variant metadata (string dictionary).
+///
+/// The metadata buffer contains a header byte followed by a dictionary of
+/// interned strings used as object field names. String views reference the
+/// raw buffer and are valid only as long as the underlying buffer is alive.
+///
+/// \note This is NOT a schema — it contains key names only, not value types.
+struct ARROW_EXPORT VariantMetadata {
+ /// Spec version (must be kVariantVersion).
+ uint8_t version = 0;
+
+ /// Whether the dictionary strings are sorted lexicographically.
+ bool is_sorted = false;
+
+ /// Number of bytes used for each offset (1, 2, 3, or 4).
+ int32_t offset_size = 0;
+
+ /// Dictionary of interned strings. Views into the raw metadata buffer.
+ std::vector<std::string_view> strings;
+};
+
+/// \brief Decode a variant metadata buffer.
+///
+/// Parses the header byte and string dictionary from the raw metadata
+/// buffer. The returned VariantMetadata contains string_views that
+/// reference the input buffer directly (zero-copy).
+///
+/// \param[in] data Pointer to the metadata buffer (must not be null)
+/// \param[in] length Length of the metadata buffer in bytes
+/// \return Parsed VariantMetadata on success, Status::Invalid on
+/// malformed input
+///
+/// \note The input buffer must outlive the returned VariantMetadata.
+ARROW_EXPORT Result<VariantMetadata> DecodeMetadata(const uint8_t* data, int64_t length);
+
+/// \brief Find the dictionary ID for a given key name.
+///
+/// Uses binary search if the metadata is sorted, otherwise linear scan.
+///
+/// \param[in] metadata Parsed metadata
+/// \param[in] key The key to search for
+/// \return The dictionary ID if found, or -1 if not present
+ARROW_EXPORT int32_t FindMetadataKey(const VariantMetadata& metadata,
+ std::string_view key);
+
+// ---------------------------------------------------------------------------
+// Header utilities
+// ---------------------------------------------------------------------------
+
+/// \brief Extract the basic type from a value header byte.
+inline BasicType GetBasicType(uint8_t header) {
+ return static_cast<BasicType>(header & 0x03);
+}
+
+/// \brief Extract the primitive type from a value header byte.
+/// Only valid when GetBasicType(header) == BasicType::kPrimitive.
+inline PrimitiveType GetPrimitiveType(uint8_t header) {
+ return static_cast<PrimitiveType>((header >> 2) & 0x3F);
+}
+
+/// \brief Get the byte size of a primitive value (excluding header).
+/// Returns -1 for variable-length types (Binary, String).
+ARROW_EXPORT int32_t PrimitiveValueSize(PrimitiveType primitive_type);
+
+/// \brief Compute the total byte size of a variant value (header + data).
+///
+/// Determines how many bytes a variant value occupies without full decoding.
+/// \param[in] data Pointer to the start of a variant value
+/// \param[in] length Maximum bytes available
+/// \return Total byte count, or Status::Invalid if truncated
+ARROW_EXPORT Result<int64_t> ValueSize(const uint8_t* data, int64_t length);
+
+/// \brief Recursively validate a variant value and all nested children.
+///
+/// Performs deep structural validation of the entire value tree:
+/// - All headers are well-formed
+/// - All field IDs reference valid dictionary entries
+/// - All offsets are within bounds and monotonically non-decreasing
+/// - All nested values are recursively valid
+/// - Nesting depth does not exceed kMaxNestingDepth
+///
+/// Use this for untrusted input where you want a single pass/fail before
+/// operating on the data. For trusted data (e.g., builder output), the
+/// per-access validation in view classes is sufficient.
+///
+/// \param[in] metadata Parsed metadata (for validating field ID references)
+/// \param[in] data Pointer to the variant value buffer
+/// \param[in] length Length of the variant value buffer in bytes
+/// \return Status::OK if valid, Status::Invalid with description of first error
+ARROW_EXPORT Status ValidateVariant(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length);
+
+// ---------------------------------------------------------------------------
+// Forward declarations
+// ---------------------------------------------------------------------------
+
+class VariantObjectView;
+class VariantArrayView;
+
+// ---------------------------------------------------------------------------
+// VariantView — non-owning view over a single variant value
+// ---------------------------------------------------------------------------
+
+/// \brief A non-owning, zero-copy view over a single variant value.
+///
+/// Construction validates the header byte is readable. Subsequent typed
+/// accessors validate type compatibility and return errors on mismatch.
+///
+/// Stack-allocated, ~32 bytes. No heap allocation.
+///
+/// \note The metadata and data buffers must outlive this view.
+class ARROW_EXPORT VariantView {
+ public:
+ /// \brief Construct a view over a variant value.
+ ///
+ /// Validates that the buffer has at least one byte and computes the
+ /// value's total size. Returns Invalid on empty/null buffers or if
+ /// the value is truncated.
+ ///
+ /// \param[in] metadata Parsed metadata (for resolving object field names)
+ /// \param[in] data Pointer to the value buffer
+ /// \param[in] length Length of the value buffer in bytes
+ static Result<VariantView> Make(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length);
+
+ /// \brief The basic type of this value.
+ BasicType type() const { return type_; }
+
+ /// \brief Whether this value is null (PrimitiveType::kNull).
+ bool is_null() const;
+
+ /// \brief Total byte size of this value (header + payload).
+ int64_t size_bytes() const { return size_; }
+
+ /// \brief Raw pointer to the value bytes.
+ const uint8_t* data() const { return data_; }
+
+ /// @name Primitive accessors
+ /// Each returns the value if the type matches, or Status::Invalid.
+ /// @{
+ Result<bool> as_bool() const;
+ Result<int8_t> as_int8() const;
+ Result<int16_t> as_int16() const;
+ Result<int32_t> as_int32() const;
+ Result<int64_t> as_int64() const;
+ Result<float> as_float() const;
+ Result<double> as_double() const;
+ Result<std::string_view> as_string() const;
+ Result<std::string_view> as_binary() const;
+ Result<int32_t> as_date() const;
+ Result<int64_t> as_timestamp_micros() const;
+ Result<int64_t> as_timestamp_micros_ntz() const;
+ Result<int64_t> as_timestamp_nanos() const;
+ Result<int64_t> as_timestamp_nanos_ntz() const;
+ Result<int64_t> as_time_ntz() const;
+ /// @}
+
+ /// @name Widening numeric accessors (Rust parity)
+ /// These coerce narrower integer types to a wider target type.
+ /// e.g., as_int64_coerced() succeeds on Int8, Int16, Int32, or Int64 values.
+ /// @{
+
+ /// \brief Read any integer variant as int64, widening if necessary.
+ /// Succeeds for Int8, Int16, Int32, and Int64 encoded values.
+ Result<int64_t> as_int64_coerced() const;
+
+ /// \brief Read any integer variant as int32, widening if necessary.
+ /// Succeeds for Int8 and Int16 and Int32 encoded values.
+ /// Returns Invalid for Int64 (would narrow).
+ Result<int32_t> as_int32_coerced() const;
+
+ /// \brief Read any numeric variant as double, coercing if necessary.
+ /// Succeeds for Int8, Int16, Int32, Int64, Float, and Double.
+ /// Note: Int64 -> double may lose precision for large values.
+ Result<double> as_double_coerced() const;
+ /// @}
+
+ /// \brief Access UUID value (16 bytes, big-endian).
+ /// Returns pointer to the 16 UUID bytes within the source buffer.
+ Result<const uint8_t*> as_uuid() const;
+
+ /// \brief Access Decimal4 (scale + 4 bytes unscaled value).
+ /// \param[out] scale Set to the decimal scale
+ /// \return Pointer to the 4 unscaled value bytes
+ Result<const uint8_t*> as_decimal4(int32_t* scale) const;
+
+ /// \brief Access Decimal8 (scale + 8 bytes unscaled value).
+ Result<const uint8_t*> as_decimal8(int32_t* scale) const;
+
+ /// \brief Access Decimal16 (scale + 16 bytes unscaled value).
+ Result<const uint8_t*> as_decimal16(int32_t* scale) const;
+ /// @}
+
+ /// @name Container accessors
+ /// @{
+
+ /// \brief Interpret this value as an object.
+ /// Returns Invalid if the basic type is not kObject.
+ Result<VariantObjectView> as_object() const;
+
+ /// \brief Interpret this value as an array.
+ /// Returns Invalid if the basic type is not kArray.
+ Result<VariantArrayView> as_array() const;
+ /// @}
+
+ /// @name Visitor traversal
+ /// @{
+
+ /// \brief Full recursive traversal via visitor pattern.
+ ///
+ /// Visits every node in the value tree, calling appropriate visitor
+ /// methods. For bulk processing of entire variant values (e.g., JSON
+ /// serialization, schema inference).
+ ///
+ /// \param[in] visitor Callback interface for decoded values
+ /// \return Status::OK on success, or first error from visitor/decode
+ Status Visit(class VariantVisitor* visitor) const;
+ /// @}
+
+ private:
+ VariantView(const VariantMetadata* metadata, const uint8_t* data, int64_t size,
+ BasicType type);
+
+ const VariantMetadata* metadata_;
+ const uint8_t* data_;
+ int64_t size_;
+ BasicType type_;
+};
+
+// ---------------------------------------------------------------------------
+// VariantObjectView — pre-parsed object for O(log n) field lookup
+// ---------------------------------------------------------------------------
+
+/// \brief A non-owning view over a variant object value with pre-parsed header.
+///
+/// Construction validates the object header structure (field counts, array
+/// bounds). After construction, field lookup is O(log n) via binary search
+/// with no redundant parsing.
+///
+/// Stack-allocated, ~72 bytes. No heap allocation.
+///
+/// \note The metadata and data buffers must outlive this view.
+class ARROW_EXPORT VariantObjectView {
+ public:
+ /// \brief Construct an object view, pre-parsing the header.
+ ///
+ /// Validates:
+ /// 1. Basic type is kObject
+ /// 2. num_fields is readable
+ /// 3. Field ID array fits in buffer
+ /// 4. Offset array fits in buffer
+ /// 5. Last offset (total data size) is within buffer
+ ///
+ /// After Make() succeeds, all field accessors are bounds-safe.
+ static Result<VariantObjectView> Make(const VariantMetadata& metadata,
+ const uint8_t* data, int64_t length);
+
+ /// \brief Number of fields in this object.
+ int32_t num_fields() const { return num_fields_; }
+
+ /// \brief Get the name of the i-th field (0-indexed).
+ /// \return The field name, or Invalid if index is out of range or
+ /// field ID references an invalid dictionary entry.
+ Result<std::string_view> field_name(int32_t index) const;
+
+ /// \brief Get the value of the i-th field (0-indexed).
+ /// \return A VariantView over the field's value.
+ Result<VariantView> field_value(int32_t index) const;
+
+ /// \brief Lookup a field by name using binary search.
+ ///
+ /// Per spec, field IDs are sorted by lexicographic order of their
+ /// corresponding key names. This enables O(log n) lookup for all
+ /// object sizes.
+ ///
+ /// \param[in] name The field name to search for
+ /// \return The field's value if found, or std::nullopt if not present.
+ ///
+ /// \note Returns std::nullopt for both "field not found" AND "malformed data"
+ /// (e.g., field ID exceeds dictionary, or field value bytes are truncated).
+ /// For untrusted data where error reporting is needed, use field_name(i)
+ /// and field_value(i) directly which return Result<T>.
+ std::optional<VariantView> get(std::string_view name) const;
+
+ /// \brief Check if a field exists by name.
+ bool contains(std::string_view name) const;
+
+ /// \brief Location of a field's value bytes within the object buffer.
+ struct FieldLocation {
+ int64_t offset; ///< Byte offset from object start to field value
+ int64_t size; ///< Byte size of the field value
+ };
+
+ /// \brief Locate a field's raw bytes by name without constructing a view.
+ ///
+ /// Useful for zero-copy extraction when you need the offset+size for
+ /// raw byte operations (e.g., shredding's UnsafeAppendEncoded).
+ ///
+ /// \param[in] name The field name to search for
+ /// \return The field location, or std::nullopt if not present.
+ std::optional<FieldLocation> locate(std::string_view name) const;
+
+ /// @name Iteration support
+ /// @{
+
+ /// \brief Iterator over (name, value) pairs.
+ ///
+ /// \warning Iteration uses ValueOrDie() internally. If the object contains
+ /// malformed field data (corrupt value bytes), dereferencing the
+ /// iterator will abort. Use field_name(i)/field_value(i) for
+ /// error-safe access on untrusted data.
+ class Iterator {
+ public:
+ using value_type = std::pair<std::string_view, VariantView>;
+
+ Iterator(const VariantObjectView* obj, int32_t index);
+
+ value_type operator*() const;
+ Iterator& operator++();
+ bool operator!=(const Iterator& other) const;
+
+ private:
+ const VariantObjectView* obj_;
+ int32_t index_;
+ };
+
+ Iterator begin() const;
+ Iterator end() const;
+ /// @}
+
+ private:
+ VariantObjectView(const VariantMetadata* metadata, const uint8_t* data, int64_t length,
+ int32_t num_fields, int8_t field_id_size, int8_t field_offset_size,
+ int64_t id_start, int64_t offset_start, int64_t data_start);
+
+ /// \brief Read the field ID at position i.
+ uint32_t field_id_at(int32_t i) const;
+
+ /// \brief Read the value offset at position i.
+ int64_t value_offset_at(int32_t i) const;
+
+ const VariantMetadata* metadata_;
+ const uint8_t* data_;
+ int64_t length_;
+ int32_t num_fields_;
+ int8_t field_id_size_;
+ int8_t field_offset_size_;
+ int64_t id_start_;
+ int64_t offset_start_;
+ int64_t data_start_;
+};
+
+// ---------------------------------------------------------------------------
+// VariantArrayView — pre-parsed array for O(1) element access
+// ---------------------------------------------------------------------------
+
+/// \brief A non-owning view over a variant array value with pre-parsed header.
+///
+/// Construction validates the array header structure. After construction,
+/// element access is O(1) via the pre-computed offset table.
+///
+/// Stack-allocated, ~56 bytes. No heap allocation.
+class ARROW_EXPORT VariantArrayView {
+ public:
+ /// \brief Construct an array view, pre-parsing the header.
+ ///
+ /// Validates:
+ /// 1. Basic type is kArray
+ /// 2. num_elements is readable
+ /// 3. Offset array fits in buffer
+ /// 4. Offsets are monotonically non-decreasing
+ /// 5. Last offset is within buffer
+ static Result<VariantArrayView> Make(const VariantMetadata& metadata,
+ const uint8_t* data, int64_t length);
+
+ /// \brief Number of elements in this array.
+ int32_t num_elements() const { return num_elements_; }
+
+ /// \brief Get the i-th element (0-indexed, O(1) access).
+ /// \return A VariantView over the element, or Invalid if index is out of range.
+ Result<VariantView> get(int32_t index) const;
+
+ /// @name Iteration support
+ /// @{
+
+ /// \warning Iteration uses ValueOrDie() internally. If the array contains
+ /// malformed element data, dereferencing the iterator will abort.
+ /// Use get(i) for error-safe access on untrusted data.
+ class Iterator {
+ public:
+ using value_type = VariantView;
+
+ Iterator(const VariantArrayView* arr, int32_t index);
+
+ value_type operator*() const;
+ Iterator& operator++();
+ bool operator!=(const Iterator& other) const;
+
+ private:
+ const VariantArrayView* arr_;
+ int32_t index_;
+ };
+
+ Iterator begin() const;
+ Iterator end() const;
+ /// @}
+
+ private:
+ VariantArrayView(const VariantMetadata* metadata, const uint8_t* data, int64_t length,
+ int32_t num_elements, int8_t offset_size, int64_t offset_start,
+ int64_t data_start);
+
+ /// \brief Read the element offset at position i.
+ int64_t element_offset_at(int32_t i) const;
+
+ const VariantMetadata* metadata_;
+ const uint8_t* data_;
+ int64_t length_;
+ int32_t num_elements_;
+ int8_t offset_size_;
+ int64_t offset_start_;
+ int64_t data_start_;
+};
+
+// ---------------------------------------------------------------------------
+// Visitor interface (for full tree traversal)
+// ---------------------------------------------------------------------------
+
+/// \brief Visitor interface for variant value traversal (SAX-style).
+///
+/// Implement this interface to receive callbacks during recursive variant
+/// value traversal. Use VariantView::Visit() to drive the traversal.
+///
+/// For point-queries (reading a specific field), use the view classes
+/// directly instead. The visitor is for bulk operations that need to
+/// process every node in the tree.
+///
+/// \note String values are raw bytes without UTF-8 validation.
+class ARROW_EXPORT VariantVisitor {
+ public:
+ virtual ~VariantVisitor() = default;
+
+ /// @name Primitive value callbacks
+ /// @{
+ virtual Status Null() = 0;
+ virtual Status Bool(bool value) = 0;
+ virtual Status Int8(int8_t value) = 0;
+ virtual Status Int16(int16_t value) = 0;
+ virtual Status Int32(int32_t value) = 0;
+ virtual Status Int64(int64_t value) = 0;
+ virtual Status Float(float value) = 0;
+ virtual Status Double(double value) = 0;
+ virtual Status Decimal4(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal8(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal16(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Date(int32_t days_since_epoch) = 0;
+ virtual Status TimestampMicros(int64_t micros_since_epoch) = 0;
+ virtual Status TimestampMicrosNTZ(int64_t micros_since_epoch) = 0;
+ virtual Status String(std::string_view value) = 0;
+ virtual Status Binary(std::string_view value) = 0;
+ virtual Status TimeNTZ(int64_t micros_since_midnight) = 0;
+ virtual Status TimestampNanos(int64_t nanos_since_epoch) = 0;
+ virtual Status TimestampNanosNTZ(int64_t nanos_since_epoch) = 0;
+ virtual Status UUID(const uint8_t* bytes) = 0;
+ /// @}
+
+ /// @name Container callbacks
+ /// @{
+ virtual Status StartObject(int32_t num_fields) = 0;
+ virtual Status FieldName(std::string_view name) = 0;
+ virtual Status EndObject() = 0;
+ virtual Status StartArray(int32_t num_elements) = 0;
+ virtual Status EndArray() = 0;
+ /// @}
+};
+
+// ---------------------------------------------------------------------------
+// VariantBuilder (encoder)
+// ---------------------------------------------------------------------------
+
+class ObjectScope;
+class ListScope;
+
+/// \brief Builder for constructing Variant binary values.
+///
+/// Provides both low-level (Offset/NextField/FinishObject) and high-level
+/// (StartObject/StartList returning RAII scopes) APIs for encoding.
+class ARROW_EXPORT VariantBuilder {
+ public:
+ VariantBuilder();
+ explicit VariantBuilder(const VariantMetadata& existing_metadata);
+ ~VariantBuilder() = default;
+
+ VariantBuilder(VariantBuilder&&) noexcept = default;
+ VariantBuilder& operator=(VariantBuilder&&) noexcept = default;
+ VariantBuilder(const VariantBuilder&) = delete;
+ VariantBuilder& operator=(const VariantBuilder&) = delete;
+
+ /// @name Primitive value setters
+ /// @{
+ Status Null();
+ Status Bool(bool value);
+ Status Int(int64_t value); ///< Auto-selects smallest int type
+ Status Int8(int8_t value);
+ Status Int16(int16_t value);
+ Status Int32(int32_t value);
+ Status Int64(int64_t value);
+ Status Float(float value);
+ Status Double(double value);
+ Status Decimal4(uint8_t scale, const uint8_t* value_bytes);
+ Status Decimal8(uint8_t scale, const uint8_t* value_bytes);
+ Status Decimal16(uint8_t scale, const uint8_t* value_bytes);
+ Status Date(int32_t days_since_epoch);
+ Status TimestampMicros(int64_t micros);
+ Status TimestampMicrosNTZ(int64_t micros);
+ Status TimeNTZ(int64_t micros);
+ Status TimestampNanos(int64_t nanos);
+ Status TimestampNanosNTZ(int64_t nanos);
+ Status String(std::string_view value); ///< Auto short-string for <=63 bytes
+ Status Binary(std::string_view value);
+ Status UUID(const uint8_t* bytes);
+ /// @}
+
+ /// @name Low-level container construction
+ /// @{
+ int64_t Offset() const;
+ int64_t NextElement(int64_t start) const;
+
+ struct FieldEntry {
+ std::string key;
+ uint32_t id;
+ int64_t offset;
+ };
+
+ FieldEntry NextField(int64_t start, std::string_view key);
+ Status FinishArray(int64_t start, const std::vector<int64_t>& offsets);
+ Status FinishObject(int64_t start, std::vector<FieldEntry>& fields);
+ /// @}
+
+ /// @name RAII container construction
+ /// @{
+
+ /// \brief Start building an object. Returns an RAII scope that auto-rolls
+ /// back if Finish() is not called (e.g., on exception/early return).
+ [[nodiscard]] ObjectScope StartObject();
+
+ /// \brief Start building a list/array. Returns an RAII scope.
+ [[nodiscard]] ListScope StartList();
+ /// @}
+
+ /// @name Output
+ /// @{
+ struct EncodedVariant {
+ std::vector<uint8_t> metadata;
+ std::vector<uint8_t> value;
+ };
+
+ Result<EncodedVariant> Finish();
+ void Reset();
+ /// @}
+
+ /// @name Internal (used by scopes and shredding)
+ /// @name Internal (used by scopes)
+ void Truncate(int64_t offset);
+ /// @}
+
+ private:
+ friend class ObjectScope;
+ friend class ListScope;
+
+ uint32_t AddKey(std::string_view key);
+
+ std::vector<uint8_t> buffer_;
+ /// \brief Dictionary mapping key names to their IDs.
+ /// Uses a custom transparent hasher to allow lookups with string_view
+ /// without constructing a std::string (C++17 heterogeneous lookup).
+ struct StringHash {
+ using is_transparent = void;
+ size_t operator()(std::string_view sv) const noexcept {
+ return std::hash<std::string_view>{}(sv);
+ }
+ };
+ struct StringEqual {
+ using is_transparent = void;
+ bool operator()(std::string_view a, std::string_view b) const noexcept {
+ return a == b;
+ }
+ };
+ std::unordered_map<std::string, uint32_t, StringHash, StringEqual> dict_;
+ std::vector<std::string> dict_keys_;
+ bool allow_duplicates_ = false;
+};
+
+// ---------------------------------------------------------------------------
+// ObjectScope — RAII scoped object builder
+// ---------------------------------------------------------------------------
+
+/// \brief RAII scope for building an object. Destructor rolls back if not committed.
+///
+/// Usage:
+/// auto obj = builder.StartObject();
+/// obj.Insert("name", "Alice");
+/// obj.Insert("age", 30);
+/// obj.Finish(); // commits
+/// // If Finish() not called, destructor truncates buffer to pre-scope state
+class ARROW_EXPORT ObjectScope {
+ public:
+ ~ObjectScope();
+
+ ObjectScope(const ObjectScope&) = delete;
+ ObjectScope& operator=(const ObjectScope&) = delete;
+ ObjectScope(ObjectScope&& other) noexcept;
+ ObjectScope& operator=(ObjectScope&&) = delete;
+
+ /// @name Insert fields (delegates to VariantBuilder primitives)
+ /// @{
+ Status Insert(std::string_view key, std::nullptr_t);
+ Status Insert(std::string_view key, bool value);
+ Status Insert(std::string_view key, int64_t value);
+ Status Insert(std::string_view key, double value);
+ Status Insert(std::string_view key, std::string_view value);
Review Comment:
Why do we have those overloaded `Insert` fields instead of using the same approach as in VariantBuilder itself? It makes the API more confusing and limits the API from conveniently inserting a UUID for example.
I propose to change the naming to `InsertNull`, `InsertBool` `InsertInt64` etc to match the root builder (as it is called either way)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Variant encoding [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50122:
URL: https://github.com/apache/arrow/pull/50122#issuecomment-4645582737
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
Thanks for opening a pull request!
If this is not a [minor PR](https://github.com/apache/arrow/blob/main/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
See also:
* [Other pull requests](https://github.com/apache/arrow/pulls/)
* [Contribution Guidelines - Contributing Overview](https://arrow.apache.org/docs/developers/overview.html)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45947 : [C++][Parquet] Variant encoding [arrow]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #50122:
URL: https://github.com/apache/arrow/pull/50122#discussion_r3485327698
##########
cpp/src/arrow/extension/variant_internal.h:
##########
@@ -0,0 +1,479 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <string>
+#include <string_view>
+#include <unordered_map>
+#include <vector>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow::extension::variant_internal {
+
+/// \file variant_internal.h
+/// \brief Utilities for Variant binary encoding/decoding.
+///
+/// Implements parsing logic per the Variant Encoding Spec:
+/// https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
+///
+/// The "internal" in the filename refers to the binary encoding internals
+/// of the Variant type, not the visibility of this header. This header is
+/// installed and provides the public C++ API for working with Variant
+/// binary data (independent of the VariantExtensionType in parquet_variant.h).
+
+// ---------------------------------------------------------------------------
+// Constants
+// ---------------------------------------------------------------------------
+
+/// Variant encoding spec version 1.
+constexpr uint8_t kVariantVersion = 1;
+
+/// Maximum nesting depth for recursive value decoding.
+/// Prevents stack overflow on deeply nested (possibly malicious) input.
+constexpr int32_t kMaxNestingDepth = 128;
+
+// ---------------------------------------------------------------------------
+// Enumerations
+// ---------------------------------------------------------------------------
+
+/// \brief Basic type codes from bits 0-1 of the value header byte.
+///
+/// Variant Encoding Spec §3: "Value encoding"
+enum class BasicType : uint8_t {
+ kPrimitive = 0,
+ kShortString = 1,
+ kObject = 2,
+ kArray = 3,
+};
+
+/// \brief Primitive type codes from bits 2-7 when basic_type == kPrimitive.
+///
+/// Variant Encoding Spec §3.1: "Primitive types"
+enum class PrimitiveType : uint8_t {
+ kNull = 0,
+ kTrue = 1,
+ kFalse = 2,
+ kInt8 = 3,
+ kInt16 = 4,
+ kInt32 = 5,
+ kInt64 = 6,
+ kDouble = 7,
+ kDecimal4 = 8,
+ kDecimal8 = 9,
+ kDecimal16 = 10,
+ kDate = 11,
+ kTimestampMicros = 12,
+ kTimestampMicrosNTZ = 13,
+ kFloat = 14,
+ kBinary = 15,
+ kString = 16,
+ kTimeNTZ = 17,
+ kTimestampNanos = 18,
+ kTimestampNanosNTZ = 19,
+ kUUID = 20,
+};
+
+// ---------------------------------------------------------------------------
+// Metadata
+// ---------------------------------------------------------------------------
+
+/// \brief Parsed variant metadata (string dictionary).
+///
+/// The metadata buffer contains a header byte followed by a dictionary of
+/// interned strings. String views reference the raw buffer and are valid
+/// only as long as the underlying buffer is alive.
+struct ARROW_EXPORT VariantMetadata {
+ /// Spec version (must be kVariantVersion).
+ uint8_t version = 0;
+
+ /// Whether the dictionary strings are sorted lexicographically.
+ bool is_sorted = false;
+
+ /// Number of bytes used for each offset (1, 2, 3, or 4).
+ int32_t offset_size = 0;
+
+ /// Dictionary of interned strings. Views into the raw metadata buffer.
+ std::vector<std::string_view> strings;
+};
+
+/// \brief Decode a variant metadata buffer.
+///
+/// Parses the header byte and string dictionary from the raw metadata
+/// buffer. The returned VariantMetadata contains string_views that
+/// reference the input buffer directly (zero-copy).
+///
+/// \param[in] data Pointer to the metadata buffer (must not be null)
+/// \param[in] length Length of the metadata buffer in bytes
+/// \return Parsed VariantMetadata on success, Status::Invalid on
+/// malformed input
+///
+/// \note The input buffer must outlive the returned VariantMetadata.
+ARROW_EXPORT Result<VariantMetadata> DecodeMetadata(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Value header utilities
+// ---------------------------------------------------------------------------
+
+/// \brief Extract the basic type from a value header byte.
+///
+/// \param[in] header The first byte of a variant value
+/// \return The BasicType (bits 0-1)
+inline BasicType GetBasicType(uint8_t header) {
+ return static_cast<BasicType>(header & 0x03);
+}
+
+/// \brief Extract the primitive type from a value header byte.
+///
+/// Only valid when GetBasicType(header) == BasicType::kPrimitive.
+///
+/// \param[in] header The first byte of a variant value
+/// \return The PrimitiveType (bits 2-7)
+inline PrimitiveType GetPrimitiveType(uint8_t header) {
+ return static_cast<PrimitiveType>((header >> 2) & 0x3F);
+}
+
+/// \brief Get the byte size of a primitive value (excluding header).
+///
+/// \param[in] primitive_type The primitive type code
+/// \return Number of bytes for the value payload, or -1 for
+/// variable-length types (Binary, String)
+ARROW_EXPORT int32_t PrimitiveValueSize(PrimitiveType primitive_type);
+
+// ---------------------------------------------------------------------------
+// Value decoding
+// ---------------------------------------------------------------------------
+
+/// \brief Visitor interface for variant value decoding.
+///
+/// Implement this interface to receive callbacks during variant value
+/// traversal. The visitor pattern avoids materializing a tree of objects,
+/// which is important when scanning millions of rows.
+///
+/// All methods return Status::OK() to continue traversal, or any error
+/// Status to abort.
+///
+/// \note String values passed to String() and FieldName() are raw bytes from
+/// the variant buffer without UTF-8 validation. Per spec, all strings
+/// must be valid UTF-8, but validation is the responsibility of a
+/// higher-level consumer (e.g., when materializing to Arrow StringArray).
+class ARROW_EXPORT VariantVisitor {
+ public:
+ virtual ~VariantVisitor() = default;
+
+ /// @name Primitive value callbacks
+ /// @{
+ virtual Status Null() = 0;
+ virtual Status Bool(bool value) = 0;
+ virtual Status Int8(int8_t value) = 0;
+ virtual Status Int16(int16_t value) = 0;
+ virtual Status Int32(int32_t value) = 0;
+ virtual Status Int64(int64_t value) = 0;
+ virtual Status Float(float value) = 0;
+ virtual Status Double(double value) = 0;
+ virtual Status Decimal4(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal8(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal16(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Date(int32_t days_since_epoch) = 0;
+ virtual Status TimestampMicros(int64_t micros_since_epoch) = 0;
+ virtual Status TimestampMicrosNTZ(int64_t micros_since_epoch) = 0;
+ virtual Status String(std::string_view value) = 0;
+ virtual Status Binary(std::string_view value) = 0;
+ virtual Status TimeNTZ(int64_t micros_since_midnight) = 0;
+ virtual Status TimestampNanos(int64_t nanos_since_epoch) = 0;
+ virtual Status TimestampNanosNTZ(int64_t nanos_since_epoch) = 0;
+ virtual Status UUID(const uint8_t* bytes) = 0;
+ /// @}
+
+ /// @name Container callbacks
+ /// @{
+
+ /// \brief Called at the start of an object with the number of fields.
+ virtual Status StartObject(int32_t num_fields) = 0;
+
+ /// \brief Called for each object field name, before the field value.
+ virtual Status FieldName(std::string_view name) = 0;
+
+ /// \brief Called after all fields of an object have been visited.
+ virtual Status EndObject() = 0;
+
+ /// \brief Called at the start of an array with the number of elements.
+ virtual Status StartArray(int32_t num_elements) = 0;
+
+ /// \brief Called after all elements of an array have been visited.
+ virtual Status EndArray() = 0;
+ /// @}
+};
+
+/// \brief Decode a variant value buffer using a visitor.
+///
+/// Recursively traverses the variant value, calling the appropriate
+/// visitor methods for each element. Objects and arrays trigger
+/// Start/End pairs with nested visits for their contents.
+///
+/// \param[in] metadata Parsed metadata (for resolving string dictionary)
+/// \param[in] data Pointer to the value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] visitor Callback interface for decoded values
+/// \return Status::OK on success, Status::Invalid on malformed input
+///
+/// \note The data buffer must remain valid for the duration of the call.
+ARROW_EXPORT Status DecodeVariantValue(const VariantMetadata& metadata,
+ const uint8_t* data, int64_t length,
+ VariantVisitor* visitor);
+
+/// \brief Get the basic type of a variant value without full decoding.
+///
+/// \param[in] data Pointer to the value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \return The BasicType of the value, or Status::Invalid if the
+/// buffer is empty
+ARROW_EXPORT Result<BasicType> GetValueBasicType(const uint8_t* data, int64_t length);
+
+/// \brief Get the number of fields in a variant object.
+///
+/// \param[in] data Pointer to the value buffer (must start with an object)
+/// \param[in] length Length of the value buffer in bytes
+/// \return The number of fields, or Status::Invalid if not an object
+ARROW_EXPORT Result<int32_t> GetObjectFieldCount(const uint8_t* data, int64_t length);
+
+/// \brief Get the number of elements in a variant array.
+///
+/// \param[in] data Pointer to the value buffer (must start with an array)
+/// \param[in] length Length of the value buffer in bytes
+/// \return The number of elements, or Status::Invalid if not an array
+ARROW_EXPORT Result<int32_t> GetArrayElementCount(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Value size computation
+// ---------------------------------------------------------------------------
+
+/// \brief Compute the total byte size of a variant value (header + data).
+///
+/// Determines how many bytes a variant value occupies by examining
+/// its header and (for containers/variable-length types) reading
+/// size information. Does NOT recursively validate the contents.
+///
+/// \param[in] data Pointer to the start of a variant value
+/// \param[in] length Maximum bytes available
+/// \return Total byte count of the value, or Status::Invalid if truncated
+ARROW_EXPORT Result<int64_t> ValueSize(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Random access utilities
+// ---------------------------------------------------------------------------
+
+/// \brief Find an object field by name and return the offset/size of its value.
+///
+/// Searches the field IDs in the object, resolving each against the
+/// metadata dictionary. Per spec, field IDs are in lexicographic order
+/// of their corresponding key names, enabling binary search for large
+/// objects (>=32 fields). For smaller objects, linear scan is used.
+///
+/// \param[in] metadata Parsed metadata (for resolving field IDs to names)
+/// \param[in] data Pointer to the object value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] field_name The field name to search for
+/// \param[out] field_offset Set to the byte offset of the field's value
+/// within data, or -1 if not found
+/// \param[out] field_size Set to the byte size of the field's value,
+/// or 0 if not found
+/// \return Status::OK if search completed (field may or may not exist),
+/// Status::Invalid if the buffer is malformed
+ARROW_EXPORT Status FindObjectField(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length, std::string_view field_name,
+ int64_t* field_offset, int64_t* field_size);
+
+/// \brief Get the i-th element of a variant array by index (O(1) access).
+///
+/// Uses the offset table for random access without traversing preceding
+/// elements.
+///
+/// \param[in] data Pointer to the array value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] index Zero-based element index
+/// \param[out] element_offset Set to the byte offset of the element within data
+/// \param[out] element_size Set to the byte size of the element
+/// \return Status::OK on success, Status::Invalid if not an array or
+/// index is out of range
+ARROW_EXPORT Status GetArrayElement(const uint8_t* data, int64_t length, int32_t index,
+ int64_t* element_offset, int64_t* element_size);
+
+/// \brief Get the i-th field of a variant object by position.
+///
+/// Returns both the field name (resolved from metadata) and a pointer
+/// to the field's value.
+///
+/// \param[in] metadata Parsed metadata
+/// \param[in] data Pointer to the object value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] index Zero-based field index
+/// \param[out] field_name Set to the field's key name
+/// \param[out] field_offset Set to the byte offset of the field's value
+/// \param[out] field_size Set to the byte size of the field's value
+/// \return Status::OK on success, Status::Invalid if not an object or
+/// index is out of range
+ARROW_EXPORT Status GetObjectFieldAt(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length, int32_t index,
+ std::string_view* field_name, int64_t* field_offset,
+ int64_t* field_size);
+
+/// \brief Find the dictionary ID for a given key name.
+///
+/// Uses binary search if the metadata is sorted, otherwise linear scan.
+///
+/// \param[in] metadata Parsed metadata
+/// \param[in] key The key to search for
+/// \return The dictionary ID if found, or -1 if not present
+ARROW_EXPORT int32_t FindMetadataKey(const VariantMetadata& metadata,
+ std::string_view key);
+
+// ---------------------------------------------------------------------------
+// Variant Builder (Encoder)
+// ---------------------------------------------------------------------------
+
+/// \brief Builder for constructing Variant binary values.
+///
+/// Mirrors the Go implementation's Builder pattern. Values are written
+/// into an internal buffer; containers (objects/arrays) use a start-offset
+/// + finish pattern that shifts data to insert headers.
+///
+/// Usage:
+/// VariantBuilder builder;
+/// auto start = builder.Offset();
+/// std::vector<VariantBuilder::FieldEntry> fields;
+/// fields.push_back(builder.NextField(start, "name"));
+/// builder.String("Alice");
+/// fields.push_back(builder.NextField(start, "age"));
+/// builder.Int(30);
+/// builder.FinishObject(start, fields);
+/// ARROW_ASSIGN_OR_RAISE(auto result, builder.Finish());
+class ARROW_EXPORT VariantBuilder {
Review Comment:
The refactored design makes this separation explicit and deliberate:
- Views: zero-copy navigation of existing bytes (stack-allocated)
- Builder: produces new bytes from scratch (dictionary preserved across `Finish()`)
"Modify existing" = read the parts you want to keep via views, write them into a new builder with `UnsafeAppendEncoded`, add/change what you need, finish.
A higher-level mutable convenience API (think `nlohmann::json`-style) could be layered on top in a follow-up. For the first implementation, keeping the primitives clean and composable felt like the right foundation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45947 : [C++][Parquet] Variant encoding [arrow]
Posted by "misiek1984 (via GitHub)" <gi...@apache.org>.
misiek1984 commented on code in PR #50122:
URL: https://github.com/apache/arrow/pull/50122#discussion_r3441835942
##########
cpp/src/arrow/extension/variant_builder_test.cc:
##########
@@ -0,0 +1,1180 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include "arrow/extension/variant_internal.h"
+#include "arrow/extension/variant_test_util.h"
+
+#include <cstring>
+#include <limits>
+#include <string>
+#include <vector>
+
+#include "arrow/testing/gtest_util.h"
+
+namespace arrow::extension::variant_internal {
+
+// ===========================================================================
+// Helper: decode an EncodedVariant and return visitor events
+// ===========================================================================
+
+/// Encode with builder, decode, return events.
+/// Note: Uses .ValueOrDie() because ASSERT_OK_AND_ASSIGN cannot be used
+/// in a non-void function. Test-only; will crash with a descriptive message
+/// on failure rather than producing a clean test failure.
+std::vector<std::string> RoundTrip(VariantBuilder& builder) {
+ auto result = builder.Finish().ValueOrDie();
+ auto metadata =
+ DecodeMetadata(result.metadata.data(), static_cast<int64_t>(result.metadata.size()))
+ .ValueOrDie();
+ RecordingVisitor visitor;
+ DecodeVariantValue(metadata, result.value.data(),
+ static_cast<int64_t>(result.value.size()), &visitor)
+ .ok();
+ return visitor.events;
+}
+
+// ===========================================================================
+// Primitive round-trip tests
+// ===========================================================================
+
+class VariantBuilderPrimitiveTest : public ::testing::Test {};
+
+TEST_F(VariantBuilderPrimitiveTest, Null) {
+ VariantBuilder b;
+ ASSERT_OK(b.Null());
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Null");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, BoolTrue) {
+ VariantBuilder b;
+ ASSERT_OK(b.Bool(true));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Bool(true)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, BoolFalse) {
+ VariantBuilder b;
+ ASSERT_OK(b.Bool(false));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Bool(false)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, IntAutoSizesInt8) {
+ VariantBuilder b;
+ ASSERT_OK(b.Int(42));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Int8(42)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, IntAutoSizesInt16) {
+ VariantBuilder b;
+ ASSERT_OK(b.Int(300));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Int16(300)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, IntAutoSizesInt32) {
+ VariantBuilder b;
+ ASSERT_OK(b.Int(100000));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Int32(100000)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, IntAutoSizesInt64) {
+ VariantBuilder b;
+ ASSERT_OK(b.Int(5000000000LL));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Int64(5000000000)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, IntNegative) {
+ VariantBuilder b;
+ ASSERT_OK(b.Int(-42));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Int8(-42)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, ShortString) {
+ VariantBuilder b;
+ ASSERT_OK(b.String("hello"));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "String(\"hello\")");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, LongString) {
+ std::string long_str(100, 'x');
+ VariantBuilder b;
+ ASSERT_OK(b.String(long_str));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "String(\"" + long_str + "\")");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, ShortStringBoundary63) {
+ std::string str63(63, 'a');
+ VariantBuilder b;
+ ASSERT_OK(b.String(str63));
+ ASSERT_OK_AND_ASSIGN(auto result, b.Finish());
+ // Should use short string encoding: 1 byte header + 63 bytes
+ ASSERT_EQ(result.value.size(), 64);
+}
+
+TEST_F(VariantBuilderPrimitiveTest, LongStringBoundary64) {
+ std::string str64(64, 'a');
+ VariantBuilder b;
+ ASSERT_OK(b.String(str64));
+ ASSERT_OK_AND_ASSIGN(auto result, b.Finish());
+ // Should use long string encoding: 1 byte header + 4 byte length + 64 bytes
+ ASSERT_EQ(result.value.size(), 69);
+}
+
+TEST_F(VariantBuilderPrimitiveTest, Date) {
+ VariantBuilder b;
+ ASSERT_OK(b.Date(19000));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "Date(19000)");
+}
+
+TEST_F(VariantBuilderPrimitiveTest, Double) {
+ VariantBuilder b;
+ ASSERT_OK(b.Double(3.14));
+ auto events = RoundTrip(b);
+ ASSERT_TRUE(events[0].find("Double(") == 0);
+}
+
+// ===========================================================================
+// Array round-trip tests
+// ===========================================================================
+
+class VariantBuilderArrayTest : public ::testing::Test {};
+
+TEST_F(VariantBuilderArrayTest, EmptyArray) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<int64_t> offsets;
+ ASSERT_OK(b.FinishArray(start, offsets));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events.size(), 2);
+ ASSERT_EQ(events[0], "StartArray(0)");
+ ASSERT_EQ(events[1], "EndArray");
+}
+
+TEST_F(VariantBuilderArrayTest, SimpleArray) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<int64_t> offsets;
+ offsets.push_back(b.NextElement(start));
+ ASSERT_OK(b.Int(1));
+ offsets.push_back(b.NextElement(start));
+ ASSERT_OK(b.Int(2));
+ offsets.push_back(b.NextElement(start));
+ ASSERT_OK(b.Int(3));
+ ASSERT_OK(b.FinishArray(start, offsets));
+
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events.size(), 5);
+ ASSERT_EQ(events[0], "StartArray(3)");
+ ASSERT_EQ(events[1], "Int8(1)");
+ ASSERT_EQ(events[2], "Int8(2)");
+ ASSERT_EQ(events[3], "Int8(3)");
+ ASSERT_EQ(events[4], "EndArray");
+}
+
+TEST_F(VariantBuilderArrayTest, NestedArray) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<int64_t> offsets;
+
+ // First element: nested array [10, 20]
+ offsets.push_back(b.NextElement(start));
+ auto inner_start = b.Offset();
+ std::vector<int64_t> inner_offsets;
+ inner_offsets.push_back(b.NextElement(inner_start));
+ ASSERT_OK(b.Int(10));
+ inner_offsets.push_back(b.NextElement(inner_start));
+ ASSERT_OK(b.Int(20));
+ ASSERT_OK(b.FinishArray(inner_start, inner_offsets));
+
+ // Second element: 30
+ offsets.push_back(b.NextElement(start));
+ ASSERT_OK(b.Int(30));
+
+ ASSERT_OK(b.FinishArray(start, offsets));
+
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "StartArray(2)");
+ ASSERT_EQ(events[1], "StartArray(2)");
+ ASSERT_EQ(events[2], "Int8(10)");
+ ASSERT_EQ(events[3], "Int8(20)");
+ ASSERT_EQ(events[4], "EndArray");
+ ASSERT_EQ(events[5], "Int8(30)");
+ ASSERT_EQ(events[6], "EndArray");
+}
+
+// ===========================================================================
+// Object round-trip tests
+// ===========================================================================
+
+class VariantBuilderObjectTest : public ::testing::Test {};
+
+TEST_F(VariantBuilderObjectTest, EmptyObject) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<VariantBuilder::FieldEntry> fields;
+ ASSERT_OK(b.FinishObject(start, fields));
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events.size(), 2);
+ ASSERT_EQ(events[0], "StartObject(0)");
+ ASSERT_EQ(events[1], "EndObject");
+}
+
+TEST_F(VariantBuilderObjectTest, SimpleObject) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<VariantBuilder::FieldEntry> fields;
+ fields.push_back(b.NextField(start, "name"));
+ ASSERT_OK(b.String("Alice"));
+ fields.push_back(b.NextField(start, "age"));
+ ASSERT_OK(b.Int(30));
+ ASSERT_OK(b.FinishObject(start, fields));
+
+ auto events = RoundTrip(b);
+ // Fields sorted by key: "age" before "name"
+ ASSERT_EQ(events[0], "StartObject(2)");
+ ASSERT_EQ(events[1], "FieldName(\"age\")");
+ ASSERT_EQ(events[2], "Int8(30)");
+ ASSERT_EQ(events[3], "FieldName(\"name\")");
+ ASSERT_EQ(events[4], "String(\"Alice\")");
+ ASSERT_EQ(events[5], "EndObject");
+}
+
+TEST_F(VariantBuilderObjectTest, NestedObject) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<VariantBuilder::FieldEntry> fields;
+ fields.push_back(b.NextField(start, "inner"));
+ {
+ auto inner_start = b.Offset();
+ std::vector<VariantBuilder::FieldEntry> inner_fields;
+ inner_fields.push_back(b.NextField(inner_start, "key"));
+ ASSERT_OK(b.String("value"));
+ ASSERT_OK(b.FinishObject(inner_start, inner_fields));
+ }
+ ASSERT_OK(b.FinishObject(start, fields));
+
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[0], "StartObject(1)");
+ ASSERT_EQ(events[1], "FieldName(\"inner\")");
+ ASSERT_EQ(events[2], "StartObject(1)");
+ ASSERT_EQ(events[3], "FieldName(\"key\")");
+ ASSERT_EQ(events[4], "String(\"value\")");
+ ASSERT_EQ(events[5], "EndObject");
+ ASSERT_EQ(events[6], "EndObject");
+}
+
+TEST_F(VariantBuilderObjectTest, DuplicateKeyError) {
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<VariantBuilder::FieldEntry> fields;
+ fields.push_back(b.NextField(start, "key"));
+ ASSERT_OK(b.Int(1));
+ fields.push_back(b.NextField(start, "key"));
+ ASSERT_OK(b.Int(2));
+ ASSERT_RAISES(Invalid, b.FinishObject(start, fields));
+}
+
+TEST_F(VariantBuilderObjectTest, FieldsSortedByKey) {
+ // Insert fields in reverse order; verify they come out sorted
+ VariantBuilder b;
+ auto start = b.Offset();
+ std::vector<VariantBuilder::FieldEntry> fields;
+ fields.push_back(b.NextField(start, "z_last"));
+ ASSERT_OK(b.Int(3));
+ fields.push_back(b.NextField(start, "a_first"));
+ ASSERT_OK(b.Int(1));
+ fields.push_back(b.NextField(start, "m_middle"));
+ ASSERT_OK(b.Int(2));
+ ASSERT_OK(b.FinishObject(start, fields));
+
+ auto events = RoundTrip(b);
+ ASSERT_EQ(events[1], "FieldName(\"a_first\")");
+ ASSERT_EQ(events[2], "Int8(1)");
+ ASSERT_EQ(events[3], "FieldName(\"m_middle\")");
+ ASSERT_EQ(events[4], "Int8(2)");
+ ASSERT_EQ(events[5], "FieldName(\"z_last\")");
+ ASSERT_EQ(events[6], "Int8(3)");
+}
+
+// ===========================================================================
+// Builder features
+// ===========================================================================
+
+class VariantBuilderFeatureTest : public ::testing::Test {};
+
+TEST_F(VariantBuilderFeatureTest, Reset) {
+ VariantBuilder b;
+ ASSERT_OK(b.Int(42));
+ auto events1 = RoundTrip(b);
+ ASSERT_EQ(events1[0], "Int8(42)");
+
+ b.Reset();
+ ASSERT_OK(b.String("hello"));
+ auto events2 = RoundTrip(b);
+ ASSERT_EQ(events2[0], "String(\"hello\")");
+}
+
+TEST_F(VariantBuilderFeatureTest, BuilderFromExistingMetadata) {
+ // First, build a variant to get metadata
+ VariantBuilder b1;
+ auto start = b1.Offset();
+ std::vector<VariantBuilder::FieldEntry> fields;
+ fields.push_back(b1.NextField(start, "name"));
+ ASSERT_OK(b1.String("Alice"));
+ ASSERT_OK(b1.FinishObject(start, fields));
+ ASSERT_OK_AND_ASSIGN(auto encoded1, b1.Finish());
+
+ // Decode the metadata
+ ASSERT_OK_AND_ASSIGN(auto meta,
+ DecodeMetadata(encoded1.metadata.data(),
+ static_cast<int64_t>(encoded1.metadata.size())));
+
+ // Build a new variant reusing the same metadata
Review Comment:
I would add a test where we reuse existing metadata but make a mistake with the data types. For example, according to the metadata, we should write a string but we write an integer instead. I think there is currently no validation for this case in VariantBuilder—is that on purpose?
Either way, the final Variant will be malformed, so the round-trip should fail.
##########
cpp/src/arrow/extension/variant_internal.h:
##########
@@ -0,0 +1,479 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <string>
+#include <string_view>
+#include <unordered_map>
+#include <vector>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow::extension::variant_internal {
+
+/// \file variant_internal.h
+/// \brief Utilities for Variant binary encoding/decoding.
+///
+/// Implements parsing logic per the Variant Encoding Spec:
+/// https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
+///
+/// The "internal" in the filename refers to the binary encoding internals
+/// of the Variant type, not the visibility of this header. This header is
+/// installed and provides the public C++ API for working with Variant
+/// binary data (independent of the VariantExtensionType in parquet_variant.h).
+
+// ---------------------------------------------------------------------------
+// Constants
+// ---------------------------------------------------------------------------
+
+/// Variant encoding spec version 1.
+constexpr uint8_t kVariantVersion = 1;
+
+/// Maximum nesting depth for recursive value decoding.
+/// Prevents stack overflow on deeply nested (possibly malicious) input.
+constexpr int32_t kMaxNestingDepth = 128;
+
+// ---------------------------------------------------------------------------
+// Enumerations
+// ---------------------------------------------------------------------------
+
+/// \brief Basic type codes from bits 0-1 of the value header byte.
+///
+/// Variant Encoding Spec §3: "Value encoding"
+enum class BasicType : uint8_t {
+ kPrimitive = 0,
+ kShortString = 1,
+ kObject = 2,
+ kArray = 3,
+};
+
+/// \brief Primitive type codes from bits 2-7 when basic_type == kPrimitive.
+///
+/// Variant Encoding Spec §3.1: "Primitive types"
+enum class PrimitiveType : uint8_t {
+ kNull = 0,
+ kTrue = 1,
+ kFalse = 2,
+ kInt8 = 3,
+ kInt16 = 4,
+ kInt32 = 5,
+ kInt64 = 6,
+ kDouble = 7,
+ kDecimal4 = 8,
+ kDecimal8 = 9,
+ kDecimal16 = 10,
+ kDate = 11,
+ kTimestampMicros = 12,
+ kTimestampMicrosNTZ = 13,
+ kFloat = 14,
+ kBinary = 15,
+ kString = 16,
+ kTimeNTZ = 17,
+ kTimestampNanos = 18,
+ kTimestampNanosNTZ = 19,
+ kUUID = 20,
+};
+
+// ---------------------------------------------------------------------------
+// Metadata
+// ---------------------------------------------------------------------------
+
+/// \brief Parsed variant metadata (string dictionary).
+///
+/// The metadata buffer contains a header byte followed by a dictionary of
+/// interned strings. String views reference the raw buffer and are valid
+/// only as long as the underlying buffer is alive.
+struct ARROW_EXPORT VariantMetadata {
+ /// Spec version (must be kVariantVersion).
+ uint8_t version = 0;
+
+ /// Whether the dictionary strings are sorted lexicographically.
+ bool is_sorted = false;
+
+ /// Number of bytes used for each offset (1, 2, 3, or 4).
+ int32_t offset_size = 0;
+
+ /// Dictionary of interned strings. Views into the raw metadata buffer.
+ std::vector<std::string_view> strings;
+};
+
+/// \brief Decode a variant metadata buffer.
+///
+/// Parses the header byte and string dictionary from the raw metadata
+/// buffer. The returned VariantMetadata contains string_views that
+/// reference the input buffer directly (zero-copy).
+///
+/// \param[in] data Pointer to the metadata buffer (must not be null)
+/// \param[in] length Length of the metadata buffer in bytes
+/// \return Parsed VariantMetadata on success, Status::Invalid on
+/// malformed input
+///
+/// \note The input buffer must outlive the returned VariantMetadata.
+ARROW_EXPORT Result<VariantMetadata> DecodeMetadata(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Value header utilities
+// ---------------------------------------------------------------------------
+
+/// \brief Extract the basic type from a value header byte.
+///
+/// \param[in] header The first byte of a variant value
+/// \return The BasicType (bits 0-1)
+inline BasicType GetBasicType(uint8_t header) {
+ return static_cast<BasicType>(header & 0x03);
+}
+
+/// \brief Extract the primitive type from a value header byte.
+///
+/// Only valid when GetBasicType(header) == BasicType::kPrimitive.
+///
+/// \param[in] header The first byte of a variant value
+/// \return The PrimitiveType (bits 2-7)
+inline PrimitiveType GetPrimitiveType(uint8_t header) {
+ return static_cast<PrimitiveType>((header >> 2) & 0x3F);
+}
+
+/// \brief Get the byte size of a primitive value (excluding header).
+///
+/// \param[in] primitive_type The primitive type code
+/// \return Number of bytes for the value payload, or -1 for
+/// variable-length types (Binary, String)
+ARROW_EXPORT int32_t PrimitiveValueSize(PrimitiveType primitive_type);
+
+// ---------------------------------------------------------------------------
+// Value decoding
+// ---------------------------------------------------------------------------
+
+/// \brief Visitor interface for variant value decoding.
+///
+/// Implement this interface to receive callbacks during variant value
+/// traversal. The visitor pattern avoids materializing a tree of objects,
+/// which is important when scanning millions of rows.
+///
+/// All methods return Status::OK() to continue traversal, or any error
+/// Status to abort.
+///
+/// \note String values passed to String() and FieldName() are raw bytes from
+/// the variant buffer without UTF-8 validation. Per spec, all strings
+/// must be valid UTF-8, but validation is the responsibility of a
+/// higher-level consumer (e.g., when materializing to Arrow StringArray).
+class ARROW_EXPORT VariantVisitor {
+ public:
+ virtual ~VariantVisitor() = default;
+
+ /// @name Primitive value callbacks
+ /// @{
+ virtual Status Null() = 0;
+ virtual Status Bool(bool value) = 0;
+ virtual Status Int8(int8_t value) = 0;
+ virtual Status Int16(int16_t value) = 0;
+ virtual Status Int32(int32_t value) = 0;
+ virtual Status Int64(int64_t value) = 0;
+ virtual Status Float(float value) = 0;
+ virtual Status Double(double value) = 0;
+ virtual Status Decimal4(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal8(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal16(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Date(int32_t days_since_epoch) = 0;
+ virtual Status TimestampMicros(int64_t micros_since_epoch) = 0;
+ virtual Status TimestampMicrosNTZ(int64_t micros_since_epoch) = 0;
+ virtual Status String(std::string_view value) = 0;
+ virtual Status Binary(std::string_view value) = 0;
+ virtual Status TimeNTZ(int64_t micros_since_midnight) = 0;
+ virtual Status TimestampNanos(int64_t nanos_since_epoch) = 0;
+ virtual Status TimestampNanosNTZ(int64_t nanos_since_epoch) = 0;
+ virtual Status UUID(const uint8_t* bytes) = 0;
+ /// @}
+
+ /// @name Container callbacks
+ /// @{
+
+ /// \brief Called at the start of an object with the number of fields.
+ virtual Status StartObject(int32_t num_fields) = 0;
+
+ /// \brief Called for each object field name, before the field value.
+ virtual Status FieldName(std::string_view name) = 0;
+
+ /// \brief Called after all fields of an object have been visited.
+ virtual Status EndObject() = 0;
+
+ /// \brief Called at the start of an array with the number of elements.
+ virtual Status StartArray(int32_t num_elements) = 0;
+
+ /// \brief Called after all elements of an array have been visited.
+ virtual Status EndArray() = 0;
+ /// @}
+};
+
+/// \brief Decode a variant value buffer using a visitor.
+///
+/// Recursively traverses the variant value, calling the appropriate
+/// visitor methods for each element. Objects and arrays trigger
+/// Start/End pairs with nested visits for their contents.
+///
+/// \param[in] metadata Parsed metadata (for resolving string dictionary)
+/// \param[in] data Pointer to the value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] visitor Callback interface for decoded values
+/// \return Status::OK on success, Status::Invalid on malformed input
+///
+/// \note The data buffer must remain valid for the duration of the call.
+ARROW_EXPORT Status DecodeVariantValue(const VariantMetadata& metadata,
+ const uint8_t* data, int64_t length,
+ VariantVisitor* visitor);
+
+/// \brief Get the basic type of a variant value without full decoding.
+///
+/// \param[in] data Pointer to the value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \return The BasicType of the value, or Status::Invalid if the
+/// buffer is empty
+ARROW_EXPORT Result<BasicType> GetValueBasicType(const uint8_t* data, int64_t length);
+
+/// \brief Get the number of fields in a variant object.
+///
+/// \param[in] data Pointer to the value buffer (must start with an object)
+/// \param[in] length Length of the value buffer in bytes
+/// \return The number of fields, or Status::Invalid if not an object
+ARROW_EXPORT Result<int32_t> GetObjectFieldCount(const uint8_t* data, int64_t length);
+
+/// \brief Get the number of elements in a variant array.
+///
+/// \param[in] data Pointer to the value buffer (must start with an array)
+/// \param[in] length Length of the value buffer in bytes
+/// \return The number of elements, or Status::Invalid if not an array
+ARROW_EXPORT Result<int32_t> GetArrayElementCount(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Value size computation
+// ---------------------------------------------------------------------------
+
+/// \brief Compute the total byte size of a variant value (header + data).
+///
+/// Determines how many bytes a variant value occupies by examining
+/// its header and (for containers/variable-length types) reading
+/// size information. Does NOT recursively validate the contents.
+///
+/// \param[in] data Pointer to the start of a variant value
+/// \param[in] length Maximum bytes available
+/// \return Total byte count of the value, or Status::Invalid if truncated
+ARROW_EXPORT Result<int64_t> ValueSize(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Random access utilities
+// ---------------------------------------------------------------------------
+
+/// \brief Find an object field by name and return the offset/size of its value.
+///
+/// Searches the field IDs in the object, resolving each against the
+/// metadata dictionary. Per spec, field IDs are in lexicographic order
+/// of their corresponding key names, enabling binary search for large
+/// objects (>=32 fields). For smaller objects, linear scan is used.
+///
+/// \param[in] metadata Parsed metadata (for resolving field IDs to names)
+/// \param[in] data Pointer to the object value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] field_name The field name to search for
+/// \param[out] field_offset Set to the byte offset of the field's value
+/// within data, or -1 if not found
+/// \param[out] field_size Set to the byte size of the field's value,
+/// or 0 if not found
+/// \return Status::OK if search completed (field may or may not exist),
+/// Status::Invalid if the buffer is malformed
+ARROW_EXPORT Status FindObjectField(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length, std::string_view field_name,
+ int64_t* field_offset, int64_t* field_size);
+
+/// \brief Get the i-th element of a variant array by index (O(1) access).
+///
+/// Uses the offset table for random access without traversing preceding
+/// elements.
+///
+/// \param[in] data Pointer to the array value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] index Zero-based element index
+/// \param[out] element_offset Set to the byte offset of the element within data
+/// \param[out] element_size Set to the byte size of the element
+/// \return Status::OK on success, Status::Invalid if not an array or
+/// index is out of range
+ARROW_EXPORT Status GetArrayElement(const uint8_t* data, int64_t length, int32_t index,
+ int64_t* element_offset, int64_t* element_size);
+
+/// \brief Get the i-th field of a variant object by position.
+///
+/// Returns both the field name (resolved from metadata) and a pointer
+/// to the field's value.
+///
+/// \param[in] metadata Parsed metadata
+/// \param[in] data Pointer to the object value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] index Zero-based field index
+/// \param[out] field_name Set to the field's key name
+/// \param[out] field_offset Set to the byte offset of the field's value
+/// \param[out] field_size Set to the byte size of the field's value
+/// \return Status::OK on success, Status::Invalid if not an object or
+/// index is out of range
+ARROW_EXPORT Status GetObjectFieldAt(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length, int32_t index,
+ std::string_view* field_name, int64_t* field_offset,
+ int64_t* field_size);
+
+/// \brief Find the dictionary ID for a given key name.
+///
+/// Uses binary search if the metadata is sorted, otherwise linear scan.
+///
+/// \param[in] metadata Parsed metadata
+/// \param[in] key The key to search for
+/// \return The dictionary ID if found, or -1 if not present
+ARROW_EXPORT int32_t FindMetadataKey(const VariantMetadata& metadata,
+ std::string_view key);
+
+// ---------------------------------------------------------------------------
+// Variant Builder (Encoder)
+// ---------------------------------------------------------------------------
+
+/// \brief Builder for constructing Variant binary values.
+///
+/// Mirrors the Go implementation's Builder pattern. Values are written
+/// into an internal buffer; containers (objects/arrays) use a start-offset
+/// + finish pattern that shifts data to insert headers.
+///
+/// Usage:
+/// VariantBuilder builder;
+/// auto start = builder.Offset();
+/// std::vector<VariantBuilder::FieldEntry> fields;
+/// fields.push_back(builder.NextField(start, "name"));
+/// builder.String("Alice");
+/// fields.push_back(builder.NextField(start, "age"));
+/// builder.Int(30);
+/// builder.FinishObject(start, fields);
+/// ARROW_ASSIGN_OR_RAISE(auto result, builder.Finish());
+class ARROW_EXPORT VariantBuilder {
+ public:
+ VariantBuilder();
+ explicit VariantBuilder(const VariantMetadata& existing_metadata);
Review Comment:
It might also be useful to pass a value buffer to `VariantBuilder` to initialize `buffer_`. This way, it will be possible to continue building an existing Variant value.
##########
cpp/src/arrow/extension/variant_internal.h:
##########
@@ -0,0 +1,479 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <string>
+#include <string_view>
+#include <unordered_map>
+#include <vector>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow::extension::variant_internal {
+
+/// \file variant_internal.h
+/// \brief Utilities for Variant binary encoding/decoding.
+///
+/// Implements parsing logic per the Variant Encoding Spec:
+/// https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
+///
+/// The "internal" in the filename refers to the binary encoding internals
+/// of the Variant type, not the visibility of this header. This header is
+/// installed and provides the public C++ API for working with Variant
+/// binary data (independent of the VariantExtensionType in parquet_variant.h).
+
+// ---------------------------------------------------------------------------
+// Constants
+// ---------------------------------------------------------------------------
+
+/// Variant encoding spec version 1.
+constexpr uint8_t kVariantVersion = 1;
+
+/// Maximum nesting depth for recursive value decoding.
+/// Prevents stack overflow on deeply nested (possibly malicious) input.
+constexpr int32_t kMaxNestingDepth = 128;
+
+// ---------------------------------------------------------------------------
+// Enumerations
+// ---------------------------------------------------------------------------
+
+/// \brief Basic type codes from bits 0-1 of the value header byte.
+///
+/// Variant Encoding Spec §3: "Value encoding"
+enum class BasicType : uint8_t {
+ kPrimitive = 0,
+ kShortString = 1,
+ kObject = 2,
+ kArray = 3,
+};
+
+/// \brief Primitive type codes from bits 2-7 when basic_type == kPrimitive.
+///
+/// Variant Encoding Spec §3.1: "Primitive types"
+enum class PrimitiveType : uint8_t {
+ kNull = 0,
+ kTrue = 1,
+ kFalse = 2,
+ kInt8 = 3,
+ kInt16 = 4,
+ kInt32 = 5,
+ kInt64 = 6,
+ kDouble = 7,
+ kDecimal4 = 8,
+ kDecimal8 = 9,
+ kDecimal16 = 10,
+ kDate = 11,
+ kTimestampMicros = 12,
+ kTimestampMicrosNTZ = 13,
+ kFloat = 14,
+ kBinary = 15,
+ kString = 16,
+ kTimeNTZ = 17,
+ kTimestampNanos = 18,
+ kTimestampNanosNTZ = 19,
+ kUUID = 20,
+};
+
+// ---------------------------------------------------------------------------
+// Metadata
+// ---------------------------------------------------------------------------
+
+/// \brief Parsed variant metadata (string dictionary).
+///
+/// The metadata buffer contains a header byte followed by a dictionary of
+/// interned strings. String views reference the raw buffer and are valid
+/// only as long as the underlying buffer is alive.
+struct ARROW_EXPORT VariantMetadata {
+ /// Spec version (must be kVariantVersion).
+ uint8_t version = 0;
+
+ /// Whether the dictionary strings are sorted lexicographically.
+ bool is_sorted = false;
+
+ /// Number of bytes used for each offset (1, 2, 3, or 4).
+ int32_t offset_size = 0;
+
+ /// Dictionary of interned strings. Views into the raw metadata buffer.
+ std::vector<std::string_view> strings;
+};
+
+/// \brief Decode a variant metadata buffer.
+///
+/// Parses the header byte and string dictionary from the raw metadata
+/// buffer. The returned VariantMetadata contains string_views that
+/// reference the input buffer directly (zero-copy).
+///
+/// \param[in] data Pointer to the metadata buffer (must not be null)
+/// \param[in] length Length of the metadata buffer in bytes
+/// \return Parsed VariantMetadata on success, Status::Invalid on
+/// malformed input
+///
+/// \note The input buffer must outlive the returned VariantMetadata.
+ARROW_EXPORT Result<VariantMetadata> DecodeMetadata(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Value header utilities
+// ---------------------------------------------------------------------------
+
+/// \brief Extract the basic type from a value header byte.
+///
+/// \param[in] header The first byte of a variant value
+/// \return The BasicType (bits 0-1)
+inline BasicType GetBasicType(uint8_t header) {
+ return static_cast<BasicType>(header & 0x03);
+}
+
+/// \brief Extract the primitive type from a value header byte.
+///
+/// Only valid when GetBasicType(header) == BasicType::kPrimitive.
+///
+/// \param[in] header The first byte of a variant value
+/// \return The PrimitiveType (bits 2-7)
+inline PrimitiveType GetPrimitiveType(uint8_t header) {
+ return static_cast<PrimitiveType>((header >> 2) & 0x3F);
+}
+
+/// \brief Get the byte size of a primitive value (excluding header).
+///
+/// \param[in] primitive_type The primitive type code
+/// \return Number of bytes for the value payload, or -1 for
+/// variable-length types (Binary, String)
+ARROW_EXPORT int32_t PrimitiveValueSize(PrimitiveType primitive_type);
+
+// ---------------------------------------------------------------------------
+// Value decoding
+// ---------------------------------------------------------------------------
+
+/// \brief Visitor interface for variant value decoding.
+///
+/// Implement this interface to receive callbacks during variant value
+/// traversal. The visitor pattern avoids materializing a tree of objects,
+/// which is important when scanning millions of rows.
+///
+/// All methods return Status::OK() to continue traversal, or any error
+/// Status to abort.
+///
+/// \note String values passed to String() and FieldName() are raw bytes from
+/// the variant buffer without UTF-8 validation. Per spec, all strings
+/// must be valid UTF-8, but validation is the responsibility of a
+/// higher-level consumer (e.g., when materializing to Arrow StringArray).
+class ARROW_EXPORT VariantVisitor {
+ public:
+ virtual ~VariantVisitor() = default;
+
+ /// @name Primitive value callbacks
+ /// @{
+ virtual Status Null() = 0;
+ virtual Status Bool(bool value) = 0;
+ virtual Status Int8(int8_t value) = 0;
+ virtual Status Int16(int16_t value) = 0;
+ virtual Status Int32(int32_t value) = 0;
+ virtual Status Int64(int64_t value) = 0;
+ virtual Status Float(float value) = 0;
+ virtual Status Double(double value) = 0;
+ virtual Status Decimal4(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal8(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal16(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Date(int32_t days_since_epoch) = 0;
+ virtual Status TimestampMicros(int64_t micros_since_epoch) = 0;
+ virtual Status TimestampMicrosNTZ(int64_t micros_since_epoch) = 0;
+ virtual Status String(std::string_view value) = 0;
+ virtual Status Binary(std::string_view value) = 0;
+ virtual Status TimeNTZ(int64_t micros_since_midnight) = 0;
+ virtual Status TimestampNanos(int64_t nanos_since_epoch) = 0;
+ virtual Status TimestampNanosNTZ(int64_t nanos_since_epoch) = 0;
+ virtual Status UUID(const uint8_t* bytes) = 0;
+ /// @}
+
+ /// @name Container callbacks
+ /// @{
+
+ /// \brief Called at the start of an object with the number of fields.
+ virtual Status StartObject(int32_t num_fields) = 0;
+
+ /// \brief Called for each object field name, before the field value.
+ virtual Status FieldName(std::string_view name) = 0;
+
+ /// \brief Called after all fields of an object have been visited.
+ virtual Status EndObject() = 0;
+
+ /// \brief Called at the start of an array with the number of elements.
+ virtual Status StartArray(int32_t num_elements) = 0;
+
+ /// \brief Called after all elements of an array have been visited.
+ virtual Status EndArray() = 0;
+ /// @}
+};
+
+/// \brief Decode a variant value buffer using a visitor.
+///
+/// Recursively traverses the variant value, calling the appropriate
+/// visitor methods for each element. Objects and arrays trigger
+/// Start/End pairs with nested visits for their contents.
+///
+/// \param[in] metadata Parsed metadata (for resolving string dictionary)
+/// \param[in] data Pointer to the value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] visitor Callback interface for decoded values
+/// \return Status::OK on success, Status::Invalid on malformed input
+///
+/// \note The data buffer must remain valid for the duration of the call.
+ARROW_EXPORT Status DecodeVariantValue(const VariantMetadata& metadata,
+ const uint8_t* data, int64_t length,
+ VariantVisitor* visitor);
+
+/// \brief Get the basic type of a variant value without full decoding.
+///
+/// \param[in] data Pointer to the value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \return The BasicType of the value, or Status::Invalid if the
+/// buffer is empty
+ARROW_EXPORT Result<BasicType> GetValueBasicType(const uint8_t* data, int64_t length);
+
+/// \brief Get the number of fields in a variant object.
+///
+/// \param[in] data Pointer to the value buffer (must start with an object)
+/// \param[in] length Length of the value buffer in bytes
+/// \return The number of fields, or Status::Invalid if not an object
+ARROW_EXPORT Result<int32_t> GetObjectFieldCount(const uint8_t* data, int64_t length);
+
+/// \brief Get the number of elements in a variant array.
+///
+/// \param[in] data Pointer to the value buffer (must start with an array)
+/// \param[in] length Length of the value buffer in bytes
+/// \return The number of elements, or Status::Invalid if not an array
+ARROW_EXPORT Result<int32_t> GetArrayElementCount(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Value size computation
+// ---------------------------------------------------------------------------
+
+/// \brief Compute the total byte size of a variant value (header + data).
+///
+/// Determines how many bytes a variant value occupies by examining
+/// its header and (for containers/variable-length types) reading
+/// size information. Does NOT recursively validate the contents.
+///
+/// \param[in] data Pointer to the start of a variant value
+/// \param[in] length Maximum bytes available
+/// \return Total byte count of the value, or Status::Invalid if truncated
+ARROW_EXPORT Result<int64_t> ValueSize(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Random access utilities
+// ---------------------------------------------------------------------------
+
+/// \brief Find an object field by name and return the offset/size of its value.
+///
+/// Searches the field IDs in the object, resolving each against the
+/// metadata dictionary. Per spec, field IDs are in lexicographic order
+/// of their corresponding key names, enabling binary search for large
+/// objects (>=32 fields). For smaller objects, linear scan is used.
+///
+/// \param[in] metadata Parsed metadata (for resolving field IDs to names)
+/// \param[in] data Pointer to the object value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] field_name The field name to search for
+/// \param[out] field_offset Set to the byte offset of the field's value
+/// within data, or -1 if not found
+/// \param[out] field_size Set to the byte size of the field's value,
+/// or 0 if not found
+/// \return Status::OK if search completed (field may or may not exist),
+/// Status::Invalid if the buffer is malformed
+ARROW_EXPORT Status FindObjectField(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length, std::string_view field_name,
+ int64_t* field_offset, int64_t* field_size);
+
+/// \brief Get the i-th element of a variant array by index (O(1) access).
+///
+/// Uses the offset table for random access without traversing preceding
+/// elements.
+///
+/// \param[in] data Pointer to the array value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] index Zero-based element index
+/// \param[out] element_offset Set to the byte offset of the element within data
+/// \param[out] element_size Set to the byte size of the element
+/// \return Status::OK on success, Status::Invalid if not an array or
+/// index is out of range
+ARROW_EXPORT Status GetArrayElement(const uint8_t* data, int64_t length, int32_t index,
+ int64_t* element_offset, int64_t* element_size);
+
+/// \brief Get the i-th field of a variant object by position.
+///
+/// Returns both the field name (resolved from metadata) and a pointer
+/// to the field's value.
+///
+/// \param[in] metadata Parsed metadata
+/// \param[in] data Pointer to the object value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] index Zero-based field index
+/// \param[out] field_name Set to the field's key name
+/// \param[out] field_offset Set to the byte offset of the field's value
+/// \param[out] field_size Set to the byte size of the field's value
+/// \return Status::OK on success, Status::Invalid if not an object or
+/// index is out of range
+ARROW_EXPORT Status GetObjectFieldAt(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length, int32_t index,
+ std::string_view* field_name, int64_t* field_offset,
+ int64_t* field_size);
+
+/// \brief Find the dictionary ID for a given key name.
+///
+/// Uses binary search if the metadata is sorted, otherwise linear scan.
+///
+/// \param[in] metadata Parsed metadata
+/// \param[in] key The key to search for
+/// \return The dictionary ID if found, or -1 if not present
+ARROW_EXPORT int32_t FindMetadataKey(const VariantMetadata& metadata,
+ std::string_view key);
+
+// ---------------------------------------------------------------------------
+// Variant Builder (Encoder)
+// ---------------------------------------------------------------------------
+
+/// \brief Builder for constructing Variant binary values.
+///
+/// Mirrors the Go implementation's Builder pattern. Values are written
+/// into an internal buffer; containers (objects/arrays) use a start-offset
+/// + finish pattern that shifts data to insert headers.
+///
+/// Usage:
+/// VariantBuilder builder;
+/// auto start = builder.Offset();
+/// std::vector<VariantBuilder::FieldEntry> fields;
+/// fields.push_back(builder.NextField(start, "name"));
+/// builder.String("Alice");
+/// fields.push_back(builder.NextField(start, "age"));
+/// builder.Int(30);
+/// builder.FinishObject(start, fields);
+/// ARROW_ASSIGN_OR_RAISE(auto result, builder.Finish());
+class ARROW_EXPORT VariantBuilder {
Review Comment:
This API is great for building new variants. Did you also consider adding an API that allows modifying existing Variant values? We would need to add a function to `VariantBuilder` similar to `FindObjectField` from the decoding PR, which would move the context of `VariantBuilder` to a specific place. Once called, you would then be able to override the existing value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45947 : [C++][Parquet] Variant encoding [arrow]
Posted by "misiek1984 (via GitHub)" <gi...@apache.org>.
misiek1984 commented on code in PR #50122:
URL: https://github.com/apache/arrow/pull/50122#discussion_r3441878179
##########
cpp/src/arrow/extension/variant_internal.h:
##########
@@ -0,0 +1,479 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <string>
+#include <string_view>
+#include <unordered_map>
+#include <vector>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow::extension::variant_internal {
+
+/// \file variant_internal.h
+/// \brief Utilities for Variant binary encoding/decoding.
+///
+/// Implements parsing logic per the Variant Encoding Spec:
+/// https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
+///
+/// The "internal" in the filename refers to the binary encoding internals
+/// of the Variant type, not the visibility of this header. This header is
+/// installed and provides the public C++ API for working with Variant
+/// binary data (independent of the VariantExtensionType in parquet_variant.h).
+
+// ---------------------------------------------------------------------------
+// Constants
+// ---------------------------------------------------------------------------
+
+/// Variant encoding spec version 1.
+constexpr uint8_t kVariantVersion = 1;
+
+/// Maximum nesting depth for recursive value decoding.
+/// Prevents stack overflow on deeply nested (possibly malicious) input.
+constexpr int32_t kMaxNestingDepth = 128;
+
+// ---------------------------------------------------------------------------
+// Enumerations
+// ---------------------------------------------------------------------------
+
+/// \brief Basic type codes from bits 0-1 of the value header byte.
+///
+/// Variant Encoding Spec §3: "Value encoding"
+enum class BasicType : uint8_t {
+ kPrimitive = 0,
+ kShortString = 1,
+ kObject = 2,
+ kArray = 3,
+};
+
+/// \brief Primitive type codes from bits 2-7 when basic_type == kPrimitive.
+///
+/// Variant Encoding Spec §3.1: "Primitive types"
+enum class PrimitiveType : uint8_t {
+ kNull = 0,
+ kTrue = 1,
+ kFalse = 2,
+ kInt8 = 3,
+ kInt16 = 4,
+ kInt32 = 5,
+ kInt64 = 6,
+ kDouble = 7,
+ kDecimal4 = 8,
+ kDecimal8 = 9,
+ kDecimal16 = 10,
+ kDate = 11,
+ kTimestampMicros = 12,
+ kTimestampMicrosNTZ = 13,
+ kFloat = 14,
+ kBinary = 15,
+ kString = 16,
+ kTimeNTZ = 17,
+ kTimestampNanos = 18,
+ kTimestampNanosNTZ = 19,
+ kUUID = 20,
+};
+
+// ---------------------------------------------------------------------------
+// Metadata
+// ---------------------------------------------------------------------------
+
+/// \brief Parsed variant metadata (string dictionary).
+///
+/// The metadata buffer contains a header byte followed by a dictionary of
+/// interned strings. String views reference the raw buffer and are valid
+/// only as long as the underlying buffer is alive.
+struct ARROW_EXPORT VariantMetadata {
+ /// Spec version (must be kVariantVersion).
+ uint8_t version = 0;
+
+ /// Whether the dictionary strings are sorted lexicographically.
+ bool is_sorted = false;
+
+ /// Number of bytes used for each offset (1, 2, 3, or 4).
+ int32_t offset_size = 0;
+
+ /// Dictionary of interned strings. Views into the raw metadata buffer.
+ std::vector<std::string_view> strings;
+};
+
+/// \brief Decode a variant metadata buffer.
+///
+/// Parses the header byte and string dictionary from the raw metadata
+/// buffer. The returned VariantMetadata contains string_views that
+/// reference the input buffer directly (zero-copy).
+///
+/// \param[in] data Pointer to the metadata buffer (must not be null)
+/// \param[in] length Length of the metadata buffer in bytes
+/// \return Parsed VariantMetadata on success, Status::Invalid on
+/// malformed input
+///
+/// \note The input buffer must outlive the returned VariantMetadata.
+ARROW_EXPORT Result<VariantMetadata> DecodeMetadata(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Value header utilities
+// ---------------------------------------------------------------------------
+
+/// \brief Extract the basic type from a value header byte.
+///
+/// \param[in] header The first byte of a variant value
+/// \return The BasicType (bits 0-1)
+inline BasicType GetBasicType(uint8_t header) {
+ return static_cast<BasicType>(header & 0x03);
+}
+
+/// \brief Extract the primitive type from a value header byte.
+///
+/// Only valid when GetBasicType(header) == BasicType::kPrimitive.
+///
+/// \param[in] header The first byte of a variant value
+/// \return The PrimitiveType (bits 2-7)
+inline PrimitiveType GetPrimitiveType(uint8_t header) {
+ return static_cast<PrimitiveType>((header >> 2) & 0x3F);
+}
+
+/// \brief Get the byte size of a primitive value (excluding header).
+///
+/// \param[in] primitive_type The primitive type code
+/// \return Number of bytes for the value payload, or -1 for
+/// variable-length types (Binary, String)
+ARROW_EXPORT int32_t PrimitiveValueSize(PrimitiveType primitive_type);
+
+// ---------------------------------------------------------------------------
+// Value decoding
+// ---------------------------------------------------------------------------
+
+/// \brief Visitor interface for variant value decoding.
+///
+/// Implement this interface to receive callbacks during variant value
+/// traversal. The visitor pattern avoids materializing a tree of objects,
+/// which is important when scanning millions of rows.
+///
+/// All methods return Status::OK() to continue traversal, or any error
+/// Status to abort.
+///
+/// \note String values passed to String() and FieldName() are raw bytes from
+/// the variant buffer without UTF-8 validation. Per spec, all strings
+/// must be valid UTF-8, but validation is the responsibility of a
+/// higher-level consumer (e.g., when materializing to Arrow StringArray).
+class ARROW_EXPORT VariantVisitor {
+ public:
+ virtual ~VariantVisitor() = default;
+
+ /// @name Primitive value callbacks
+ /// @{
+ virtual Status Null() = 0;
+ virtual Status Bool(bool value) = 0;
+ virtual Status Int8(int8_t value) = 0;
+ virtual Status Int16(int16_t value) = 0;
+ virtual Status Int32(int32_t value) = 0;
+ virtual Status Int64(int64_t value) = 0;
+ virtual Status Float(float value) = 0;
+ virtual Status Double(double value) = 0;
+ virtual Status Decimal4(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal8(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal16(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Date(int32_t days_since_epoch) = 0;
+ virtual Status TimestampMicros(int64_t micros_since_epoch) = 0;
+ virtual Status TimestampMicrosNTZ(int64_t micros_since_epoch) = 0;
+ virtual Status String(std::string_view value) = 0;
+ virtual Status Binary(std::string_view value) = 0;
+ virtual Status TimeNTZ(int64_t micros_since_midnight) = 0;
+ virtual Status TimestampNanos(int64_t nanos_since_epoch) = 0;
+ virtual Status TimestampNanosNTZ(int64_t nanos_since_epoch) = 0;
+ virtual Status UUID(const uint8_t* bytes) = 0;
+ /// @}
+
+ /// @name Container callbacks
+ /// @{
+
+ /// \brief Called at the start of an object with the number of fields.
+ virtual Status StartObject(int32_t num_fields) = 0;
+
+ /// \brief Called for each object field name, before the field value.
+ virtual Status FieldName(std::string_view name) = 0;
+
+ /// \brief Called after all fields of an object have been visited.
+ virtual Status EndObject() = 0;
+
+ /// \brief Called at the start of an array with the number of elements.
+ virtual Status StartArray(int32_t num_elements) = 0;
+
+ /// \brief Called after all elements of an array have been visited.
+ virtual Status EndArray() = 0;
+ /// @}
+};
+
+/// \brief Decode a variant value buffer using a visitor.
+///
+/// Recursively traverses the variant value, calling the appropriate
+/// visitor methods for each element. Objects and arrays trigger
+/// Start/End pairs with nested visits for their contents.
+///
+/// \param[in] metadata Parsed metadata (for resolving string dictionary)
+/// \param[in] data Pointer to the value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] visitor Callback interface for decoded values
+/// \return Status::OK on success, Status::Invalid on malformed input
+///
+/// \note The data buffer must remain valid for the duration of the call.
+ARROW_EXPORT Status DecodeVariantValue(const VariantMetadata& metadata,
+ const uint8_t* data, int64_t length,
+ VariantVisitor* visitor);
+
+/// \brief Get the basic type of a variant value without full decoding.
+///
+/// \param[in] data Pointer to the value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \return The BasicType of the value, or Status::Invalid if the
+/// buffer is empty
+ARROW_EXPORT Result<BasicType> GetValueBasicType(const uint8_t* data, int64_t length);
+
+/// \brief Get the number of fields in a variant object.
+///
+/// \param[in] data Pointer to the value buffer (must start with an object)
+/// \param[in] length Length of the value buffer in bytes
+/// \return The number of fields, or Status::Invalid if not an object
+ARROW_EXPORT Result<int32_t> GetObjectFieldCount(const uint8_t* data, int64_t length);
+
+/// \brief Get the number of elements in a variant array.
+///
+/// \param[in] data Pointer to the value buffer (must start with an array)
+/// \param[in] length Length of the value buffer in bytes
+/// \return The number of elements, or Status::Invalid if not an array
+ARROW_EXPORT Result<int32_t> GetArrayElementCount(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Value size computation
+// ---------------------------------------------------------------------------
+
+/// \brief Compute the total byte size of a variant value (header + data).
+///
+/// Determines how many bytes a variant value occupies by examining
+/// its header and (for containers/variable-length types) reading
+/// size information. Does NOT recursively validate the contents.
+///
+/// \param[in] data Pointer to the start of a variant value
+/// \param[in] length Maximum bytes available
+/// \return Total byte count of the value, or Status::Invalid if truncated
+ARROW_EXPORT Result<int64_t> ValueSize(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Random access utilities
+// ---------------------------------------------------------------------------
+
+/// \brief Find an object field by name and return the offset/size of its value.
+///
+/// Searches the field IDs in the object, resolving each against the
+/// metadata dictionary. Per spec, field IDs are in lexicographic order
+/// of their corresponding key names, enabling binary search for large
+/// objects (>=32 fields). For smaller objects, linear scan is used.
+///
+/// \param[in] metadata Parsed metadata (for resolving field IDs to names)
+/// \param[in] data Pointer to the object value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] field_name The field name to search for
+/// \param[out] field_offset Set to the byte offset of the field's value
+/// within data, or -1 if not found
+/// \param[out] field_size Set to the byte size of the field's value,
+/// or 0 if not found
+/// \return Status::OK if search completed (field may or may not exist),
+/// Status::Invalid if the buffer is malformed
+ARROW_EXPORT Status FindObjectField(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length, std::string_view field_name,
+ int64_t* field_offset, int64_t* field_size);
+
+/// \brief Get the i-th element of a variant array by index (O(1) access).
+///
+/// Uses the offset table for random access without traversing preceding
+/// elements.
+///
+/// \param[in] data Pointer to the array value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] index Zero-based element index
+/// \param[out] element_offset Set to the byte offset of the element within data
+/// \param[out] element_size Set to the byte size of the element
+/// \return Status::OK on success, Status::Invalid if not an array or
+/// index is out of range
+ARROW_EXPORT Status GetArrayElement(const uint8_t* data, int64_t length, int32_t index,
+ int64_t* element_offset, int64_t* element_size);
+
+/// \brief Get the i-th field of a variant object by position.
+///
+/// Returns both the field name (resolved from metadata) and a pointer
+/// to the field's value.
+///
+/// \param[in] metadata Parsed metadata
+/// \param[in] data Pointer to the object value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] index Zero-based field index
+/// \param[out] field_name Set to the field's key name
+/// \param[out] field_offset Set to the byte offset of the field's value
+/// \param[out] field_size Set to the byte size of the field's value
+/// \return Status::OK on success, Status::Invalid if not an object or
+/// index is out of range
+ARROW_EXPORT Status GetObjectFieldAt(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length, int32_t index,
+ std::string_view* field_name, int64_t* field_offset,
+ int64_t* field_size);
+
+/// \brief Find the dictionary ID for a given key name.
+///
+/// Uses binary search if the metadata is sorted, otherwise linear scan.
+///
+/// \param[in] metadata Parsed metadata
+/// \param[in] key The key to search for
+/// \return The dictionary ID if found, or -1 if not present
+ARROW_EXPORT int32_t FindMetadataKey(const VariantMetadata& metadata,
+ std::string_view key);
+
+// ---------------------------------------------------------------------------
+// Variant Builder (Encoder)
+// ---------------------------------------------------------------------------
+
+/// \brief Builder for constructing Variant binary values.
+///
+/// Mirrors the Go implementation's Builder pattern. Values are written
+/// into an internal buffer; containers (objects/arrays) use a start-offset
+/// + finish pattern that shifts data to insert headers.
+///
+/// Usage:
+/// VariantBuilder builder;
+/// auto start = builder.Offset();
+/// std::vector<VariantBuilder::FieldEntry> fields;
+/// fields.push_back(builder.NextField(start, "name"));
+/// builder.String("Alice");
+/// fields.push_back(builder.NextField(start, "age"));
+/// builder.Int(30);
+/// builder.FinishObject(start, fields);
+/// ARROW_ASSIGN_OR_RAISE(auto result, builder.Finish());
+class ARROW_EXPORT VariantBuilder {
Review Comment:
This API is great for building new variants. Did you also consider adding an API that allows modifying existing Variant values? We would need to add a function to `VariantBuilder` similar to `FindObjectField` from the decoding PR, which would "move"| the context of `VariantBuilder` to a specific place/field. Once called, you would then be able to override the existing value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45947 : [C++][Parquet] Variant encoding [arrow]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on code in PR #50122:
URL: https://github.com/apache/arrow/pull/50122#discussion_r3485326021
##########
cpp/src/arrow/extension/variant_internal.h:
##########
@@ -0,0 +1,479 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#pragma once
+
+#include <cstdint>
+#include <string>
+#include <string_view>
+#include <unordered_map>
+#include <vector>
+
+#include "arrow/result.h"
+#include "arrow/status.h"
+#include "arrow/util/visibility.h"
+
+namespace arrow::extension::variant_internal {
+
+/// \file variant_internal.h
+/// \brief Utilities for Variant binary encoding/decoding.
+///
+/// Implements parsing logic per the Variant Encoding Spec:
+/// https://github.com/apache/parquet-format/blob/master/VariantEncoding.md
+///
+/// The "internal" in the filename refers to the binary encoding internals
+/// of the Variant type, not the visibility of this header. This header is
+/// installed and provides the public C++ API for working with Variant
+/// binary data (independent of the VariantExtensionType in parquet_variant.h).
+
+// ---------------------------------------------------------------------------
+// Constants
+// ---------------------------------------------------------------------------
+
+/// Variant encoding spec version 1.
+constexpr uint8_t kVariantVersion = 1;
+
+/// Maximum nesting depth for recursive value decoding.
+/// Prevents stack overflow on deeply nested (possibly malicious) input.
+constexpr int32_t kMaxNestingDepth = 128;
+
+// ---------------------------------------------------------------------------
+// Enumerations
+// ---------------------------------------------------------------------------
+
+/// \brief Basic type codes from bits 0-1 of the value header byte.
+///
+/// Variant Encoding Spec §3: "Value encoding"
+enum class BasicType : uint8_t {
+ kPrimitive = 0,
+ kShortString = 1,
+ kObject = 2,
+ kArray = 3,
+};
+
+/// \brief Primitive type codes from bits 2-7 when basic_type == kPrimitive.
+///
+/// Variant Encoding Spec §3.1: "Primitive types"
+enum class PrimitiveType : uint8_t {
+ kNull = 0,
+ kTrue = 1,
+ kFalse = 2,
+ kInt8 = 3,
+ kInt16 = 4,
+ kInt32 = 5,
+ kInt64 = 6,
+ kDouble = 7,
+ kDecimal4 = 8,
+ kDecimal8 = 9,
+ kDecimal16 = 10,
+ kDate = 11,
+ kTimestampMicros = 12,
+ kTimestampMicrosNTZ = 13,
+ kFloat = 14,
+ kBinary = 15,
+ kString = 16,
+ kTimeNTZ = 17,
+ kTimestampNanos = 18,
+ kTimestampNanosNTZ = 19,
+ kUUID = 20,
+};
+
+// ---------------------------------------------------------------------------
+// Metadata
+// ---------------------------------------------------------------------------
+
+/// \brief Parsed variant metadata (string dictionary).
+///
+/// The metadata buffer contains a header byte followed by a dictionary of
+/// interned strings. String views reference the raw buffer and are valid
+/// only as long as the underlying buffer is alive.
+struct ARROW_EXPORT VariantMetadata {
+ /// Spec version (must be kVariantVersion).
+ uint8_t version = 0;
+
+ /// Whether the dictionary strings are sorted lexicographically.
+ bool is_sorted = false;
+
+ /// Number of bytes used for each offset (1, 2, 3, or 4).
+ int32_t offset_size = 0;
+
+ /// Dictionary of interned strings. Views into the raw metadata buffer.
+ std::vector<std::string_view> strings;
+};
+
+/// \brief Decode a variant metadata buffer.
+///
+/// Parses the header byte and string dictionary from the raw metadata
+/// buffer. The returned VariantMetadata contains string_views that
+/// reference the input buffer directly (zero-copy).
+///
+/// \param[in] data Pointer to the metadata buffer (must not be null)
+/// \param[in] length Length of the metadata buffer in bytes
+/// \return Parsed VariantMetadata on success, Status::Invalid on
+/// malformed input
+///
+/// \note The input buffer must outlive the returned VariantMetadata.
+ARROW_EXPORT Result<VariantMetadata> DecodeMetadata(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Value header utilities
+// ---------------------------------------------------------------------------
+
+/// \brief Extract the basic type from a value header byte.
+///
+/// \param[in] header The first byte of a variant value
+/// \return The BasicType (bits 0-1)
+inline BasicType GetBasicType(uint8_t header) {
+ return static_cast<BasicType>(header & 0x03);
+}
+
+/// \brief Extract the primitive type from a value header byte.
+///
+/// Only valid when GetBasicType(header) == BasicType::kPrimitive.
+///
+/// \param[in] header The first byte of a variant value
+/// \return The PrimitiveType (bits 2-7)
+inline PrimitiveType GetPrimitiveType(uint8_t header) {
+ return static_cast<PrimitiveType>((header >> 2) & 0x3F);
+}
+
+/// \brief Get the byte size of a primitive value (excluding header).
+///
+/// \param[in] primitive_type The primitive type code
+/// \return Number of bytes for the value payload, or -1 for
+/// variable-length types (Binary, String)
+ARROW_EXPORT int32_t PrimitiveValueSize(PrimitiveType primitive_type);
+
+// ---------------------------------------------------------------------------
+// Value decoding
+// ---------------------------------------------------------------------------
+
+/// \brief Visitor interface for variant value decoding.
+///
+/// Implement this interface to receive callbacks during variant value
+/// traversal. The visitor pattern avoids materializing a tree of objects,
+/// which is important when scanning millions of rows.
+///
+/// All methods return Status::OK() to continue traversal, or any error
+/// Status to abort.
+///
+/// \note String values passed to String() and FieldName() are raw bytes from
+/// the variant buffer without UTF-8 validation. Per spec, all strings
+/// must be valid UTF-8, but validation is the responsibility of a
+/// higher-level consumer (e.g., when materializing to Arrow StringArray).
+class ARROW_EXPORT VariantVisitor {
+ public:
+ virtual ~VariantVisitor() = default;
+
+ /// @name Primitive value callbacks
+ /// @{
+ virtual Status Null() = 0;
+ virtual Status Bool(bool value) = 0;
+ virtual Status Int8(int8_t value) = 0;
+ virtual Status Int16(int16_t value) = 0;
+ virtual Status Int32(int32_t value) = 0;
+ virtual Status Int64(int64_t value) = 0;
+ virtual Status Float(float value) = 0;
+ virtual Status Double(double value) = 0;
+ virtual Status Decimal4(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal8(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Decimal16(const uint8_t* bytes, int32_t scale) = 0;
+ virtual Status Date(int32_t days_since_epoch) = 0;
+ virtual Status TimestampMicros(int64_t micros_since_epoch) = 0;
+ virtual Status TimestampMicrosNTZ(int64_t micros_since_epoch) = 0;
+ virtual Status String(std::string_view value) = 0;
+ virtual Status Binary(std::string_view value) = 0;
+ virtual Status TimeNTZ(int64_t micros_since_midnight) = 0;
+ virtual Status TimestampNanos(int64_t nanos_since_epoch) = 0;
+ virtual Status TimestampNanosNTZ(int64_t nanos_since_epoch) = 0;
+ virtual Status UUID(const uint8_t* bytes) = 0;
+ /// @}
+
+ /// @name Container callbacks
+ /// @{
+
+ /// \brief Called at the start of an object with the number of fields.
+ virtual Status StartObject(int32_t num_fields) = 0;
+
+ /// \brief Called for each object field name, before the field value.
+ virtual Status FieldName(std::string_view name) = 0;
+
+ /// \brief Called after all fields of an object have been visited.
+ virtual Status EndObject() = 0;
+
+ /// \brief Called at the start of an array with the number of elements.
+ virtual Status StartArray(int32_t num_elements) = 0;
+
+ /// \brief Called after all elements of an array have been visited.
+ virtual Status EndArray() = 0;
+ /// @}
+};
+
+/// \brief Decode a variant value buffer using a visitor.
+///
+/// Recursively traverses the variant value, calling the appropriate
+/// visitor methods for each element. Objects and arrays trigger
+/// Start/End pairs with nested visits for their contents.
+///
+/// \param[in] metadata Parsed metadata (for resolving string dictionary)
+/// \param[in] data Pointer to the value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] visitor Callback interface for decoded values
+/// \return Status::OK on success, Status::Invalid on malformed input
+///
+/// \note The data buffer must remain valid for the duration of the call.
+ARROW_EXPORT Status DecodeVariantValue(const VariantMetadata& metadata,
+ const uint8_t* data, int64_t length,
+ VariantVisitor* visitor);
+
+/// \brief Get the basic type of a variant value without full decoding.
+///
+/// \param[in] data Pointer to the value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \return The BasicType of the value, or Status::Invalid if the
+/// buffer is empty
+ARROW_EXPORT Result<BasicType> GetValueBasicType(const uint8_t* data, int64_t length);
+
+/// \brief Get the number of fields in a variant object.
+///
+/// \param[in] data Pointer to the value buffer (must start with an object)
+/// \param[in] length Length of the value buffer in bytes
+/// \return The number of fields, or Status::Invalid if not an object
+ARROW_EXPORT Result<int32_t> GetObjectFieldCount(const uint8_t* data, int64_t length);
+
+/// \brief Get the number of elements in a variant array.
+///
+/// \param[in] data Pointer to the value buffer (must start with an array)
+/// \param[in] length Length of the value buffer in bytes
+/// \return The number of elements, or Status::Invalid if not an array
+ARROW_EXPORT Result<int32_t> GetArrayElementCount(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Value size computation
+// ---------------------------------------------------------------------------
+
+/// \brief Compute the total byte size of a variant value (header + data).
+///
+/// Determines how many bytes a variant value occupies by examining
+/// its header and (for containers/variable-length types) reading
+/// size information. Does NOT recursively validate the contents.
+///
+/// \param[in] data Pointer to the start of a variant value
+/// \param[in] length Maximum bytes available
+/// \return Total byte count of the value, or Status::Invalid if truncated
+ARROW_EXPORT Result<int64_t> ValueSize(const uint8_t* data, int64_t length);
+
+// ---------------------------------------------------------------------------
+// Random access utilities
+// ---------------------------------------------------------------------------
+
+/// \brief Find an object field by name and return the offset/size of its value.
+///
+/// Searches the field IDs in the object, resolving each against the
+/// metadata dictionary. Per spec, field IDs are in lexicographic order
+/// of their corresponding key names, enabling binary search for large
+/// objects (>=32 fields). For smaller objects, linear scan is used.
+///
+/// \param[in] metadata Parsed metadata (for resolving field IDs to names)
+/// \param[in] data Pointer to the object value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] field_name The field name to search for
+/// \param[out] field_offset Set to the byte offset of the field's value
+/// within data, or -1 if not found
+/// \param[out] field_size Set to the byte size of the field's value,
+/// or 0 if not found
+/// \return Status::OK if search completed (field may or may not exist),
+/// Status::Invalid if the buffer is malformed
+ARROW_EXPORT Status FindObjectField(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length, std::string_view field_name,
+ int64_t* field_offset, int64_t* field_size);
+
+/// \brief Get the i-th element of a variant array by index (O(1) access).
+///
+/// Uses the offset table for random access without traversing preceding
+/// elements.
+///
+/// \param[in] data Pointer to the array value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] index Zero-based element index
+/// \param[out] element_offset Set to the byte offset of the element within data
+/// \param[out] element_size Set to the byte size of the element
+/// \return Status::OK on success, Status::Invalid if not an array or
+/// index is out of range
+ARROW_EXPORT Status GetArrayElement(const uint8_t* data, int64_t length, int32_t index,
+ int64_t* element_offset, int64_t* element_size);
+
+/// \brief Get the i-th field of a variant object by position.
+///
+/// Returns both the field name (resolved from metadata) and a pointer
+/// to the field's value.
+///
+/// \param[in] metadata Parsed metadata
+/// \param[in] data Pointer to the object value buffer
+/// \param[in] length Length of the value buffer in bytes
+/// \param[in] index Zero-based field index
+/// \param[out] field_name Set to the field's key name
+/// \param[out] field_offset Set to the byte offset of the field's value
+/// \param[out] field_size Set to the byte size of the field's value
+/// \return Status::OK on success, Status::Invalid if not an object or
+/// index is out of range
+ARROW_EXPORT Status GetObjectFieldAt(const VariantMetadata& metadata, const uint8_t* data,
+ int64_t length, int32_t index,
+ std::string_view* field_name, int64_t* field_offset,
+ int64_t* field_size);
+
+/// \brief Find the dictionary ID for a given key name.
+///
+/// Uses binary search if the metadata is sorted, otherwise linear scan.
+///
+/// \param[in] metadata Parsed metadata
+/// \param[in] key The key to search for
+/// \return The dictionary ID if found, or -1 if not present
+ARROW_EXPORT int32_t FindMetadataKey(const VariantMetadata& metadata,
+ std::string_view key);
+
+// ---------------------------------------------------------------------------
+// Variant Builder (Encoder)
+// ---------------------------------------------------------------------------
+
+/// \brief Builder for constructing Variant binary values.
+///
+/// Mirrors the Go implementation's Builder pattern. Values are written
+/// into an internal buffer; containers (objects/arrays) use a start-offset
+/// + finish pattern that shifts data to insert headers.
+///
+/// Usage:
+/// VariantBuilder builder;
+/// auto start = builder.Offset();
+/// std::vector<VariantBuilder::FieldEntry> fields;
+/// fields.push_back(builder.NextField(start, "name"));
+/// builder.String("Alice");
+/// fields.push_back(builder.NextField(start, "age"));
+/// builder.Int(30);
+/// builder.FinishObject(start, fields);
+/// ARROW_ASSIGN_OR_RAISE(auto result, builder.Finish());
+class ARROW_EXPORT VariantBuilder {
+ public:
+ VariantBuilder();
+ explicit VariantBuilder(const VariantMetadata& existing_metadata);
Review Comment:
Thanks for the valid point. The variant binary format is immutable by design, inserting a field into an existing object requires rewriting the header (field IDs and offsets are packed arrays, not linked structures). So "modify in place" isn't feasible at the format level.
The refactored design handles the "start from existing data" use case through:
- `VariantBuilder(const VariantMetadata& existing_metadata)`: constructor that pre-populates the key dictionary from an existing metadata buffer.
- Read->rebuild pattern: `VariantObjectView` (read existing) + `ObjectScope` + `UnsafeAppendEncoded` (write new). This enables zero-copy field transfer between variants. It's exactly what the shredding reconstruction path uses.
This matches Rust's `Variant` (read-only) vs `VariantBuilder` (write-only) architecture, and is the standard immutable-format pattern (FlatBuffers, Cap'n Proto, etc.).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45947 : [C++][Parquet] Variant encoding [arrow]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on PR #50122:
URL: https://github.com/apache/arrow/pull/50122#issuecomment-4815343351
Force-pushed a refactored version. The initial implementation carried over some Go patterns (manual start/finish without safety guarantees, separate lookup buffer). After reviewing the feedback — particularly around initialization from existing buffers and the question about modifying existing variants, I reworked the builder with C++ idiom
at the center of the design.
Key changes:
- **RAII scopes** (`ObjectScope`/`ListScope`) with auto-rollback on destruction replace
the previous unguarded start/finish pattern
- **`[[nodiscard]]`** on scope-returning functions prevents silent discard
- **Transparent hasher** on `dict_` eliminates the old `lookup_buf_` member variable
- **Sorted-check optimization** in `FinishObject` (skip sort when fields already ordered)
- Move-only builder (copy deleted) enforces single ownership
Regarding the earlier review questions about modifying existing variants and type mismatch testing, I've addressed these in replies above with architectural context from the refactored design.
All 335 tests pass end-to-end with `BUILD_WARNING_LEVEL=CHECKIN` (87 encoder-specific).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-45947 : [C++][Parquet] Variant encoding [arrow]
Posted by "qzyu999 (via GitHub)" <gi...@apache.org>.
qzyu999 commented on PR #50122:
URL: https://github.com/apache/arrow/pull/50122#issuecomment-5137544817
Hi all, I'm seeing here: https://github.com/apache/arrow/issues/45937#issuecomment-5112253045 that @wgtmac is considering this PR abandoned in favor of https://github.com/apache/arrow/pull/50252. I think it makes sense given that he had been assigned first and I started working on it under the assumption that it was abandoned abandoned after a period of no PR. I should've confirmed with @wgtmac the status of the PR prior to starting work on it. Apologies for having wasted any other developers' hours.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org