You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "wgtmac (via GitHub)" <gi...@apache.org> on 2026/02/10 16:07:29 UTC

[PR] GH-49217: [C++][Parquet] Fix map type to preserve key-value metadata [arrow]

wgtmac opened a new pull request, #49218:
URL: https://github.com/apache/arrow/pull/49218

   ### Rationale for this change
   
   Previously, when converting Parquet schemas back to Arrow schemas with serialized ARROW:schema metadata, the key-value metadata on map nested fields (key/value fields) was lost. The GetNestedFactory() function lacked a MAP case, preventing ApplyOriginalStorageMetadata from recursively restoring metadata to map children. This fix adds MAP support to the factory function, enabling proper metadata preservation for maps during schema roundtrips.
   
   ### What changes are included in this PR?
   
   Added a MAP case to GetNestedFactory() that returns a lambda to reconstruct MapType while preserving the keys_sorted property from the original type.
   
   ### Are these changes tested?
   
   Yes. Added a test case to verify the issue has been fixed.
   
   ### Are there any user-facing changes?
   
   No.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-49217: [C++][Parquet] Fix map type to preserve key-value metadata [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #49218:
URL: https://github.com/apache/arrow/pull/49218#discussion_r2792010061


##########
cpp/src/parquet/arrow/arrow_schema_test.cc:
##########
@@ -2018,6 +2019,50 @@ TEST_F(TestConvertRoundTrip, FieldIdPreserveAllColumnTypes) {
   ASSERT_EQ(thrift_field_ids, expected_field_ids);
 }
 
+TEST_F(TestConvertRoundTrip, MapNestedFieldMetadataPreserved) {
+  auto key_meta = ::arrow::key_value_metadata({"k"}, {"v"});
+  auto inner_meta = ::arrow::key_value_metadata({"inner_k"}, {"inner_v"});
+
+  auto map_key = ::arrow::field("key", UTF8, /*nullable=*/false, key_meta);
+  auto map_value = ::arrow::field(
+      "value",
+      ::arrow::struct_({::arrow::field("inner", INT64, /*nullable=*/true, inner_meta)}),
+      /*nullable=*/true, inner_meta);
+  auto sorted_map =
+      std::make_shared<::arrow::MapType>(map_key, map_value, /*keys_sorted=*/true);
+  auto arrow_schema = ::arrow::schema(
+      {::arrow::field("m", sorted_map, /*nullable=*/true, FieldIdMetadata(99))});
+
+  std::shared_ptr<SchemaDescriptor> parquet_schema;
+  ASSERT_OK(ToParquetSchema(arrow_schema.get(), *::parquet::default_writer_properties(),
+                            &parquet_schema));
+
+  std::shared_ptr<KeyValueMetadata> kv_metadata;
+  ASSERT_OK(ArrowSchemaToParquetMetadata(arrow_schema, kv_metadata));
+
+  std::shared_ptr<::arrow::Schema> restored_schema;
+  ASSERT_OK(FromParquetSchema(parquet_schema.get(), ArrowReaderProperties(), kv_metadata,
+                              &restored_schema));
+  ASSERT_EQ(restored_schema->num_fields(), 1);
+
+  auto restored_map = ::arrow::internal::checked_pointer_cast<::arrow::MapType>(
+      restored_schema->field(0)->type());
+  ASSERT_EQ(GetFieldId(*restored_schema->field(0)), 99);
+
+  ASSERT_TRUE(restored_map->keys_sorted());

Review Comment:
   Can we just call `AssertTypeEqual(*sorted_map, *restored_map, /*check_metadata=*/true);`?
   It doesn't seem useful to spelunk the nested type hierarchy manually.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-49217: [C++][Parquet] Fix map type to preserve key-value metadata [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #49218:
URL: https://github.com/apache/arrow/pull/49218#issuecomment-3881714105

   Could you help review this? @pitrou @adamreeve @mapleFU


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-49217: [C++][Parquet] Fix map type to preserve key-value metadata [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on PR #49218:
URL: https://github.com/apache/arrow/pull/49218#issuecomment-3883995738

   Thanks all for the review!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-49217: [C++][Parquet] Fix map type to preserve key-value metadata [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #49218:
URL: https://github.com/apache/arrow/pull/49218#discussion_r2792428197


##########
cpp/src/parquet/arrow/arrow_schema_test.cc:
##########
@@ -2018,6 +2019,50 @@ TEST_F(TestConvertRoundTrip, FieldIdPreserveAllColumnTypes) {
   ASSERT_EQ(thrift_field_ids, expected_field_ids);
 }
 
+TEST_F(TestConvertRoundTrip, MapNestedFieldMetadataPreserved) {
+  auto key_meta = ::arrow::key_value_metadata({"k"}, {"v"});
+  auto inner_meta = ::arrow::key_value_metadata({"inner_k"}, {"inner_v"});
+
+  auto map_key = ::arrow::field("key", UTF8, /*nullable=*/false, key_meta);
+  auto map_value = ::arrow::field(
+      "value",
+      ::arrow::struct_({::arrow::field("inner", INT64, /*nullable=*/true, inner_meta)}),
+      /*nullable=*/true, inner_meta);
+  auto sorted_map =
+      std::make_shared<::arrow::MapType>(map_key, map_value, /*keys_sorted=*/true);
+  auto arrow_schema = ::arrow::schema(
+      {::arrow::field("m", sorted_map, /*nullable=*/true, FieldIdMetadata(99))});
+
+  std::shared_ptr<SchemaDescriptor> parquet_schema;
+  ASSERT_OK(ToParquetSchema(arrow_schema.get(), *::parquet::default_writer_properties(),
+                            &parquet_schema));
+
+  std::shared_ptr<KeyValueMetadata> kv_metadata;
+  ASSERT_OK(ArrowSchemaToParquetMetadata(arrow_schema, kv_metadata));
+
+  std::shared_ptr<::arrow::Schema> restored_schema;
+  ASSERT_OK(FromParquetSchema(parquet_schema.get(), ArrowReaderProperties(), kv_metadata,
+                              &restored_schema));
+  ASSERT_EQ(restored_schema->num_fields(), 1);
+
+  auto restored_map = ::arrow::internal::checked_pointer_cast<::arrow::MapType>(
+      restored_schema->field(0)->type());
+  ASSERT_EQ(GetFieldId(*restored_schema->field(0)), 99);
+
+  ASSERT_TRUE(restored_map->keys_sorted());

Review Comment:
   Added!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-49217: [C++][Parquet] Fix map type to preserve key-value metadata [arrow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #49218:
URL: https://github.com/apache/arrow/pull/49218#issuecomment-3878940389

   :warning: GitHub issue #49217 **has been automatically assigned in GitHub** to PR creator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-49217: [C++][Parquet] Fix map type to preserve key-value metadata [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #49218:
URL: https://github.com/apache/arrow/pull/49218#discussion_r2791143211


##########
cpp/src/parquet/arrow/schema.cc:
##########
@@ -991,6 +991,16 @@ std::function<std::shared_ptr<::arrow::DataType>(FieldVector)> GetNestedFactory(
         };
       }
       break;
+    case ::arrow::Type::MAP:
+      if (origin_type.id() == ::arrow::Type::MAP) {
+        const bool keys_sorted =
+            checked_cast<const ::arrow::MapType&>(origin_type).keys_sorted();

Review Comment:
   Yes, that's why I made it sorted in this test case.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-49217: [C++][Parquet] Fix map type to preserve key-value metadata [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac merged PR #49218:
URL: https://github.com/apache/arrow/pull/49218


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-49217: [C++][Parquet] Fix map type to preserve key-value metadata [arrow]

Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on code in PR #49218:
URL: https://github.com/apache/arrow/pull/49218#discussion_r2791135240


##########
cpp/src/parquet/arrow/schema.cc:
##########
@@ -991,6 +991,16 @@ std::function<std::shared_ptr<::arrow::DataType>(FieldVector)> GetNestedFactory(
         };
       }
       break;
+    case ::arrow::Type::MAP:
+      if (origin_type.id() == ::arrow::Type::MAP) {
+        const bool keys_sorted =
+            checked_cast<const ::arrow::MapType&>(origin_type).keys_sorted();

Review Comment:
   According to `MapToSchemaField`, is `keys_sorted` always false?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-49217: [C++][Parquet] Fix map type to preserve key-value metadata [arrow]

Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #49218:
URL: https://github.com/apache/arrow/pull/49218#discussion_r2792406974


##########
cpp/src/parquet/arrow/arrow_schema_test.cc:
##########
@@ -2018,6 +2019,50 @@ TEST_F(TestConvertRoundTrip, FieldIdPreserveAllColumnTypes) {
   ASSERT_EQ(thrift_field_ids, expected_field_ids);
 }
 
+TEST_F(TestConvertRoundTrip, MapNestedFieldMetadataPreserved) {
+  auto key_meta = ::arrow::key_value_metadata({"k"}, {"v"});
+  auto inner_meta = ::arrow::key_value_metadata({"inner_k"}, {"inner_v"});
+
+  auto map_key = ::arrow::field("key", UTF8, /*nullable=*/false, key_meta);
+  auto map_value = ::arrow::field(
+      "value",
+      ::arrow::struct_({::arrow::field("inner", INT64, /*nullable=*/true, inner_meta)}),
+      /*nullable=*/true, inner_meta);
+  auto sorted_map =
+      std::make_shared<::arrow::MapType>(map_key, map_value, /*keys_sorted=*/true);
+  auto arrow_schema = ::arrow::schema(
+      {::arrow::field("m", sorted_map, /*nullable=*/true, FieldIdMetadata(99))});
+
+  std::shared_ptr<SchemaDescriptor> parquet_schema;
+  ASSERT_OK(ToParquetSchema(arrow_schema.get(), *::parquet::default_writer_properties(),
+                            &parquet_schema));
+
+  std::shared_ptr<KeyValueMetadata> kv_metadata;
+  ASSERT_OK(ArrowSchemaToParquetMetadata(arrow_schema, kv_metadata));
+
+  std::shared_ptr<::arrow::Schema> restored_schema;
+  ASSERT_OK(FromParquetSchema(parquet_schema.get(), ArrowReaderProperties(), kv_metadata,
+                              &restored_schema));
+  ASSERT_EQ(restored_schema->num_fields(), 1);
+
+  auto restored_map = ::arrow::internal::checked_pointer_cast<::arrow::MapType>(
+      restored_schema->field(0)->type());
+  ASSERT_EQ(GetFieldId(*restored_schema->field(0)), 99);
+
+  ASSERT_TRUE(restored_map->keys_sorted());

Review Comment:
   Oh, I see. Can you add a comment about that, then?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] GH-49217: [C++][Parquet] Fix map type to preserve key-value metadata [arrow]

Posted by "wgtmac (via GitHub)" <gi...@apache.org>.
wgtmac commented on code in PR #49218:
URL: https://github.com/apache/arrow/pull/49218#discussion_r2792376765


##########
cpp/src/parquet/arrow/arrow_schema_test.cc:
##########
@@ -2018,6 +2019,50 @@ TEST_F(TestConvertRoundTrip, FieldIdPreserveAllColumnTypes) {
   ASSERT_EQ(thrift_field_ids, expected_field_ids);
 }
 
+TEST_F(TestConvertRoundTrip, MapNestedFieldMetadataPreserved) {
+  auto key_meta = ::arrow::key_value_metadata({"k"}, {"v"});
+  auto inner_meta = ::arrow::key_value_metadata({"inner_k"}, {"inner_v"});
+
+  auto map_key = ::arrow::field("key", UTF8, /*nullable=*/false, key_meta);
+  auto map_value = ::arrow::field(
+      "value",
+      ::arrow::struct_({::arrow::field("inner", INT64, /*nullable=*/true, inner_meta)}),
+      /*nullable=*/true, inner_meta);
+  auto sorted_map =
+      std::make_shared<::arrow::MapType>(map_key, map_value, /*keys_sorted=*/true);
+  auto arrow_schema = ::arrow::schema(
+      {::arrow::field("m", sorted_map, /*nullable=*/true, FieldIdMetadata(99))});
+
+  std::shared_ptr<SchemaDescriptor> parquet_schema;
+  ASSERT_OK(ToParquetSchema(arrow_schema.get(), *::parquet::default_writer_properties(),
+                            &parquet_schema));
+
+  std::shared_ptr<KeyValueMetadata> kv_metadata;
+  ASSERT_OK(ArrowSchemaToParquetMetadata(arrow_schema, kv_metadata));
+
+  std::shared_ptr<::arrow::Schema> restored_schema;
+  ASSERT_OK(FromParquetSchema(parquet_schema.get(), ArrowReaderProperties(), kv_metadata,
+                              &restored_schema));
+  ASSERT_EQ(restored_schema->num_fields(), 1);
+
+  auto restored_map = ::arrow::internal::checked_pointer_cast<::arrow::MapType>(
+      restored_schema->field(0)->type());
+  ASSERT_EQ(GetFieldId(*restored_schema->field(0)), 99);
+
+  ASSERT_TRUE(restored_map->keys_sorted());

Review Comment:
   I didn't use this approach because it will fail at comparing map type metadata fingerprint. The reason is that
   
   `"key_value"` is always used when converting arrow map type to parquet map type:
    https://github.com/apache/arrow/blob/bc489218164218387c9728b89497c80d41eb1c00/cpp/src/parquet/arrow/schema.cc#L126  
   
   but Arrow map type uses `"entries"`: https://github.com/apache/arrow/blob/bc489218164218387c9728b89497c80d41eb1c00/cpp/src/arrow/type.cc#L1041
   
   Fixing this requires larger effort and may break existing applications.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org