You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "CurtHagenlocher (via GitHub)" <gi...@apache.org> on 2026/10/18 21:57:30 UTC

[I] C Data Interface puts a dictionary-encoded extension type's metadata on the wrong schema [arrow-dotnet]

CurtHagenlocher opened a new issue, #473:
URL: https://github.com/apache/arrow-dotnet/issues/473

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   For a dictionary-encoded extension type, such as `dictionary<int32, arrow.uuid>`, the C Data Interface exporter and importer disagree with Arrow C++ about which `ArrowSchema` carries `ARROW:extension:name`. As a result, such a field can't be exchanged in either direction, and it doesn't round-trip within .NET either.
   
   **Where the metadata goes**
   
   | Producer | Field schema (format `i`) | `dictionary` child (format `w:16`) |
   |---|---|---|
   | pyarrow 24.0.0 | none | `ARROW:extension:name=arrow.uuid` |
   | Apache.Arrow (`main`) | `ARROW:extension:name=arrow.uuid` | none |
   
   Arrow C++ treats the dictionary's value type as the extension type, so the metadata belongs on the `dictionary` child. `CArrowSchemaExporter.ExportField` instead moves a dictionary value's extension type up to the field (`src/Apache.Arrow/C/CArrowSchemaExporter.cs`, the `field.DataType is DictionaryType dt && dt.ValueType is ExtensionType dext` branch). That matches IPC, where a dictionary-encoded field's metadata describes its value type, but not the C Data Interface.
   
   **Effects** (with `GuidExtensionDefinition` registered)
   
   1. **.NET → pyarrow:** `pa.Field._import_from_c` fails with `Invalid storage type for UuidType: dictionary<values=fixed_size_binary[16], indices=int32, ordered=0>`. C++ tries to apply the extension to the field's whole type, which is the dictionary.
   2. **pyarrow → .NET:** `CArrowSchemaImporter.ImportField(ptr, registry)` returns `dictionary<int32, fixed_size_binary[16]>`, and the imported `DictionaryArray.Dictionary` is a `FixedSizeBinaryArray`. The importer never reads the `dictionary` child's metadata.
   3. **.NET → .NET:** this also comes back as `dictionary<int32, fixed_size_binary[16]>` with the extension metadata left on the field. `ImportedArrowSchema.GetAsField` calls `TryCreateType` with the `DictionaryType` as the storage type, which `GuidExtensionDefinition` rejects.
   
   Repro for (1):
   
   ```csharp
   var field = new Field("f", new DictionaryType(Int32Type.Default, GuidType.Default, false), true);
   CArrowSchema* c = CArrowSchema.Create();
   CArrowSchemaExporter.ExportField(field, c);
   // c->metadata has ARROW:extension:name; c->dictionary->metadata is null
   // pyarrow: pa.Field._import_from_c(ptr) -> ArrowInvalid: Invalid storage type for UuidType
   ```
   
   Repro for (2):
   
   ```python
   pa.field("f", pa.dictionary(pa.int32(), pa.uuid()))._export_to_c(ptr)
   # then CArrowSchemaImporter.ImportField(ptr, registry) -> value type is FixedSizeBinaryType
   ```
   
   **Suggested fix**
   
   - **Export:** write the value type's extension metadata on the `dictionary` child rather than the field. Any field-level metadata the user supplied should stay on the field.
   - **Import:** resolve the `dictionary` child's own extension metadata against the registry, so its value type becomes the extension type. For compatibility with schemas already exported by older Apache.Arrow versions, the importer could also accept the extension name on the field when the field is dictionary-encoded and the definition rejects the dictionary as storage.
   
   #472 doesn't fix this. It passes the registry into the dictionary's nested fields, but it doesn't read the `dictionary` child's own metadata.
   
   Found while working on #466 / #472.
   
   ### Component(s)
   
   C#
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] C Data Interface puts a dictionary-encoded extension type's metadata on the wrong schema [arrow-dotnet]

Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher closed issue #473: C Data Interface puts a dictionary-encoded extension type's metadata on the wrong schema
URL: https://github.com/apache/arrow-dotnet/issues/473


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org