You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "CurtHagenlocher (via GitHub)" <gi...@apache.org> on 2026/10/18 21:44:05 UTC
[PR] Resolve extension types when importing C array streams [arrow-dotnet]
CurtHagenlocher opened a new pull request, #472:
URL: https://github.com/apache/arrow-dotnet/pull/472
## What's Changed
Adds overloads that take an `ExtensionTypeRegistry`:
```csharp
CArrowArrayStreamImporter.ImportArrayStream(CArrowArrayStream* ptr, ExtensionTypeRegistry extensionRegistry)
CArrowDeviceArrayStreamImporter.ImportDeviceArrayStream(CArrowDeviceArrayStream* ptr, ExtensionTypeRegistry extensionRegistry)
```
These import the stream's schema with the registry. The record batch importer already builds extension arrays when a field's type is an `ExtensionType`, so a `arrow.uuid` column from a native producer now comes back as a `GuidType` field with a `GuidArray`, rather than a `FixedSizeBinaryArray` that only has the extension name in its metadata. The existing overloads behave as before. The device stream importer had the same gap, so it gets the same overload.
### Nested fields
`CArrowSchemaImporter` previously applied the registry only to top-level fields, so `list<arrow.uuid>` still imported as a list of `FixedSizeBinary` even with a registry. The registry is now threaded through child fields of list, large list, list view, large list view, fixed-size list, struct, map, union and run-end encoded types, and dictionary value types. This affects `ImportSchema` and `ImportField` with a registry as well as the new stream overloads. As a side effect, `GetAsSchema` no longer parses the top-level fields twice when a registry is given.
### Tests
- Stream import with and without a registry, for both the array stream and device array stream importers.
- Schema import resolving `arrow.uuid` nested inside each of the types listed above, including `list<list<uuid>>`.
- A `list<uuid>` stream end to end, including `GetSlicedValues` returning `GuidArray`s (#468).
- A pyarrow-produced stream with top-level and list-nested `arrow.uuid` columns.
Closes #466.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 5.5 <no...@anthropic.com>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve extension types when importing C array streams [arrow-dotnet]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher commented on code in PR #472:
URL: https://github.com/apache/arrow-dotnet/pull/472#discussion_r4309084398
##########
src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -372,7 +381,7 @@ public Field GetAsField(ExtensionTypeRegistry extensionRegistry)
bool nullable = _cSchema->GetFlag(CArrowSchema.ArrowFlagNullable);
- IArrowType type = GetAsType();
+ IArrowType type = GetAsType(extensionRegistry);
Review Comment:
Good catch, but this is pre-existing rather than introduced here, and it's tracked in #473. The root cause is on the export side: Arrow C++ puts a dictionary value type's extension metadata on the `dictionary` child schema, whereas `CArrowSchemaExporter` puts it on the field. Resolving the field-level metadata against `ValueType` on import would make .NET-to-.NET round trips work, but it would also make the non-standard placement look supported, and pyarrow interop would still be broken in both directions. I'd rather fix the export and import sides together, plus a direct `dictionary<int32, uuid>` test, in a follow-up for #473.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve extension types when importing C array streams [arrow-dotnet]
Posted by "Copilot (via GitHub)" <gi...@apache.org>.
Copilot commented on code in PR #472:
URL: https://github.com/apache/arrow-dotnet/pull/472#discussion_r4308946531
##########
src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -372,7 +381,7 @@ public Field GetAsField(ExtensionTypeRegistry extensionRegistry)
bool nullable = _cSchema->GetFlag(CArrowSchema.ArrowFlagNullable);
- IArrowType type = GetAsType();
+ IArrowType type = GetAsType(extensionRegistry);
Review Comment:
Direct extension-valued dictionaries still lose their extension type with the new registry overloads. `CArrowSchemaExporter.ExportField` places the extension metadata on the outer dictionary field, but `TryCreateType` receives the entire `DictionaryType`; `GuidExtensionDefinition` rejects it because it expects fixed-size binary storage. Resolve the extension against `DictionaryType.ValueType`, then rebuild the dictionary with its original index type and ordering. Add a direct `dictionary<int32, uuid>` round-trip test: the existing `dictionary<int32, list<uuid>>` test only exercises metadata on a nested list child.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Resolve extension types when importing C array streams [arrow-dotnet]
Posted by "CurtHagenlocher (via GitHub)" <gi...@apache.org>.
CurtHagenlocher merged PR #472:
URL: https://github.com/apache/arrow-dotnet/pull/472
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org