You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "wjones127 (via GitHub)" <gi...@apache.org> on 2023/02/10 22:22:46 UTC
[GitHub] [arrow] wjones127 opened a new pull request, #34133: GH-33856: [C#] c data interface for schemas and types
wjones127 opened a new pull request, #34133:
URL: https://github.com/apache/arrow/pull/34133
<!--
Thanks for opening a pull request!
If this is your first pull request you can find detailed information on how
to contribute here:
* [New Contributor's Guide](https://arrow.apache.org/docs/dev/developers/guide/step_by_step/pr_lifecycle.html#reviews-and-merge-of-the-pull-request)
* [Contributing Overview](https://arrow.apache.org/docs/dev/developers/overview.html)
If this is not a [minor PR](https://github.com/apache/arrow/blob/master/CONTRIBUTING.md#Minor-Fixes). Could you open an issue for this pull request on GitHub? https://github.com/apache/arrow/issues/new/choose
Opening GitHub issues ahead of time contributes to the [Openness](http://theapacheway.com/open/#:~:text=Openness%20allows%20new%20users%20the,must%20happen%20in%20the%20open.) of the Apache Arrow project.
Then could you also rename the pull request title in the following format?
GH-${GITHUB_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
or
MINOR: [${COMPONENT}] ${SUMMARY}
In the case of PARQUET issues on JIRA the title also supports:
PARQUET-${JIRA_ISSUE_ID}: [${COMPONENT}] ${SUMMARY}
-->
### Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your changes and offer better suggestions for fixes.
-->
### What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is sometimes worth providing a summary of the individual changes in this PR.
-->
### Are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are they covered by existing tests)?
-->
### Are there any user-facing changes?
<!--
If there are user-facing changes then we may require documentation to be updated before approving the PR.
-->
<!--
If there are any breaking changes to public APIs, please uncomment the line below and explain which changes are breaking.
-->
<!-- **This PR includes breaking changes to public APIs.** -->
<!--
Please uncomment the line below (and provide explanation) if the changes fix either (a) a security vulnerability, (b) a bug that caused incorrect or invalid data to be produced, or (c) a bug that causes a crash (even when the API contract is upheld). We use this to highlight fixes to issues that may affect users without their knowledge. For this reason, fixing bugs that cause errors don't count, since those are usually obvious.
-->
<!-- **This PR contains a "Critical Fix".** -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1124941545
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
Review Comment:
We shouldn't be using `IntPtr` for data that is actually pointers. Instead, use the correct pointer type.
```suggestion
public byte* format;
```
See https://learn.microsoft.com/en-us/dotnet/api/system.intptr?view=net-7.0#remarks. An `IntPtr` is
> Represents a signed integer where the bit-width is the same as a pointer.
We should use pointers when we really mean pointers, and `IntPtr` (or `nint` in newer C# versions) to mean "a number that is the same bit-width as a pointer", for example `size_t`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1124957069
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
Review Comment:
> What type would you suggest for children and dictionary?
```C#
public CArrowSchema** children;
public CArrowSchema* dictionary;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1124965341
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
Review Comment:
Such a type issues a warning:
```
csharp(CS8500): This takes the address of, gets the size of, or declares a pointer to a managed type ('CArrowSchema')
```
Also is this how to allocate such a type?
```csharp
var pointerList = (CArrowSchema**)Marshal.AllocHGlobal(numFields * IntPtr.Size);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on pull request #34133: GH-34737: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on PR #34133:
URL: https://github.com/apache/arrow/pull/34133#issuecomment-1487029662
Re-running the macOS 12 Python 3 leg which failed. Once it passes, I'll merge this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] adamreeve commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1119262146
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
Review Comment:
Should format and name be `UnmanagedType.LPUTF8Str`?
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+
+ public ReleaseCArrowSchema release;
+ // Check this out: https://github.com/G-Research/ParquetSharp/blob/386d91bd5e6fe6cb81583803447023c1359957c8/csharp/ParquetHandle.cs#L8
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.formatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType)
+ {
+ if (((DictionaryType)datatype).Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype)
+ {
+ if (datatype is NestedType)
+ {
+ IReadOnlyList<Field> fields = ((NestedType)datatype).Fields;
Review Comment:
You can merge the type check and cast:
```suggestion
if (datatype is NestedType nestedType)
{
IReadOnlyList<Field> fields = nestedType.Fields;
```
(same for `ConstructDictionary` below)
##########
csharp/test/Apache.Arrow.Tests/CDataInterfaceSchemaTests.cs:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.C;
+using Apache.Arrow.Types;
+using Python.Runtime;
+using Xunit;
+
+namespace Apache.Arrow.Tests
+{
+ public class CDataSchemaTest
Review Comment:
Would it make sense to skip these tests if `PYTHONNET_PYDLL` isn't set? That way people can still run the tests normally in an IDE without having to configure the Python integration.
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+
+ public ReleaseCArrowSchema release;
+ // Check this out: https://github.com/G-Research/ParquetSharp/blob/386d91bd5e6fe6cb81583803447023c1359957c8/csharp/ParquetHandle.cs#L8
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.formatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType)
+ {
+ if (((DictionaryType)datatype).Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype)
+ {
+ if (datatype is NestedType)
+ {
+ IReadOnlyList<Field> fields = ((NestedType)datatype).Fields;
+ int numFields = fields.Count;
+
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * sizeof(IntPtr));
+
+ for (var i = 0; i < numFields; i++)
+ {
+ var cSchema = new CArrowSchema();
+ CArrowSchema.ExportField(fields[i], out cSchema);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return pointerList;
+ }
+ else
+ {
+ return (IntPtr*)IntPtr.Zero;
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType)
+ {
+ var cSchema = new CArrowSchema();
+ IArrowType valueType = ((DictionaryType)datatype).ValueType;
+ CArrowSchema.ExportDataType(valueType, out cSchema);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ /// <param name="schema">An uninitialized CArrowSchema.</param>
+ public static void ExportDataType(IArrowType datatype, out CArrowSchema schema)
+ {
+ schema.format = GetFormat(datatype);
+ schema.name = null;
+ schema.metadata = null;
+ schema.flags = GetFlags(datatype);
+
+ schema.children = ConstructChildren(datatype);
+ schema.n_children = datatype is NestedType ? ((NestedType)datatype).Fields.Count : 0;
+
+ schema.dictionary = ConstructDictionary(datatype);
+
+ schema.release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.children[i]);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ schema.private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ /// <param name="schema">An uninitialized CArrowSchema.</param>
+ public static void ExportField(Field field, out CArrowSchema schema)
+ {
+ ExportDataType(field.DataType, out schema);
+ schema.name = field.Name;
+ // TODO: field metadata
+ schema.metadata = null;
+ schema.flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ /// <param name="outSchema">An uninitialized CArrowSchema</param>
+ public static void ExportSchema(Schema schema, out CArrowSchema outSchema)
+ {
+ // TODO: top-level metadata
+ var structType = new StructType(schema.Fields.Values.ToList());
+ ExportDataType(structType, out outSchema);
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public IntPtr Export(IntPtr ptr)
Review Comment:
It seems a bit surprising that this returns an IntPtr, should this just be a void method?
##########
csharp/src/Apache.Arrow/C/ImportedArrowSchema.cs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// A <see cref="CArrowSchema"/> imported from somewhere else.
+ /// </summary>
+ ///
+ /// <example>
+ /// Typically, when importing a schema we will allocate an uninitialized
+ /// <see cref="CArrowSchema"/>, pass the pointer to the foreign function,
+ /// then construct this class with the initialized pointer.
+ ///
+ /// <code>
+ /// var cSchema = new CArrowSchema();
+ /// IntPtr importedPtr = cSchema.AllocateAsPtr();
Review Comment:
Would it tidy things up slightly to have a static method that allocated an uninitialized schema without needing to create a `CArrowSchema` instance first?
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
Review Comment:
Should `children` just be `IntPtr` rather than `IntPtr*`? That would allow the struct itself to not need to be unsafe, although might make dealing with `children` internally a bit more fiddly.
##########
csharp/test/Apache.Arrow.Tests/CDataInterfaceSchemaTests.cs:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.C;
+using Apache.Arrow.Types;
+using Python.Runtime;
+using Xunit;
+
+namespace Apache.Arrow.Tests
+{
+ public class CDataSchemaTest
+ {
+ public CDataSchemaTest()
+ {
+ PythonEngine.Initialize();
+ }
+
+ public static Schema GetTestSchema()
Review Comment:
`GetTestSchema`, `GetPythonFields` and `GetPythonSchema` could be private
##########
csharp/test/Apache.Arrow.Tests/CDataInterfaceSchemaTests.cs:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.C;
+using Apache.Arrow.Types;
+using Python.Runtime;
+using Xunit;
+
+namespace Apache.Arrow.Tests
+{
+ public class CDataSchemaTest
+ {
+ public CDataSchemaTest()
+ {
+ PythonEngine.Initialize();
+ }
+
+ public static Schema GetTestSchema()
+ {
+ // TODO: Add more types
+ using (Py.GIL())
+ {
+ var schema = new Schema.Builder()
+ .Field(f => f.Name("null").DataType(NullType.Default).Nullable(true))
+ .Field(f => f.Name("bool").DataType(BooleanType.Default).Nullable(true))
+ .Field(f => f.Name("i8").DataType(Int8Type.Default).Nullable(true))
+ .Field(f => f.Name("u8").DataType(UInt8Type.Default).Nullable(true))
+ .Field(f => f.Name("i16").DataType(Int16Type.Default).Nullable(true))
+ .Field(f => f.Name("u16").DataType(UInt16Type.Default).Nullable(true))
+ .Field(f => f.Name("i32").DataType(Int32Type.Default).Nullable(true))
+ .Field(f => f.Name("u32").DataType(UInt32Type.Default).Nullable(true))
+ .Field(f => f.Name("i64").DataType(Int64Type.Default).Nullable(true))
+ .Field(f => f.Name("u64").DataType(UInt64Type.Default).Nullable(true))
+
+ .Field(f => f.Name("f16").DataType(HalfFloatType.Default).Nullable(true))
+ .Field(f => f.Name("f32").DataType(FloatType.Default).Nullable(true))
+ .Field(f => f.Name("f64").DataType(DoubleType.Default).Nullable(true))
+
+ .Field(f => f.Name("binary").DataType(BinaryType.Default).Nullable(false))
+ .Field(f => f.Name("string").DataType(StringType.Default).Nullable(false))
+
+ .Field(f => f.Name("date32").DataType(Date32Type.Default).Nullable(false))
+ .Field(f => f.Name("date64").DataType(Date64Type.Default).Nullable(false))
+ .Field(f => f.Name("time32_s").DataType(new Time32Type(TimeUnit.Second)).Nullable(false))
+ .Field(f => f.Name("time32_ms").DataType(new Time32Type(TimeUnit.Millisecond)).Nullable(false))
+ .Field(f => f.Name("time64_us").DataType(new Time64Type(TimeUnit.Microsecond)).Nullable(false))
+ .Field(f => f.Name("time64_ns").DataType(new Time64Type(TimeUnit.Nanosecond)).Nullable(false))
+
+ .Field(f => f.Name("list_string").DataType(new ListType(StringType.Default)).Nullable(false))
+ .Field(f => f.Name("list_list_i32").DataType(new ListType(new ListType(Int32Type.Default))).Nullable(false))
+
+ .Field(f => f.Name("dict_string").DataType(new DictionaryType(Int32Type.Default, StringType.Default, false)).Nullable(false))
+ .Field(f => f.Name("dict_string_ordered").DataType(new DictionaryType(Int32Type.Default, StringType.Default, true)).Nullable(false))
+ .Field(f => f.Name("list_dict_string").DataType(new ListType(new DictionaryType(Int32Type.Default, StringType.Default, false))).Nullable(false))
+
+ .Build();
+ return schema;
+ }
+ }
+
+ public static IEnumerable<dynamic> GetPythonFields()
+ {
+ using (Py.GIL())
+ {
+ dynamic pa = Py.Import("pyarrow");
+ yield return pa.field("null", pa.GetAttr("null").Invoke(), true);
+ yield return pa.field("bool", pa.bool_(), true);
+ yield return pa.field("i8", pa.int8(), true);
+ yield return pa.field("u8", pa.uint8(), true);
+ yield return pa.field("i16", pa.int16(), true);
+ yield return pa.field("u16", pa.uint16(), true);
+ yield return pa.field("i32", pa.int32(), true);
+ yield return pa.field("u32", pa.uint32(), true);
+ yield return pa.field("i64", pa.int64(), true);
+ yield return pa.field("u64", pa.uint64(), true);
+
+ yield return pa.field("f16", pa.float16(), true);
+ yield return pa.field("f32", pa.float32(), true);
+ yield return pa.field("f64", pa.float64(), true);
+
+ yield return pa.field("binary", pa.binary(), false);
+ yield return pa.field("string", pa.utf8(), false);
+
+ yield return pa.field("date32", pa.date32(), false);
+ yield return pa.field("date64", pa.date64(), false);
+ yield return pa.field("time32_s", pa.time32("s"), false);
+ yield return pa.field("time32_ms", pa.time32("ms"), false);
+ yield return pa.field("time64_us", pa.time64("us"), false);
+ yield return pa.field("time64_ns", pa.time64("ns"), false);
+
+ yield return pa.field("list_string", pa.list_(pa.utf8()), false);
+ yield return pa.field("list_list_i32", pa.list_(pa.list_(pa.int32())), false);
+
+ yield return pa.field("dict_string", pa.dictionary(pa.int32(), pa.utf8(), false), false);
+ yield return pa.field("dict_string_ordered", pa.dictionary(pa.int32(), pa.utf8(), true), false);
+ yield return pa.field("list_dict_string", pa.list_(pa.dictionary(pa.int32(), pa.utf8(), false)), false);
+ }
+ }
+
+ public static dynamic GetPythonSchema()
+ {
+ using (Py.GIL())
+ {
+ dynamic pa = Py.Import("pyarrow");
+ return pa.schema(GetPythonFields().ToList());
+ }
+ }
+
+ // Schemas created in Python, used in CSharp
+ [Fact]
+ public void ImportType()
+ {
+ PythonEngine.Initialize();
Review Comment:
The extra calls to `PythonEngine.Initialize` look unnecessary as it's already initialized in the constructor.
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
Review Comment:
From the C Data interface docs it looks like `metadata` isn't a string but is a binary array. C# stores strings as UTF-16 so they can't store arbitrary binary data. Maybe representing this as an `IntPtr` would be most appropriate?
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+
+ public ReleaseCArrowSchema release;
+ // Check this out: https://github.com/G-Research/ParquetSharp/blob/386d91bd5e6fe6cb81583803447023c1359957c8/csharp/ParquetHandle.cs#L8
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.formatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType)
+ {
+ if (((DictionaryType)datatype).Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype)
+ {
+ if (datatype is NestedType)
+ {
+ IReadOnlyList<Field> fields = ((NestedType)datatype).Fields;
+ int numFields = fields.Count;
+
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * sizeof(IntPtr));
+
+ for (var i = 0; i < numFields; i++)
+ {
+ var cSchema = new CArrowSchema();
+ CArrowSchema.ExportField(fields[i], out cSchema);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return pointerList;
+ }
+ else
+ {
+ return (IntPtr*)IntPtr.Zero;
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType)
+ {
+ var cSchema = new CArrowSchema();
+ IArrowType valueType = ((DictionaryType)datatype).ValueType;
+ CArrowSchema.ExportDataType(valueType, out cSchema);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ /// <param name="schema">An uninitialized CArrowSchema.</param>
+ public static void ExportDataType(IArrowType datatype, out CArrowSchema schema)
Review Comment:
Is there a reason that these `Export*` methods all use an out parameter rather than just returning a `CArrowSchema`?
##########
csharp/src/Apache.Arrow/C/ImportedArrowSchema.cs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// A <see cref="CArrowSchema"/> imported from somewhere else.
+ /// </summary>
+ ///
+ /// <example>
+ /// Typically, when importing a schema we will allocate an uninitialized
+ /// <see cref="CArrowSchema"/>, pass the pointer to the foreign function,
+ /// then construct this class with the initialized pointer.
+ ///
+ /// <code>
+ /// var cSchema = new CArrowSchema();
+ /// IntPtr importedPtr = cSchema.AllocateAsPtr();
+ /// foreign_export_function(importedPtr);
+ /// var importedType = new ImportedArrowSchema(importedPtr);
+ /// ArrowType arrowType = importedType.GetAsType();
+ /// <code>
+ /// </example>
+ public sealed class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema _data;
+ private readonly IntPtr _handle;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(IntPtr handle)
+ {
+ _data = Marshal.PtrToStructure<CArrowSchema>(handle);
+ if (_data.release == null)
+ {
+ throw new Exception("Tried to import a schema that has already been released.");
+ }
+ _handle = handle;
+ _isRoot = true;
+ }
+
+ private ImportedArrowSchema(IntPtr handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot)
+ {
+ _data.release(_handle);
+ }
+ }
+
+ public ArrowType GetAsType()
+ {
+ if (_data.dictionary != IntPtr.Zero)
+ {
+ ArrowType indices_type = _data.format switch
Review Comment:
```suggestion
ArrowType indicesType = _data.format switch
```
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,519 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+using System;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+
+ public ReleaseCArrowSchema release;
+ // Check this out: https://github.com/G-Research/ParquetSharp/blob/386d91bd5e6fe6cb81583803447023c1359957c8/csharp/ParquetHandle.cs#L8
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.format_string;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ARROW_FLAG_NULLABLE;
+ }
+
+ if (datatype is DictionaryType)
+ {
+ if (((DictionaryType)datatype).Ordered)
+ {
+ flags |= ARROW_FLAG_DICTIONARY_ORDERED;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype)
+ {
+ if (datatype is NestedType)
+ {
+ var fields = ((NestedType)datatype).Fields;
+ int n_fields = fields.Count;
+
+ IntPtr* pointer_list = (IntPtr*)Marshal.AllocHGlobal(n_fields * sizeof(IntPtr));
+
+ for (var i = 0; i < n_fields; i++)
+ {
+ var c_schema = new CArrowSchema();
+ CArrowSchema.ExportField(fields[i], out c_schema);
+ IntPtr exported_schema = c_schema.AllocateAsPtr();
+ pointer_list[i] = exported_schema;
+ }
+
+ return pointer_list;
+ }
+ else
+ {
+ return (IntPtr*)IntPtr.Zero;
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType)
+ {
+ var c_schema = new CArrowSchema();
+ var value_type = ((DictionaryType)datatype).ValueType;
+ CArrowSchema.ExportDataType(value_type, out c_schema);
+ return c_schema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ /// <param name="schema">An uninitialized CArrowSchema.</param>
+ public static void ExportDataType(IArrowType datatype, out CArrowSchema schema)
+ {
+ schema.format = GetFormat(datatype);
+ schema.name = null;
+ schema.metadata = null;
+ schema.flags = GetFlags(datatype);
+
+ schema.children = ConstructChildren(datatype);
+ schema.n_children = datatype is NestedType ? ((NestedType)datatype).Fields.Count : 0;
Review Comment:
You could pass the schema by reference instead:
```C#
void SetChildren(datatype, ref CArrowSchema schema);
```
Or another alternative:
```C#
schema.children = ConstructChildren(datatype, out var numChildren);
schema.children = numChildren
```
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+
+ public ReleaseCArrowSchema release;
+ // Check this out: https://github.com/G-Research/ParquetSharp/blob/386d91bd5e6fe6cb81583803447023c1359957c8/csharp/ParquetHandle.cs#L8
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.formatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType)
+ {
+ if (((DictionaryType)datatype).Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype)
+ {
+ if (datatype is NestedType)
+ {
+ IReadOnlyList<Field> fields = ((NestedType)datatype).Fields;
+ int numFields = fields.Count;
+
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * sizeof(IntPtr));
+
+ for (var i = 0; i < numFields; i++)
+ {
+ var cSchema = new CArrowSchema();
+ CArrowSchema.ExportField(fields[i], out cSchema);
Review Comment:
You can inline the variable declaration, and there's no need to include the class name when calling a static method from within the class:
```suggestion
ExportField(fields[i], out CArrowSchema cSchema);
```
or
```suggestion
ExportField(fields[i], out var cSchema);
```
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+
+ public ReleaseCArrowSchema release;
+ // Check this out: https://github.com/G-Research/ParquetSharp/blob/386d91bd5e6fe6cb81583803447023c1359957c8/csharp/ParquetHandle.cs#L8
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.formatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType)
+ {
+ if (((DictionaryType)datatype).Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype)
+ {
+ if (datatype is NestedType)
+ {
+ IReadOnlyList<Field> fields = ((NestedType)datatype).Fields;
+ int numFields = fields.Count;
+
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * sizeof(IntPtr));
+
+ for (var i = 0; i < numFields; i++)
+ {
+ var cSchema = new CArrowSchema();
+ CArrowSchema.ExportField(fields[i], out cSchema);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return pointerList;
+ }
+ else
+ {
+ return (IntPtr*)IntPtr.Zero;
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType)
+ {
+ var cSchema = new CArrowSchema();
+ IArrowType valueType = ((DictionaryType)datatype).ValueType;
+ CArrowSchema.ExportDataType(valueType, out cSchema);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ /// <param name="schema">An uninitialized CArrowSchema.</param>
+ public static void ExportDataType(IArrowType datatype, out CArrowSchema schema)
+ {
+ schema.format = GetFormat(datatype);
+ schema.name = null;
+ schema.metadata = null;
+ schema.flags = GetFlags(datatype);
+
+ schema.children = ConstructChildren(datatype);
+ schema.n_children = datatype is NestedType ? ((NestedType)datatype).Fields.Count : 0;
+
+ schema.dictionary = ConstructDictionary(datatype);
+
+ schema.release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.children[i]);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ schema.private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ /// <param name="schema">An uninitialized CArrowSchema.</param>
+ public static void ExportField(Field field, out CArrowSchema schema)
+ {
+ ExportDataType(field.DataType, out schema);
+ schema.name = field.Name;
+ // TODO: field metadata
+ schema.metadata = null;
+ schema.flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ /// <param name="outSchema">An uninitialized CArrowSchema</param>
+ public static void ExportSchema(Schema schema, out CArrowSchema outSchema)
+ {
+ // TODO: top-level metadata
+ var structType = new StructType(schema.Fields.Values.ToList());
+ ExportDataType(structType, out outSchema);
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public IntPtr Export(IntPtr ptr)
+ {
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ public const int ArrowFlagDictionaryOrdered = 1;
+ public const int ArrowFlagNullable = 2;
+ public const int ArrowFlagMapKeysSorted = 4;
+
+ private class TypeFormatter :
+ IArrowTypeVisitor<NullType>,
+ IArrowTypeVisitor<BooleanType>,
+ IArrowTypeVisitor<Int8Type>,
+ IArrowTypeVisitor<Int16Type>,
+ IArrowTypeVisitor<Int32Type>,
+ IArrowTypeVisitor<Int64Type>,
+ IArrowTypeVisitor<UInt8Type>,
+ IArrowTypeVisitor<UInt16Type>,
+ IArrowTypeVisitor<UInt32Type>,
+ IArrowTypeVisitor<UInt64Type>,
+ IArrowTypeVisitor<HalfFloatType>,
+ IArrowTypeVisitor<FloatType>,
+ IArrowTypeVisitor<DoubleType>,
+ IArrowTypeVisitor<Decimal128Type>,
+ IArrowTypeVisitor<Decimal256Type>,
+ IArrowTypeVisitor<Date32Type>,
+ IArrowTypeVisitor<Date64Type>,
+ IArrowTypeVisitor<Time32Type>,
+ IArrowTypeVisitor<Time64Type>,
+ IArrowTypeVisitor<TimestampType>,
+ IArrowTypeVisitor<StringType>,
+ IArrowTypeVisitor<BinaryType>,
+ IArrowTypeVisitor<FixedSizeBinaryType>,
+ IArrowTypeVisitor<ListType>,
+ IArrowTypeVisitor<StructType>,
+ IArrowTypeVisitor<DictionaryType>
+ {
+ public string formatString;
Review Comment:
```suggestion
public string FormatString;
```
##########
csharp/src/Apache.Arrow/C/ImportedArrowSchema.cs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// A <see cref="CArrowSchema"/> imported from somewhere else.
+ /// </summary>
+ ///
+ /// <example>
+ /// Typically, when importing a schema we will allocate an uninitialized
+ /// <see cref="CArrowSchema"/>, pass the pointer to the foreign function,
+ /// then construct this class with the initialized pointer.
+ ///
+ /// <code>
+ /// var cSchema = new CArrowSchema();
+ /// IntPtr importedPtr = cSchema.AllocateAsPtr();
+ /// foreign_export_function(importedPtr);
+ /// var importedType = new ImportedArrowSchema(importedPtr);
+ /// ArrowType arrowType = importedType.GetAsType();
+ /// <code>
+ /// </example>
+ public sealed class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema _data;
+ private readonly IntPtr _handle;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(IntPtr handle)
+ {
+ _data = Marshal.PtrToStructure<CArrowSchema>(handle);
+ if (_data.release == null)
+ {
+ throw new Exception("Tried to import a schema that has already been released.");
+ }
+ _handle = handle;
+ _isRoot = true;
+ }
+
+ private ImportedArrowSchema(IntPtr handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot)
+ {
+ _data.release(_handle);
+ }
+ }
+
+ public ArrowType GetAsType()
+ {
+ if (_data.dictionary != IntPtr.Zero)
+ {
+ ArrowType indices_type = _data.format switch
+ {
+ "c" => new Int8Type(),
+ "C" => new UInt8Type(),
+ "s" => new Int16Type(),
+ "S" => new UInt16Type(),
+ "i" => new Int32Type(),
+ "I" => new UInt32Type(),
+ "l" => new Int64Type(),
+ "L" => new UInt64Type(),
+ _ => throw new InvalidDataException($"Indices must be an integer, but got format string {_data.format}"),
+ };
+
+ var dictionarySchema = new ImportedArrowSchema(_data.dictionary, /*is_root*/ false);
+ ArrowType dictionaryType = dictionarySchema.GetAsType();
+
+ bool ordered = (_data.flags & CArrowSchema.ArrowFlagNullable) == CArrowSchema.ArrowFlagNullable;
+
+ return new DictionaryType(indices_type, dictionaryType, ordered);
+ }
+
+ // Special handling for nested types
+ if (_data.format == "+l")
+ {
+ if (_data.n_children != 1)
+ {
+ throw new Exception("Expected list type to have exactly one child.");
+ }
+ ImportedArrowSchema childSchema;
+ unsafe
+ {
+ if (_data.children[0] == null)
+ {
+ throw new Exception("Expected list type child to be non-null.");
+ }
+ childSchema = new ImportedArrowSchema(_data.children[0]);
+ }
+
+ Field childField = childSchema.GetAsField();
+
+ return new ListType(childField);
+ }
+ else if (_data.format == "+s")
+ {
+ var child_schemas = new ImportedArrowSchema[_data.n_children];
+ unsafe
+ {
+ for (int i = 0; i < _data.n_children; i++)
+ {
+ if (_data.children[i] == null)
+ {
+ throw new Exception("Expected struct type child to be non-null.");
+ }
+ child_schemas[i] = new ImportedArrowSchema(_data.children[i]);
+ }
+
+ }
+
+ List<Field> childFields = child_schemas.Select(schema => schema.GetAsField()).ToList();
+
+ return new StructType(childFields);
+ }
+ // TODO: Map type and large list type
+
+ return _data.format switch
+ {
+ // Primitives
+ "n" => new NullType(),
+ "b" => new BooleanType(),
+ "c" => new Int8Type(),
+ "C" => new UInt8Type(),
+ "s" => new Int16Type(),
+ "S" => new UInt16Type(),
+ "i" => new Int32Type(),
+ "I" => new UInt32Type(),
+ "l" => new Int64Type(),
+ "L" => new UInt64Type(),
+ "e" => new HalfFloatType(),
+ "f" => new FloatType(),
+ "g" => new DoubleType(),
+ // Binary data
+ "z" => new BinaryType(),
+ //"Z" => new LargeBinaryType() // Not yet implemented
+ "u" => new StringType(),
+ //"U" => new LargeStringType(), // Not yet implemented
+ // TODO: decimal
+ // TODO: fixed-width binary
+ // Date and time
+ "tdD" => new Date32Type(),
+ "tdm" => new Date64Type(),
+ "tts" => new Time32Type(TimeUnit.Second),
+ "ttm" => new Time32Type(TimeUnit.Millisecond),
+ "ttu" => new Time64Type(TimeUnit.Microsecond),
+ "ttn" => new Time64Type(TimeUnit.Nanosecond),
+ // TODO: timestamp with timezone,
+ // TODO: duration not yet implemented
+ "tiM" => new IntervalType(IntervalUnit.YearMonth),
+ "tiD" => new IntervalType(IntervalUnit.DayTime),
+ //"tin" => new IntervalType(IntervalUnit.MonthDayNanosecond), // Not yet implemented
+ _ => throw new NotSupportedException("Data type is not yet supported in import.")
+ };
+ }
+
+ public Field GetAsField()
+ {
+ string fieldName = string.IsNullOrEmpty(_data.name) ? "" : _data.name;
+
+ bool nullable = (_data.flags & CArrowSchema.ArrowFlagNullable) == CArrowSchema.ArrowFlagNullable;
+
+ return new Field(fieldName, GetAsType(), nullable);
+ }
+
+ public Schema GetAsSchema()
+ {
+ ArrowType fullType = GetAsType();
+ if (fullType is StructType)
+ {
+ StructType structType = (StructType)fullType;
Review Comment:
```suggestion
if (fullType is StructType structType)
{
```
##########
csharp/src/Apache.Arrow/C/ImportedArrowSchema.cs:
##########
@@ -0,0 +1,205 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// A <see cref="CArrowSchema"/> imported from somewhere else.
+ /// </summary>
+ ///
+ /// <example>
+ /// Typically, when importing a schema we will allocate an uninitialized
+ /// <see cref="CArrowSchema"/>, pass the pointer to the foreign function,
+ /// then construct this class with the initialized pointer.
+ ///
+ /// <code>
+ /// var cSchema = new CArrowSchema();
+ /// IntPtr importedPtr = cSchema.AllocateAsPtr();
+ /// foreign_export_function(importedPtr);
+ /// var importedType = new ImportedArrowSchema(importedPtr);
Review Comment:
```suggestion
/// using var importedType = new ImportedArrowSchema(importedPtr);
```
? Although then you need to be careful that you don't free the pointer before `Dispose` is called, so maybe it's better to do:
```c#
using (var importedType = new ImportedArrowSchema(importedPtr)) {
ArrowType arrowType = importedType.GetAsType();
}
```
It looks like we no longer need the data after we have a C# field/schema/type though, so could this class be replaced with a set of static methods that could be used something like:
```C#
ArrowType type = CDataInterface.ImportType(importedPtr);
```
That should simplify the API and make it less likely we try to call release after the pointer data has been freed.
##########
csharp/test/Apache.Arrow.Tests/CDataInterfaceSchemaTests.cs:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.C;
+using Apache.Arrow.Types;
+using Python.Runtime;
+using Xunit;
+
+namespace Apache.Arrow.Tests
+{
+ public class CDataSchemaTest
+ {
+ public CDataSchemaTest()
+ {
+ PythonEngine.Initialize();
+ }
+
+ public static Schema GetTestSchema()
+ {
+ // TODO: Add more types
+ using (Py.GIL())
+ {
+ var schema = new Schema.Builder()
+ .Field(f => f.Name("null").DataType(NullType.Default).Nullable(true))
+ .Field(f => f.Name("bool").DataType(BooleanType.Default).Nullable(true))
+ .Field(f => f.Name("i8").DataType(Int8Type.Default).Nullable(true))
+ .Field(f => f.Name("u8").DataType(UInt8Type.Default).Nullable(true))
+ .Field(f => f.Name("i16").DataType(Int16Type.Default).Nullable(true))
+ .Field(f => f.Name("u16").DataType(UInt16Type.Default).Nullable(true))
+ .Field(f => f.Name("i32").DataType(Int32Type.Default).Nullable(true))
+ .Field(f => f.Name("u32").DataType(UInt32Type.Default).Nullable(true))
+ .Field(f => f.Name("i64").DataType(Int64Type.Default).Nullable(true))
+ .Field(f => f.Name("u64").DataType(UInt64Type.Default).Nullable(true))
+
+ .Field(f => f.Name("f16").DataType(HalfFloatType.Default).Nullable(true))
+ .Field(f => f.Name("f32").DataType(FloatType.Default).Nullable(true))
+ .Field(f => f.Name("f64").DataType(DoubleType.Default).Nullable(true))
+
+ .Field(f => f.Name("binary").DataType(BinaryType.Default).Nullable(false))
+ .Field(f => f.Name("string").DataType(StringType.Default).Nullable(false))
+
+ .Field(f => f.Name("date32").DataType(Date32Type.Default).Nullable(false))
+ .Field(f => f.Name("date64").DataType(Date64Type.Default).Nullable(false))
+ .Field(f => f.Name("time32_s").DataType(new Time32Type(TimeUnit.Second)).Nullable(false))
+ .Field(f => f.Name("time32_ms").DataType(new Time32Type(TimeUnit.Millisecond)).Nullable(false))
+ .Field(f => f.Name("time64_us").DataType(new Time64Type(TimeUnit.Microsecond)).Nullable(false))
+ .Field(f => f.Name("time64_ns").DataType(new Time64Type(TimeUnit.Nanosecond)).Nullable(false))
+
+ .Field(f => f.Name("list_string").DataType(new ListType(StringType.Default)).Nullable(false))
+ .Field(f => f.Name("list_list_i32").DataType(new ListType(new ListType(Int32Type.Default))).Nullable(false))
+
+ .Field(f => f.Name("dict_string").DataType(new DictionaryType(Int32Type.Default, StringType.Default, false)).Nullable(false))
+ .Field(f => f.Name("dict_string_ordered").DataType(new DictionaryType(Int32Type.Default, StringType.Default, true)).Nullable(false))
+ .Field(f => f.Name("list_dict_string").DataType(new ListType(new DictionaryType(Int32Type.Default, StringType.Default, false))).Nullable(false))
+
+ .Build();
+ return schema;
+ }
+ }
+
+ public static IEnumerable<dynamic> GetPythonFields()
+ {
+ using (Py.GIL())
+ {
+ dynamic pa = Py.Import("pyarrow");
+ yield return pa.field("null", pa.GetAttr("null").Invoke(), true);
+ yield return pa.field("bool", pa.bool_(), true);
+ yield return pa.field("i8", pa.int8(), true);
+ yield return pa.field("u8", pa.uint8(), true);
+ yield return pa.field("i16", pa.int16(), true);
+ yield return pa.field("u16", pa.uint16(), true);
+ yield return pa.field("i32", pa.int32(), true);
+ yield return pa.field("u32", pa.uint32(), true);
+ yield return pa.field("i64", pa.int64(), true);
+ yield return pa.field("u64", pa.uint64(), true);
+
+ yield return pa.field("f16", pa.float16(), true);
+ yield return pa.field("f32", pa.float32(), true);
+ yield return pa.field("f64", pa.float64(), true);
+
+ yield return pa.field("binary", pa.binary(), false);
+ yield return pa.field("string", pa.utf8(), false);
+
+ yield return pa.field("date32", pa.date32(), false);
+ yield return pa.field("date64", pa.date64(), false);
+ yield return pa.field("time32_s", pa.time32("s"), false);
+ yield return pa.field("time32_ms", pa.time32("ms"), false);
+ yield return pa.field("time64_us", pa.time64("us"), false);
+ yield return pa.field("time64_ns", pa.time64("ns"), false);
+
+ yield return pa.field("list_string", pa.list_(pa.utf8()), false);
+ yield return pa.field("list_list_i32", pa.list_(pa.list_(pa.int32())), false);
+
+ yield return pa.field("dict_string", pa.dictionary(pa.int32(), pa.utf8(), false), false);
+ yield return pa.field("dict_string_ordered", pa.dictionary(pa.int32(), pa.utf8(), true), false);
+ yield return pa.field("list_dict_string", pa.list_(pa.dictionary(pa.int32(), pa.utf8(), false)), false);
+ }
+ }
+
+ public static dynamic GetPythonSchema()
+ {
+ using (Py.GIL())
+ {
+ dynamic pa = Py.Import("pyarrow");
+ return pa.schema(GetPythonFields().ToList());
+ }
+ }
+
+ // Schemas created in Python, used in CSharp
+ [Fact]
+ public void ImportType()
+ {
+ PythonEngine.Initialize();
+ Schema schema = GetTestSchema();
+ IEnumerable<dynamic> pyFields = GetPythonFields();
+
+ foreach ((Field field, dynamic pyField) in schema.Fields.Values.AsEnumerable()
+ .Zip(pyFields))
+ {
+ var cSchema = new CArrowSchema();
+ IntPtr importedPtr = cSchema.AllocateAsPtr();
+
+ using (Py.GIL())
+ {
+ dynamic py_datatype = pyField.type;
Review Comment:
```suggestion
dynamic pyDatatype = pyField.type;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1121013836
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
Review Comment:
What about [`IntPtr.Size`](https://learn.microsoft.com/en-us/dotnet/api/system.intptr.size?view=net-7.0#system-intptr-size)?
```
Marshal.AllocHGlobal(numFields * IntPtr.Size);
```
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
Review Comment:
Why `StdCall`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1120682053
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+
+ public ReleaseCArrowSchema release;
+ // Check this out: https://github.com/G-Research/ParquetSharp/blob/386d91bd5e6fe6cb81583803447023c1359957c8/csharp/ParquetHandle.cs#L8
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.formatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType)
+ {
+ if (((DictionaryType)datatype).Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype)
+ {
+ if (datatype is NestedType)
+ {
+ IReadOnlyList<Field> fields = ((NestedType)datatype).Fields;
Review Comment:
Thanks, that is nifty.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1124965225
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
Review Comment:
We should use function pointers here, not managed delegates.
https://learn.microsoft.com/en-us/dotnet/csharp/language-reference/unsafe-code#function-pointers
(Coding in GitHub, I'm not 100% sure this code will compile/run, but hopefully you get the idea):
```suggestion
[MarshalAs(UnmanagedType.FunctionPtr)]
public delegate* unmanaged<CArrowSchema*, void> release;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1124987830
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
Review Comment:
That makes sense. The syntax here is for C# 9.0 and above, so I'll need to figure out the equivalent for C# 8.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1138787644
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
Review Comment:
Here, we're implicitly assuming that `foreign_import_function` called `release` already and is done with it. Because we allocated the `CArrowSchema*` pointer, we must free it. It's supposed to be that when exporting the consumer will allocate the `CArrowSchema*`, but that seems impossible in cases like Python.
Although when we get to exporting Arrays, I'm not sure how the lifetime is handled. Because the buffers do need to be kept alive, so the call to `Free` in that case needs to be deferred until after the Python process is done with it. But how does C# know when that is?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #34133: GH-34737: [C#] C Data interface for schemas and types
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1210661356
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
Review Comment:
@westonpace Arrow C++ does build on 32-bit Linux (we have nightly crossbow tests for that).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1148422940
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchemaImporter.ImportSchema(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Schema ImportSchema(CArrowSchema* ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ private sealed unsafe class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema* _data;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(CArrowSchema* handle)
+ {
+ _data = (CArrowSchema*)handle;
+ if (_data->release == null)
+ {
+ throw new ArgumentException("Tried to import a schema that has already been released.");
+ }
+ _isRoot = true;
+ }
+
+ public ImportedArrowSchema(CArrowSchema* handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot && _data->release != null)
+ {
+ _data->release(_data);
Review Comment:
I have 2 questions about this:
> they need to know when you are finished using those resources.
1. Isn't the fact that the function call returned to the caller enough to know that I'm finished using those resources?
> it may decrement a dedicated reference count
2. How do I "increment a dedicated reference count"? For example, if I received a `CArrowSchema*` from somewhere, and I need to pass it into a C# Importer and a C++ importer method. How could I tell the `CArrowSchema*` that it needs to allow `release` to be called twice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #34133:
URL: https://github.com/apache/arrow/pull/34133#issuecomment-1458960023
I have refactored to make `CArrowSchema` simply use real pointers for its members, and separated out the import and export functions to their own classes. So now instead of passing around `IntPtr`, users will be passing around `CArrowSchema*`. I like that this makes the API more strongly typed, as IntPtr drops the information about what type it is pointing to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1133025208
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+
+ ptr->format = null;
+ ptr->name = null;
+ ptr->metadata = null;
+ ptr->flags = 0;
+ ptr->n_children = 0;
+ ptr->children = null;
+ ptr->dictionary = null;
+ ptr->release = null;
+ ptr->private_data = null;
+
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
Review Comment:
That should be referencing `Create`. Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1133026715
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchemaImporter.ImportSchema(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Schema ImportSchema(CArrowSchema* ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ private sealed unsafe class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema* _data;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(CArrowSchema* handle)
+ {
+ _data = (CArrowSchema*)handle;
+ if (_data->release == null)
+ {
+ throw new ArgumentException("Tried to import a schema that has already been released.");
+ }
+ _isRoot = true;
+ }
+
+ public ImportedArrowSchema(CArrowSchema* handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot && _data->release != null)
+ {
+ _data->release(_data);
Review Comment:
I've added a remark to these methods to indicate the release callback will be called. And I've added a unit test to make sure that it is called, even if the import fails.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] adamreeve commented on pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on PR #34133:
URL: https://github.com/apache/arrow/pull/34133#issuecomment-1450735965
> Thanks for the feedback @adamreeve. I've realized I could simplify the API by changing the `Export*` methods into constructors for `CArrowSchema`. And I've moved the `ImportedArrowSchema` into an internal class within `CArrowSchema` so that it can handle the common import and disposal logic. Let me know what you think.
:+1: nice, the new API looks a lot more straightforward to work with
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1142814166
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchemaImporter.ImportSchema(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Schema ImportSchema(CArrowSchema* ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ private sealed unsafe class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema* _data;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(CArrowSchema* handle)
+ {
+ _data = (CArrowSchema*)handle;
+ if (_data->release == null)
+ {
+ throw new ArgumentException("Tried to import a schema that has already been released.");
+ }
+ _isRoot = true;
+ }
+
+ public ImportedArrowSchema(CArrowSchema* handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot && _data->release != null)
+ {
+ _data->release(_data);
Review Comment:
> I feel this is a bad pattern - releasing resources that you don't own/didn't create.
To be clear: releasing doesn't necessarily mean freeing memory. It could also mean decrementing a reference count on a shared pointer. The release callback is just a way for the consumer to inform the producer that it is no longer using those resources. If the producer also isn't using those resources, it will free them. That's commonly the case with `ArrowSchema`, since it's all just small temporary values we created for exchange. But in other cases it might keep them around. That will be the case with `ArrowArray`, where the buffers will be behind reference counted pointers that the producer might still have references to.
> It would be really surprising to me that I couldn't get a CArrowSchema* from somewhere, and I couldn't call CArrowSchemaImporter.ImportSchema multiple times on it.
I think it's basically just a trade off. Either you can only import once, and the release callback is handled for you. Or you are allowed to import multiple times, but you are responsible for also calling the release callback (or else risking memory leaks). It seems the current design expects that most situations people only care to import something once, and that it's more important to prevent accidental memory leaks than it is support multiple imports.
> Can you show me in the code where release is called?
Indeed it's a little complicated:
1. When you import, you create a `SchemaImporter`: https://github.com/apache/arrow/blob/08fb86107b001e2e1903ae0e416ff51386d4239e/cpp/src/arrow/c/bridge.cc#L1212-L1213
2. When the schema importer is passed the `ArrowSchema`, it passes that to an `ExportGuard` https://github.com/apache/arrow/blob/08fb86107b001e2e1903ae0e416ff51386d4239e/cpp/src/arrow/c/bridge.cc#L831
3. When the `SchemaImporter` destructor is called, the `ExportGuard` destructor is called, and that calls the release callback https://github.com/apache/arrow/blob/08fb86107b001e2e1903ae0e416ff51386d4239e/cpp/src/arrow/c/util_internal.h#L61
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1142700326
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchemaImporter.ImportSchema(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Schema ImportSchema(CArrowSchema* ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ private sealed unsafe class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema* _data;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(CArrowSchema* handle)
+ {
+ _data = (CArrowSchema*)handle;
+ if (_data->release == null)
+ {
+ throw new ArgumentException("Tried to import a schema that has already been released.");
+ }
+ _isRoot = true;
+ }
+
+ public ImportedArrowSchema(CArrowSchema* handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot && _data->release != null)
+ {
+ _data->release(_data);
Review Comment:
Can you show me in the code where `release` is called? I'm not seeing the code for it. (Yes, I do see the comment, but I don't see the backing code, nor the place in https://arrow.apache.org/docs/format/CDataInterface.html where it says that when you "import" a C Data schema object, the incoming pointer is released.
I feel this is a bad pattern - releasing resources that you don't own/didn't create. It would be really surprising to me that I couldn't get a `CArrowSchema*` from somewhere, and I couldn't call `CArrowSchemaImporter.ImportSchema` multiple times on it.
cc @pitrou who has the "git blame" on the above bridge.h file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1138852146
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
Review Comment:
https://github.com/apache/arrow/blob/427f1b649e36ef3c64dd61ed86808463a440eb94/cpp/src/arrow/c/helpers.h#L70-L75
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1140740742
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
Review Comment:
I think there are two hard rules:
* `CArrowSchema` should only be free'd if `CArrowSchema.release` is null.
* `CArrowSchema.release` should only be set to null when either the release callback is called or the data is moved. (So I think that line you added is not ideal)
If the `foreign_import_function` doesn't set it null, then a different pattern should be used. For example, the foreign importer should be the one to allocate the CArrowSchema and be responsible for freeing it. I think this is the reason that it's intended that the consumers allocate and own `CArrowSchema`, rather than the producer. But when interacting with R and Python, it's sometimes convenient to have the C++/Rust/... producer make these structs, so the move pattern is nice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #34133: GH-34737: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #34133:
URL: https://github.com/apache/arrow/pull/34133#issuecomment-1490820937
@eerhardt I think we can ignore that error for now. The test is flakey and we are still looking into the root cause.
https://github.com/apache/arrow/issues/34743
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] ursabot commented on pull request #34133: GH-34737: [C#] C Data interface for schemas and types
Posted by "ursabot (via GitHub)" <gi...@apache.org>.
ursabot commented on PR #34133:
URL: https://github.com/apache/arrow/pull/34133#issuecomment-1495398512
Benchmark runs are scheduled for baseline = 7c307b14dccdfb5852cbafb48635df73f03f16c9 and contender = f02d35119ae6bd31ae02f776ac00fc27b80f42da. f02d35119ae6bd31ae02f776ac00fc27b80f42da is a master commit associated with this PR. Results will be available as each benchmark for each run completes.
Conbench compare runs links:
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ec2-t3-xlarge-us-east-2](https://conbench.ursa.dev/compare/runs/2897688431f94da69efdc95f82d3cf0e...ba45b8a5d2fc49e2837ff0a4bafd15fc/)
[Failed :arrow_down:0.0% :arrow_up:0.0%] [test-mac-arm](https://conbench.ursa.dev/compare/runs/28c349306e994fa39b07e9f57649e707...a5117e4992be41aa8d9c796e132291d2/)
[Finished :arrow_down:0.0% :arrow_up:0.0%] [ursa-i9-9960x](https://conbench.ursa.dev/compare/runs/161927f948814f4dae280cbfcd05ee94...fd8bb796d15b417ab27537e831fbc1ba/)
[Failed :arrow_down:0.0% :arrow_up:0.0%] [ursa-thinkcentre-m75q](https://conbench.ursa.dev/compare/runs/5779f137a5ae46cb9404a5fae7624304...6fb68170d8834c44893dd2fa4616daf2/)
Buildkite builds:
[Finished] [`f02d3511` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2620)
[Failed] [`f02d3511` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2652)
[Finished] [`f02d3511` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2618)
[Failed] [`f02d3511` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2643)
[Finished] [`7c307b14` ec2-t3-xlarge-us-east-2](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ec2-t3-xlarge-us-east-2/builds/2619)
[Failed] [`7c307b14` test-mac-arm](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-test-mac-arm/builds/2651)
[Finished] [`7c307b14` ursa-i9-9960x](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-i9-9960x/builds/2617)
[Failed] [`7c307b14` ursa-thinkcentre-m75q](https://buildkite.com/apache-arrow/arrow-bci-benchmark-on-ursa-thinkcentre-m75q/builds/2642)
Supported benchmarks:
ec2-t3-xlarge-us-east-2: Supported benchmark langs: Python, R. Runs only benchmarks with cloud = True
test-mac-arm: Supported benchmark langs: C++, Python, R
ursa-i9-9960x: Supported benchmark langs: Python, R, JavaScript
ursa-thinkcentre-m75q: Supported benchmark langs: C++, Java
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #34133:
URL: https://github.com/apache/arrow/pull/34133#issuecomment-1448992357
Thanks for the feedback @adamreeve. I've realized I could simplify the API by changing the `Export*` methods into constructors for `CArrowSchema`. And I've moved the `ImportedArrowSchema` into an internal class within `CArrowSchema` so that it can handle the common import and disposal logic. Let me know what you think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #34133: GH-34737: [C#] C Data interface for schemas and types
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34133:
URL: https://github.com/apache/arrow/pull/34133#issuecomment-1484382223
* Closes: #34737
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-34737: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1148720269
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,278 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportType(IArrowType datatype, CArrowSchema* schema)
+ {
+ if (datatype == null)
+ {
+ throw new ArgumentNullException(nameof(datatype));
+ }
+ if (schema == null)
+ {
+ throw new ArgumentNullException(nameof(schema));
+ }
+ if (schema->release != null)
+ {
+ throw new ArgumentException("Cannot export schema to a struct that is already initialized.");
+ }
+
+ schema->format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ schema->name = null;
+ schema->metadata = null;
+ schema->flags = GetFlags(datatype);
+
+ schema->children = ConstructChildren(datatype, out var numChildren);
+ schema->n_children = numChildren;
+
+ schema->dictionary = ConstructDictionary(datatype);
+
+ schema->release = (delegate* unmanaged[Stdcall]<CArrowSchema*, void>)Marshal.GetFunctionPointerForDelegate(
+ ReleaseCArrowSchema);
+
+ schema->private_data = null;
+ }
+
+ /// <summary>
+ /// Export a field to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="field">The field to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(field, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportField(Field field, CArrowSchema* schema)
+ {
+ ExportType(field.DataType, schema);
+ schema->name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
Review Comment:
I've created an umbrella issue to track the remaining C data interface tasks (https://github.com/apache/arrow/issues/33856) and another one to track remaining data types to implement (https://github.com/apache/arrow/issues/34736).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #34133: GH-33856: [C#] c data interface for schemas and types
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34133:
URL: https://github.com/apache/arrow/pull/34133#issuecomment-1426416928
:warning: GitHub issue #33856 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] kou commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "kou (via GitHub)" <gi...@apache.org>.
kou commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1117982638
##########
ci/scripts/csharp_test.sh:
##########
@@ -21,6 +21,17 @@ set -ex
source_dir=${1}/csharp
+# Python and PyArrow are required for C Data Interface tests.
+if [ -z "${PYTHON}" ]; then
+ if [ ! which python3 > /dev/null 2>&1 ]; then
+ export PYTHON=python
+ else
+ export PYTHON=python3
+ fi
Review Comment:
How about simplifying this?
```suggestion
if which python3 > /dev/null 2>&1; then
export PYTHON=python3
else
export PYTHON=python
fi
```
BTW, `type` is more portable than `which`.
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,519 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+using System;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+
+ public ReleaseCArrowSchema release;
+ // Check this out: https://github.com/G-Research/ParquetSharp/blob/386d91bd5e6fe6cb81583803447023c1359957c8/csharp/ParquetHandle.cs#L8
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.format_string;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ARROW_FLAG_NULLABLE;
+ }
+
+ if (datatype is DictionaryType)
+ {
+ if (((DictionaryType)datatype).Ordered)
+ {
+ flags |= ARROW_FLAG_DICTIONARY_ORDERED;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype)
+ {
+ if (datatype is NestedType)
+ {
+ var fields = ((NestedType)datatype).Fields;
+ int n_fields = fields.Count;
+
+ IntPtr* pointer_list = (IntPtr*)Marshal.AllocHGlobal(n_fields * sizeof(IntPtr));
+
+ for (var i = 0; i < n_fields; i++)
+ {
+ var c_schema = new CArrowSchema();
+ CArrowSchema.ExportField(fields[i], out c_schema);
+ IntPtr exported_schema = c_schema.AllocateAsPtr();
+ pointer_list[i] = exported_schema;
+ }
+
+ return pointer_list;
+ }
+ else
+ {
+ return (IntPtr*)IntPtr.Zero;
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType)
+ {
+ var c_schema = new CArrowSchema();
+ var value_type = ((DictionaryType)datatype).ValueType;
+ CArrowSchema.ExportDataType(value_type, out c_schema);
+ return c_schema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ /// <param name="schema">An uninitialized CArrowSchema.</param>
+ public static void ExportDataType(IArrowType datatype, out CArrowSchema schema)
+ {
+ schema.format = GetFormat(datatype);
+ schema.name = null;
+ schema.metadata = null;
+ schema.flags = GetFlags(datatype);
+
+ schema.children = ConstructChildren(datatype);
+ schema.n_children = datatype is NestedType ? ((NestedType)datatype).Fields.Count : 0;
Review Comment:
How about setting them at once in `ConstructChildren()` because they are related?
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,519 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+using System;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+
+ public ReleaseCArrowSchema release;
+ // Check this out: https://github.com/G-Research/ParquetSharp/blob/386d91bd5e6fe6cb81583803447023c1359957c8/csharp/ParquetHandle.cs#L8
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.format_string;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ARROW_FLAG_NULLABLE;
+ }
+
+ if (datatype is DictionaryType)
+ {
+ if (((DictionaryType)datatype).Ordered)
+ {
+ flags |= ARROW_FLAG_DICTIONARY_ORDERED;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype)
+ {
+ if (datatype is NestedType)
+ {
+ var fields = ((NestedType)datatype).Fields;
+ int n_fields = fields.Count;
+
+ IntPtr* pointer_list = (IntPtr*)Marshal.AllocHGlobal(n_fields * sizeof(IntPtr));
+
+ for (var i = 0; i < n_fields; i++)
+ {
+ var c_schema = new CArrowSchema();
+ CArrowSchema.ExportField(fields[i], out c_schema);
+ IntPtr exported_schema = c_schema.AllocateAsPtr();
+ pointer_list[i] = exported_schema;
+ }
+
+ return pointer_list;
+ }
+ else
+ {
+ return (IntPtr*)IntPtr.Zero;
Review Comment:
I'm not familiar with C# but is it safe?
Is there `nullptr` in C#?
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,519 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+using System;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+
+ public ReleaseCArrowSchema release;
+ // Check this out: https://github.com/G-Research/ParquetSharp/blob/386d91bd5e6fe6cb81583803447023c1359957c8/csharp/ParquetHandle.cs#L8
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.format_string;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ARROW_FLAG_NULLABLE;
+ }
+
+ if (datatype is DictionaryType)
+ {
+ if (((DictionaryType)datatype).Ordered)
+ {
+ flags |= ARROW_FLAG_DICTIONARY_ORDERED;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype)
+ {
+ if (datatype is NestedType)
+ {
+ var fields = ((NestedType)datatype).Fields;
+ int n_fields = fields.Count;
+
+ IntPtr* pointer_list = (IntPtr*)Marshal.AllocHGlobal(n_fields * sizeof(IntPtr));
+
+ for (var i = 0; i < n_fields; i++)
+ {
+ var c_schema = new CArrowSchema();
+ CArrowSchema.ExportField(fields[i], out c_schema);
+ IntPtr exported_schema = c_schema.AllocateAsPtr();
+ pointer_list[i] = exported_schema;
+ }
+
+ return pointer_list;
+ }
+ else
+ {
+ return (IntPtr*)IntPtr.Zero;
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType)
+ {
+ var c_schema = new CArrowSchema();
+ var value_type = ((DictionaryType)datatype).ValueType;
+ CArrowSchema.ExportDataType(value_type, out c_schema);
+ return c_schema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ /// <param name="schema">An uninitialized CArrowSchema.</param>
+ public static void ExportDataType(IArrowType datatype, out CArrowSchema schema)
+ {
+ schema.format = GetFormat(datatype);
+ schema.name = null;
+ schema.metadata = null;
+ schema.flags = GetFlags(datatype);
+
+ schema.children = ConstructChildren(datatype);
+ schema.n_children = datatype is NestedType ? ((NestedType)datatype).Fields.Count : 0;
+
+ schema.dictionary = ConstructDictionary(datatype);
+
+ schema.release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.children[i]);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ schema.private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ /// <param name="schema">An uninitialized CArrowSchema.</param>
+ public static void ExportField(Field field, out CArrowSchema schema)
+ {
+ ExportDataType(field.DataType, out schema);
+ schema.name = field.Name;
+ // TODO: field metadata
+ schema.metadata = null;
+ schema.flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ /// <param name="out_schema">An uninitialized CArrowSchema</param>
+ public static void ExportSchema(Schema schema, out CArrowSchema out_schema)
+ {
+ // TODO: top-level metadata
+ var struct_type = new StructType(schema.Fields.Values.ToList());
+ ExportDataType(struct_type, out out_schema);
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr"></param>
+ /// <returns></returns>
Review Comment:
Should we fill them?
##########
csharp/test/Apache.Arrow.Tests/CDataInterfaceSchemaTests.cs:
##########
@@ -0,0 +1,302 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Collections;
+using System.Collections.Generic;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.C;
+using Apache.Arrow.Types;
+using Python.Runtime;
+using Xunit;
+
+namespace Apache.Arrow.Tests
+{
+ public class CDataSchemaTest
+ {
+ public CDataSchemaTest()
+ {
+ PythonEngine.Initialize();
+ }
+
+ public static Schema GetTestSchema()
+ {
+ // TODO: Add more types
+ using (Py.GIL())
+ {
+ var schema = new Schema.Builder()
+ .Field(f => f.Name("null").DataType(NullType.Default).Nullable(true))
+ .Field(f => f.Name("bool").DataType(BooleanType.Default).Nullable(true))
+ .Field(f => f.Name("i8").DataType(Int8Type.Default).Nullable(true))
+ .Field(f => f.Name("u8").DataType(UInt8Type.Default).Nullable(true))
+ .Field(f => f.Name("i16").DataType(Int16Type.Default).Nullable(true))
+ .Field(f => f.Name("u16").DataType(UInt16Type.Default).Nullable(true))
+ .Field(f => f.Name("i32").DataType(Int32Type.Default).Nullable(true))
+ .Field(f => f.Name("u32").DataType(UInt32Type.Default).Nullable(true))
+ .Field(f => f.Name("i64").DataType(Int64Type.Default).Nullable(true))
+ .Field(f => f.Name("u64").DataType(UInt64Type.Default).Nullable(true))
+
+ .Field(f => f.Name("f16").DataType(HalfFloatType.Default).Nullable(true))
+ .Field(f => f.Name("f32").DataType(FloatType.Default).Nullable(true))
+ .Field(f => f.Name("f64").DataType(DoubleType.Default).Nullable(true))
+
+ .Field(f => f.Name("binary").DataType(BinaryType.Default).Nullable(false))
+ .Field(f => f.Name("string").DataType(StringType.Default).Nullable(false))
+
+ .Field(f => f.Name("date32").DataType(Date32Type.Default).Nullable(false))
+ .Field(f => f.Name("date64").DataType(Date64Type.Default).Nullable(false))
+ .Field(f => f.Name("time32_s").DataType(new Time32Type(TimeUnit.Second)).Nullable(false))
+ .Field(f => f.Name("time32_ms").DataType(new Time32Type(TimeUnit.Millisecond)).Nullable(false))
+ .Field(f => f.Name("time64_us").DataType(new Time64Type(TimeUnit.Microsecond)).Nullable(false))
+ .Field(f => f.Name("time64_ns").DataType(new Time64Type(TimeUnit.Nanosecond)).Nullable(false))
+
+ .Field(f => f.Name("list_string").DataType(new ListType(StringType.Default)).Nullable(false))
+ .Field(f => f.Name("list_list_i32").DataType(new ListType(new ListType(Int32Type.Default))).Nullable(false))
+
+ .Field(f => f.Name("dict_string").DataType(new DictionaryType(Int32Type.Default, StringType.Default, false)).Nullable(false))
+ .Field(f => f.Name("dict_string_ordered").DataType(new DictionaryType(Int32Type.Default, StringType.Default, true)).Nullable(false))
+ .Field(f => f.Name("list_dict_string").DataType(new ListType(new DictionaryType(Int32Type.Default, StringType.Default, false))).Nullable(false))
+
+ .Build();
+ return schema;
+ }
+ }
+
+ public static IEnumerable<dynamic> GetPythonFields()
+ {
+ using (Py.GIL())
+ {
+ dynamic pa = Py.Import("pyarrow");
+ yield return pa.field("null", pa.GetAttr("null").Invoke(), true);
+ yield return pa.field("bool", pa.bool_(), true);
+ yield return pa.field("i8", pa.int8(), true);
+ yield return pa.field("u8", pa.uint8(), true);
+ yield return pa.field("i16", pa.int16(), true);
+ yield return pa.field("u16", pa.uint16(), true);
+ yield return pa.field("i32", pa.int32(), true);
+ yield return pa.field("u32", pa.uint32(), true);
+ yield return pa.field("i64", pa.int64(), true);
+ yield return pa.field("u64", pa.uint64(), true);
+
+ yield return pa.field("f16", pa.float16(), true);
+ yield return pa.field("f32", pa.float32(), true);
+ yield return pa.field("f64", pa.float64(), true);
+
+ yield return pa.field("binary", pa.binary(), false);
+ yield return pa.field("string", pa.utf8(), false);
+
+ yield return pa.field("date32", pa.date32(), false);
+ yield return pa.field("date64", pa.date64(), false);
+ yield return pa.field("time32_s", pa.time32("s"), false);
+ yield return pa.field("time32_ms", pa.time32("ms"), false);
+ yield return pa.field("time64_us", pa.time64("us"), false);
+ yield return pa.field("time64_ns", pa.time64("ns"), false);
+
+ yield return pa.field("list_string", pa.list_(pa.utf8()), false);
+ yield return pa.field("list_list_i32", pa.list_(pa.list_(pa.int32())), false);
+
+ yield return pa.field("dict_string", pa.dictionary(pa.int32(), pa.utf8(), false), false);
+ yield return pa.field("dict_string_ordered", pa.dictionary(pa.int32(), pa.utf8(), true), false);
+ yield return pa.field("list_dict_string", pa.list_(pa.dictionary(pa.int32(), pa.utf8(), false)), false);
+ }
+ }
+
+ public static dynamic GetPythonSchema()
+ {
+ using (Py.GIL())
+ {
+ dynamic pa = Py.Import("pyarrow");
+ return pa.schema(GetPythonFields().ToList());
+ }
+ }
+
+ // Schemas created in Python, used in CSharp
+ [Fact]
+ public void ImportType()
+ {
+ PythonEngine.Initialize();
+ Schema schema = GetTestSchema();
+ IEnumerable<dynamic> py_fields = GetPythonFields();
+
+ foreach ((Field field, dynamic py_field) in schema.Fields.Values.AsEnumerable()
+ .Zip(py_fields))
+ {
+ var c_schema = new CArrowSchema();
+ IntPtr imported_ptr = c_schema.AllocateAsPtr();
+
+ using (Py.GIL())
+ {
+ dynamic py_datatype = py_field.type;
+ py_datatype._export_to_c(imported_ptr.ToInt64());
+ }
+
+ var dataTypeComparer = new ArrayTypeComparer(field.DataType);
+ var imported_type = new ImportedArrowSchema(imported_ptr);
Review Comment:
How about using the same naming convention? (`camelCase` v.s. `snake_case`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1123542945
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
+ {
+ format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ name = IntPtr.Zero;
+ metadata = IntPtr.Zero;
+ flags = GetFlags(datatype);
+
+ children = ConstructChildren(datatype, out var numChildren);
+ n_children = numChildren;
+
+ dictionary = ConstructDictionary(datatype);
+
+ release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+
+ Marshal.FreeHGlobal(schema.format);
+ Marshal.FreeHGlobal(schema.name);
+ Marshal.FreeHGlobal(schema.metadata);
+
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.GetChild(i));
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ public CArrowSchema(Field field) : this(field.DataType)
+ {
+ name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ metadata = IntPtr.Zero;
+ flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ public CArrowSchema(Schema schema) : this(new StructType(schema.Fields.Values.ToList()))
+ {
+ // TODO: top-level metadata
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public void Export(IntPtr ptr)
+ {
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ }
+
+ /// <summary>
+ /// Allocated a new pointer to an uninitialized CArrowSchema.
+ /// </summary>
+ /// <remarks>This is used to import schemas. See <see cref="ImportType"/>,
+ /// <see cref="ImportField"/>, and <see cref="ImportSchema"/>. Once data
+ /// is imported, this pointer must be cleaned up with <see cref="FreePtr"/>.
+ /// </remarks>
+ public static IntPtr AllocateUninitialized()
+ {
+ return Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchema.ImportType(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static ArrowType ImportType(IntPtr ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchema.ImportField(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Field ImportField(IntPtr ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchema.ImportSchema(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Schema ImportSchema(IntPtr ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ public const int ArrowFlagDictionaryOrdered = 1;
+ public const int ArrowFlagNullable = 2;
+ public const int ArrowFlagMapKeysSorted = 4;
+
+ private class TypeFormatter :
+ IArrowTypeVisitor<NullType>,
+ IArrowTypeVisitor<BooleanType>,
+ IArrowTypeVisitor<Int8Type>,
+ IArrowTypeVisitor<Int16Type>,
+ IArrowTypeVisitor<Int32Type>,
+ IArrowTypeVisitor<Int64Type>,
+ IArrowTypeVisitor<UInt8Type>,
+ IArrowTypeVisitor<UInt16Type>,
+ IArrowTypeVisitor<UInt32Type>,
+ IArrowTypeVisitor<UInt64Type>,
+ IArrowTypeVisitor<HalfFloatType>,
+ IArrowTypeVisitor<FloatType>,
+ IArrowTypeVisitor<DoubleType>,
+ IArrowTypeVisitor<Decimal128Type>,
+ IArrowTypeVisitor<Decimal256Type>,
+ IArrowTypeVisitor<Date32Type>,
+ IArrowTypeVisitor<Date64Type>,
+ IArrowTypeVisitor<Time32Type>,
+ IArrowTypeVisitor<Time64Type>,
+ IArrowTypeVisitor<TimestampType>,
+ IArrowTypeVisitor<StringType>,
+ IArrowTypeVisitor<BinaryType>,
+ IArrowTypeVisitor<FixedSizeBinaryType>,
+ IArrowTypeVisitor<ListType>,
+ IArrowTypeVisitor<StructType>,
+ IArrowTypeVisitor<DictionaryType>
+ {
+ public string FormatString;
+ public void Visit(NullType _) => FormatString = "n";
+ public void Visit(BooleanType _) => FormatString = "b";
+ // Integers
+ public void Visit(Int8Type _) => FormatString = "c";
+ public void Visit(UInt8Type _) => FormatString = "C";
+ public void Visit(Int16Type _) => FormatString = "s";
+ public void Visit(UInt16Type _) => FormatString = "S";
+ public void Visit(Int32Type _) => FormatString = "i";
+ public void Visit(UInt32Type _) => FormatString = "I";
+ public void Visit(Int64Type _) => FormatString = "l";
+ public void Visit(UInt64Type _) => FormatString = "L";
+ // Floats
+ public void Visit(HalfFloatType _) => FormatString = "e";
+ public void Visit(FloatType _) => FormatString = "f";
+ public void Visit(DoubleType _) => FormatString = "g";
+ // Binary
+ public void Visit(BinaryType _) => FormatString = "z";
+ public void Visit(StringType _) => FormatString = "u";
+ public void Visit(FixedSizeBinaryType datatype)
+ {
+ FormatString = $"w:{datatype.ByteWidth}";
+ }
+ // Decimal
+ public void Visit(Decimal128Type datatype)
+ {
+ FormatString = $"d:{datatype.Precision},{datatype.Scale}";
+ }
+ public void Visit(Decimal256Type datatype)
+ {
+ FormatString = $"w:{datatype.Precision},{datatype.Scale},256";
+ }
+ // Date
+ public void Visit(Date32Type _) => FormatString = "tdD";
+ public void Visit(Date64Type _) => FormatString = "tdm";
+
+ private char TimeUnitComponent(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+ // Time
+ public void Visit(Time32Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ public void Visit(Time64Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ // Timestamp type
+ public void Visit(TimestampType datatype)
+ {
+ FormatString = String.Format("ts{0}:{1}", TimeUnitComponent(datatype.Unit), datatype.Timezone);
+ }
+ // Nested
+ public void Visit(ListType _) => FormatString = "+l";
+ public void Visit(StructType _) => FormatString = "+s";
+ // Dictionary
+ public void Visit(DictionaryType datatype)
+ {
+ // format string is that of the indices
+ datatype.IndexType.Accept(this);
+ }
+
+ // Unsupported
+ public void Visit(IArrowType type)
+ {
+ throw new NotImplementedException($"Exporting {type.Name} not implemented");
+ }
+ }
+
+ private sealed class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema _data;
+ private readonly IntPtr _handle;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(IntPtr handle)
+ {
+ _data = Marshal.PtrToStructure<CArrowSchema>(handle);
+ if (_data.release == null)
+ {
+ throw new Exception("Tried to import a schema that has already been released.");
+ }
+ _handle = handle;
+ _isRoot = true;
+ }
+
+ private ImportedArrowSchema(IntPtr handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot)
+ {
+ _data.release(_handle);
+ }
+ }
Review Comment:
Made it idempotent by checking whether `release` is null, which I think we should always be doing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1139433788
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
Review Comment:
So let's split these into two points.
1. I agree that `CArrowSchema*` needs to be freed. I'm not sure it has to be heap-allocated. I think it could be stack-allocated but I don't remember if `struct` was guaranteed to be stack allocated or not. It's probably large enough (with all the strings) to justify heap allocation so I think this is good.
2. I'm not sure that it is safe to assume that `release` has been set to null. I do agree it is safe when calling Arrow-C++'s import function. I just don't think this is part of the spec. And, the spec is very clear that it is the consumer's responsibility to call the top-level `release` function. So, while Arrow does copy everything into its own memory during it's import function I don't think this is spec-mandated. Maybe it would be a good idea to add this mandate to the spec?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1138851201
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
Review Comment:
Okay read up on it. In the C++ bridge, the consumer moves the data a struct it owns by memcpy'ing it to the struct and then calling release on the source. So all import functions are expected to call `release` on the original struct (but might copy data off of it before then).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1140719743
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
Review Comment:
Well, the problem is that you really DO need to release the `CArrowSchema*` regardless of what the foreign import function does. So, is there a way to do that without calling `release` if it exists?
Maybe the pattern could be:
```
/// CArrowSchema* exportPtr = CArrowSchema.Create();
/// CArrowSchemaExporter.ExportType(dataType, exportPtr);
/// foreign_import_function(exportPtr);
/// exportPtr->release = IntPtr.Zero;
/// CArrowSchema.Free(exportPtr);
```
However, this seems potentially error prone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1148513875
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,268 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.New();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportType(IArrowType datatype, CArrowSchema* schema)
+ {
+ if (schema == null)
+ {
+ throw new ArgumentNullException("Cannot export schema to a null pointer.");
+ }
+ if (schema->release != null)
+ {
+ throw new ArgumentException("Cannot export schema to a struct that is already initialized.");
+ }
+
+ schema->format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ schema->name = null;
+ schema->metadata = null;
+ schema->flags = GetFlags(datatype);
+
+ schema->children = ConstructChildren(datatype, out var numChildren);
+ schema->n_children = numChildren;
+
+ schema->dictionary = ConstructDictionary(datatype);
+
+ schema->release = (delegate* unmanaged[Stdcall]<CArrowSchema*, void>)Marshal.GetFunctionPointerForDelegate(
+ ReleaseCArrowSchema);
+
+ schema->private_data = null;
+ }
+
+ /// <summary>
+ /// Export a field to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="field">The field to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.New();
+ /// CArrowSchemaExporter.ExportType(field, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportField(Field field, CArrowSchema* schema)
+ {
+ ExportType(field.DataType, schema);
+ schema->name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ schema->metadata = null;
+ schema->flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Export a schema to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="schema">The schema to export</param>
+ /// <param name="out_schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.New();
+ /// CArrowSchemaExporter.ExportType(schema, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportSchema(Schema schema, CArrowSchema* out_schema)
+ {
+ var structType = new StructType(schema.FieldsList);
+ // TODO: top-level metadata
+ ExportType(structType, out_schema);
+ }
+
+ private static char FormatTimeUnit(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ switch (datatype)
+ {
+ case NullType _: return "n";
+ case BooleanType _: return "b";
+ // Integers
+ case Int8Type _: return "c";
+ case UInt8Type _: return "C";
+ case Int16Type _: return "s";
+ case UInt16Type _: return "S";
+ case Int32Type _: return "i";
+ case UInt32Type _: return "I";
+ case Int64Type _: return "l";
+ case UInt64Type _: return "L";
+ // Floats
+ case HalfFloatType _: return "e";
+ case FloatType _: return "f";
+ case DoubleType _: return "g";
+ // Decimal
+ case Decimal128Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale}";
+ case Decimal256Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale},256";
+ // Binary
+ case BinaryType _: return "z";
+ case StringType _: return "u";
+ case FixedSizeBinaryType binaryType:
+ return $"w:{binaryType.ByteWidth}";
+ // Date
+ case Date32Type _: return "tdD";
+ case Date64Type _: return "tdm";
+ // Time
+ case Time32Type timeType:
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ case Time64Type timeType:
+ // Same prefix as Time32, but allowed time units are different.
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ // Timestamp
+ case TimestampType timestampType:
+ return String.Format("ts{0}:{1}", FormatTimeUnit(timestampType.Unit), timestampType.Timezone);
+ // Nested
+ case ListType _: return "+l";
+ case StructType _: return "+s";
+ // Dictionary
+ case DictionaryType dictionaryType:
+ return GetFormat(dictionaryType.IndexType);
+ default: throw new NotImplementedException($"Exporting {datatype.Name} not implemented");
+ };
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= CArrowSchema.ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= CArrowSchema.ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static unsafe CArrowSchema** ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ var pointerList = (CArrowSchema**)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ CArrowSchema* cSchema = CArrowSchema.New();
+ ExportField(fields[i], cSchema);
+ pointerList[i] = cSchema;
+ }
+
+ return pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return null;
+ }
+ }
+
+ private static unsafe CArrowSchema* ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ CArrowSchema* cSchema = CArrowSchema.New();
+ ExportType(dictType.ValueType, cSchema);
+ return cSchema;
+ }
+ else
+ {
+ return null;
+ }
+ }
+
+ private static unsafe void ReleaseCArrowSchema(CArrowSchema* schema)
+ {
+ if (schema == null) return;
+ if (schema->release == null) return;
Review Comment:
Well, if the release callback is already null, the consumer cannot have called this function...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #34133: GH-34737: [C#] C Data interface for schemas and types
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34133:
URL: https://github.com/apache/arrow/pull/34133#issuecomment-1484382267
:warning: GitHub issue #34737 **has been automatically assigned in GitHub** to PR creator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1122349814
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
Review Comment:
Okay I've narrowed it so `unsafe` only appears when constructing and accessing children. Does that seem like the improvement we are looking for?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] adamreeve commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1122470032
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
Review Comment:
👍 yeah I think that is preferable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1124991301
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
Review Comment:
https://github.com/apache/arrow/blob/48df467eb5c684aafbd168ad056c64bb2f6706e8/csharp/Directory.Build.props#L37
just update this to be `latest` instead. Not sure why we are setting `8.0` explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1123510339
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
Review Comment:
Thanks. That is a lot cleaner :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1124955152
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
Review Comment:
That makes sense for the string fields. What type would you suggest for `children` and `dictionary`?
https://github.com/apache/arrow/blob/54f8fe377099ab9904e9858aa26c63caa199e3df/cpp/src/arrow/c/abi.h#L40-L41
Originally I had
```
public IntPtr* children;
public IntPtr dictionary;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] adamreeve commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1122474881
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
+ {
+ format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ name = IntPtr.Zero;
+ metadata = IntPtr.Zero;
+ flags = GetFlags(datatype);
+
+ children = ConstructChildren(datatype, out var numChildren);
+ n_children = numChildren;
+
+ dictionary = ConstructDictionary(datatype);
+
+ release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+
+ Marshal.FreeHGlobal(schema.format);
+ Marshal.FreeHGlobal(schema.name);
+ Marshal.FreeHGlobal(schema.metadata);
+
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.GetChild(i));
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ public CArrowSchema(Field field) : this(field.DataType)
+ {
+ name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ metadata = IntPtr.Zero;
+ flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ public CArrowSchema(Schema schema) : this(new StructType(schema.Fields.Values.ToList()))
+ {
+ // TODO: top-level metadata
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
Review Comment:
Maybe a nitpick but I think `ExportAsPtr` or maybe just having an override of `Export` would more clearly describe the main intent of this method. I guess `AllocateAndExport` would be more descriptive but is getting a little verbose
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1148513105
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
Review Comment:
I think this discussion is a bit pedantic as it all depends how the consumer API (`foreign_import_function`) is defined, and that is not specified by the C Data Interface.
The C Data Interface only says what should happen with the _contents_ of the C struct. How the struct itself is allocated is left for the producer or consumer to decide (and may indeed depend on which language those are implemented in).
I would generally suggest that the producer handles the lifetime of the struct it populates, and the consumer moves the contents of the producer-populated struct to its own managed area.
(also, in real-world examples you would hope that `foreign_import_function` is able to return an error)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on pull request #34133: GH-34737: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on PR #34133:
URL: https://github.com/apache/arrow/pull/34133#issuecomment-1490788564
@westonpace - I see you still have changes requested. Is that stale? or are you still requesting changes on the current proposal?
The macOS 12 Python 3 leg is still failing. Does anyone know if that is related to these changes? @wjones127, can you merge with `main` to see if the errors go away?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1132921630
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
Review Comment:
```suggestion
public static CArrowSchema* Create()
```
1. The whole struct is unsafe. Don't need this on every method.
2. I'd prefer `Create` over `New` to not confuse this with a constructor.
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+
+ ptr->format = null;
+ ptr->name = null;
+ ptr->metadata = null;
+ ptr->flags = 0;
+ ptr->n_children = 0;
+ ptr->children = null;
+ ptr->dictionary = null;
+ ptr->release = null;
+ ptr->private_data = null;
+
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static unsafe void Free(CArrowSchema* schema)
+ {
+ if (schema->release != null)
+ {
+ // Call release if not already called.
+ schema->release(schema);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema);
+ }
+
+
+ /// <summary>
+ /// Whether this field is semantically nullable (regardless of whether it actually has null values)
+ /// </summary>
+ public const long ArrowFlagDictionaryOrdered = 1;
+ /// <summary>
+ /// For dictionary-encoded types, whether the ordering of dictionary indices is semantically meaningful.
+ /// </summary>
+ public const long ArrowFlagNullable = 2;
+ /// <summary>
+ /// For map types, whether the keys within each map value are sorted.
+ /// </summary>
+ public const long ArrowFlagMapKeysSorted = 4;
+
+ /// <summary>
+ /// Get the value of a particular flag.
+ /// </summary>
+ /// <remarks>
+ /// Known valid flags are <see cref="ArrowFlagDictionaryOrdered" />,
+ /// <see cref="ArrowFlagNullable" />, and <see cref="ArrowFlagMapKeysSorted" />.
+ /// </remarks>
+ public bool GetFlag(long flag)
+ {
+ return (flags & flag) == flag;
+ }
+
+ internal CArrowSchema* GetChild(int i)
Review Comment:
```suggestion
internal readonly CArrowSchema* GetChild(int i)
```
##########
csharp/src/Apache.Arrow/C/StringUtil.cs:
##########
@@ -0,0 +1,66 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Runtime.InteropServices;
+
+// Adapted from:
+// https://github.com/G-Research/ParquetSharp/blob/467d99298fb5a5b9d5935b9c8dbde95e63954dd3/csharp/StringUtil.cs
+
+namespace Apache.Arrow.C
+{
+
+ internal static class StringUtil
+ {
+ public static unsafe byte* ToCStringUtf8(string str)
+ {
+ var utf8 = System.Text.Encoding.UTF8;
+ int byteCount = utf8.GetByteCount(str);
+ byte* byteArray = (byte*)Marshal.AllocHGlobal(byteCount + 1);
+
+ fixed (char* chars = str)
+ {
+ utf8.GetBytes(chars, str.Length, byteArray, byteCount);
+ }
+
+ // Need to make sure it is nul-terminated.
+ byteArray[byteCount] = 0;
+
+ return byteArray;
+ }
+
+ public static unsafe string PtrToStringUtf8(byte* ptr)
+ {
+#if NETSTANDARD2_1_OR_GREATER
+ return Marshal.PtrToStringUTF8(ptr);
+#else
+ if (ptr == null)
+ {
+ return null;
+ }
+
+ unsafe
+ {
Review Comment:
```suggestion
```
The method is already unsafe.
##########
csharp/src/Apache.Arrow/C/StringUtil.cs:
##########
@@ -0,0 +1,66 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Runtime.InteropServices;
+
+// Adapted from:
+// https://github.com/G-Research/ParquetSharp/blob/467d99298fb5a5b9d5935b9c8dbde95e63954dd3/csharp/StringUtil.cs
+
+namespace Apache.Arrow.C
+{
+
+ internal static class StringUtil
+ {
+ public static unsafe byte* ToCStringUtf8(string str)
+ {
+ var utf8 = System.Text.Encoding.UTF8;
+ int byteCount = utf8.GetByteCount(str);
+ byte* byteArray = (byte*)Marshal.AllocHGlobal(byteCount + 1);
+
+ fixed (char* chars = str)
+ {
+ utf8.GetBytes(chars, str.Length, byteArray, byteCount);
+ }
+
+ // Need to make sure it is nul-terminated.
Review Comment:
```suggestion
// Need to make sure it is null-terminated.
```
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+
+ ptr->format = null;
+ ptr->name = null;
+ ptr->metadata = null;
+ ptr->flags = 0;
+ ptr->n_children = 0;
+ ptr->children = null;
+ ptr->dictionary = null;
+ ptr->release = null;
+ ptr->private_data = null;
+
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static unsafe void Free(CArrowSchema* schema)
+ {
+ if (schema->release != null)
+ {
+ // Call release if not already called.
+ schema->release(schema);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema);
+ }
+
+
+ /// <summary>
+ /// Whether this field is semantically nullable (regardless of whether it actually has null values)
+ /// </summary>
+ public const long ArrowFlagDictionaryOrdered = 1;
+ /// <summary>
+ /// For dictionary-encoded types, whether the ordering of dictionary indices is semantically meaningful.
+ /// </summary>
+ public const long ArrowFlagNullable = 2;
+ /// <summary>
+ /// For map types, whether the keys within each map value are sorted.
+ /// </summary>
+ public const long ArrowFlagMapKeysSorted = 4;
+
+ /// <summary>
+ /// Get the value of a particular flag.
+ /// </summary>
+ /// <remarks>
+ /// Known valid flags are <see cref="ArrowFlagDictionaryOrdered" />,
+ /// <see cref="ArrowFlagNullable" />, and <see cref="ArrowFlagMapKeysSorted" />.
+ /// </remarks>
+ public bool GetFlag(long flag)
+ {
+ return (flags & flag) == flag;
+ }
+
+ internal CArrowSchema* GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new ArgumentOutOfRangeException("Child index out of bounds.");
+ }
+ if (children == null)
+ {
+ throw new Exception("Children array is null but n_children is not zero.");
+ }
+ unsafe
+ {
+ return children[i];
+ }
Review Comment:
```suggestion
return children[i];
```
Whole struct is unsafe.
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
Review Comment:
```suggestion
```
No need for this blank line.
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
Review Comment:
```suggestion
var ptr = (CArrowSchema*)Marshal.AllocHGlobal(sizeof(CArrowSchema));
```
Using `sizeof` keyword here is better because it becomes a constant, and not a runtime call. Also it is "safe" with NativeAOT and Trimming.
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
Review Comment:
```suggestion
```
(nit) - only one blank line between members.
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+
+ ptr->format = null;
+ ptr->name = null;
+ ptr->metadata = null;
+ ptr->flags = 0;
+ ptr->n_children = 0;
+ ptr->children = null;
+ ptr->dictionary = null;
+ ptr->release = null;
+ ptr->private_data = null;
+
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static unsafe void Free(CArrowSchema* schema)
+ {
+ if (schema->release != null)
+ {
+ // Call release if not already called.
+ schema->release(schema);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema);
+ }
+
+
+ /// <summary>
+ /// Whether this field is semantically nullable (regardless of whether it actually has null values)
+ /// </summary>
+ public const long ArrowFlagDictionaryOrdered = 1;
+ /// <summary>
+ /// For dictionary-encoded types, whether the ordering of dictionary indices is semantically meaningful.
+ /// </summary>
+ public const long ArrowFlagNullable = 2;
+ /// <summary>
+ /// For map types, whether the keys within each map value are sorted.
+ /// </summary>
+ public const long ArrowFlagMapKeysSorted = 4;
+
+ /// <summary>
+ /// Get the value of a particular flag.
+ /// </summary>
+ /// <remarks>
+ /// Known valid flags are <see cref="ArrowFlagDictionaryOrdered" />,
+ /// <see cref="ArrowFlagNullable" />, and <see cref="ArrowFlagMapKeysSorted" />.
+ /// </remarks>
+ public bool GetFlag(long flag)
+ {
+ return (flags & flag) == flag;
+ }
+
+ internal CArrowSchema* GetChild(int i)
Review Comment:
```suggestion
internal CArrowSchema* GetChild(long i)
```
Since n_children is a `long`, does it make sense for this to be a `long`?
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+
+ ptr->format = null;
+ ptr->name = null;
+ ptr->metadata = null;
+ ptr->flags = 0;
+ ptr->n_children = 0;
+ ptr->children = null;
+ ptr->dictionary = null;
+ ptr->release = null;
+ ptr->private_data = null;
+
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static unsafe void Free(CArrowSchema* schema)
+ {
+ if (schema->release != null)
+ {
+ // Call release if not already called.
+ schema->release(schema);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema);
+ }
+
+
+ /// <summary>
+ /// Whether this field is semantically nullable (regardless of whether it actually has null values)
+ /// </summary>
+ public const long ArrowFlagDictionaryOrdered = 1;
+ /// <summary>
+ /// For dictionary-encoded types, whether the ordering of dictionary indices is semantically meaningful.
+ /// </summary>
+ public const long ArrowFlagNullable = 2;
+ /// <summary>
+ /// For map types, whether the keys within each map value are sorted.
+ /// </summary>
+ public const long ArrowFlagMapKeysSorted = 4;
+
+ /// <summary>
+ /// Get the value of a particular flag.
+ /// </summary>
+ /// <remarks>
+ /// Known valid flags are <see cref="ArrowFlagDictionaryOrdered" />,
+ /// <see cref="ArrowFlagNullable" />, and <see cref="ArrowFlagMapKeysSorted" />.
+ /// </remarks>
+ public bool GetFlag(long flag)
+ {
+ return (flags & flag) == flag;
+ }
+
+ internal CArrowSchema* GetChild(int i)
+ {
+ if (i >= n_children)
Review Comment:
```suggestion
if ((ulong)i >= (ulong)n_children)
```
This is a trick we use a lot in the dotnet/runtime. It does 2 checks at once:
1. Checks if the index is larger than the n_children.
2. Checks if the index is negative.
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+
+ ptr->format = null;
+ ptr->name = null;
+ ptr->metadata = null;
+ ptr->flags = 0;
+ ptr->n_children = 0;
+ ptr->children = null;
+ ptr->dictionary = null;
+ ptr->release = null;
+ ptr->private_data = null;
+
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static unsafe void Free(CArrowSchema* schema)
+ {
+ if (schema->release != null)
+ {
+ // Call release if not already called.
+ schema->release(schema);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema);
+ }
+
+
+ /// <summary>
+ /// Whether this field is semantically nullable (regardless of whether it actually has null values)
+ /// </summary>
+ public const long ArrowFlagDictionaryOrdered = 1;
+ /// <summary>
+ /// For dictionary-encoded types, whether the ordering of dictionary indices is semantically meaningful.
+ /// </summary>
+ public const long ArrowFlagNullable = 2;
+ /// <summary>
+ /// For map types, whether the keys within each map value are sorted.
+ /// </summary>
+ public const long ArrowFlagMapKeysSorted = 4;
+
+ /// <summary>
+ /// Get the value of a particular flag.
+ /// </summary>
+ /// <remarks>
+ /// Known valid flags are <see cref="ArrowFlagDictionaryOrdered" />,
+ /// <see cref="ArrowFlagNullable" />, and <see cref="ArrowFlagMapKeysSorted" />.
+ /// </remarks>
+ public bool GetFlag(long flag)
+ {
+ return (flags & flag) == flag;
+ }
+
+ internal CArrowSchema* GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new ArgumentOutOfRangeException("Child index out of bounds.");
Review Comment:
```suggestion
throw new ArgumentOutOfRangeException($"Child index '{i}' out of bounds.");
```
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+
+ ptr->format = null;
+ ptr->name = null;
+ ptr->metadata = null;
+ ptr->flags = 0;
+ ptr->n_children = 0;
+ ptr->children = null;
+ ptr->dictionary = null;
+ ptr->release = null;
+ ptr->private_data = null;
+
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static unsafe void Free(CArrowSchema* schema)
+ {
+ if (schema->release != null)
+ {
+ // Call release if not already called.
+ schema->release(schema);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema);
+ }
+
+
+ /// <summary>
+ /// Whether this field is semantically nullable (regardless of whether it actually has null values)
+ /// </summary>
+ public const long ArrowFlagDictionaryOrdered = 1;
+ /// <summary>
+ /// For dictionary-encoded types, whether the ordering of dictionary indices is semantically meaningful.
+ /// </summary>
+ public const long ArrowFlagNullable = 2;
+ /// <summary>
+ /// For map types, whether the keys within each map value are sorted.
+ /// </summary>
+ public const long ArrowFlagMapKeysSorted = 4;
+
+ /// <summary>
+ /// Get the value of a particular flag.
+ /// </summary>
+ /// <remarks>
+ /// Known valid flags are <see cref="ArrowFlagDictionaryOrdered" />,
+ /// <see cref="ArrowFlagNullable" />, and <see cref="ArrowFlagMapKeysSorted" />.
+ /// </remarks>
+ public bool GetFlag(long flag)
Review Comment:
```suggestion
public readonly bool GetFlag(long flag)
```
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+
+ ptr->format = null;
+ ptr->name = null;
+ ptr->metadata = null;
+ ptr->flags = 0;
+ ptr->n_children = 0;
+ ptr->children = null;
+ ptr->dictionary = null;
+ ptr->release = null;
+ ptr->private_data = null;
+
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static unsafe void Free(CArrowSchema* schema)
+ {
+ if (schema->release != null)
+ {
+ // Call release if not already called.
+ schema->release(schema);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema);
+ }
+
+
+ /// <summary>
+ /// Whether this field is semantically nullable (regardless of whether it actually has null values)
+ /// </summary>
+ public const long ArrowFlagDictionaryOrdered = 1;
+ /// <summary>
+ /// For dictionary-encoded types, whether the ordering of dictionary indices is semantically meaningful.
+ /// </summary>
+ public const long ArrowFlagNullable = 2;
+ /// <summary>
+ /// For map types, whether the keys within each map value are sorted.
+ /// </summary>
+ public const long ArrowFlagMapKeysSorted = 4;
+
+ /// <summary>
+ /// Get the value of a particular flag.
+ /// </summary>
+ /// <remarks>
+ /// Known valid flags are <see cref="ArrowFlagDictionaryOrdered" />,
+ /// <see cref="ArrowFlagNullable" />, and <see cref="ArrowFlagMapKeysSorted" />.
+ /// </remarks>
+ public bool GetFlag(long flag)
+ {
+ return (flags & flag) == flag;
+ }
+
+ internal CArrowSchema* GetChild(int i)
Review Comment:
Any reason this is `internal`?
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+
+ ptr->format = null;
+ ptr->name = null;
+ ptr->metadata = null;
+ ptr->flags = 0;
+ ptr->n_children = 0;
+ ptr->children = null;
+ ptr->dictionary = null;
+ ptr->release = null;
+ ptr->private_data = null;
+
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
Review Comment:
What is `AllocateAsPtr`?
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchemaImporter.ImportSchema(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Schema ImportSchema(CArrowSchema* ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ private sealed unsafe class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema* _data;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(CArrowSchema* handle)
+ {
+ _data = (CArrowSchema*)handle;
+ if (_data->release == null)
+ {
+ throw new ArgumentException("Tried to import a schema that has already been released.");
+ }
+ _isRoot = true;
+ }
+
+ public ImportedArrowSchema(CArrowSchema* handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot && _data->release != null)
+ {
+ _data->release(_data);
+ }
+ }
+
+ public ArrowType GetAsType()
+ {
+ var format = StringUtil.PtrToStringUtf8(_data->format);
+ if (_data->dictionary != null)
+ {
+ ArrowType indicesType = format switch
+ {
+ "c" => new Int8Type(),
+ "C" => new UInt8Type(),
+ "s" => new Int16Type(),
+ "S" => new UInt16Type(),
+ "i" => new Int32Type(),
+ "I" => new UInt32Type(),
+ "l" => new Int64Type(),
+ "L" => new UInt64Type(),
+ _ => throw new InvalidDataException($"Indices must be an integer, but got format string {format}"),
+ };
+
+ var dictionarySchema = new ImportedArrowSchema(_data->dictionary, isRoot: false);
+ ArrowType dictionaryType = dictionarySchema.GetAsType();
+
+ bool ordered = _data->GetFlag(CArrowSchema.ArrowFlagDictionaryOrdered);
+
+ return new DictionaryType(indicesType, dictionaryType, ordered);
+ }
+
+ // Special handling for nested types
+ if (format == "+l")
+ {
+ if (_data->n_children != 1)
+ {
+ throw new InvalidDataException("Expected list type to have exactly one child.");
+ }
+ ImportedArrowSchema childSchema;
+ if (_data->GetChild(0) == null)
+ {
+ throw new InvalidDataException("Expected list type child to be non-null.");
+ }
+ childSchema = new ImportedArrowSchema(_data->GetChild(0), isRoot: false);
+
+ Field childField = childSchema.GetAsField();
+
+ return new ListType(childField);
+ }
+ else if (format == "+s")
+ {
+ var child_schemas = new ImportedArrowSchema[_data->n_children];
+
+ for (int i = 0; i < _data->n_children; i++)
+ {
+ if (_data->GetChild(i) == null)
+ {
+ throw new InvalidDataException("Expected struct type child to be non-null.");
+ }
+ child_schemas[i] = new ImportedArrowSchema(_data->GetChild(i), isRoot: false);
+ }
+
+
+ List<Field> childFields = child_schemas.Select(schema => schema.GetAsField()).ToList();
+
+ return new StructType(childFields);
+ }
+ // TODO: Map type and large list type
+
+ // Decimals
+ if (format.StartsWith("d:"))
+ {
+ bool is256 = format.EndsWith(",256");
+ string parameters_part = format.Remove(0, 2);
+ if (is256) parameters_part.Substring(0, parameters_part.Length - 5);
+ string[] parameters = parameters_part.Split(',');
+ int precision = Int32.Parse(parameters[0]);
+ int scale = Int32.Parse(parameters[1]);
+ if (is256)
+ {
+ return new Decimal256Type(precision, scale);
+ }
+ else
+ {
+ return new Decimal128Type(precision, scale);
+ }
+ }
+
+ // Timestamps
+ if (format.StartsWith("ts"))
+ {
+ TimeUnit timeUnit = format[2] switch
+ {
+ 's' => TimeUnit.Second,
+ 'm' => TimeUnit.Millisecond,
+ 'u' => TimeUnit.Microsecond,
+ 'n' => TimeUnit.Nanosecond,
+ _ => throw new InvalidDataException($"Unsupported time unit for import: {format[2]}"),
+ };
+
+ string timezone = format.Split(':')[1];
+ return new TimestampType(timeUnit, timezone);
+ }
+
+ // Fixed-width binary
+ if (format.StartsWith("w:"))
+ {
+ int width = Int32.Parse(format.Substring(2));
+ return new FixedSizeBinaryType(width);
+ }
+
+ return format switch
+ {
+ // Primitives
+ "n" => new NullType(),
+ "b" => new BooleanType(),
+ "c" => new Int8Type(),
+ "C" => new UInt8Type(),
+ "s" => new Int16Type(),
+ "S" => new UInt16Type(),
+ "i" => new Int32Type(),
+ "I" => new UInt32Type(),
+ "l" => new Int64Type(),
+ "L" => new UInt64Type(),
+ "e" => new HalfFloatType(),
+ "f" => new FloatType(),
+ "g" => new DoubleType(),
Review Comment:
These should all use the `.Default` property, where applicable. No reason to allocate extra objects here.
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,268 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.New();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportType(IArrowType datatype, CArrowSchema* schema)
+ {
+ if (schema == null)
+ {
+ throw new ArgumentNullException("Cannot export schema to a null pointer.");
+ }
+ if (schema->release != null)
+ {
+ throw new ArgumentException("Cannot export schema to a struct that is already initialized.");
+ }
+
+ schema->format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ schema->name = null;
+ schema->metadata = null;
+ schema->flags = GetFlags(datatype);
+
+ schema->children = ConstructChildren(datatype, out var numChildren);
+ schema->n_children = numChildren;
+
+ schema->dictionary = ConstructDictionary(datatype);
+
+ schema->release = (delegate* unmanaged[Stdcall]<CArrowSchema*, void>)Marshal.GetFunctionPointerForDelegate(
+ ReleaseCArrowSchema);
+
+ schema->private_data = null;
+ }
+
+ /// <summary>
+ /// Export a field to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="field">The field to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.New();
+ /// CArrowSchemaExporter.ExportType(field, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportField(Field field, CArrowSchema* schema)
+ {
+ ExportType(field.DataType, schema);
+ schema->name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ schema->metadata = null;
+ schema->flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Export a schema to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="schema">The schema to export</param>
+ /// <param name="out_schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.New();
+ /// CArrowSchemaExporter.ExportType(schema, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportSchema(Schema schema, CArrowSchema* out_schema)
+ {
+ var structType = new StructType(schema.FieldsList);
+ // TODO: top-level metadata
+ ExportType(structType, out_schema);
+ }
+
+ private static char FormatTimeUnit(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ switch (datatype)
+ {
+ case NullType _: return "n";
+ case BooleanType _: return "b";
+ // Integers
+ case Int8Type _: return "c";
+ case UInt8Type _: return "C";
+ case Int16Type _: return "s";
+ case UInt16Type _: return "S";
+ case Int32Type _: return "i";
+ case UInt32Type _: return "I";
+ case Int64Type _: return "l";
+ case UInt64Type _: return "L";
+ // Floats
+ case HalfFloatType _: return "e";
+ case FloatType _: return "f";
+ case DoubleType _: return "g";
+ // Decimal
+ case Decimal128Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale}";
+ case Decimal256Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale},256";
+ // Binary
+ case BinaryType _: return "z";
+ case StringType _: return "u";
+ case FixedSizeBinaryType binaryType:
+ return $"w:{binaryType.ByteWidth}";
+ // Date
+ case Date32Type _: return "tdD";
+ case Date64Type _: return "tdm";
+ // Time
+ case Time32Type timeType:
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ case Time64Type timeType:
+ // Same prefix as Time32, but allowed time units are different.
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ // Timestamp
+ case TimestampType timestampType:
+ return String.Format("ts{0}:{1}", FormatTimeUnit(timestampType.Unit), timestampType.Timezone);
+ // Nested
+ case ListType _: return "+l";
+ case StructType _: return "+s";
+ // Dictionary
+ case DictionaryType dictionaryType:
+ return GetFormat(dictionaryType.IndexType);
+ default: throw new NotImplementedException($"Exporting {datatype.Name} not implemented");
+ };
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= CArrowSchema.ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= CArrowSchema.ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static unsafe CArrowSchema** ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ var pointerList = (CArrowSchema**)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ CArrowSchema* cSchema = CArrowSchema.New();
+ ExportField(fields[i], cSchema);
+ pointerList[i] = cSchema;
+ }
+
+ return pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return null;
+ }
+ }
+
+ private static unsafe CArrowSchema* ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ CArrowSchema* cSchema = CArrowSchema.New();
+ ExportType(dictType.ValueType, cSchema);
+ return cSchema;
+ }
+ else
+ {
+ return null;
+ }
+ }
+
+ private static unsafe void ReleaseCArrowSchema(CArrowSchema* schema)
+ {
+ if (schema == null) return;
+ if (schema->release == null) return;
Review Comment:
Why is this check necessary?
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchemaImporter.ImportSchema(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Schema ImportSchema(CArrowSchema* ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ private sealed unsafe class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema* _data;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(CArrowSchema* handle)
+ {
+ _data = (CArrowSchema*)handle;
Review Comment:
(nit) The ctor parameter should be the same name as the backing field. It makes it easier to read and know that these things are the same thing.
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchemaImporter.ImportSchema(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Schema ImportSchema(CArrowSchema* ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ private sealed unsafe class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema* _data;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(CArrowSchema* handle)
Review Comment:
Are we checking the `handle` pointer for `null`? This is passed directly here from a public API.
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchemaImporter.ImportSchema(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Schema ImportSchema(CArrowSchema* ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ private sealed unsafe class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema* _data;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(CArrowSchema* handle)
+ {
+ _data = (CArrowSchema*)handle;
Review Comment:
```suggestion
_data = handle;
```
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchemaImporter.ImportSchema(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Schema ImportSchema(CArrowSchema* ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ private sealed unsafe class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema* _data;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(CArrowSchema* handle)
+ {
+ _data = (CArrowSchema*)handle;
+ if (_data->release == null)
+ {
+ throw new ArgumentException("Tried to import a schema that has already been released.");
+ }
+ _isRoot = true;
+ }
+
+ public ImportedArrowSchema(CArrowSchema* handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot && _data->release != null)
+ {
+ _data->release(_data);
Review Comment:
I don't think we should be releasing anything that we don't own. `_data` was passed by the public API, right? We don't own it, we shouldn't dispose of it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1132957233
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,268 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.New();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportType(IArrowType datatype, CArrowSchema* schema)
+ {
+ if (schema == null)
+ {
+ throw new ArgumentNullException("Cannot export schema to a null pointer.");
+ }
+ if (schema->release != null)
+ {
+ throw new ArgumentException("Cannot export schema to a struct that is already initialized.");
+ }
+
+ schema->format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ schema->name = null;
+ schema->metadata = null;
+ schema->flags = GetFlags(datatype);
+
+ schema->children = ConstructChildren(datatype, out var numChildren);
+ schema->n_children = numChildren;
+
+ schema->dictionary = ConstructDictionary(datatype);
+
+ schema->release = (delegate* unmanaged[Stdcall]<CArrowSchema*, void>)Marshal.GetFunctionPointerForDelegate(
+ ReleaseCArrowSchema);
+
+ schema->private_data = null;
+ }
+
+ /// <summary>
+ /// Export a field to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="field">The field to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.New();
+ /// CArrowSchemaExporter.ExportType(field, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportField(Field field, CArrowSchema* schema)
+ {
+ ExportType(field.DataType, schema);
+ schema->name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ schema->metadata = null;
+ schema->flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Export a schema to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="schema">The schema to export</param>
+ /// <param name="out_schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.New();
+ /// CArrowSchemaExporter.ExportType(schema, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportSchema(Schema schema, CArrowSchema* out_schema)
+ {
+ var structType = new StructType(schema.FieldsList);
+ // TODO: top-level metadata
+ ExportType(structType, out_schema);
+ }
+
+ private static char FormatTimeUnit(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ switch (datatype)
+ {
+ case NullType _: return "n";
+ case BooleanType _: return "b";
+ // Integers
+ case Int8Type _: return "c";
+ case UInt8Type _: return "C";
+ case Int16Type _: return "s";
+ case UInt16Type _: return "S";
+ case Int32Type _: return "i";
+ case UInt32Type _: return "I";
+ case Int64Type _: return "l";
+ case UInt64Type _: return "L";
+ // Floats
+ case HalfFloatType _: return "e";
+ case FloatType _: return "f";
+ case DoubleType _: return "g";
+ // Decimal
+ case Decimal128Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale}";
+ case Decimal256Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale},256";
+ // Binary
+ case BinaryType _: return "z";
+ case StringType _: return "u";
+ case FixedSizeBinaryType binaryType:
+ return $"w:{binaryType.ByteWidth}";
+ // Date
+ case Date32Type _: return "tdD";
+ case Date64Type _: return "tdm";
+ // Time
+ case Time32Type timeType:
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ case Time64Type timeType:
+ // Same prefix as Time32, but allowed time units are different.
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ // Timestamp
+ case TimestampType timestampType:
+ return String.Format("ts{0}:{1}", FormatTimeUnit(timestampType.Unit), timestampType.Timezone);
+ // Nested
+ case ListType _: return "+l";
+ case StructType _: return "+s";
+ // Dictionary
+ case DictionaryType dictionaryType:
+ return GetFormat(dictionaryType.IndexType);
+ default: throw new NotImplementedException($"Exporting {datatype.Name} not implemented");
+ };
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= CArrowSchema.ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= CArrowSchema.ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static unsafe CArrowSchema** ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ var pointerList = (CArrowSchema**)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ CArrowSchema* cSchema = CArrowSchema.New();
+ ExportField(fields[i], cSchema);
+ pointerList[i] = cSchema;
+ }
+
+ return pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return null;
+ }
+ }
+
+ private static unsafe CArrowSchema* ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ CArrowSchema* cSchema = CArrowSchema.New();
+ ExportType(dictType.ValueType, cSchema);
+ return cSchema;
+ }
+ else
+ {
+ return null;
+ }
+ }
+
+ private static unsafe void ReleaseCArrowSchema(CArrowSchema* schema)
+ {
+ if (schema == null) return;
+ if (schema->release == null) return;
Review Comment:
`release` is set to null in a release function to indicate it has already been released. This check makes it idempotent.
See: https://arrow.apache.org/docs/format/CDataInterface.html#release-callback-semantics-for-producers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1120559599
##########
csharp/test/Apache.Arrow.Tests/CDataInterfaceSchemaTests.cs:
##########
@@ -0,0 +1,300 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.C;
+using Apache.Arrow.Types;
+using Python.Runtime;
+using Xunit;
+
+namespace Apache.Arrow.Tests
+{
+ public class CDataSchemaTest
Review Comment:
That's a good point. I will add that, but also want to make sure that we don't accidentally skip this in CI without realizing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1120673139
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
Review Comment:
This doesn't seem to be recognized by `netstandard1.3` and `netstandard2.0`. 😞
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1122546409
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
+ {
+ format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ name = IntPtr.Zero;
+ metadata = IntPtr.Zero;
+ flags = GetFlags(datatype);
+
+ children = ConstructChildren(datatype, out var numChildren);
+ n_children = numChildren;
+
+ dictionary = ConstructDictionary(datatype);
+
+ release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+
+ Marshal.FreeHGlobal(schema.format);
+ Marshal.FreeHGlobal(schema.name);
+ Marshal.FreeHGlobal(schema.metadata);
+
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.GetChild(i));
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ public CArrowSchema(Field field) : this(field.DataType)
+ {
+ name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ metadata = IntPtr.Zero;
+ flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ public CArrowSchema(Schema schema) : this(new StructType(schema.Fields.Values.ToList()))
+ {
+ // TODO: top-level metadata
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public void Export(IntPtr ptr)
+ {
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ }
+
+ /// <summary>
+ /// Allocated a new pointer to an uninitialized CArrowSchema.
+ /// </summary>
+ /// <remarks>This is used to import schemas. See <see cref="ImportType"/>,
+ /// <see cref="ImportField"/>, and <see cref="ImportSchema"/>. Once data
+ /// is imported, this pointer must be cleaned up with <see cref="FreePtr"/>.
+ /// </remarks>
+ public static IntPtr AllocateUninitialized()
+ {
+ return Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchema.ImportType(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static ArrowType ImportType(IntPtr ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchema.ImportField(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Field ImportField(IntPtr ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchema.ImportSchema(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Schema ImportSchema(IntPtr ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ public const int ArrowFlagDictionaryOrdered = 1;
Review Comment:
I doubt it will matter but maybe use longs here given the flags field is a long?
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
Review Comment:
Not something that necessarily needs fixing but this pattern is probably obsolete. Nowadays you can just [switch on the type](https://systemoutofmemory.com/blogs/the-programmer-blog/c-sharp-switch-on-type).
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
Review Comment:
Why not?
```
public string name;
```
Here and elsewhere.
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
+ {
+ format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ name = IntPtr.Zero;
+ metadata = IntPtr.Zero;
+ flags = GetFlags(datatype);
+
+ children = ConstructChildren(datatype, out var numChildren);
+ n_children = numChildren;
+
+ dictionary = ConstructDictionary(datatype);
+
+ release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+
+ Marshal.FreeHGlobal(schema.format);
+ Marshal.FreeHGlobal(schema.name);
+ Marshal.FreeHGlobal(schema.metadata);
+
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.GetChild(i));
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ public CArrowSchema(Field field) : this(field.DataType)
+ {
+ name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ metadata = IntPtr.Zero;
+ flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ public CArrowSchema(Schema schema) : this(new StructType(schema.Fields.Values.ToList()))
+ {
+ // TODO: top-level metadata
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public void Export(IntPtr ptr)
+ {
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ }
+
+ /// <summary>
+ /// Allocated a new pointer to an uninitialized CArrowSchema.
+ /// </summary>
+ /// <remarks>This is used to import schemas. See <see cref="ImportType"/>,
+ /// <see cref="ImportField"/>, and <see cref="ImportSchema"/>. Once data
+ /// is imported, this pointer must be cleaned up with <see cref="FreePtr"/>.
+ /// </remarks>
+ public static IntPtr AllocateUninitialized()
+ {
+ return Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchema.ImportType(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static ArrowType ImportType(IntPtr ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchema.ImportField(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Field ImportField(IntPtr ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchema.ImportSchema(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Schema ImportSchema(IntPtr ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ public const int ArrowFlagDictionaryOrdered = 1;
+ public const int ArrowFlagNullable = 2;
+ public const int ArrowFlagMapKeysSorted = 4;
+
+ private class TypeFormatter :
+ IArrowTypeVisitor<NullType>,
+ IArrowTypeVisitor<BooleanType>,
+ IArrowTypeVisitor<Int8Type>,
+ IArrowTypeVisitor<Int16Type>,
+ IArrowTypeVisitor<Int32Type>,
+ IArrowTypeVisitor<Int64Type>,
+ IArrowTypeVisitor<UInt8Type>,
+ IArrowTypeVisitor<UInt16Type>,
+ IArrowTypeVisitor<UInt32Type>,
+ IArrowTypeVisitor<UInt64Type>,
+ IArrowTypeVisitor<HalfFloatType>,
+ IArrowTypeVisitor<FloatType>,
+ IArrowTypeVisitor<DoubleType>,
+ IArrowTypeVisitor<Decimal128Type>,
+ IArrowTypeVisitor<Decimal256Type>,
+ IArrowTypeVisitor<Date32Type>,
+ IArrowTypeVisitor<Date64Type>,
+ IArrowTypeVisitor<Time32Type>,
+ IArrowTypeVisitor<Time64Type>,
+ IArrowTypeVisitor<TimestampType>,
+ IArrowTypeVisitor<StringType>,
+ IArrowTypeVisitor<BinaryType>,
+ IArrowTypeVisitor<FixedSizeBinaryType>,
+ IArrowTypeVisitor<ListType>,
+ IArrowTypeVisitor<StructType>,
+ IArrowTypeVisitor<DictionaryType>
+ {
+ public string FormatString;
+ public void Visit(NullType _) => FormatString = "n";
+ public void Visit(BooleanType _) => FormatString = "b";
+ // Integers
+ public void Visit(Int8Type _) => FormatString = "c";
+ public void Visit(UInt8Type _) => FormatString = "C";
+ public void Visit(Int16Type _) => FormatString = "s";
+ public void Visit(UInt16Type _) => FormatString = "S";
+ public void Visit(Int32Type _) => FormatString = "i";
+ public void Visit(UInt32Type _) => FormatString = "I";
+ public void Visit(Int64Type _) => FormatString = "l";
+ public void Visit(UInt64Type _) => FormatString = "L";
+ // Floats
+ public void Visit(HalfFloatType _) => FormatString = "e";
+ public void Visit(FloatType _) => FormatString = "f";
+ public void Visit(DoubleType _) => FormatString = "g";
+ // Binary
+ public void Visit(BinaryType _) => FormatString = "z";
+ public void Visit(StringType _) => FormatString = "u";
+ public void Visit(FixedSizeBinaryType datatype)
+ {
+ FormatString = $"w:{datatype.ByteWidth}";
+ }
+ // Decimal
+ public void Visit(Decimal128Type datatype)
+ {
+ FormatString = $"d:{datatype.Precision},{datatype.Scale}";
+ }
+ public void Visit(Decimal256Type datatype)
+ {
+ FormatString = $"w:{datatype.Precision},{datatype.Scale},256";
Review Comment:
```suggestion
FormatString = $"d:{datatype.Precision},{datatype.Scale},256";
```
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
+ {
+ format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ name = IntPtr.Zero;
+ metadata = IntPtr.Zero;
+ flags = GetFlags(datatype);
+
+ children = ConstructChildren(datatype, out var numChildren);
+ n_children = numChildren;
+
+ dictionary = ConstructDictionary(datatype);
+
+ release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+
+ Marshal.FreeHGlobal(schema.format);
+ Marshal.FreeHGlobal(schema.name);
+ Marshal.FreeHGlobal(schema.metadata);
+
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.GetChild(i));
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ public CArrowSchema(Field field) : this(field.DataType)
+ {
+ name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ metadata = IntPtr.Zero;
+ flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ public CArrowSchema(Schema schema) : this(new StructType(schema.Fields.Values.ToList()))
+ {
+ // TODO: top-level metadata
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public void Export(IntPtr ptr)
+ {
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ }
+
+ /// <summary>
+ /// Allocated a new pointer to an uninitialized CArrowSchema.
+ /// </summary>
+ /// <remarks>This is used to import schemas. See <see cref="ImportType"/>,
+ /// <see cref="ImportField"/>, and <see cref="ImportSchema"/>. Once data
+ /// is imported, this pointer must be cleaned up with <see cref="FreePtr"/>.
+ /// </remarks>
+ public static IntPtr AllocateUninitialized()
+ {
+ return Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchema.ImportType(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static ArrowType ImportType(IntPtr ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchema.ImportField(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Field ImportField(IntPtr ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchema.ImportSchema(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Schema ImportSchema(IntPtr ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ public const int ArrowFlagDictionaryOrdered = 1;
+ public const int ArrowFlagNullable = 2;
+ public const int ArrowFlagMapKeysSorted = 4;
+
+ private class TypeFormatter :
+ IArrowTypeVisitor<NullType>,
+ IArrowTypeVisitor<BooleanType>,
+ IArrowTypeVisitor<Int8Type>,
+ IArrowTypeVisitor<Int16Type>,
+ IArrowTypeVisitor<Int32Type>,
+ IArrowTypeVisitor<Int64Type>,
+ IArrowTypeVisitor<UInt8Type>,
+ IArrowTypeVisitor<UInt16Type>,
+ IArrowTypeVisitor<UInt32Type>,
+ IArrowTypeVisitor<UInt64Type>,
+ IArrowTypeVisitor<HalfFloatType>,
+ IArrowTypeVisitor<FloatType>,
+ IArrowTypeVisitor<DoubleType>,
+ IArrowTypeVisitor<Decimal128Type>,
+ IArrowTypeVisitor<Decimal256Type>,
+ IArrowTypeVisitor<Date32Type>,
+ IArrowTypeVisitor<Date64Type>,
+ IArrowTypeVisitor<Time32Type>,
+ IArrowTypeVisitor<Time64Type>,
+ IArrowTypeVisitor<TimestampType>,
+ IArrowTypeVisitor<StringType>,
+ IArrowTypeVisitor<BinaryType>,
+ IArrowTypeVisitor<FixedSizeBinaryType>,
+ IArrowTypeVisitor<ListType>,
+ IArrowTypeVisitor<StructType>,
+ IArrowTypeVisitor<DictionaryType>
+ {
+ public string FormatString;
+ public void Visit(NullType _) => FormatString = "n";
+ public void Visit(BooleanType _) => FormatString = "b";
+ // Integers
+ public void Visit(Int8Type _) => FormatString = "c";
+ public void Visit(UInt8Type _) => FormatString = "C";
+ public void Visit(Int16Type _) => FormatString = "s";
+ public void Visit(UInt16Type _) => FormatString = "S";
+ public void Visit(Int32Type _) => FormatString = "i";
+ public void Visit(UInt32Type _) => FormatString = "I";
+ public void Visit(Int64Type _) => FormatString = "l";
+ public void Visit(UInt64Type _) => FormatString = "L";
+ // Floats
+ public void Visit(HalfFloatType _) => FormatString = "e";
+ public void Visit(FloatType _) => FormatString = "f";
+ public void Visit(DoubleType _) => FormatString = "g";
+ // Binary
+ public void Visit(BinaryType _) => FormatString = "z";
+ public void Visit(StringType _) => FormatString = "u";
+ public void Visit(FixedSizeBinaryType datatype)
+ {
+ FormatString = $"w:{datatype.ByteWidth}";
+ }
+ // Decimal
+ public void Visit(Decimal128Type datatype)
+ {
+ FormatString = $"d:{datatype.Precision},{datatype.Scale}";
+ }
+ public void Visit(Decimal256Type datatype)
+ {
+ FormatString = $"w:{datatype.Precision},{datatype.Scale},256";
+ }
+ // Date
+ public void Visit(Date32Type _) => FormatString = "tdD";
+ public void Visit(Date64Type _) => FormatString = "tdm";
+
+ private char TimeUnitComponent(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+ // Time
+ public void Visit(Time32Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ public void Visit(Time64Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ // Timestamp type
+ public void Visit(TimestampType datatype)
+ {
+ FormatString = String.Format("ts{0}:{1}", TimeUnitComponent(datatype.Unit), datatype.Timezone);
+ }
+ // Nested
+ public void Visit(ListType _) => FormatString = "+l";
+ public void Visit(StructType _) => FormatString = "+s";
+ // Dictionary
+ public void Visit(DictionaryType datatype)
+ {
+ // format string is that of the indices
+ datatype.IndexType.Accept(this);
+ }
+
+ // Unsupported
+ public void Visit(IArrowType type)
+ {
+ throw new NotImplementedException($"Exporting {type.Name} not implemented");
+ }
+ }
+
+ private sealed class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema _data;
+ private readonly IntPtr _handle;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(IntPtr handle)
+ {
+ _data = Marshal.PtrToStructure<CArrowSchema>(handle);
+ if (_data.release == null)
+ {
+ throw new Exception("Tried to import a schema that has already been released.");
+ }
+ _handle = handle;
+ _isRoot = true;
+ }
+
+ private ImportedArrowSchema(IntPtr handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot)
+ {
+ _data.release(_handle);
+ }
+ }
Review Comment:
`Dispose` should be idempotent. Also, there is [a pattern](https://learn.microsoft.com/en-us/dotnet/standard/garbage-collection/implementing-dispose) that is typically used with `Dispose()` and `Dispose(bool)` though that is probably not strictly needed here since there is no managed state.
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
+ {
+ format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ name = IntPtr.Zero;
+ metadata = IntPtr.Zero;
+ flags = GetFlags(datatype);
+
+ children = ConstructChildren(datatype, out var numChildren);
+ n_children = numChildren;
+
+ dictionary = ConstructDictionary(datatype);
+
+ release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+
+ Marshal.FreeHGlobal(schema.format);
+ Marshal.FreeHGlobal(schema.name);
+ Marshal.FreeHGlobal(schema.metadata);
+
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.GetChild(i));
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ public CArrowSchema(Field field) : this(field.DataType)
+ {
+ name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ metadata = IntPtr.Zero;
+ flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ public CArrowSchema(Schema schema) : this(new StructType(schema.Fields.Values.ToList()))
+ {
+ // TODO: top-level metadata
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public void Export(IntPtr ptr)
+ {
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ }
+
+ /// <summary>
+ /// Allocated a new pointer to an uninitialized CArrowSchema.
+ /// </summary>
+ /// <remarks>This is used to import schemas. See <see cref="ImportType"/>,
+ /// <see cref="ImportField"/>, and <see cref="ImportSchema"/>. Once data
+ /// is imported, this pointer must be cleaned up with <see cref="FreePtr"/>.
+ /// </remarks>
+ public static IntPtr AllocateUninitialized()
+ {
+ return Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchema.ImportType(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static ArrowType ImportType(IntPtr ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchema.ImportField(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Field ImportField(IntPtr ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchema.ImportSchema(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Schema ImportSchema(IntPtr ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ public const int ArrowFlagDictionaryOrdered = 1;
+ public const int ArrowFlagNullable = 2;
+ public const int ArrowFlagMapKeysSorted = 4;
+
+ private class TypeFormatter :
+ IArrowTypeVisitor<NullType>,
+ IArrowTypeVisitor<BooleanType>,
+ IArrowTypeVisitor<Int8Type>,
+ IArrowTypeVisitor<Int16Type>,
+ IArrowTypeVisitor<Int32Type>,
+ IArrowTypeVisitor<Int64Type>,
+ IArrowTypeVisitor<UInt8Type>,
+ IArrowTypeVisitor<UInt16Type>,
+ IArrowTypeVisitor<UInt32Type>,
+ IArrowTypeVisitor<UInt64Type>,
+ IArrowTypeVisitor<HalfFloatType>,
+ IArrowTypeVisitor<FloatType>,
+ IArrowTypeVisitor<DoubleType>,
+ IArrowTypeVisitor<Decimal128Type>,
+ IArrowTypeVisitor<Decimal256Type>,
+ IArrowTypeVisitor<Date32Type>,
+ IArrowTypeVisitor<Date64Type>,
+ IArrowTypeVisitor<Time32Type>,
+ IArrowTypeVisitor<Time64Type>,
+ IArrowTypeVisitor<TimestampType>,
+ IArrowTypeVisitor<StringType>,
+ IArrowTypeVisitor<BinaryType>,
+ IArrowTypeVisitor<FixedSizeBinaryType>,
+ IArrowTypeVisitor<ListType>,
+ IArrowTypeVisitor<StructType>,
+ IArrowTypeVisitor<DictionaryType>
+ {
+ public string FormatString;
+ public void Visit(NullType _) => FormatString = "n";
+ public void Visit(BooleanType _) => FormatString = "b";
+ // Integers
+ public void Visit(Int8Type _) => FormatString = "c";
+ public void Visit(UInt8Type _) => FormatString = "C";
+ public void Visit(Int16Type _) => FormatString = "s";
+ public void Visit(UInt16Type _) => FormatString = "S";
+ public void Visit(Int32Type _) => FormatString = "i";
+ public void Visit(UInt32Type _) => FormatString = "I";
+ public void Visit(Int64Type _) => FormatString = "l";
+ public void Visit(UInt64Type _) => FormatString = "L";
+ // Floats
+ public void Visit(HalfFloatType _) => FormatString = "e";
+ public void Visit(FloatType _) => FormatString = "f";
+ public void Visit(DoubleType _) => FormatString = "g";
+ // Binary
+ public void Visit(BinaryType _) => FormatString = "z";
+ public void Visit(StringType _) => FormatString = "u";
+ public void Visit(FixedSizeBinaryType datatype)
+ {
+ FormatString = $"w:{datatype.ByteWidth}";
+ }
+ // Decimal
+ public void Visit(Decimal128Type datatype)
+ {
+ FormatString = $"d:{datatype.Precision},{datatype.Scale}";
+ }
+ public void Visit(Decimal256Type datatype)
+ {
+ FormatString = $"w:{datatype.Precision},{datatype.Scale},256";
+ }
+ // Date
+ public void Visit(Date32Type _) => FormatString = "tdD";
+ public void Visit(Date64Type _) => FormatString = "tdm";
+
+ private char TimeUnitComponent(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+ // Time
+ public void Visit(Time32Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ public void Visit(Time64Type datatype)
Review Comment:
Maybe a comment here explaining that `tt`is reused because `Time32` and `Time64` will never have the same units?
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
+ {
+ format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ name = IntPtr.Zero;
+ metadata = IntPtr.Zero;
+ flags = GetFlags(datatype);
+
+ children = ConstructChildren(datatype, out var numChildren);
+ n_children = numChildren;
+
+ dictionary = ConstructDictionary(datatype);
+
+ release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+
+ Marshal.FreeHGlobal(schema.format);
+ Marshal.FreeHGlobal(schema.name);
+ Marshal.FreeHGlobal(schema.metadata);
+
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.GetChild(i));
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ public CArrowSchema(Field field) : this(field.DataType)
+ {
+ name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ metadata = IntPtr.Zero;
+ flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ public CArrowSchema(Schema schema) : this(new StructType(schema.Fields.Values.ToList()))
+ {
+ // TODO: top-level metadata
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public void Export(IntPtr ptr)
+ {
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ }
+
+ /// <summary>
+ /// Allocated a new pointer to an uninitialized CArrowSchema.
+ /// </summary>
+ /// <remarks>This is used to import schemas. See <see cref="ImportType"/>,
+ /// <see cref="ImportField"/>, and <see cref="ImportSchema"/>. Once data
+ /// is imported, this pointer must be cleaned up with <see cref="FreePtr"/>.
+ /// </remarks>
+ public static IntPtr AllocateUninitialized()
+ {
+ return Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchema.ImportType(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static ArrowType ImportType(IntPtr ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchema.ImportField(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Field ImportField(IntPtr ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchema.ImportSchema(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Schema ImportSchema(IntPtr ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ public const int ArrowFlagDictionaryOrdered = 1;
+ public const int ArrowFlagNullable = 2;
+ public const int ArrowFlagMapKeysSorted = 4;
+
+ private class TypeFormatter :
+ IArrowTypeVisitor<NullType>,
+ IArrowTypeVisitor<BooleanType>,
+ IArrowTypeVisitor<Int8Type>,
+ IArrowTypeVisitor<Int16Type>,
+ IArrowTypeVisitor<Int32Type>,
+ IArrowTypeVisitor<Int64Type>,
+ IArrowTypeVisitor<UInt8Type>,
+ IArrowTypeVisitor<UInt16Type>,
+ IArrowTypeVisitor<UInt32Type>,
+ IArrowTypeVisitor<UInt64Type>,
+ IArrowTypeVisitor<HalfFloatType>,
+ IArrowTypeVisitor<FloatType>,
+ IArrowTypeVisitor<DoubleType>,
+ IArrowTypeVisitor<Decimal128Type>,
+ IArrowTypeVisitor<Decimal256Type>,
+ IArrowTypeVisitor<Date32Type>,
+ IArrowTypeVisitor<Date64Type>,
+ IArrowTypeVisitor<Time32Type>,
+ IArrowTypeVisitor<Time64Type>,
+ IArrowTypeVisitor<TimestampType>,
+ IArrowTypeVisitor<StringType>,
+ IArrowTypeVisitor<BinaryType>,
+ IArrowTypeVisitor<FixedSizeBinaryType>,
+ IArrowTypeVisitor<ListType>,
+ IArrowTypeVisitor<StructType>,
+ IArrowTypeVisitor<DictionaryType>
+ {
+ public string FormatString;
+ public void Visit(NullType _) => FormatString = "n";
+ public void Visit(BooleanType _) => FormatString = "b";
+ // Integers
+ public void Visit(Int8Type _) => FormatString = "c";
+ public void Visit(UInt8Type _) => FormatString = "C";
+ public void Visit(Int16Type _) => FormatString = "s";
+ public void Visit(UInt16Type _) => FormatString = "S";
+ public void Visit(Int32Type _) => FormatString = "i";
+ public void Visit(UInt32Type _) => FormatString = "I";
+ public void Visit(Int64Type _) => FormatString = "l";
+ public void Visit(UInt64Type _) => FormatString = "L";
+ // Floats
+ public void Visit(HalfFloatType _) => FormatString = "e";
+ public void Visit(FloatType _) => FormatString = "f";
+ public void Visit(DoubleType _) => FormatString = "g";
+ // Binary
+ public void Visit(BinaryType _) => FormatString = "z";
+ public void Visit(StringType _) => FormatString = "u";
+ public void Visit(FixedSizeBinaryType datatype)
+ {
+ FormatString = $"w:{datatype.ByteWidth}";
+ }
+ // Decimal
+ public void Visit(Decimal128Type datatype)
+ {
+ FormatString = $"d:{datatype.Precision},{datatype.Scale}";
+ }
+ public void Visit(Decimal256Type datatype)
+ {
+ FormatString = $"w:{datatype.Precision},{datatype.Scale},256";
+ }
+ // Date
+ public void Visit(Date32Type _) => FormatString = "tdD";
+ public void Visit(Date64Type _) => FormatString = "tdm";
+
+ private char TimeUnitComponent(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+ // Time
+ public void Visit(Time32Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ public void Visit(Time64Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ // Timestamp type
+ public void Visit(TimestampType datatype)
+ {
+ FormatString = String.Format("ts{0}:{1}", TimeUnitComponent(datatype.Unit), datatype.Timezone);
+ }
+ // Nested
+ public void Visit(ListType _) => FormatString = "+l";
+ public void Visit(StructType _) => FormatString = "+s";
+ // Dictionary
+ public void Visit(DictionaryType datatype)
+ {
+ // format string is that of the indices
+ datatype.IndexType.Accept(this);
+ }
+
+ // Unsupported
+ public void Visit(IArrowType type)
+ {
+ throw new NotImplementedException($"Exporting {type.Name} not implemented");
+ }
+ }
+
+ private sealed class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema _data;
+ private readonly IntPtr _handle;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(IntPtr handle)
+ {
+ _data = Marshal.PtrToStructure<CArrowSchema>(handle);
+ if (_data.release == null)
+ {
+ throw new Exception("Tried to import a schema that has already been released.");
+ }
+ _handle = handle;
+ _isRoot = true;
+ }
+
+ private ImportedArrowSchema(IntPtr handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot)
+ {
+ _data.release(_handle);
+ }
+ }
+
+ public ArrowType GetAsType()
+ {
+ var format = StringUtil.PtrToStringUtf8(_data.format);
+ if (_data.dictionary != IntPtr.Zero)
+ {
+ ArrowType indicesType = format switch
+ {
+ "c" => new Int8Type(),
+ "C" => new UInt8Type(),
+ "s" => new Int16Type(),
+ "S" => new UInt16Type(),
+ "i" => new Int32Type(),
+ "I" => new UInt32Type(),
+ "l" => new Int64Type(),
+ "L" => new UInt64Type(),
+ _ => throw new InvalidDataException($"Indices must be an integer, but got format string {format}"),
+ };
+
+ var dictionarySchema = new ImportedArrowSchema(_data.dictionary, /*is_root*/ false);
+ ArrowType dictionaryType = dictionarySchema.GetAsType();
+
+ bool ordered = (_data.flags & CArrowSchema.ArrowFlagNullable) == CArrowSchema.ArrowFlagNullable;
Review Comment:
ArrowFlagDictionaryOrdered?
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
+ {
+ format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ name = IntPtr.Zero;
+ metadata = IntPtr.Zero;
+ flags = GetFlags(datatype);
+
+ children = ConstructChildren(datatype, out var numChildren);
+ n_children = numChildren;
+
+ dictionary = ConstructDictionary(datatype);
+
+ release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+
+ Marshal.FreeHGlobal(schema.format);
+ Marshal.FreeHGlobal(schema.name);
+ Marshal.FreeHGlobal(schema.metadata);
+
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.GetChild(i));
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ public CArrowSchema(Field field) : this(field.DataType)
+ {
+ name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ metadata = IntPtr.Zero;
+ flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ public CArrowSchema(Schema schema) : this(new StructType(schema.Fields.Values.ToList()))
+ {
+ // TODO: top-level metadata
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public void Export(IntPtr ptr)
+ {
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ }
+
+ /// <summary>
+ /// Allocated a new pointer to an uninitialized CArrowSchema.
+ /// </summary>
+ /// <remarks>This is used to import schemas. See <see cref="ImportType"/>,
+ /// <see cref="ImportField"/>, and <see cref="ImportSchema"/>. Once data
+ /// is imported, this pointer must be cleaned up with <see cref="FreePtr"/>.
+ /// </remarks>
+ public static IntPtr AllocateUninitialized()
+ {
+ return Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchema.ImportType(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static ArrowType ImportType(IntPtr ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchema.ImportField(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Field ImportField(IntPtr ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchema.ImportSchema(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Schema ImportSchema(IntPtr ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ public const int ArrowFlagDictionaryOrdered = 1;
+ public const int ArrowFlagNullable = 2;
+ public const int ArrowFlagMapKeysSorted = 4;
+
+ private class TypeFormatter :
+ IArrowTypeVisitor<NullType>,
+ IArrowTypeVisitor<BooleanType>,
+ IArrowTypeVisitor<Int8Type>,
+ IArrowTypeVisitor<Int16Type>,
+ IArrowTypeVisitor<Int32Type>,
+ IArrowTypeVisitor<Int64Type>,
+ IArrowTypeVisitor<UInt8Type>,
+ IArrowTypeVisitor<UInt16Type>,
+ IArrowTypeVisitor<UInt32Type>,
+ IArrowTypeVisitor<UInt64Type>,
+ IArrowTypeVisitor<HalfFloatType>,
+ IArrowTypeVisitor<FloatType>,
+ IArrowTypeVisitor<DoubleType>,
+ IArrowTypeVisitor<Decimal128Type>,
+ IArrowTypeVisitor<Decimal256Type>,
+ IArrowTypeVisitor<Date32Type>,
+ IArrowTypeVisitor<Date64Type>,
+ IArrowTypeVisitor<Time32Type>,
+ IArrowTypeVisitor<Time64Type>,
+ IArrowTypeVisitor<TimestampType>,
+ IArrowTypeVisitor<StringType>,
+ IArrowTypeVisitor<BinaryType>,
+ IArrowTypeVisitor<FixedSizeBinaryType>,
+ IArrowTypeVisitor<ListType>,
+ IArrowTypeVisitor<StructType>,
+ IArrowTypeVisitor<DictionaryType>
+ {
+ public string FormatString;
+ public void Visit(NullType _) => FormatString = "n";
+ public void Visit(BooleanType _) => FormatString = "b";
+ // Integers
+ public void Visit(Int8Type _) => FormatString = "c";
+ public void Visit(UInt8Type _) => FormatString = "C";
+ public void Visit(Int16Type _) => FormatString = "s";
+ public void Visit(UInt16Type _) => FormatString = "S";
+ public void Visit(Int32Type _) => FormatString = "i";
+ public void Visit(UInt32Type _) => FormatString = "I";
+ public void Visit(Int64Type _) => FormatString = "l";
+ public void Visit(UInt64Type _) => FormatString = "L";
+ // Floats
+ public void Visit(HalfFloatType _) => FormatString = "e";
+ public void Visit(FloatType _) => FormatString = "f";
+ public void Visit(DoubleType _) => FormatString = "g";
+ // Binary
+ public void Visit(BinaryType _) => FormatString = "z";
+ public void Visit(StringType _) => FormatString = "u";
+ public void Visit(FixedSizeBinaryType datatype)
+ {
+ FormatString = $"w:{datatype.ByteWidth}";
+ }
+ // Decimal
+ public void Visit(Decimal128Type datatype)
+ {
+ FormatString = $"d:{datatype.Precision},{datatype.Scale}";
+ }
+ public void Visit(Decimal256Type datatype)
+ {
+ FormatString = $"w:{datatype.Precision},{datatype.Scale},256";
+ }
+ // Date
+ public void Visit(Date32Type _) => FormatString = "tdD";
+ public void Visit(Date64Type _) => FormatString = "tdm";
+
+ private char TimeUnitComponent(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+ // Time
+ public void Visit(Time32Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ public void Visit(Time64Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ // Timestamp type
+ public void Visit(TimestampType datatype)
+ {
+ FormatString = String.Format("ts{0}:{1}", TimeUnitComponent(datatype.Unit), datatype.Timezone);
+ }
+ // Nested
+ public void Visit(ListType _) => FormatString = "+l";
+ public void Visit(StructType _) => FormatString = "+s";
+ // Dictionary
+ public void Visit(DictionaryType datatype)
+ {
+ // format string is that of the indices
+ datatype.IndexType.Accept(this);
+ }
+
+ // Unsupported
+ public void Visit(IArrowType type)
+ {
+ throw new NotImplementedException($"Exporting {type.Name} not implemented");
+ }
+ }
+
+ private sealed class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema _data;
+ private readonly IntPtr _handle;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(IntPtr handle)
+ {
+ _data = Marshal.PtrToStructure<CArrowSchema>(handle);
+ if (_data.release == null)
+ {
+ throw new Exception("Tried to import a schema that has already been released.");
+ }
+ _handle = handle;
+ _isRoot = true;
+ }
+
+ private ImportedArrowSchema(IntPtr handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot)
+ {
+ _data.release(_handle);
+ }
+ }
+
+ public ArrowType GetAsType()
+ {
+ var format = StringUtil.PtrToStringUtf8(_data.format);
+ if (_data.dictionary != IntPtr.Zero)
+ {
+ ArrowType indicesType = format switch
+ {
+ "c" => new Int8Type(),
+ "C" => new UInt8Type(),
+ "s" => new Int16Type(),
+ "S" => new UInt16Type(),
+ "i" => new Int32Type(),
+ "I" => new UInt32Type(),
+ "l" => new Int64Type(),
+ "L" => new UInt64Type(),
+ _ => throw new InvalidDataException($"Indices must be an integer, but got format string {format}"),
+ };
+
+ var dictionarySchema = new ImportedArrowSchema(_data.dictionary, /*is_root*/ false);
Review Comment:
```suggestion
var dictionarySchema = new ImportedArrowSchema(_data.dictionary, isRoot: false);
```
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
+ {
+ format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ name = IntPtr.Zero;
+ metadata = IntPtr.Zero;
+ flags = GetFlags(datatype);
+
+ children = ConstructChildren(datatype, out var numChildren);
+ n_children = numChildren;
+
+ dictionary = ConstructDictionary(datatype);
+
+ release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+
+ Marshal.FreeHGlobal(schema.format);
+ Marshal.FreeHGlobal(schema.name);
+ Marshal.FreeHGlobal(schema.metadata);
+
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.GetChild(i));
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ public CArrowSchema(Field field) : this(field.DataType)
+ {
+ name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ metadata = IntPtr.Zero;
+ flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ public CArrowSchema(Schema schema) : this(new StructType(schema.Fields.Values.ToList()))
+ {
+ // TODO: top-level metadata
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public void Export(IntPtr ptr)
+ {
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ }
+
+ /// <summary>
+ /// Allocated a new pointer to an uninitialized CArrowSchema.
+ /// </summary>
+ /// <remarks>This is used to import schemas. See <see cref="ImportType"/>,
+ /// <see cref="ImportField"/>, and <see cref="ImportSchema"/>. Once data
+ /// is imported, this pointer must be cleaned up with <see cref="FreePtr"/>.
+ /// </remarks>
+ public static IntPtr AllocateUninitialized()
+ {
+ return Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchema.ImportType(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static ArrowType ImportType(IntPtr ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchema.ImportField(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Field ImportField(IntPtr ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchema.ImportSchema(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Schema ImportSchema(IntPtr ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ public const int ArrowFlagDictionaryOrdered = 1;
+ public const int ArrowFlagNullable = 2;
+ public const int ArrowFlagMapKeysSorted = 4;
+
+ private class TypeFormatter :
+ IArrowTypeVisitor<NullType>,
+ IArrowTypeVisitor<BooleanType>,
+ IArrowTypeVisitor<Int8Type>,
+ IArrowTypeVisitor<Int16Type>,
+ IArrowTypeVisitor<Int32Type>,
+ IArrowTypeVisitor<Int64Type>,
+ IArrowTypeVisitor<UInt8Type>,
+ IArrowTypeVisitor<UInt16Type>,
+ IArrowTypeVisitor<UInt32Type>,
+ IArrowTypeVisitor<UInt64Type>,
+ IArrowTypeVisitor<HalfFloatType>,
+ IArrowTypeVisitor<FloatType>,
+ IArrowTypeVisitor<DoubleType>,
+ IArrowTypeVisitor<Decimal128Type>,
+ IArrowTypeVisitor<Decimal256Type>,
+ IArrowTypeVisitor<Date32Type>,
+ IArrowTypeVisitor<Date64Type>,
+ IArrowTypeVisitor<Time32Type>,
+ IArrowTypeVisitor<Time64Type>,
+ IArrowTypeVisitor<TimestampType>,
+ IArrowTypeVisitor<StringType>,
+ IArrowTypeVisitor<BinaryType>,
+ IArrowTypeVisitor<FixedSizeBinaryType>,
+ IArrowTypeVisitor<ListType>,
+ IArrowTypeVisitor<StructType>,
+ IArrowTypeVisitor<DictionaryType>
+ {
+ public string FormatString;
+ public void Visit(NullType _) => FormatString = "n";
+ public void Visit(BooleanType _) => FormatString = "b";
+ // Integers
+ public void Visit(Int8Type _) => FormatString = "c";
+ public void Visit(UInt8Type _) => FormatString = "C";
+ public void Visit(Int16Type _) => FormatString = "s";
+ public void Visit(UInt16Type _) => FormatString = "S";
+ public void Visit(Int32Type _) => FormatString = "i";
+ public void Visit(UInt32Type _) => FormatString = "I";
+ public void Visit(Int64Type _) => FormatString = "l";
+ public void Visit(UInt64Type _) => FormatString = "L";
+ // Floats
+ public void Visit(HalfFloatType _) => FormatString = "e";
+ public void Visit(FloatType _) => FormatString = "f";
+ public void Visit(DoubleType _) => FormatString = "g";
+ // Binary
+ public void Visit(BinaryType _) => FormatString = "z";
+ public void Visit(StringType _) => FormatString = "u";
+ public void Visit(FixedSizeBinaryType datatype)
+ {
+ FormatString = $"w:{datatype.ByteWidth}";
+ }
+ // Decimal
+ public void Visit(Decimal128Type datatype)
+ {
+ FormatString = $"d:{datatype.Precision},{datatype.Scale}";
+ }
+ public void Visit(Decimal256Type datatype)
+ {
+ FormatString = $"w:{datatype.Precision},{datatype.Scale},256";
+ }
+ // Date
+ public void Visit(Date32Type _) => FormatString = "tdD";
+ public void Visit(Date64Type _) => FormatString = "tdm";
+
+ private char TimeUnitComponent(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+ // Time
+ public void Visit(Time32Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ public void Visit(Time64Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ // Timestamp type
+ public void Visit(TimestampType datatype)
+ {
+ FormatString = String.Format("ts{0}:{1}", TimeUnitComponent(datatype.Unit), datatype.Timezone);
+ }
+ // Nested
+ public void Visit(ListType _) => FormatString = "+l";
+ public void Visit(StructType _) => FormatString = "+s";
+ // Dictionary
+ public void Visit(DictionaryType datatype)
+ {
+ // format string is that of the indices
+ datatype.IndexType.Accept(this);
+ }
+
+ // Unsupported
+ public void Visit(IArrowType type)
+ {
+ throw new NotImplementedException($"Exporting {type.Name} not implemented");
+ }
+ }
+
+ private sealed class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema _data;
+ private readonly IntPtr _handle;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(IntPtr handle)
+ {
+ _data = Marshal.PtrToStructure<CArrowSchema>(handle);
+ if (_data.release == null)
+ {
+ throw new Exception("Tried to import a schema that has already been released.");
+ }
+ _handle = handle;
+ _isRoot = true;
+ }
+
+ private ImportedArrowSchema(IntPtr handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot)
+ {
+ _data.release(_handle);
+ }
+ }
+
+ public ArrowType GetAsType()
+ {
+ var format = StringUtil.PtrToStringUtf8(_data.format);
+ if (_data.dictionary != IntPtr.Zero)
+ {
+ ArrowType indicesType = format switch
+ {
+ "c" => new Int8Type(),
+ "C" => new UInt8Type(),
+ "s" => new Int16Type(),
+ "S" => new UInt16Type(),
+ "i" => new Int32Type(),
+ "I" => new UInt32Type(),
+ "l" => new Int64Type(),
+ "L" => new UInt64Type(),
+ _ => throw new InvalidDataException($"Indices must be an integer, but got format string {format}"),
+ };
+
+ var dictionarySchema = new ImportedArrowSchema(_data.dictionary, /*is_root*/ false);
+ ArrowType dictionaryType = dictionarySchema.GetAsType();
+
+ bool ordered = (_data.flags & CArrowSchema.ArrowFlagNullable) == CArrowSchema.ArrowFlagNullable;
+
+ return new DictionaryType(indicesType, dictionaryType, ordered);
+ }
+
+ // Special handling for nested types
+ if (format == "+l")
+ {
+ if (_data.n_children != 1)
+ {
+ throw new Exception("Expected list type to have exactly one child.");
+ }
+ ImportedArrowSchema childSchema;
+ if (_data.GetChild(0) == IntPtr.Zero)
+ {
+ throw new Exception("Expected list type child to be non-null.");
+ }
+ childSchema = new ImportedArrowSchema(_data.GetChild(0));
Review Comment:
`is_root=false`?
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
Review Comment:
Ah, I read up some more on this. I don't think `StdCall` is the default (that would be `WinApi` which defaults to `StdCall` on Windows and `CDecl` on Linux).
That being said, I don't think we need to worry about calling convention anywhere as it is only relevant when working with 32-bit binaries: https://stackoverflow.com/questions/34832679/is-the-callingconvention-ignored-in-64-bit-net-applications
I'm pretty sure Arrow-C++ doesn't even build on 32-bit (https://github.com/apache/arrow/issues/32111) so this is probably a non-issue. With that in mind, perhaps skip the `UnamangedFunctionPointer` entirely? It seems its [only purpose](https://stackoverflow.com/questions/17364738/c-pinvoke-what-does-unmanagedfunctionpointer-do) is to specify the calling convention.
##########
csharp/src/Apache.Arrow/C/StringUtil.cs:
##########
@@ -0,0 +1,67 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Runtime.InteropServices;
+
+// Adapted from:
+// https://github.com/G-Research/ParquetSharp/blob/467d99298fb5a5b9d5935b9c8dbde95e63954dd3/csharp/StringUtil.cs
+
+namespace Apache.Arrow.C
+{
+
+ internal static class StringUtil
+ {
+ public static unsafe IntPtr ToCStringUtf8(string str)
+ {
+ var utf8 = System.Text.Encoding.UTF8;
+ int byteCount = utf8.GetByteCount(str);
+ byte* byteArray = (byte*)Marshal.AllocHGlobal(byteCount + 1);
+
+ fixed (char* chars = str)
+ {
+ utf8.GetBytes(chars, str.Length, byteArray, byteCount);
+ }
+
+ // Need to make sure it is nul-terminated.
+ byteArray[byteCount] = 0;
+
+ return (IntPtr)byteArray;
+ }
+
+ public static string PtrToStringUtf8(IntPtr ptr)
+ {
+#if NETSTANDARD2_1_OR_GREATER
+ return Marshal.PtrToStringUTF8(ptr);
+#else
+ if (ptr == IntPtr.Zero)
+ {
+ return null;
+ }
+
+ unsafe
+ {
+ var s = (byte*)ptr;
+ int length;
+ for (length = 0; s[length] != '\0'; ++length)
+ {
+ }
+
+ return System.Text.Encoding.UTF8.GetString(s, length);
+ }
+#endif
+ }
+ }
+}
Review Comment:
newline
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
Review Comment:
Ah, nevermind. It seems that .NET tries to take ownership of the memory and free the string when it's done.
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
+ {
+ format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ name = IntPtr.Zero;
+ metadata = IntPtr.Zero;
+ flags = GetFlags(datatype);
+
+ children = ConstructChildren(datatype, out var numChildren);
+ n_children = numChildren;
+
+ dictionary = ConstructDictionary(datatype);
+
+ release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+
+ Marshal.FreeHGlobal(schema.format);
+ Marshal.FreeHGlobal(schema.name);
+ Marshal.FreeHGlobal(schema.metadata);
+
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.GetChild(i));
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ public CArrowSchema(Field field) : this(field.DataType)
+ {
+ name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ metadata = IntPtr.Zero;
+ flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ public CArrowSchema(Schema schema) : this(new StructType(schema.Fields.Values.ToList()))
+ {
+ // TODO: top-level metadata
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public void Export(IntPtr ptr)
+ {
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ }
+
+ /// <summary>
+ /// Allocated a new pointer to an uninitialized CArrowSchema.
+ /// </summary>
+ /// <remarks>This is used to import schemas. See <see cref="ImportType"/>,
+ /// <see cref="ImportField"/>, and <see cref="ImportSchema"/>. Once data
+ /// is imported, this pointer must be cleaned up with <see cref="FreePtr"/>.
+ /// </remarks>
+ public static IntPtr AllocateUninitialized()
+ {
+ return Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchema.ImportType(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static ArrowType ImportType(IntPtr ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchema.ImportField(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Field ImportField(IntPtr ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchema.ImportSchema(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Schema ImportSchema(IntPtr ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ public const int ArrowFlagDictionaryOrdered = 1;
+ public const int ArrowFlagNullable = 2;
+ public const int ArrowFlagMapKeysSorted = 4;
+
+ private class TypeFormatter :
+ IArrowTypeVisitor<NullType>,
+ IArrowTypeVisitor<BooleanType>,
+ IArrowTypeVisitor<Int8Type>,
+ IArrowTypeVisitor<Int16Type>,
+ IArrowTypeVisitor<Int32Type>,
+ IArrowTypeVisitor<Int64Type>,
+ IArrowTypeVisitor<UInt8Type>,
+ IArrowTypeVisitor<UInt16Type>,
+ IArrowTypeVisitor<UInt32Type>,
+ IArrowTypeVisitor<UInt64Type>,
+ IArrowTypeVisitor<HalfFloatType>,
+ IArrowTypeVisitor<FloatType>,
+ IArrowTypeVisitor<DoubleType>,
+ IArrowTypeVisitor<Decimal128Type>,
+ IArrowTypeVisitor<Decimal256Type>,
+ IArrowTypeVisitor<Date32Type>,
+ IArrowTypeVisitor<Date64Type>,
+ IArrowTypeVisitor<Time32Type>,
+ IArrowTypeVisitor<Time64Type>,
+ IArrowTypeVisitor<TimestampType>,
+ IArrowTypeVisitor<StringType>,
+ IArrowTypeVisitor<BinaryType>,
+ IArrowTypeVisitor<FixedSizeBinaryType>,
+ IArrowTypeVisitor<ListType>,
+ IArrowTypeVisitor<StructType>,
+ IArrowTypeVisitor<DictionaryType>
+ {
+ public string FormatString;
+ public void Visit(NullType _) => FormatString = "n";
+ public void Visit(BooleanType _) => FormatString = "b";
+ // Integers
+ public void Visit(Int8Type _) => FormatString = "c";
+ public void Visit(UInt8Type _) => FormatString = "C";
+ public void Visit(Int16Type _) => FormatString = "s";
+ public void Visit(UInt16Type _) => FormatString = "S";
+ public void Visit(Int32Type _) => FormatString = "i";
+ public void Visit(UInt32Type _) => FormatString = "I";
+ public void Visit(Int64Type _) => FormatString = "l";
+ public void Visit(UInt64Type _) => FormatString = "L";
+ // Floats
+ public void Visit(HalfFloatType _) => FormatString = "e";
+ public void Visit(FloatType _) => FormatString = "f";
+ public void Visit(DoubleType _) => FormatString = "g";
+ // Binary
+ public void Visit(BinaryType _) => FormatString = "z";
+ public void Visit(StringType _) => FormatString = "u";
+ public void Visit(FixedSizeBinaryType datatype)
+ {
+ FormatString = $"w:{datatype.ByteWidth}";
+ }
+ // Decimal
+ public void Visit(Decimal128Type datatype)
+ {
+ FormatString = $"d:{datatype.Precision},{datatype.Scale}";
+ }
+ public void Visit(Decimal256Type datatype)
+ {
+ FormatString = $"w:{datatype.Precision},{datatype.Scale},256";
+ }
+ // Date
+ public void Visit(Date32Type _) => FormatString = "tdD";
+ public void Visit(Date64Type _) => FormatString = "tdm";
+
+ private char TimeUnitComponent(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+ // Time
+ public void Visit(Time32Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ public void Visit(Time64Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ // Timestamp type
+ public void Visit(TimestampType datatype)
+ {
+ FormatString = String.Format("ts{0}:{1}", TimeUnitComponent(datatype.Unit), datatype.Timezone);
+ }
+ // Nested
+ public void Visit(ListType _) => FormatString = "+l";
+ public void Visit(StructType _) => FormatString = "+s";
+ // Dictionary
+ public void Visit(DictionaryType datatype)
+ {
+ // format string is that of the indices
+ datatype.IndexType.Accept(this);
+ }
+
+ // Unsupported
+ public void Visit(IArrowType type)
+ {
+ throw new NotImplementedException($"Exporting {type.Name} not implemented");
+ }
+ }
+
+ private sealed class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema _data;
+ private readonly IntPtr _handle;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(IntPtr handle)
+ {
+ _data = Marshal.PtrToStructure<CArrowSchema>(handle);
+ if (_data.release == null)
+ {
+ throw new Exception("Tried to import a schema that has already been released.");
+ }
+ _handle = handle;
+ _isRoot = true;
+ }
+
+ private ImportedArrowSchema(IntPtr handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot)
+ {
+ _data.release(_handle);
+ }
+ }
+
+ public ArrowType GetAsType()
+ {
+ var format = StringUtil.PtrToStringUtf8(_data.format);
+ if (_data.dictionary != IntPtr.Zero)
+ {
+ ArrowType indicesType = format switch
+ {
+ "c" => new Int8Type(),
+ "C" => new UInt8Type(),
+ "s" => new Int16Type(),
+ "S" => new UInt16Type(),
+ "i" => new Int32Type(),
+ "I" => new UInt32Type(),
+ "l" => new Int64Type(),
+ "L" => new UInt64Type(),
+ _ => throw new InvalidDataException($"Indices must be an integer, but got format string {format}"),
+ };
+
+ var dictionarySchema = new ImportedArrowSchema(_data.dictionary, /*is_root*/ false);
+ ArrowType dictionaryType = dictionarySchema.GetAsType();
+
+ bool ordered = (_data.flags & CArrowSchema.ArrowFlagNullable) == CArrowSchema.ArrowFlagNullable;
+
+ return new DictionaryType(indicesType, dictionaryType, ordered);
+ }
+
+ // Special handling for nested types
+ if (format == "+l")
+ {
+ if (_data.n_children != 1)
+ {
+ throw new Exception("Expected list type to have exactly one child.");
+ }
+ ImportedArrowSchema childSchema;
+ if (_data.GetChild(0) == IntPtr.Zero)
+ {
+ throw new Exception("Expected list type child to be non-null.");
+ }
+ childSchema = new ImportedArrowSchema(_data.GetChild(0));
+
+ Field childField = childSchema.GetAsField();
+
+ return new ListType(childField);
+ }
+ else if (format == "+s")
+ {
+ var child_schemas = new ImportedArrowSchema[_data.n_children];
+
+ for (int i = 0; i < _data.n_children; i++)
+ {
+ if (_data.GetChild(i) == IntPtr.Zero)
+ {
+ throw new Exception("Expected struct type child to be non-null.");
+ }
+ child_schemas[i] = new ImportedArrowSchema(_data.GetChild(i));
Review Comment:
`is_root=false`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on PR #34133:
URL: https://github.com/apache/arrow/pull/34133#issuecomment-1446687108
@adamreeve would you be interested in reviewing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1118948166
##########
csharp/test/Apache.Arrow.Tests/CDataInterfaceSchemaTests.cs:
##########
@@ -0,0 +1,302 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Collections;
+using System.Collections.Generic;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.C;
+using Apache.Arrow.Types;
+using Python.Runtime;
+using Xunit;
+
+namespace Apache.Arrow.Tests
+{
+ public class CDataSchemaTest
+ {
+ public CDataSchemaTest()
+ {
+ PythonEngine.Initialize();
+ }
+
+ public static Schema GetTestSchema()
+ {
+ // TODO: Add more types
+ using (Py.GIL())
+ {
+ var schema = new Schema.Builder()
+ .Field(f => f.Name("null").DataType(NullType.Default).Nullable(true))
+ .Field(f => f.Name("bool").DataType(BooleanType.Default).Nullable(true))
+ .Field(f => f.Name("i8").DataType(Int8Type.Default).Nullable(true))
+ .Field(f => f.Name("u8").DataType(UInt8Type.Default).Nullable(true))
+ .Field(f => f.Name("i16").DataType(Int16Type.Default).Nullable(true))
+ .Field(f => f.Name("u16").DataType(UInt16Type.Default).Nullable(true))
+ .Field(f => f.Name("i32").DataType(Int32Type.Default).Nullable(true))
+ .Field(f => f.Name("u32").DataType(UInt32Type.Default).Nullable(true))
+ .Field(f => f.Name("i64").DataType(Int64Type.Default).Nullable(true))
+ .Field(f => f.Name("u64").DataType(UInt64Type.Default).Nullable(true))
+
+ .Field(f => f.Name("f16").DataType(HalfFloatType.Default).Nullable(true))
+ .Field(f => f.Name("f32").DataType(FloatType.Default).Nullable(true))
+ .Field(f => f.Name("f64").DataType(DoubleType.Default).Nullable(true))
+
+ .Field(f => f.Name("binary").DataType(BinaryType.Default).Nullable(false))
+ .Field(f => f.Name("string").DataType(StringType.Default).Nullable(false))
+
+ .Field(f => f.Name("date32").DataType(Date32Type.Default).Nullable(false))
+ .Field(f => f.Name("date64").DataType(Date64Type.Default).Nullable(false))
+ .Field(f => f.Name("time32_s").DataType(new Time32Type(TimeUnit.Second)).Nullable(false))
+ .Field(f => f.Name("time32_ms").DataType(new Time32Type(TimeUnit.Millisecond)).Nullable(false))
+ .Field(f => f.Name("time64_us").DataType(new Time64Type(TimeUnit.Microsecond)).Nullable(false))
+ .Field(f => f.Name("time64_ns").DataType(new Time64Type(TimeUnit.Nanosecond)).Nullable(false))
+
+ .Field(f => f.Name("list_string").DataType(new ListType(StringType.Default)).Nullable(false))
+ .Field(f => f.Name("list_list_i32").DataType(new ListType(new ListType(Int32Type.Default))).Nullable(false))
+
+ .Field(f => f.Name("dict_string").DataType(new DictionaryType(Int32Type.Default, StringType.Default, false)).Nullable(false))
+ .Field(f => f.Name("dict_string_ordered").DataType(new DictionaryType(Int32Type.Default, StringType.Default, true)).Nullable(false))
+ .Field(f => f.Name("list_dict_string").DataType(new ListType(new DictionaryType(Int32Type.Default, StringType.Default, false))).Nullable(false))
+
+ .Build();
+ return schema;
+ }
+ }
+
+ public static IEnumerable<dynamic> GetPythonFields()
+ {
+ using (Py.GIL())
+ {
+ dynamic pa = Py.Import("pyarrow");
+ yield return pa.field("null", pa.GetAttr("null").Invoke(), true);
+ yield return pa.field("bool", pa.bool_(), true);
+ yield return pa.field("i8", pa.int8(), true);
+ yield return pa.field("u8", pa.uint8(), true);
+ yield return pa.field("i16", pa.int16(), true);
+ yield return pa.field("u16", pa.uint16(), true);
+ yield return pa.field("i32", pa.int32(), true);
+ yield return pa.field("u32", pa.uint32(), true);
+ yield return pa.field("i64", pa.int64(), true);
+ yield return pa.field("u64", pa.uint64(), true);
+
+ yield return pa.field("f16", pa.float16(), true);
+ yield return pa.field("f32", pa.float32(), true);
+ yield return pa.field("f64", pa.float64(), true);
+
+ yield return pa.field("binary", pa.binary(), false);
+ yield return pa.field("string", pa.utf8(), false);
+
+ yield return pa.field("date32", pa.date32(), false);
+ yield return pa.field("date64", pa.date64(), false);
+ yield return pa.field("time32_s", pa.time32("s"), false);
+ yield return pa.field("time32_ms", pa.time32("ms"), false);
+ yield return pa.field("time64_us", pa.time64("us"), false);
+ yield return pa.field("time64_ns", pa.time64("ns"), false);
+
+ yield return pa.field("list_string", pa.list_(pa.utf8()), false);
+ yield return pa.field("list_list_i32", pa.list_(pa.list_(pa.int32())), false);
+
+ yield return pa.field("dict_string", pa.dictionary(pa.int32(), pa.utf8(), false), false);
+ yield return pa.field("dict_string_ordered", pa.dictionary(pa.int32(), pa.utf8(), true), false);
+ yield return pa.field("list_dict_string", pa.list_(pa.dictionary(pa.int32(), pa.utf8(), false)), false);
+ }
+ }
+
+ public static dynamic GetPythonSchema()
+ {
+ using (Py.GIL())
+ {
+ dynamic pa = Py.Import("pyarrow");
+ return pa.schema(GetPythonFields().ToList());
+ }
+ }
+
+ // Schemas created in Python, used in CSharp
+ [Fact]
+ public void ImportType()
+ {
+ PythonEngine.Initialize();
+ Schema schema = GetTestSchema();
+ IEnumerable<dynamic> py_fields = GetPythonFields();
+
+ foreach ((Field field, dynamic py_field) in schema.Fields.Values.AsEnumerable()
+ .Zip(py_fields))
+ {
+ var c_schema = new CArrowSchema();
+ IntPtr imported_ptr = c_schema.AllocateAsPtr();
+
+ using (Py.GIL())
+ {
+ dynamic py_datatype = py_field.type;
+ py_datatype._export_to_c(imported_ptr.ToInt64());
+ }
+
+ var dataTypeComparer = new ArrayTypeComparer(field.DataType);
+ var imported_type = new ImportedArrowSchema(imported_ptr);
Review Comment:
That's a good point. I need to fix the code style.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1118987851
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,519 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+using System;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+
+ public ReleaseCArrowSchema release;
+ // Check this out: https://github.com/G-Research/ParquetSharp/blob/386d91bd5e6fe6cb81583803447023c1359957c8/csharp/ParquetHandle.cs#L8
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.format_string;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ARROW_FLAG_NULLABLE;
+ }
+
+ if (datatype is DictionaryType)
+ {
+ if (((DictionaryType)datatype).Ordered)
+ {
+ flags |= ARROW_FLAG_DICTIONARY_ORDERED;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype)
+ {
+ if (datatype is NestedType)
+ {
+ var fields = ((NestedType)datatype).Fields;
+ int n_fields = fields.Count;
+
+ IntPtr* pointer_list = (IntPtr*)Marshal.AllocHGlobal(n_fields * sizeof(IntPtr));
+
+ for (var i = 0; i < n_fields; i++)
+ {
+ var c_schema = new CArrowSchema();
+ CArrowSchema.ExportField(fields[i], out c_schema);
+ IntPtr exported_schema = c_schema.AllocateAsPtr();
+ pointer_list[i] = exported_schema;
+ }
+
+ return pointer_list;
+ }
+ else
+ {
+ return (IntPtr*)IntPtr.Zero;
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType)
+ {
+ var c_schema = new CArrowSchema();
+ var value_type = ((DictionaryType)datatype).ValueType;
+ CArrowSchema.ExportDataType(value_type, out c_schema);
+ return c_schema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ /// <param name="schema">An uninitialized CArrowSchema.</param>
+ public static void ExportDataType(IArrowType datatype, out CArrowSchema schema)
+ {
+ schema.format = GetFormat(datatype);
+ schema.name = null;
+ schema.metadata = null;
+ schema.flags = GetFlags(datatype);
+
+ schema.children = ConstructChildren(datatype);
+ schema.n_children = datatype is NestedType ? ((NestedType)datatype).Fields.Count : 0;
+
+ schema.dictionary = ConstructDictionary(datatype);
+
+ schema.release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.children[i]);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ schema.private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ /// <param name="schema">An uninitialized CArrowSchema.</param>
+ public static void ExportField(Field field, out CArrowSchema schema)
+ {
+ ExportDataType(field.DataType, out schema);
+ schema.name = field.Name;
+ // TODO: field metadata
+ schema.metadata = null;
+ schema.flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ /// <param name="out_schema">An uninitialized CArrowSchema</param>
+ public static void ExportSchema(Schema schema, out CArrowSchema out_schema)
+ {
+ // TODO: top-level metadata
+ var struct_type = new StructType(schema.Fields.Values.ToList());
+ ExportDataType(struct_type, out out_schema);
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr"></param>
+ /// <returns></returns>
Review Comment:
Yes, filled this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1138629019
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+ /// <summary>
+ /// Allocate and zero-initialize an unmanaged pointer of this type.
+ /// </summary>
+ /// <remarks>
+ /// This pointer must later be freed by <see cref="Free"/>.
+ /// </remarks>
+ public static CArrowSchema* Create()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(sizeof(CArrowSchema));
+
+ ptr->format = null;
+ ptr->name = null;
+ ptr->metadata = null;
+ ptr->flags = 0;
+ ptr->n_children = 0;
+ ptr->children = null;
+ ptr->dictionary = null;
+ ptr->release = null;
+ ptr->private_data = null;
+
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="Create"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void Free(CArrowSchema* schema)
+ {
+ if (schema->release != null)
+ {
+ // Call release if not already called.
+ schema->release(schema);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema);
+ }
+
+
+ /// <summary>
+ /// Whether this field is semantically nullable (regardless of whether it actually has null values)
+ /// </summary>
+ public const long ArrowFlagDictionaryOrdered = 1;
+ /// <summary>
+ /// For dictionary-encoded types, whether the ordering of dictionary indices is semantically meaningful.
+ /// </summary>
+ public const long ArrowFlagNullable = 2;
Review Comment:
```suggestion
/// <summary>
/// For dictionary-encoded types, whether the ordering of dictionary indices is semantically meaningful.
/// </summary>
public const long ArrowFlagDictionaryOrdered = 1;
/// <summary>
/// Whether this field is semantically nullable (regardless of whether it actually has null values)
/// </summary>
public const long ArrowFlagNullable = 2;
```
Looks like these comments got swapped
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,301 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <remarks>
+ /// This will call the release callback on the passed struct, even if
+ /// this function fails.
+ /// </remarks>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
Review Comment:
```suggestion
/// Typically, you will allocate an uninitialized CArrowSchema pointer,
```
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
Review Comment:
Won't this trigger a call to `exportPtr->release`? Is that safe given that you've already passed ownership during the `foreign_import_function`.
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportType(IArrowType datatype, CArrowSchema* schema)
+ {
+ if (schema == null)
+ {
+ throw new ArgumentNullException("Cannot export schema to a null pointer.");
+ }
+ if (schema->release != null)
+ {
+ throw new ArgumentException("Cannot export schema to a struct that is already initialized.");
+ }
+
+ schema->format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ schema->name = null;
+ schema->metadata = null;
+ schema->flags = GetFlags(datatype);
+
+ schema->children = ConstructChildren(datatype, out var numChildren);
+ schema->n_children = numChildren;
+
+ schema->dictionary = ConstructDictionary(datatype);
+
+ schema->release = (delegate* unmanaged[Stdcall]<CArrowSchema*, void>)Marshal.GetFunctionPointerForDelegate(
+ ReleaseCArrowSchema);
+
+ schema->private_data = null;
+ }
+
+ /// <summary>
+ /// Export a field to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="field">The field to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(field, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportField(Field field, CArrowSchema* schema)
+ {
+ ExportType(field.DataType, schema);
+ schema->name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ schema->metadata = null;
+ schema->flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Export a schema to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="schema">The schema to export</param>
+ /// <param name="out_schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(schema, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportSchema(Schema schema, CArrowSchema* out_schema)
+ {
+ var structType = new StructType(schema.FieldsList);
+ // TODO: top-level metadata
+ ExportType(structType, out_schema);
+ }
+
+ private static char FormatTimeUnit(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ switch (datatype)
+ {
+ case NullType _: return "n";
+ case BooleanType _: return "b";
+ // Integers
+ case Int8Type _: return "c";
+ case UInt8Type _: return "C";
+ case Int16Type _: return "s";
+ case UInt16Type _: return "S";
+ case Int32Type _: return "i";
+ case UInt32Type _: return "I";
+ case Int64Type _: return "l";
+ case UInt64Type _: return "L";
+ // Floats
+ case HalfFloatType _: return "e";
+ case FloatType _: return "f";
+ case DoubleType _: return "g";
+ // Decimal
+ case Decimal128Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale}";
+ case Decimal256Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale},256";
+ // Binary
+ case BinaryType _: return "z";
+ case StringType _: return "u";
+ case FixedSizeBinaryType binaryType:
+ return $"w:{binaryType.ByteWidth}";
+ // Date
+ case Date32Type _: return "tdD";
+ case Date64Type _: return "tdm";
+ // Time
+ case Time32Type timeType:
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ case Time64Type timeType:
+ // Same prefix as Time32, but allowed time units are different.
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ // Timestamp
+ case TimestampType timestampType:
+ return String.Format("ts{0}:{1}", FormatTimeUnit(timestampType.Unit), timestampType.Timezone);
+ // Nested
+ case ListType _: return "+l";
+ case StructType _: return "+s";
+ // Dictionary
+ case DictionaryType dictionaryType:
+ return GetFormat(dictionaryType.IndexType);
+ default: throw new NotImplementedException($"Exporting {datatype.Name} not implemented");
+ };
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= CArrowSchema.ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= CArrowSchema.ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
Review Comment:
Could you check if the type id is a map type and throw a "not supported" exception? This way, when map type is eventually added, it won't silently do the wrong thing if we miss this spot.
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,301 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <remarks>
+ /// This will call the release callback on the passed struct, even if
+ /// this function fails.
+ /// </remarks>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.Create();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
Review Comment:
```suggestion
/// Import C pointer as a <see cref="Field"/>.
```
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportType(IArrowType datatype, CArrowSchema* schema)
+ {
+ if (schema == null)
+ {
+ throw new ArgumentNullException("Cannot export schema to a null pointer.");
+ }
+ if (schema->release != null)
+ {
+ throw new ArgumentException("Cannot export schema to a struct that is already initialized.");
+ }
+
+ schema->format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ schema->name = null;
+ schema->metadata = null;
+ schema->flags = GetFlags(datatype);
+
+ schema->children = ConstructChildren(datatype, out var numChildren);
+ schema->n_children = numChildren;
+
+ schema->dictionary = ConstructDictionary(datatype);
+
+ schema->release = (delegate* unmanaged[Stdcall]<CArrowSchema*, void>)Marshal.GetFunctionPointerForDelegate(
+ ReleaseCArrowSchema);
+
+ schema->private_data = null;
+ }
+
+ /// <summary>
+ /// Export a field to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="field">The field to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(field, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportField(Field field, CArrowSchema* schema)
+ {
+ ExportType(field.DataType, schema);
+ schema->name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ schema->metadata = null;
+ schema->flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Export a schema to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="schema">The schema to export</param>
+ /// <param name="out_schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(schema, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportSchema(Schema schema, CArrowSchema* out_schema)
+ {
+ var structType = new StructType(schema.FieldsList);
+ // TODO: top-level metadata
+ ExportType(structType, out_schema);
+ }
+
+ private static char FormatTimeUnit(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ switch (datatype)
+ {
+ case NullType _: return "n";
+ case BooleanType _: return "b";
+ // Integers
+ case Int8Type _: return "c";
+ case UInt8Type _: return "C";
+ case Int16Type _: return "s";
+ case UInt16Type _: return "S";
+ case Int32Type _: return "i";
+ case UInt32Type _: return "I";
+ case Int64Type _: return "l";
+ case UInt64Type _: return "L";
+ // Floats
+ case HalfFloatType _: return "e";
+ case FloatType _: return "f";
+ case DoubleType _: return "g";
+ // Decimal
+ case Decimal128Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale}";
+ case Decimal256Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale},256";
+ // Binary
+ case BinaryType _: return "z";
+ case StringType _: return "u";
+ case FixedSizeBinaryType binaryType:
+ return $"w:{binaryType.ByteWidth}";
+ // Date
+ case Date32Type _: return "tdD";
+ case Date64Type _: return "tdm";
+ // Time
+ case Time32Type timeType:
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ case Time64Type timeType:
+ // Same prefix as Time32, but allowed time units are different.
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ // Timestamp
+ case TimestampType timestampType:
+ return String.Format("ts{0}:{1}", FormatTimeUnit(timestampType.Unit), timestampType.Timezone);
+ // Nested
+ case ListType _: return "+l";
+ case StructType _: return "+s";
+ // Dictionary
+ case DictionaryType dictionaryType:
+ return GetFormat(dictionaryType.IndexType);
+ default: throw new NotImplementedException($"Exporting {datatype.Name} not implemented");
+ };
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= CArrowSchema.ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= CArrowSchema.ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static unsafe CArrowSchema** ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
Review Comment:
Is there a chance that `numChildren` could be 0? Does `AllocHGlobal` return `IntPtr.Zero` in that case?
Maybe more of a concern on import but I seem to recall a zero-field union being a useless, but technically possible, data type.
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,301 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <remarks>
+ /// This will call the release callback on the passed struct, even if
+ /// this function fails.
+ /// </remarks>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.Create();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <remarks>
+ /// This will call the release callback on the passed struct, even if
+ /// this function fails.
+ /// </remarks>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
Review Comment:
```suggestion
/// Typically, you will allocate an uninitialized CArrowSchema pointer,
```
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,301 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <remarks>
+ /// This will call the release callback on the passed struct, even if
+ /// this function fails.
+ /// </remarks>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.Create();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <remarks>
+ /// This will call the release callback on the passed struct, even if
+ /// this function fails.
+ /// </remarks>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.Create();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
Review Comment:
```suggestion
/// Import C pointer as a <see cref="Schema"/>.
```
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,301 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <remarks>
+ /// This will call the release callback on the passed struct, even if
+ /// this function fails.
+ /// </remarks>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.Create();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <remarks>
+ /// This will call the release callback on the passed struct, even if
+ /// this function fails.
+ /// </remarks>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.Create();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <remarks>
+ /// This will call the release callback on the passed struct, even if
+ /// this function fails.
+ /// </remarks>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
Review Comment:
```suggestion
/// Typically, you will allocate an uninitialized CArrowSchema pointer,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt merged pull request #34133: GH-34737: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt merged PR #34133:
URL: https://github.com/apache/arrow/pull/34133
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1132958403
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+
+ ptr->format = null;
+ ptr->name = null;
+ ptr->metadata = null;
+ ptr->flags = 0;
+ ptr->n_children = 0;
+ ptr->children = null;
+ ptr->dictionary = null;
+ ptr->release = null;
+ ptr->private_data = null;
+
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static unsafe void Free(CArrowSchema* schema)
+ {
+ if (schema->release != null)
+ {
+ // Call release if not already called.
+ schema->release(schema);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema);
+ }
+
+
+ /// <summary>
+ /// Whether this field is semantically nullable (regardless of whether it actually has null values)
+ /// </summary>
+ public const long ArrowFlagDictionaryOrdered = 1;
+ /// <summary>
+ /// For dictionary-encoded types, whether the ordering of dictionary indices is semantically meaningful.
+ /// </summary>
+ public const long ArrowFlagNullable = 2;
+ /// <summary>
+ /// For map types, whether the keys within each map value are sorted.
+ /// </summary>
+ public const long ArrowFlagMapKeysSorted = 4;
+
+ /// <summary>
+ /// Get the value of a particular flag.
+ /// </summary>
+ /// <remarks>
+ /// Known valid flags are <see cref="ArrowFlagDictionaryOrdered" />,
+ /// <see cref="ArrowFlagNullable" />, and <see cref="ArrowFlagMapKeysSorted" />.
+ /// </remarks>
+ public bool GetFlag(long flag)
+ {
+ return (flags & flag) == flag;
+ }
+
+ internal CArrowSchema* GetChild(int i)
Review Comment:
I thought about exposing it, but got worried it could be misused easily. If someone where to try to import a child schema instead of the root `CArrowSchema`, they would call the wrong release callback. Consumers "MUST not call any of its children’s release callbacks (including the optional dictionary)". See https://arrow.apache.org/docs/format/CDataInterface.html#release-callback-semantics-for-consumers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-34737: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1148736430
##########
csharp/test/Apache.Arrow.Tests/CDataInterfaceSchemaTests.cs:
##########
@@ -0,0 +1,427 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.C;
+using Apache.Arrow.Types;
+using Python.Runtime;
+using Xunit;
+
+namespace Apache.Arrow.Tests
+{
+ public class CDataSchemaTest
+ {
+ [Fact]
+ public unsafe void InitializeZeroed()
+ {
+ CArrowSchema* cSchema = CArrowSchema.Create();
+
+ Assert.True(cSchema->format == null);
+ Assert.True(cSchema->name == null);
+ Assert.True(cSchema->metadata == null);
+ Assert.Equal(0, cSchema->flags);
+ Assert.Equal(0, cSchema->n_children);
+ Assert.True(cSchema->children == null);
+ Assert.True(cSchema->dictionary == null);
+ Assert.True(cSchema->release == null);
+ Assert.True(cSchema->private_data == null);
+
+ CArrowSchema.Free(cSchema);
+ }
+
+ [Fact]
+ public unsafe void FlagsSet()
+ {
+ // Non-nullable field
+ {
+ var nonNullField = new Field("non_null", Int32Type.Default, false);
+ CArrowSchema* cSchema = CArrowSchema.Create();
+ CArrowSchemaExporter.ExportField(nonNullField, cSchema);
+ Assert.False(cSchema->GetFlag(CArrowSchema.ArrowFlagNullable));
+ CArrowSchema.Free(cSchema);
+ }
+
+ // Nullable field
+ {
+ var nullableField = new Field("nullable", Int32Type.Default, true);
+ CArrowSchema* cSchema = CArrowSchema.Create();
+ CArrowSchemaExporter.ExportField(nullableField, cSchema);
+ Assert.True(cSchema->GetFlag(CArrowSchema.ArrowFlagNullable));
+ CArrowSchema.Free(cSchema);
+ }
+
+ // dictionary ordered
+ {
+ var orderedDictionary = new DictionaryType(Int32Type.Default, StringType.Default, true);
+ CArrowSchema* cSchema = CArrowSchema.Create();
+ CArrowSchemaExporter.ExportType(orderedDictionary, cSchema);
+ Assert.True(cSchema->GetFlag(CArrowSchema.ArrowFlagDictionaryOrdered));
+ CArrowSchema.Free(cSchema);
+ }
+
+ // dictionary unordered
+ {
+ var unorderedDictionary = new DictionaryType(Int32Type.Default, StringType.Default, false);
+ CArrowSchema* cSchema = CArrowSchema.Create();
+ CArrowSchemaExporter.ExportType(unorderedDictionary, cSchema);
+ Assert.False(cSchema->GetFlag(CArrowSchema.ArrowFlagDictionaryOrdered));
+ CArrowSchema.Free(cSchema);
+ }
+ }
+
+ [Fact]
+ public unsafe void CallsReleaseForValid()
+ {
+ CArrowSchema* cSchema = CArrowSchema.Create();
+ CArrowSchemaExporter.ExportType(Int32Type.Default, cSchema);
+ Assert.False(cSchema->release == null);
+ CArrowSchemaImporter.ImportType(cSchema);
+ Assert.True(cSchema->release == null);
+ CArrowSchema.Free(cSchema);
+ }
+
+ [Fact]
+ public unsafe void CallsReleaseForInvalid()
+ {
+ // Make sure we call release callback, even if the imported schema
+ // is invalid.
+ CArrowSchema* cSchema = CArrowSchema.Create();
+
+ bool wasCalled = false;
+ var releaseCallback = (CArrowSchema* cSchema) =>
+ {
+ wasCalled = true;
+ cSchema->release = null;
+ };
+ cSchema->release = (delegate* unmanaged[Stdcall]<CArrowSchema*, void>)Marshal.GetFunctionPointerForDelegate(
+ releaseCallback);
+
+ Assert.Throws<NullReferenceException>(() =>
+ {
+ CArrowSchemaImporter.ImportType(cSchema);
+ });
+ Assert.True(wasCalled);
+ CArrowSchema.Free(cSchema);
+ }
+ }
+
+ public class CDataSchemaPythonTest
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1133026549
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+
+ ptr->format = null;
+ ptr->name = null;
+ ptr->metadata = null;
+ ptr->flags = 0;
+ ptr->n_children = 0;
+ ptr->children = null;
+ ptr->dictionary = null;
+ ptr->release = null;
+ ptr->private_data = null;
+
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static unsafe void Free(CArrowSchema* schema)
+ {
+ if (schema->release != null)
+ {
+ // Call release if not already called.
+ schema->release(schema);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema);
+ }
+
+
+ /// <summary>
+ /// Whether this field is semantically nullable (regardless of whether it actually has null values)
+ /// </summary>
+ public const long ArrowFlagDictionaryOrdered = 1;
+ /// <summary>
+ /// For dictionary-encoded types, whether the ordering of dictionary indices is semantically meaningful.
+ /// </summary>
+ public const long ArrowFlagNullable = 2;
+ /// <summary>
+ /// For map types, whether the keys within each map value are sorted.
+ /// </summary>
+ public const long ArrowFlagMapKeysSorted = 4;
+
+ /// <summary>
+ /// Get the value of a particular flag.
+ /// </summary>
+ /// <remarks>
+ /// Known valid flags are <see cref="ArrowFlagDictionaryOrdered" />,
+ /// <see cref="ArrowFlagNullable" />, and <see cref="ArrowFlagMapKeysSorted" />.
+ /// </remarks>
+ public bool GetFlag(long flag)
+ {
+ return (flags & flag) == flag;
+ }
+
+ internal CArrowSchema* GetChild(int i)
+ {
+ if (i >= n_children)
Review Comment:
Cool!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] adamreeve commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1120748625
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
Review Comment:
Ah yeah you may need to handle these manually then, eg. see https://github.com/G-Research/ParquetSharp/blob/467d99298fb5a5b9d5935b9c8dbde95e63954dd3/csharp/StringUtil.cs#L22
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1119020773
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,519 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+using System;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+
+ public ReleaseCArrowSchema release;
+ // Check this out: https://github.com/G-Research/ParquetSharp/blob/386d91bd5e6fe6cb81583803447023c1359957c8/csharp/ParquetHandle.cs#L8
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.format_string;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ARROW_FLAG_NULLABLE;
+ }
+
+ if (datatype is DictionaryType)
+ {
+ if (((DictionaryType)datatype).Ordered)
+ {
+ flags |= ARROW_FLAG_DICTIONARY_ORDERED;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype)
+ {
+ if (datatype is NestedType)
+ {
+ var fields = ((NestedType)datatype).Fields;
+ int n_fields = fields.Count;
+
+ IntPtr* pointer_list = (IntPtr*)Marshal.AllocHGlobal(n_fields * sizeof(IntPtr));
+
+ for (var i = 0; i < n_fields; i++)
+ {
+ var c_schema = new CArrowSchema();
+ CArrowSchema.ExportField(fields[i], out c_schema);
+ IntPtr exported_schema = c_schema.AllocateAsPtr();
+ pointer_list[i] = exported_schema;
+ }
+
+ return pointer_list;
+ }
+ else
+ {
+ return (IntPtr*)IntPtr.Zero;
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType)
+ {
+ var c_schema = new CArrowSchema();
+ var value_type = ((DictionaryType)datatype).ValueType;
+ CArrowSchema.ExportDataType(value_type, out c_schema);
+ return c_schema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ /// <param name="schema">An uninitialized CArrowSchema.</param>
+ public static void ExportDataType(IArrowType datatype, out CArrowSchema schema)
+ {
+ schema.format = GetFormat(datatype);
+ schema.name = null;
+ schema.metadata = null;
+ schema.flags = GetFlags(datatype);
+
+ schema.children = ConstructChildren(datatype);
+ schema.n_children = datatype is NestedType ? ((NestedType)datatype).Fields.Count : 0;
Review Comment:
I was originally trying to see if I could do something like
```csharp
(schema.children, schema.n_children) = ConstructChildren(datatype);
```
but couldn't figure out how to import support for tuples. I also tried a signature like:
```csharp
void SetChildren(datatype, out CArrowSchema schema);
```
But then C# wants me to set every field of `schema` within `SetChildren` (it doesn't know it's only being called from a function that does).
Open to suggestions :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1118947702
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,519 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+
+using System;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+
+ public ReleaseCArrowSchema release;
+ // Check this out: https://github.com/G-Research/ParquetSharp/blob/386d91bd5e6fe6cb81583803447023c1359957c8/csharp/ParquetHandle.cs#L8
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.format_string;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ARROW_FLAG_NULLABLE;
+ }
+
+ if (datatype is DictionaryType)
+ {
+ if (((DictionaryType)datatype).Ordered)
+ {
+ flags |= ARROW_FLAG_DICTIONARY_ORDERED;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype)
+ {
+ if (datatype is NestedType)
+ {
+ var fields = ((NestedType)datatype).Fields;
+ int n_fields = fields.Count;
+
+ IntPtr* pointer_list = (IntPtr*)Marshal.AllocHGlobal(n_fields * sizeof(IntPtr));
+
+ for (var i = 0; i < n_fields; i++)
+ {
+ var c_schema = new CArrowSchema();
+ CArrowSchema.ExportField(fields[i], out c_schema);
+ IntPtr exported_schema = c_schema.AllocateAsPtr();
+ pointer_list[i] = exported_schema;
+ }
+
+ return pointer_list;
+ }
+ else
+ {
+ return (IntPtr*)IntPtr.Zero;
Review Comment:
There's two nulls in C#: `null`, for C# objects, and `IntPtr.Zero`, for C interop. I think this is the correct null here, but open to feedback from those who know C# more deeply than I.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] adamreeve commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1125151308
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
Review Comment:
It's an open source project so we try to support a broad range of platforms, I don't know of specific users that would need .netstandard 2.1 specifically that couldn't use net6.0 though. And no we wouldn't need this supported in net471. So after considering this some more it's probably fine if the C data interface support is netcoreapp only as we could provide arrow integration as a separate library or add a new target and limit support to that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1124979427
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
Review Comment:
From https://learn.microsoft.com/en-us/dotnet/csharp/misc/cs0208 (a similar warning):
> A managed type is:
> any reference type
> any struct that contains a reference type as a field or property
We need to make sure `CArrowSchema` doesn't contain any reference types as a field or property. i.e. the managed delegate that I pointed out in the other comment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] adamreeve commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1122197932
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
Review Comment:
Yeah I think `IntPtr.Size` works here, for other struct types you'd generally want to use `Marshal.SizeOf` in this scenario
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1140770334
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
Review Comment:
I was all set to disagree with you. However, reading in detail, I think you're correct.
I don't like this but it might be the safest general pattern:
```
/// CArrowSchema* exportPtr = CArrowSchema.Create();
/// CArrowSchemaExporter.ExportType(dataType, exportPtr);
/// foreign_import_function(exportPtr);
/// if (exportPtr->release != IntPtr.Zero)
/// {
/// // Generally a foreign import function should copy the release callback
/// // and set this to IntPtr.Zero. If they do not then it will not be possible for
/// // them to call the release callback. Perhaps the imported expected to allocate
/// // the CArrowSchema instance itself?
/// CArrowSchema.Free(exportPtr);
/// throw new InvalidOperationException("Foreign import function failed to fully consume schema");
/// }
/// CArrowSchema.Free(exportPtr);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1139446814
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
Review Comment:
I was thinking that stack-allocated values in C# would be managed memory, and that wouldn't be accessible from C/other languages. But I might be mistaken on that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1139011131
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportType(IArrowType datatype, CArrowSchema* schema)
+ {
+ if (schema == null)
+ {
+ throw new ArgumentNullException("Cannot export schema to a null pointer.");
+ }
+ if (schema->release != null)
+ {
+ throw new ArgumentException("Cannot export schema to a struct that is already initialized.");
+ }
+
+ schema->format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ schema->name = null;
+ schema->metadata = null;
+ schema->flags = GetFlags(datatype);
+
+ schema->children = ConstructChildren(datatype, out var numChildren);
+ schema->n_children = numChildren;
+
+ schema->dictionary = ConstructDictionary(datatype);
+
+ schema->release = (delegate* unmanaged[Stdcall]<CArrowSchema*, void>)Marshal.GetFunctionPointerForDelegate(
+ ReleaseCArrowSchema);
+
+ schema->private_data = null;
+ }
+
+ /// <summary>
+ /// Export a field to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="field">The field to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(field, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportField(Field field, CArrowSchema* schema)
+ {
+ ExportType(field.DataType, schema);
+ schema->name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ schema->metadata = null;
+ schema->flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Export a schema to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="schema">The schema to export</param>
+ /// <param name="out_schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(schema, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportSchema(Schema schema, CArrowSchema* out_schema)
+ {
+ var structType = new StructType(schema.FieldsList);
+ // TODO: top-level metadata
+ ExportType(structType, out_schema);
+ }
+
+ private static char FormatTimeUnit(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ switch (datatype)
+ {
+ case NullType _: return "n";
+ case BooleanType _: return "b";
+ // Integers
+ case Int8Type _: return "c";
+ case UInt8Type _: return "C";
+ case Int16Type _: return "s";
+ case UInt16Type _: return "S";
+ case Int32Type _: return "i";
+ case UInt32Type _: return "I";
+ case Int64Type _: return "l";
+ case UInt64Type _: return "L";
+ // Floats
+ case HalfFloatType _: return "e";
+ case FloatType _: return "f";
+ case DoubleType _: return "g";
+ // Decimal
+ case Decimal128Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale}";
+ case Decimal256Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale},256";
+ // Binary
+ case BinaryType _: return "z";
+ case StringType _: return "u";
+ case FixedSizeBinaryType binaryType:
+ return $"w:{binaryType.ByteWidth}";
+ // Date
+ case Date32Type _: return "tdD";
+ case Date64Type _: return "tdm";
+ // Time
+ case Time32Type timeType:
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ case Time64Type timeType:
+ // Same prefix as Time32, but allowed time units are different.
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ // Timestamp
+ case TimestampType timestampType:
+ return String.Format("ts{0}:{1}", FormatTimeUnit(timestampType.Unit), timestampType.Timezone);
+ // Nested
+ case ListType _: return "+l";
+ case StructType _: return "+s";
+ // Dictionary
+ case DictionaryType dictionaryType:
+ return GetFormat(dictionaryType.IndexType);
+ default: throw new NotImplementedException($"Exporting {datatype.Name} not implemented");
+ };
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= CArrowSchema.ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= CArrowSchema.ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static unsafe CArrowSchema** ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
Review Comment:
It is guarded here:
https://github.com/apache/arrow/blob/f32c27b470beb495c06bae1219b0256998728780/csharp/src/Apache.Arrow/Types/NestedType.cs#L28-L35
So not possible to construct right now (Union types aren't implemented). But I think like your other comment, it would be good to raise an exception if we do see this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1148414553
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchemaImporter.ImportSchema(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Schema ImportSchema(CArrowSchema* ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ private sealed unsafe class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema* _data;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(CArrowSchema* handle)
+ {
+ _data = (CArrowSchema*)handle;
+ if (_data->release == null)
+ {
+ throw new ArgumentException("Tried to import a schema that has already been released.");
+ }
+ _isRoot = true;
+ }
+
+ public ImportedArrowSchema(CArrowSchema* handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot && _data->release != null)
+ {
+ _data->release(_data);
Review Comment:
> I feel this is a bad pattern - releasing resources that you don't own/didn't create
This is the entire point. Someone (the producer) decided to share those resources with you, and they need to know when you are finished using those resources.
As @wjones127 said, the release callback will generally not be a straight deallocation; instead, it may decrement a dedicated reference count (or unpin the backing object if the producer is implemented in a GC-based language).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1122350740
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
Review Comment:
To be honest, I'm not sure on this. It was the default and seemed to work with the integration tests fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1119021194
##########
ci/scripts/csharp_test.sh:
##########
@@ -21,6 +21,17 @@ set -ex
source_dir=${1}/csharp
+# Python and PyArrow are required for C Data Interface tests.
+if [ -z "${PYTHON}" ]; then
+ if [ ! which python3 > /dev/null 2>&1 ]; then
+ export PYTHON=python
+ else
+ export PYTHON=python3
+ fi
Review Comment:
Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1122577319
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
Review Comment:
Also see: https://github.com/apache/arrow/pull/34133#discussion_r1119262146
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1120681607
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
Review Comment:
Yeah looking at this, I don't know how I would do this without unsafe:
```csharp
Marshal.AllocHGlobal(numFields * sizeof(IntPtr))
```
And that's called in the constructor. (Compiler says that you can't call `sizeof(IntPtr)` outside of an unsafe context; not sure why.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1148412180
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
Review Comment:
Sorry I missed this question earlier.
See https://github.com/dotnet/runtime/issues/33802 for a lot of good discussion on this topic. In this case, `sizeof(CArrowSchema)` and `Marshal.SizeOf<CArrowSchema>()` will always return the same size. It is just that `sizeof(CArrowSchema)` is more efficient and is guaranteed to work in trimed/NativeAOT apps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] adamreeve commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1133322009
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,122 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
+ public void* private_data;
+
+
+ public static unsafe CArrowSchema* New()
+ {
+ var ptr = (CArrowSchema*)Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
Review Comment:
Hi @eerhardt, would you mind explaining this a bit more? I would have thought that `Marshal.SizeOf` is the correct approach here as we want to know the size of the marshaled representation. In this case we might be lucky that this matches the size of the struct in managed memory as the struct only contains pointers and longs, and we don't need to worry about any ordering differences as all the members have the same size. I'd be concerned about future changes altering the struct representation and breaking this assumption, although that does seem unlikely. And I don't think we can assume applications will use NativeAOT or Trimming?
We could store the size in a static member if we want to avoid the overhead of the `Marshal.SizeOf` call each time, or just hard-code the size as a constant and add a check that this matches the marshaled size somewhere. I guess even just adding a test that `sizeof(CArrowSchema) == Marshal.SizeOf<CArrowSchema>()` would alleviate my concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1123511088
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
+ {
+ format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ name = IntPtr.Zero;
+ metadata = IntPtr.Zero;
+ flags = GetFlags(datatype);
+
+ children = ConstructChildren(datatype, out var numChildren);
+ n_children = numChildren;
+
+ dictionary = ConstructDictionary(datatype);
+
+ release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+
+ Marshal.FreeHGlobal(schema.format);
+ Marshal.FreeHGlobal(schema.name);
+ Marshal.FreeHGlobal(schema.metadata);
+
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.GetChild(i));
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ public CArrowSchema(Field field) : this(field.DataType)
+ {
+ name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ metadata = IntPtr.Zero;
+ flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ public CArrowSchema(Schema schema) : this(new StructType(schema.Fields.Values.ToList()))
+ {
+ // TODO: top-level metadata
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public void Export(IntPtr ptr)
+ {
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ }
+
+ /// <summary>
+ /// Allocated a new pointer to an uninitialized CArrowSchema.
+ /// </summary>
+ /// <remarks>This is used to import schemas. See <see cref="ImportType"/>,
+ /// <see cref="ImportField"/>, and <see cref="ImportSchema"/>. Once data
+ /// is imported, this pointer must be cleaned up with <see cref="FreePtr"/>.
+ /// </remarks>
+ public static IntPtr AllocateUninitialized()
+ {
+ return Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchema.ImportType(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static ArrowType ImportType(IntPtr ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchema.ImportField(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Field ImportField(IntPtr ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchema.ImportSchema(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Schema ImportSchema(IntPtr ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ public const int ArrowFlagDictionaryOrdered = 1;
+ public const int ArrowFlagNullable = 2;
+ public const int ArrowFlagMapKeysSorted = 4;
+
+ private class TypeFormatter :
+ IArrowTypeVisitor<NullType>,
+ IArrowTypeVisitor<BooleanType>,
+ IArrowTypeVisitor<Int8Type>,
+ IArrowTypeVisitor<Int16Type>,
+ IArrowTypeVisitor<Int32Type>,
+ IArrowTypeVisitor<Int64Type>,
+ IArrowTypeVisitor<UInt8Type>,
+ IArrowTypeVisitor<UInt16Type>,
+ IArrowTypeVisitor<UInt32Type>,
+ IArrowTypeVisitor<UInt64Type>,
+ IArrowTypeVisitor<HalfFloatType>,
+ IArrowTypeVisitor<FloatType>,
+ IArrowTypeVisitor<DoubleType>,
+ IArrowTypeVisitor<Decimal128Type>,
+ IArrowTypeVisitor<Decimal256Type>,
+ IArrowTypeVisitor<Date32Type>,
+ IArrowTypeVisitor<Date64Type>,
+ IArrowTypeVisitor<Time32Type>,
+ IArrowTypeVisitor<Time64Type>,
+ IArrowTypeVisitor<TimestampType>,
+ IArrowTypeVisitor<StringType>,
+ IArrowTypeVisitor<BinaryType>,
+ IArrowTypeVisitor<FixedSizeBinaryType>,
+ IArrowTypeVisitor<ListType>,
+ IArrowTypeVisitor<StructType>,
+ IArrowTypeVisitor<DictionaryType>
+ {
+ public string FormatString;
+ public void Visit(NullType _) => FormatString = "n";
+ public void Visit(BooleanType _) => FormatString = "b";
+ // Integers
+ public void Visit(Int8Type _) => FormatString = "c";
+ public void Visit(UInt8Type _) => FormatString = "C";
+ public void Visit(Int16Type _) => FormatString = "s";
+ public void Visit(UInt16Type _) => FormatString = "S";
+ public void Visit(Int32Type _) => FormatString = "i";
+ public void Visit(UInt32Type _) => FormatString = "I";
+ public void Visit(Int64Type _) => FormatString = "l";
+ public void Visit(UInt64Type _) => FormatString = "L";
+ // Floats
+ public void Visit(HalfFloatType _) => FormatString = "e";
+ public void Visit(FloatType _) => FormatString = "f";
+ public void Visit(DoubleType _) => FormatString = "g";
+ // Binary
+ public void Visit(BinaryType _) => FormatString = "z";
+ public void Visit(StringType _) => FormatString = "u";
+ public void Visit(FixedSizeBinaryType datatype)
+ {
+ FormatString = $"w:{datatype.ByteWidth}";
+ }
+ // Decimal
+ public void Visit(Decimal128Type datatype)
+ {
+ FormatString = $"d:{datatype.Precision},{datatype.Scale}";
+ }
+ public void Visit(Decimal256Type datatype)
+ {
+ FormatString = $"w:{datatype.Precision},{datatype.Scale},256";
Review Comment:
Thanks for catching this. Reminds me I meant to test decimals and timestamps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1125035558
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
Review Comment:
Can you elaborate on your reasons for targeting `netstandard2.1` instead of `net6.0`? I see that ParquetSharp also targets .NET Framework (`net471`). Would you need the C Data interface on .NET Framework?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1148438305
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchemaImporter.ImportSchema(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Schema ImportSchema(CArrowSchema* ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ private sealed unsafe class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema* _data;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(CArrowSchema* handle)
+ {
+ _data = (CArrowSchema*)handle;
+ if (_data->release == null)
+ {
+ throw new ArgumentException("Tried to import a schema that has already been released.");
+ }
+ _isRoot = true;
+ }
+
+ public ImportedArrowSchema(CArrowSchema* handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot && _data->release != null)
+ {
+ _data->release(_data);
Review Comment:
> Isn't the fact that the function call returned to the caller enough to know that I'm finished using those resources?
I think this will make more sense when you look at array import rather than schema import. The main point of the C data interface is to allow one implementation (the producer) to share array buffers with another implementation (the consumer). If the lifetime of those buffers was limited to the import function call, then that means the consumer would either be limited to using the arrays during the import or else have to copy the buffers (at which point you might as well use IPC).
As an example, imagine we used the C data interface to create bindings to DataFusion to run queries.
```csharp
RecordBatch batch = ...
DataFusionContext context;
// Uses C data interface to import batch (as a struct array)
context.ImportBatch("my_table", batch);
// This query is applied to the imported array
var result = context.Sql("SELECT sum(x * y) FROM my_table");
```
Here, if we assumed it was safe to cleanup the batch after calling `ImportBatch`, then we couldn't call the subsequent query to do any computation on it. We need those buffers to live as long as the consumer still needs them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] westonpace commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "westonpace (via GitHub)" <gi...@apache.org>.
westonpace commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1139434660
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
Review Comment:
Ah, regarding point 1, the strings are just pointers in the struct so...if you could mandate it is stack-allocated, I think that would be fine too. But still, this is unlikely to really matter. These structs/functions should be more or less invisible to the user most of the time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #34133: GH-33856: [C#] c data interface for schemas and types
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34133:
URL: https://github.com/apache/arrow/pull/34133#issuecomment-1426416891
* Closes: #33856
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1140492811
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
Review Comment:
> I'm not sure that it is safe to assume that release has been set to null. I do agree it is safe when calling Arrow-C++'s import function. I just don't think this is part of the spec
You're right this isn't guaranteed in the spec. It's more something that needs to be negotiated in each API. So perhaps I just need to say something like that in the examples?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1132956320
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchemaImporter.ImportSchema(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Schema ImportSchema(CArrowSchema* ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ private sealed unsafe class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema* _data;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(CArrowSchema* handle)
+ {
+ _data = (CArrowSchema*)handle;
+ if (_data->release == null)
+ {
+ throw new ArgumentException("Tried to import a schema that has already been released.");
+ }
+ _isRoot = true;
+ }
+
+ public ImportedArrowSchema(CArrowSchema* handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot && _data->release != null)
+ {
+ _data->release(_data);
Review Comment:
Is there a way in C# to indicate that something is meant to take ownership of a pointer? TBH I think I would prefer that over relying on the user to remember to call the release callback in the appropriate situation.
For reference, that's how the C++ import functions work:
https://github.com/apache/arrow/blob/5b2fbade23eda9bc95b1e3854b19efff177cd0bd/cpp/src/arrow/c/bridge.h#L86-L94
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] adamreeve commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1122208927
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,559 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr* children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr* ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * sizeof(IntPtr));
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return pointerList;
+ }
+ else
+ {
+ numChildren = 0;
+ return (IntPtr*)IntPtr.Zero;
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
+ {
+ format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ name = IntPtr.Zero;
+ metadata = IntPtr.Zero;
+ flags = GetFlags(datatype);
+
+ children = ConstructChildren(datatype, out var numChildren);
+ n_children = numChildren;
+
+ dictionary = ConstructDictionary(datatype);
+
+ release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+
+ Marshal.FreeHGlobal(schema.format);
+ Marshal.FreeHGlobal(schema.name);
+ Marshal.FreeHGlobal(schema.metadata);
+
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.children[i]);
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ public CArrowSchema(Field field) : this(field.DataType)
+ {
+ name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ metadata = IntPtr.Zero;
+ flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ public CArrowSchema(Schema schema) : this(new StructType(schema.Fields.Values.ToList()))
+ {
+ // TODO: top-level metadata
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public void Export(IntPtr ptr)
+ {
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ }
+
+ public static IntPtr AllocateUninitialized()
Review Comment:
This method needs documentation, and should mention needing to call `FreePtr`. The examples in the `Import` methods below should probably also include calls to `FreePtr` for completeness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1120548598
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,339 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ unsafe public struct CArrowSchema
+ {
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string format;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string name;
+ [MarshalAs(UnmanagedType.LPStr)]
+ public string metadata;
Review Comment:
Good point. This isn't nul-terminated so I think there's probably no default marshaller for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1124983227
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static char FormatTimeUnit(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ switch (datatype)
+ {
+ case NullType _: return "n";
+ case BooleanType _: return "b";
+ // Integers
+ case Int8Type _: return "c";
+ case UInt8Type _: return "C";
+ case Int16Type _: return "s";
+ case UInt16Type _: return "S";
+ case Int32Type _: return "i";
+ case UInt32Type _: return "I";
+ case Int64Type _: return "l";
+ case UInt64Type _: return "L";
+ // Floats
+ case HalfFloatType _: return "e";
+ case FloatType _: return "f";
+ case DoubleType _: return "g";
+ // Decimal
+ case Decimal128Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale}";
+ case Decimal256Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale},256";
+ // Binary
+ case BinaryType _: return "z";
+ case StringType _: return "u";
+ case FixedSizeBinaryType binaryType:
+ return $"w:{binaryType.ByteWidth}";
+ // Date
+ case Date32Type _: return "tdD";
+ case Date64Type _: return "tdm";
+ // Time
+ case Time32Type timeType:
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ case Time64Type timeType:
+ // Same prefix as Time32, but allowed time units are different.
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ // Timestamp
+ case TimestampType timestampType:
+ return String.Format("ts{0}:{1}", FormatTimeUnit(timestampType.Unit), timestampType.Timezone);
+ // Nested
+ case ListType _: return "+l";
+ case StructType _: return "+s";
+ // Dictionary
+ case DictionaryType dictionaryType:
+ return GetFormat(dictionaryType.IndexType);
+ default: throw new NotImplementedException($"Exporting {datatype.Name} not implemented");
+ };
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ /// <summary>
+ /// Whether this field is semantically nullable (regardless of whether it actually has null values)
+ /// </summary>
+ public const long ArrowFlagDictionaryOrdered = 1;
+ /// <summary>
+ /// For dictionary-encoded types, whether the ordering of dictionary indices is semantically meaningful.
+ /// </summary>
+ public const long ArrowFlagNullable = 2;
+ /// <summary>
+ /// For map types, whether the keys within each map value are sorted.
+ /// </summary>
+ public const long ArrowFlagMapKeysSorted = 4;
+
+ /// <summary>
+ /// Get the value of a particular flag.
+ /// </summary>
+ /// <remarks>
+ /// Known valid flags are <see cref="ArrowFlagDictionaryOrdered" />,
+ /// <see cref="ArrowFlagNullable" />, and <see cref="ArrowFlagMapKeysSorted" />.
+ /// </remarks>
+ public bool GetFlag(long flag)
+ {
+ return (flags & flag) == flag;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
Review Comment:
The intention is both ways: Import C data to C#, and export C# data to C.
This would enable reading / writing Parquet to/from Arrow in ParquetSharp, for instance.
Reading: Parquet -> C++ Arrow -> C data interface -> C# Arrow
Writing: C# Arrow -> C data interface -> C++ Arrow -> Parquet
We could instead have an `Importer` and `Exporter` class
```csharp
public class CDataImporter
{
public static ArrowType ImportType(IntPtr ptr);
public static Field ImportField(IntPtr ptr);
public static Schema ImportSchema(IntPtr ptr);
}
```
Or simple have methods on various types:
```csharp
public class ArrowType
{
public IntPtr ExportToC();
}
public class Field
{
public IntPtr ExportToC();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] adamreeve commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1125013879
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
Review Comment:
We would like to consume this in a netstandard 2.1 library (ParquetSharp). Although Arrow integration might be a separate package, but it would definitely be a library rather than a standalone application.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1123535339
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,584 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ TypeFormatter formatter = new TypeFormatter();
+ datatype.Accept(formatter);
+ return formatter.FormatString;
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
+ {
+ format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ name = IntPtr.Zero;
+ metadata = IntPtr.Zero;
+ flags = GetFlags(datatype);
+
+ children = ConstructChildren(datatype, out var numChildren);
+ n_children = numChildren;
+
+ dictionary = ConstructDictionary(datatype);
+
+ release = (IntPtr self) =>
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(self);
+
+ Marshal.FreeHGlobal(schema.format);
+ Marshal.FreeHGlobal(schema.name);
+ Marshal.FreeHGlobal(schema.metadata);
+
+ if (schema.n_children > 0)
+ {
+ for (int i = 0; i < schema.n_children; i++)
+ {
+ FreePtr(schema.GetChild(i));
+ }
+ Marshal.FreeHGlobal((IntPtr)schema.children);
+ }
+
+ if (schema.dictionary != IntPtr.Zero)
+ {
+ FreePtr(schema.dictionary);
+ }
+ Marshal.DestroyStructure<CArrowSchema>(self);
+ };
+
+ private_data = IntPtr.Zero;
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a field.
+ /// </summary>
+ /// <param name="field">Field to export.</param>
+ public CArrowSchema(Field field) : this(field.DataType)
+ {
+ name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
+ metadata = IntPtr.Zero;
+ flags = GetFlags(field.DataType, field.IsNullable);
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as a schema.
+ /// </summary>
+ /// <param name="schema">Schema to export.</param>
+ public CArrowSchema(Schema schema) : this(new StructType(schema.Fields.Values.ToList()))
+ {
+ // TODO: top-level metadata
+ }
+
+ /// <summary>
+ /// Allocate an unmanaged pointer and copy this instances data to it.
+ /// </summary>
+ /// <remarks>
+ /// To avoid a memory leak, you must call <see cref="FreePtr"/> on this
+ /// pointer when done using it.
+ /// </remarks>
+ public IntPtr AllocateAsPtr()
+ {
+ IntPtr ptr = Marshal.AllocHGlobal(Marshal.SizeOf(this));
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ return ptr;
+ }
+
+ /// <summary>
+ /// Free a pointer that was allocated in <see cref="AllocateAsPtr"/>.
+ /// </summary>
+ /// <remarks>
+ /// Do not call this on a pointer that was allocated elsewhere.
+ /// </remarks>
+ public static void FreePtr(IntPtr ptr)
+ {
+ var schema = Marshal.PtrToStructure<CArrowSchema>(ptr);
+ if (schema.release != null)
+ {
+ // Call release if not already called.
+ schema.release(ptr);
+ }
+ Marshal.FreeHGlobal(ptr);
+ }
+
+ /// <summary>
+ /// Export to an existing pointer
+ /// </summary>
+ /// <param name="ptr">An allocated but uninitialized pointer.</param>
+ public void Export(IntPtr ptr)
+ {
+ Marshal.StructureToPtr<CArrowSchema>(this, ptr, false);
+ }
+
+ /// <summary>
+ /// Allocated a new pointer to an uninitialized CArrowSchema.
+ /// </summary>
+ /// <remarks>This is used to import schemas. See <see cref="ImportType"/>,
+ /// <see cref="ImportField"/>, and <see cref="ImportSchema"/>. Once data
+ /// is imported, this pointer must be cleaned up with <see cref="FreePtr"/>.
+ /// </remarks>
+ public static IntPtr AllocateUninitialized()
+ {
+ return Marshal.AllocHGlobal(Marshal.SizeOf<CArrowSchema>());
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchema.ImportType(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static ArrowType ImportType(IntPtr ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchema.ImportField(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Field ImportField(IntPtr ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchema.ImportSchema(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
+ /// </code>
+ /// </examples>
+ public static Schema ImportSchema(IntPtr ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ public const int ArrowFlagDictionaryOrdered = 1;
+ public const int ArrowFlagNullable = 2;
+ public const int ArrowFlagMapKeysSorted = 4;
+
+ private class TypeFormatter :
+ IArrowTypeVisitor<NullType>,
+ IArrowTypeVisitor<BooleanType>,
+ IArrowTypeVisitor<Int8Type>,
+ IArrowTypeVisitor<Int16Type>,
+ IArrowTypeVisitor<Int32Type>,
+ IArrowTypeVisitor<Int64Type>,
+ IArrowTypeVisitor<UInt8Type>,
+ IArrowTypeVisitor<UInt16Type>,
+ IArrowTypeVisitor<UInt32Type>,
+ IArrowTypeVisitor<UInt64Type>,
+ IArrowTypeVisitor<HalfFloatType>,
+ IArrowTypeVisitor<FloatType>,
+ IArrowTypeVisitor<DoubleType>,
+ IArrowTypeVisitor<Decimal128Type>,
+ IArrowTypeVisitor<Decimal256Type>,
+ IArrowTypeVisitor<Date32Type>,
+ IArrowTypeVisitor<Date64Type>,
+ IArrowTypeVisitor<Time32Type>,
+ IArrowTypeVisitor<Time64Type>,
+ IArrowTypeVisitor<TimestampType>,
+ IArrowTypeVisitor<StringType>,
+ IArrowTypeVisitor<BinaryType>,
+ IArrowTypeVisitor<FixedSizeBinaryType>,
+ IArrowTypeVisitor<ListType>,
+ IArrowTypeVisitor<StructType>,
+ IArrowTypeVisitor<DictionaryType>
+ {
+ public string FormatString;
+ public void Visit(NullType _) => FormatString = "n";
+ public void Visit(BooleanType _) => FormatString = "b";
+ // Integers
+ public void Visit(Int8Type _) => FormatString = "c";
+ public void Visit(UInt8Type _) => FormatString = "C";
+ public void Visit(Int16Type _) => FormatString = "s";
+ public void Visit(UInt16Type _) => FormatString = "S";
+ public void Visit(Int32Type _) => FormatString = "i";
+ public void Visit(UInt32Type _) => FormatString = "I";
+ public void Visit(Int64Type _) => FormatString = "l";
+ public void Visit(UInt64Type _) => FormatString = "L";
+ // Floats
+ public void Visit(HalfFloatType _) => FormatString = "e";
+ public void Visit(FloatType _) => FormatString = "f";
+ public void Visit(DoubleType _) => FormatString = "g";
+ // Binary
+ public void Visit(BinaryType _) => FormatString = "z";
+ public void Visit(StringType _) => FormatString = "u";
+ public void Visit(FixedSizeBinaryType datatype)
+ {
+ FormatString = $"w:{datatype.ByteWidth}";
+ }
+ // Decimal
+ public void Visit(Decimal128Type datatype)
+ {
+ FormatString = $"d:{datatype.Precision},{datatype.Scale}";
+ }
+ public void Visit(Decimal256Type datatype)
+ {
+ FormatString = $"w:{datatype.Precision},{datatype.Scale},256";
+ }
+ // Date
+ public void Visit(Date32Type _) => FormatString = "tdD";
+ public void Visit(Date64Type _) => FormatString = "tdm";
+
+ private char TimeUnitComponent(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+ // Time
+ public void Visit(Time32Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ public void Visit(Time64Type datatype)
+ {
+ FormatString = String.Format("tt{0}", TimeUnitComponent(datatype.Unit));
+ }
+ // Timestamp type
+ public void Visit(TimestampType datatype)
+ {
+ FormatString = String.Format("ts{0}:{1}", TimeUnitComponent(datatype.Unit), datatype.Timezone);
+ }
+ // Nested
+ public void Visit(ListType _) => FormatString = "+l";
+ public void Visit(StructType _) => FormatString = "+s";
+ // Dictionary
+ public void Visit(DictionaryType datatype)
+ {
+ // format string is that of the indices
+ datatype.IndexType.Accept(this);
+ }
+
+ // Unsupported
+ public void Visit(IArrowType type)
+ {
+ throw new NotImplementedException($"Exporting {type.Name} not implemented");
+ }
+ }
+
+ private sealed class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema _data;
+ private readonly IntPtr _handle;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(IntPtr handle)
+ {
+ _data = Marshal.PtrToStructure<CArrowSchema>(handle);
+ if (_data.release == null)
+ {
+ throw new Exception("Tried to import a schema that has already been released.");
+ }
+ _handle = handle;
+ _isRoot = true;
+ }
+
+ private ImportedArrowSchema(IntPtr handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot)
+ {
+ _data.release(_handle);
+ }
+ }
+
+ public ArrowType GetAsType()
+ {
+ var format = StringUtil.PtrToStringUtf8(_data.format);
+ if (_data.dictionary != IntPtr.Zero)
+ {
+ ArrowType indicesType = format switch
+ {
+ "c" => new Int8Type(),
+ "C" => new UInt8Type(),
+ "s" => new Int16Type(),
+ "S" => new UInt16Type(),
+ "i" => new Int32Type(),
+ "I" => new UInt32Type(),
+ "l" => new Int64Type(),
+ "L" => new UInt64Type(),
+ _ => throw new InvalidDataException($"Indices must be an integer, but got format string {format}"),
+ };
+
+ var dictionarySchema = new ImportedArrowSchema(_data.dictionary, /*is_root*/ false);
+ ArrowType dictionaryType = dictionarySchema.GetAsType();
+
+ bool ordered = (_data.flags & CArrowSchema.ArrowFlagNullable) == CArrowSchema.ArrowFlagNullable;
Review Comment:
Thanks. I've updated the tests to catch this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1124973175
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
+ public IntPtr private_data;
+
+ private static char FormatTimeUnit(TimeUnit unit) => unit switch
+ {
+ TimeUnit.Second => 's',
+ TimeUnit.Millisecond => 'm',
+ TimeUnit.Microsecond => 'u',
+ TimeUnit.Nanosecond => 'n',
+ _ => throw new InvalidDataException($"Unsupported time unit for export: {unit}"),
+ };
+
+ private static string GetFormat(IArrowType datatype)
+ {
+ switch (datatype)
+ {
+ case NullType _: return "n";
+ case BooleanType _: return "b";
+ // Integers
+ case Int8Type _: return "c";
+ case UInt8Type _: return "C";
+ case Int16Type _: return "s";
+ case UInt16Type _: return "S";
+ case Int32Type _: return "i";
+ case UInt32Type _: return "I";
+ case Int64Type _: return "l";
+ case UInt64Type _: return "L";
+ // Floats
+ case HalfFloatType _: return "e";
+ case FloatType _: return "f";
+ case DoubleType _: return "g";
+ // Decimal
+ case Decimal128Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale}";
+ case Decimal256Type decimalType:
+ return $"d:{decimalType.Precision},{decimalType.Scale},256";
+ // Binary
+ case BinaryType _: return "z";
+ case StringType _: return "u";
+ case FixedSizeBinaryType binaryType:
+ return $"w:{binaryType.ByteWidth}";
+ // Date
+ case Date32Type _: return "tdD";
+ case Date64Type _: return "tdm";
+ // Time
+ case Time32Type timeType:
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ case Time64Type timeType:
+ // Same prefix as Time32, but allowed time units are different.
+ return String.Format("tt{0}", FormatTimeUnit(timeType.Unit));
+ // Timestamp
+ case TimestampType timestampType:
+ return String.Format("ts{0}:{1}", FormatTimeUnit(timestampType.Unit), timestampType.Timezone);
+ // Nested
+ case ListType _: return "+l";
+ case StructType _: return "+s";
+ // Dictionary
+ case DictionaryType dictionaryType:
+ return GetFormat(dictionaryType.IndexType);
+ default: throw new NotImplementedException($"Exporting {datatype.Name} not implemented");
+ };
+ }
+
+ private static long GetFlags(IArrowType datatype, bool nullable = true)
+ {
+ long flags = 0;
+
+ if (nullable)
+ {
+ flags |= ArrowFlagNullable;
+ }
+
+ if (datatype is DictionaryType dictionaryType)
+ {
+ if (dictionaryType.Ordered)
+ {
+ flags |= ArrowFlagDictionaryOrdered;
+ }
+ }
+
+ // TODO: when we implement MapType, make sure to set the KEYS_SORTED flag.
+ return flags;
+ }
+
+ /// <summary>
+ /// Whether this field is semantically nullable (regardless of whether it actually has null values)
+ /// </summary>
+ public const long ArrowFlagDictionaryOrdered = 1;
+ /// <summary>
+ /// For dictionary-encoded types, whether the ordering of dictionary indices is semantically meaningful.
+ /// </summary>
+ public const long ArrowFlagNullable = 2;
+ /// <summary>
+ /// For map types, whether the keys within each map value are sorted.
+ /// </summary>
+ public const long ArrowFlagMapKeysSorted = 4;
+
+ /// <summary>
+ /// Get the value of a particular flag.
+ /// </summary>
+ /// <remarks>
+ /// Known valid flags are <see cref="ArrowFlagDictionaryOrdered" />,
+ /// <see cref="ArrowFlagNullable" />, and <see cref="ArrowFlagMapKeysSorted" />.
+ /// </remarks>
+ public bool GetFlag(long flag)
+ {
+ return (flags & flag) == flag;
+ }
+
+ private static IntPtr ConstructChildren(IArrowType datatype, out long numChildren)
+ {
+ if (datatype is NestedType nestedType)
+ {
+ IReadOnlyList<Field> fields = nestedType.Fields;
+ int numFields = fields.Count;
+ numChildren = numFields;
+
+ unsafe
+ {
+ IntPtr* pointerList = (IntPtr*)Marshal.AllocHGlobal(numFields * IntPtr.Size);
+
+ for (var i = 0; i < numChildren; i++)
+ {
+ var cSchema = new CArrowSchema(fields[i]);
+ IntPtr exportedSchema = cSchema.AllocateAsPtr();
+ pointerList[i] = exportedSchema;
+ }
+
+ return (IntPtr)pointerList;
+ }
+
+ }
+ else
+ {
+ numChildren = 0;
+ return IntPtr.Zero;
+ }
+ }
+
+ private IntPtr GetChild(int i)
+ {
+ if (i >= n_children)
+ {
+ throw new Exception("Child index out of bounds.");
+ }
+ if (children == IntPtr.Zero)
+ {
+ throw new Exception("Children array is null.");
+ }
+ unsafe
+ {
+ return ((IntPtr*)children)[i];
+ }
+ }
+
+ private static IntPtr ConstructDictionary(IArrowType datatype)
+ {
+ if (datatype is DictionaryType dictType)
+ {
+ var cSchema = new CArrowSchema(dictType.ValueType);
+ return cSchema.AllocateAsPtr();
+ }
+ else
+ {
+ return IntPtr.Zero;
+ }
+ }
+
+ /// <summary>
+ /// Initialize the exported C schema as an Arrow type.
+ /// </summary>
+ /// <param name="datatype">The Arrow type to export.</param>
+ public CArrowSchema(IArrowType datatype)
Review Comment:
I'm a bit confused by the intention/use case here. I assumed the intention was so C# code could consume "C Data Interface" memory already in memory created by some other system.
Is the use case here the other way? Maybe instead of putting these as methods on the `CArrowSchema` struct, it would be better to have a separate class/API/factory that created these. And then `CArrowSchema` struct would stay about defining the data structure, and not contain the logic of turning C# objects into the data structure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1124999826
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,599 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+[UnmanagedFunctionPointer(CallingConvention.StdCall)]
+public delegate void ReleaseCArrowSchema(IntPtr schema);
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ ///
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public struct CArrowSchema
+ {
+ public IntPtr format;
+ public IntPtr name;
+ public IntPtr metadata;
+ public long flags;
+ public long n_children;
+ public IntPtr children;
+ public IntPtr dictionary;
+ [MarshalAs(UnmanagedType.FunctionPtr)]
+ public ReleaseCArrowSchema release;
Review Comment:
Also, if it makes it easier, we can make the C Data Interface only available on NETCOREAPP. I don't think we need to support it on `netstandard` at all. At least not until someone actually needs it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] adamreeve commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "adamreeve (via GitHub)" <gi...@apache.org>.
adamreeve commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1128785618
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public sealed class CArrowSchemaImporter
Review Comment:
This class should probably be static too
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,231 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public sealed class CArrowSchemaExporter
Review Comment:
This class should be static if it's only going to have static methods
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public sealed class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// IntPtr importedPtr = CArrowSchema.AllocateUninitialized();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchema.ImportType(importedPtr);
+ /// CArrowSchema.FreePtr(importedPtr);
Review Comment:
Should be `Free` rather than `FreePtr`, and similarly for docs in other methods
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] wjones127 commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "wjones127 (via GitHub)" <gi...@apache.org>.
wjones127 commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1148438532
##########
csharp/src/Apache.Arrow/C/CArrowSchemaImporter.cs:
##########
@@ -0,0 +1,285 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaImporter
+ {
+ /// <summary>
+ /// Import C pointer as an <see cref="ArrowType"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// ArrowType importedType = CArrowSchemaImporter.ImportType(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe ArrowType ImportType(CArrowSchema* ptr)
+ {
+ using var importedType = new ImportedArrowSchema(ptr);
+ return importedType.GetAsType();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Field"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedField = CArrowSchemaImporter.ImportField(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Field ImportField(CArrowSchema* ptr)
+ {
+ using var importedField = new ImportedArrowSchema(ptr);
+ return importedField.GetAsField();
+ }
+
+ /// <summary>
+ /// Import C pointer as an <see cref="Schema"/>.
+ /// </summary>
+ /// <examples>
+ /// Typically, you will allocate a uninitialized CArrowSchema pointer,
+ /// pass that to external function, and then use this method to import
+ /// the result.
+ ///
+ /// <code>
+ /// CArrowSchema* importedPtr = CArrowSchema.New();
+ /// foreign_export_function(importedPtr);
+ /// Field importedSchema = CArrowSchemaImporter.ImportSchema(importedPtr);
+ /// CArrowSchema.Free(importedPtr);
+ /// </code>
+ /// </examples>
+ public static unsafe Schema ImportSchema(CArrowSchema* ptr)
+ {
+ using var importedSchema = new ImportedArrowSchema(ptr);
+ return importedSchema.GetAsSchema();
+ }
+
+ private sealed unsafe class ImportedArrowSchema : IDisposable
+ {
+ private readonly CArrowSchema* _data;
+ private readonly bool _isRoot;
+
+ public ImportedArrowSchema(CArrowSchema* handle)
+ {
+ _data = (CArrowSchema*)handle;
+ if (_data->release == null)
+ {
+ throw new ArgumentException("Tried to import a schema that has already been released.");
+ }
+ _isRoot = true;
+ }
+
+ public ImportedArrowSchema(CArrowSchema* handle, bool isRoot) : this(handle)
+ {
+ _isRoot = isRoot;
+ }
+
+ public void Dispose()
+ {
+ // We only call release on a root-level schema, not child ones.
+ if (_isRoot && _data->release != null)
+ {
+ _data->release(_data);
Review Comment:
> How do I "increment a dedicated reference count"? For example, if I received a CArrowSchema* from somewhere, and I need to pass it into a C# Importer and a C++ importer method. How could I tell the CArrowSchema* that it needs to allow release to be called twice?
I don't think you can import a `CArrowSchema*` more than once for that reason. So either the original producer would have to produce a `CArrowSchema` for each consumer, or else you could chain them, so that the C# importer imports the original one, and then the C# exporter creates a `CArrowSchema` that could be imported by C++.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] eerhardt commented on a diff in pull request #34133: GH-33856: [C#] C Data interface for schemas and types
Posted by "eerhardt (via GitHub)" <gi...@apache.org>.
eerhardt commented on code in PR #34133:
URL: https://github.com/apache/arrow/pull/34133#discussion_r1148397028
##########
csharp/test/Apache.Arrow.Tests/CDataInterfaceSchemaTests.cs:
##########
@@ -0,0 +1,427 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+using System;
+using System.Collections.Generic;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.C;
+using Apache.Arrow.Types;
+using Python.Runtime;
+using Xunit;
+
+namespace Apache.Arrow.Tests
+{
+ public class CDataSchemaTest
+ {
+ [Fact]
+ public unsafe void InitializeZeroed()
+ {
+ CArrowSchema* cSchema = CArrowSchema.Create();
+
+ Assert.True(cSchema->format == null);
+ Assert.True(cSchema->name == null);
+ Assert.True(cSchema->metadata == null);
+ Assert.Equal(0, cSchema->flags);
+ Assert.Equal(0, cSchema->n_children);
+ Assert.True(cSchema->children == null);
+ Assert.True(cSchema->dictionary == null);
+ Assert.True(cSchema->release == null);
+ Assert.True(cSchema->private_data == null);
+
+ CArrowSchema.Free(cSchema);
+ }
+
+ [Fact]
+ public unsafe void FlagsSet()
+ {
+ // Non-nullable field
+ {
+ var nonNullField = new Field("non_null", Int32Type.Default, false);
+ CArrowSchema* cSchema = CArrowSchema.Create();
+ CArrowSchemaExporter.ExportField(nonNullField, cSchema);
+ Assert.False(cSchema->GetFlag(CArrowSchema.ArrowFlagNullable));
+ CArrowSchema.Free(cSchema);
+ }
+
+ // Nullable field
+ {
+ var nullableField = new Field("nullable", Int32Type.Default, true);
+ CArrowSchema* cSchema = CArrowSchema.Create();
+ CArrowSchemaExporter.ExportField(nullableField, cSchema);
+ Assert.True(cSchema->GetFlag(CArrowSchema.ArrowFlagNullable));
+ CArrowSchema.Free(cSchema);
+ }
+
+ // dictionary ordered
+ {
+ var orderedDictionary = new DictionaryType(Int32Type.Default, StringType.Default, true);
+ CArrowSchema* cSchema = CArrowSchema.Create();
+ CArrowSchemaExporter.ExportType(orderedDictionary, cSchema);
+ Assert.True(cSchema->GetFlag(CArrowSchema.ArrowFlagDictionaryOrdered));
+ CArrowSchema.Free(cSchema);
+ }
+
+ // dictionary unordered
+ {
+ var unorderedDictionary = new DictionaryType(Int32Type.Default, StringType.Default, false);
+ CArrowSchema* cSchema = CArrowSchema.Create();
+ CArrowSchemaExporter.ExportType(unorderedDictionary, cSchema);
+ Assert.False(cSchema->GetFlag(CArrowSchema.ArrowFlagDictionaryOrdered));
+ CArrowSchema.Free(cSchema);
+ }
+ }
+
+ [Fact]
+ public unsafe void CallsReleaseForValid()
+ {
+ CArrowSchema* cSchema = CArrowSchema.Create();
+ CArrowSchemaExporter.ExportType(Int32Type.Default, cSchema);
+ Assert.False(cSchema->release == null);
+ CArrowSchemaImporter.ImportType(cSchema);
+ Assert.True(cSchema->release == null);
+ CArrowSchema.Free(cSchema);
+ }
+
+ [Fact]
+ public unsafe void CallsReleaseForInvalid()
+ {
+ // Make sure we call release callback, even if the imported schema
+ // is invalid.
+ CArrowSchema* cSchema = CArrowSchema.Create();
+
+ bool wasCalled = false;
+ var releaseCallback = (CArrowSchema* cSchema) =>
+ {
+ wasCalled = true;
+ cSchema->release = null;
+ };
+ cSchema->release = (delegate* unmanaged[Stdcall]<CArrowSchema*, void>)Marshal.GetFunctionPointerForDelegate(
+ releaseCallback);
+
+ Assert.Throws<NullReferenceException>(() =>
+ {
+ CArrowSchemaImporter.ImportType(cSchema);
+ });
+ Assert.True(wasCalled);
+ CArrowSchema.Free(cSchema);
+ }
+ }
+
+ public class CDataSchemaPythonTest
Review Comment:
Can we split this into a new file? Having a single class per file makes it easier to navigate the code and find what you are looking for.
##########
csharp/src/Apache.Arrow/C/CArrowSchemaExporter.cs:
##########
@@ -0,0 +1,278 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements. See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ public static class CArrowSchemaExporter
+ {
+ /// <summary>
+ /// Export a type to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="datatype">The datatype to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(dataType, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportType(IArrowType datatype, CArrowSchema* schema)
+ {
+ if (datatype == null)
+ {
+ throw new ArgumentNullException(nameof(datatype));
+ }
+ if (schema == null)
+ {
+ throw new ArgumentNullException(nameof(schema));
+ }
+ if (schema->release != null)
+ {
+ throw new ArgumentException("Cannot export schema to a struct that is already initialized.");
+ }
+
+ schema->format = StringUtil.ToCStringUtf8(GetFormat(datatype));
+ schema->name = null;
+ schema->metadata = null;
+ schema->flags = GetFlags(datatype);
+
+ schema->children = ConstructChildren(datatype, out var numChildren);
+ schema->n_children = numChildren;
+
+ schema->dictionary = ConstructDictionary(datatype);
+
+ schema->release = (delegate* unmanaged[Stdcall]<CArrowSchema*, void>)Marshal.GetFunctionPointerForDelegate(
+ ReleaseCArrowSchema);
+
+ schema->private_data = null;
+ }
+
+ /// <summary>
+ /// Export a field to a <see cref="CArrowSchema"/>.
+ /// </summary>
+ /// <param name="field">The field to export</param>
+ /// <param name="schema">An allocated but uninitialized CArrowSchema pointer.</param>
+ /// <example>
+ /// <code>
+ /// CArrowSchema* exportPtr = CArrowSchema.Create();
+ /// CArrowSchemaExporter.ExportType(field, exportPtr);
+ /// foreign_import_function(exportPtr);
+ /// CArrowSchema.Free(exportPtr);
+ /// </code>
+ /// </example>
+ public static unsafe void ExportField(Field field, CArrowSchema* schema)
+ {
+ ExportType(field.DataType, schema);
+ schema->name = StringUtil.ToCStringUtf8(field.Name);
+ // TODO: field metadata
Review Comment:
Can you log issues for these TODOs?
##########
csharp/src/Apache.Arrow/C/CArrowSchema.cs:
##########
@@ -0,0 +1,124 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+using System;
+using System.Runtime.InteropServices;
+using Apache.Arrow.Types;
+
+namespace Apache.Arrow.C
+{
+ /// <summary>
+ /// An Arrow C Data Interface Schema, which represents a type, field, or schema.
+ /// </summary>
+ /// <remarks>
+ /// This is used to export <see cref="ArrowType"/>, <see cref="Field"/>, or
+ /// <see cref="Schema"/> to other languages. It matches the layout of the
+ /// ArrowSchema struct described in https://github.com/apache/arrow/blob/main/cpp/src/arrow/c/abi.h.
+ /// </remarks>
+ [StructLayout(LayoutKind.Sequential)]
+ public unsafe struct CArrowSchema
+ {
+ public byte* format;
+ public byte* name;
+ public byte* metadata;
+ public long flags;
+ public long n_children;
+ public CArrowSchema** children;
+ public CArrowSchema* dictionary;
+ public delegate* unmanaged[Stdcall]<CArrowSchema*, void> release;
Review Comment:
In the future (when we start targeting net6.0+), I think we will want to drop the `[Stdcall]` here and just use the default:
```
public delegate* unmanaged<CArrowSchema*, void> release;
```
But that doesn't work in the older TFMs, so what we have now is fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org