You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2023/02/23 19:07:38 UTC
[GitHub] [arrow-nanoarrow] paleolimbot opened a new pull request, #117: feat(python): Python schema wrapper
paleolimbot opened a new pull request, #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117
Just some preliminary thoughts based on the R bindings and some work profiling the IPC reader that I thought might live best in PR form.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1132717128
##########
python/tests/test_nanoarrow.py:
##########
@@ -1,27 +1,155 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
import numpy as np
import pyarrow as pa
-import nanoarrow
+import nanoarrow as na
import pytest
+def test_version():
+ assert(na.version() == "0.2.0-SNAPSHOT")
def test_as_numpy_array():
-
+
arr = pa.array([1, 2, 3])
- result = nanoarrow.as_numpy_array(arr)
+ result = na.as_numpy_array(arr)
expected = arr.to_numpy()
np.testing.assert_array_equal(result, expected)
arr = pa.array([1, 2, 3], pa.uint8())
- result = nanoarrow.as_numpy_array(arr)
+ result = na.as_numpy_array(arr)
expected = arr.to_numpy()
np.testing.assert_array_equal(result, expected)
arr = pa.array([1, 2, None])
with pytest.raises(ValueError, match="Cannot convert array with nulls"):
- nanoarrow.as_numpy_array(arr)
+ na.as_numpy_array(arr)
arr = pa.array([[1], [2, 3]])
with pytest.raises(TypeError, match="Cannot convert a non-primitive array"):
- nanoarrow.as_numpy_array(arr)
+ na.as_numpy_array(arr)
+
+def test_schema_basic():
+ # Blank invalid schema
+ schema = na.CSchema.Empty()
+ assert(schema.is_valid() is False)
+ assert(repr(schema) == "[invalid: schema is released]")
+
+ pa_schema = pa.schema([pa.field("some_name", pa.int32())])
+ pa_schema._export_to_c(schema._addr())
+
+ assert(schema.format == "+s")
+ assert(schema.flags == 0)
+ assert(len(schema.children) == 1)
+ assert(schema.children[0].format == "i")
+ assert(schema.children[0].name == "some_name")
+ assert(repr(schema.children[0]) == "int32")
+
+ with pytest.raises(IndexError):
+ schema.children[1]
+
+def test_schema_parse():
+ schema = na.CSchema.Empty()
+ with pytest.raises(RuntimeError):
+ schema.parse()
+
+ pa.schema([pa.field("col1", pa.int32())])._export_to_c(schema._addr())
+
+ info = schema.parse()
+ assert(info['type'] == 'struct')
+ assert(info['storage_type'] == 'struct')
+ assert(info['name'] == '')
+
+ # Check on the child
+ child = schema.children[0]
+ child_info = child.parse()
+ assert(child_info['type'] == 'int32')
+ assert(child_info['storage_type'] == 'int32')
+ assert(child_info['name'] == 'col1')
+
+def test_schema_info_params():
+ schema = na.CSchema.Empty()
+ pa.binary(12)._export_to_c(schema._addr())
Review Comment:
I'm going to punt on pyarrow integration just this second...for the tests it's only one extra line to `_export_to_c()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1231694016
##########
python/nanoarrow/_lib.pyx:
##########
@@ -730,9 +761,13 @@ cdef class ArrayViewBuffers:
cdef ArrayView _array_view
cdef int64_t _length
- def __init__(self, ArrayView array_view):
+ def __cinit__(self, ArrayView array_view):
self._array_view = array_view
- self._length = array_view._array._ptr.n_buffers
+ self._length = 3
Review Comment:
It's just to remove the reference to the `_array`, since that no longer exists. The `ArrowArrayView` in C doesn't have an explicit `n_buffers` member (but maybe should since the shenanigan on the following lines gets used quite a bit in C as well).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1132428639
##########
python/setup.py:
##########
@@ -17,29 +17,23 @@
# specific language governing permissions and limitations
# under the License.
-import shutil
-from pathlib import Path
-
+import os
+import sys
+import subprocess
from setuptools import Extension, setup
-
import numpy as np
-
-# setuptools gets confused by relative paths that extend above the project root
-target = Path(__file__).parent / "src" / "nanoarrow"
-shutil.copy(
- Path(__file__).parent / "../dist/nanoarrow.c", target / "nanoarrow.c"
-)
-shutil.copy(
- Path(__file__).parent / "../dist/nanoarrow.h", target / "nanoarrow.h"
-)
+# Run bootstrap.py to run cmake generating a fresh bundle based on this
+# checkout or copy from ../dist if the caller doesn't have cmake available
+this_dir = os.path.dirname(__file__)
+subprocess.run([sys.executable, os.path.join(this_dir, 'bootstrap.py')])
Review Comment:
There's a bootstrap.R file too and I really like keep the logic separate since it's rather extensive and maybe should even be tested separately. I forgot about sdist...I made a few changes to make sure that (1) bootstrap.py wouldn't be included in sdist and (2) make sure not to error if it isn't there. Probably this will need some workshopping come 'we have to build 39348 wheels' time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1132433740
##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -84,3 +85,329 @@ def as_numpy_array(arr):
# TODO set base
return result
+
+
+def version():
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class CSchemaHolder:
+ cdef ArrowSchema c_schema
+
+ def __init__(self):
+ self.c_schema.release = NULL
+
+ def __del__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class CArrayHolder:
+ cdef ArrowArray c_array
+
+ def __init__(self):
+ self.c_array.release = NULL
+
+ def __del__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class CArrayViewHolder:
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __del__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+cdef class CSchema:
Review Comment:
No good reason! I forget where I got the CSomething pattern (maybe pyarrow?). After starting with it I thought it might make room for a small nanoarrow.Schema class that has better autocomplete/is safer (maybe whose `@property`s are things like `.type`, `.precision`, `.scale`, `.unit`, etc. instead of the lower-level `.format` and friends). Maybe that is also planning for something that will never exist, too 🤷
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1132722310
##########
python/.gitignore:
##########
@@ -18,7 +18,8 @@
src/nanoarrow/nanoarrow.c
src/nanoarrow/nanoarrow.h
-src/nanoarrow/*.cpp
+src/nanoarrow/nanoarrow_c.pxd
Review Comment:
Checking it into dist/ would be fine because users of that would just copy the whole folder into their python project and the .h/.pxd would stay together?
For local development and CI it's essential to have cmake generate the bundle (or else you could make changes in the C library that break the Python package and CI wouldn't catch it). Also important to include copy from ../dist as a fallback so that installing by URL works via pip. I tested this on Mac and Windows and Linux but it probably needs some flushing out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] jorisvandenbossche commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1231935807
##########
python/nanoarrow/_lib.pyx:
##########
@@ -730,9 +761,13 @@ cdef class ArrayViewBuffers:
cdef ArrayView _array_view
cdef int64_t _length
- def __init__(self, ArrayView array_view):
+ def __cinit__(self, ArrayView array_view):
self._array_view = array_view
- self._length = array_view._array._ptr.n_buffers
+ self._length = 3
Review Comment:
Thanks for the explanation! And yes, seeing this code here, adding `n_buffers` to the view object might be useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] jorisvandenbossche commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1231441990
##########
python/nanoarrow/_lib.pyx:
##########
@@ -730,9 +761,13 @@ cdef class ArrayViewBuffers:
cdef ArrayView _array_view
cdef int64_t _length
- def __init__(self, ArrayView array_view):
+ def __cinit__(self, ArrayView array_view):
self._array_view = array_view
- self._length = array_view._array._ptr.n_buffers
+ self._length = 3
Review Comment:
Just out of curiosity, the `n_buffers` member of ArrowArray is not always accurate (or not what we want here)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] jorisvandenbossche commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1131521826
##########
python/.gitignore:
##########
@@ -18,7 +18,8 @@
src/nanoarrow/nanoarrow.c
src/nanoarrow/nanoarrow.h
-src/nanoarrow/*.cpp
+src/nanoarrow/nanoarrow_c.pxd
Review Comment:
Another option could be to actually include / commit this file? Although that might be annoying if a non-python related PR updates the c files, this gets out of sync (if we only update it in PRs that touch python). And in the end maybe not much different than the .c and .h files we also ignore
##########
python/bootstrap.py:
##########
@@ -0,0 +1,197 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+import re
+import os
+import shutil
+
+# Generate the nanoarrow_c.pxd file used by the Cython extension
+class NanoarrowPxdGenerator:
+
+ def __init__(self):
+ self._define_regexes()
+
+ def generate_nanoarrow_pxd(self, file_in, file_out):
+ file_in_name = os.path.basename(file_in)
+
+ # Read the nanoarrow.h header
+ content = None
+ with open(file_in, 'r') as input:
+ content = input.read()
+
+ # Strip comments
+ content = self.re_comment.sub('', content)
+
+ # Find types and function definitions
+ types = self._find_types(content)
+ func_defs = self._find_func_defs(content)
+
+ # Make corresponding cython definitions
+ types_cython = [self._type_to_cython(t, ' ') for t in types]
+ func_defs_cython = [self._func_def_to_cython(d, ' ') for d in func_defs]
+
+ # Unindent the header
+ header = self.re_newline_plus_indent.sub('\n', self._pxd_header())
+
+ # Write nanoarrow_c.pxd
+ with open(file_out, 'wb') as output:
+ output.write(header.encode('UTF-8'))
+
+ output.write(f'\ncdef extern from "{file_in_name}":\n'.encode("UTF-8"))
+
+ # A few things we add in manually
+ output.write(b'\n')
+ output.write(b' ctypedef int ArrowErrorCode\n')
+ output.write(b' cdef int NANOARROW_OK\n')
+ output.write(b'\n')
+
+ for type in types_cython:
+ output.write(type.encode('UTF-8'))
+ output.write(b'\n\n')
+
+ for func_def in func_defs_cython:
+ output.write(func_def.encode('UTF-8'))
+ output.write(b'\n')
+
+ output.write(b'\n')
+
+ def _define_regexes(self):
+ self.re_comment = re.compile(r'\s*//[^\n]*')
+ self.re_type = re.compile(r'(?P<type>struct|union|enum) (?P<name>Arrow[^ ]+) {(?P<body>[^}]*)}')
+ self.re_func_def = re.compile(r'\n(static inline )?(?P<const>const )?(struct|enum )?(?P<return_type>[A-Za-z0-9_*]+) (?P<name>Arrow[A-Za-z]+)\((?P<args>[^\)]*)\);')
+ self.re_tagged_type = re.compile(r'(?P<type>struct|union|enum) (?P<name>Arrow[A-Za-z]+)')
+ self.re_struct_delim = re.compile(r';\s*')
+ self.re_enum_delim = re.compile(r',\s*')
+ self.re_whitespace = re.compile(r'\s+')
+ self.re_newline_plus_indent = re.compile(r'\n +')
+
+ def _strip_comments(self, content):
+ return self.re_comment.sub('', content)
+
+ def _find_types(self, content):
+ return [m.groupdict() for m in self.re_type.finditer(content)]
+
+ def _find_func_defs(self, content):
+ return [m.groupdict() for m in self.re_func_def.finditer(content)]
+
+ def _type_to_cython(self, t, indent=''):
+ type = t['type']
+ name = t['name']
+ body = self.re_tagged_type.sub(r'\2', t['body'].strip())
+ if type == 'enum':
+ items = [item for item in self.re_enum_delim.split(body) if item]
+ else:
+ items = [item for item in self.re_struct_delim.split(body) if item]
+
+ cython_body = f'\n{indent} '.join([''] + items)
+ return f'{indent}cdef {type} {name}:{cython_body}'
Review Comment:
Strictly speaking the `cdef` is unnecessary here (and the same for func below)
The `cdef extern from ..` ensures cython already assumes this
##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -84,3 +85,329 @@ def as_numpy_array(arr):
# TODO set base
return result
+
+
+def version():
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class CSchemaHolder:
+ cdef ArrowSchema c_schema
+
+ def __init__(self):
+ self.c_schema.release = NULL
+
+ def __del__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class CArrayHolder:
+ cdef ArrowArray c_array
+
+ def __init__(self):
+ self.c_array.release = NULL
+
+ def __del__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class CArrayViewHolder:
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __del__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+cdef class CSchema:
Review Comment:
Is the reason you call it CSchema (with C prefix) because this represents a "C" Data Interface schema?
##########
python/setup.py:
##########
@@ -17,29 +17,23 @@
# specific language governing permissions and limitations
# under the License.
-import shutil
-from pathlib import Path
-
+import os
+import sys
+import subprocess
from setuptools import Extension, setup
-
import numpy as np
-
-# setuptools gets confused by relative paths that extend above the project root
-target = Path(__file__).parent / "src" / "nanoarrow"
-shutil.copy(
- Path(__file__).parent / "../dist/nanoarrow.c", target / "nanoarrow.c"
-)
-shutil.copy(
- Path(__file__).parent / "../dist/nanoarrow.h", target / "nanoarrow.h"
-)
+# Run bootstrap.py to run cmake generating a fresh bundle based on this
+# checkout or copy from ../dist if the caller doesn't have cmake available
+this_dir = os.path.dirname(__file__)
+subprocess.run([sys.executable, os.path.join(this_dir, 'bootstrap.py')])
Review Comment:
I would maybe move the full content of `bootstrap.py` into `setup.py`, and then there is no need to call it here with a subprocess.
In the end, bootstrap.py is ran every time we build/install the project, and that is essentially the logic that typically belongs in setup.py (I also don't think there is a need to be able to call the bootstrap separately?)
##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -84,3 +85,329 @@ def as_numpy_array(arr):
# TODO set base
return result
+
+
+def version():
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class CSchemaHolder:
Review Comment:
Those "Holder" classes are only for when we "own" the pointer, and then to ensure t is properly released? (to serve as the "base" object in that case, while if you create the CSchema/CArray class from another object that owns the pointer, this other object is the `_base`)
##########
python/setup.py:
##########
@@ -17,29 +17,23 @@
# specific language governing permissions and limitations
# under the License.
-import shutil
-from pathlib import Path
-
+import os
+import sys
+import subprocess
from setuptools import Extension, setup
-
import numpy as np
-
-# setuptools gets confused by relative paths that extend above the project root
-target = Path(__file__).parent / "src" / "nanoarrow"
-shutil.copy(
- Path(__file__).parent / "../dist/nanoarrow.c", target / "nanoarrow.c"
-)
-shutil.copy(
- Path(__file__).parent / "../dist/nanoarrow.h", target / "nanoarrow.h"
-)
+# Run bootstrap.py to run cmake generating a fresh bundle based on this
+# checkout or copy from ../dist if the caller doesn't have cmake available
+this_dir = os.path.dirname(__file__)
+subprocess.run([sys.executable, os.path.join(this_dir, 'bootstrap.py')])
Review Comment:
Eventually (later), we will also want to have some better logic to determine if this step needs to be run or not, for example when distributing sdists (see eg https://github.com/apache/arrow-adbc/blob/923e0408fe5a32cc6501b997fafa8316ace25fe0/python/adbc_driver_manager/setup.py#L33-L50)
##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -84,3 +85,329 @@ def as_numpy_array(arr):
# TODO set base
return result
+
+
+def version():
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class CSchemaHolder:
+ cdef ArrowSchema c_schema
+
+ def __init__(self):
+ self.c_schema.release = NULL
+
+ def __del__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class CArrayHolder:
+ cdef ArrowArray c_array
+
+ def __init__(self):
+ self.c_array.release = NULL
+
+ def __del__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class CArrayViewHolder:
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __del__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+cdef class CSchema:
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def Empty():
+ base = CSchemaHolder()
+ return CSchema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return CSchemaChildren(self)
+
+ def parse(self):
+ self._assert_valid()
+
+ cdef ArrowError error
+ cdef ArrowSchemaView schema_view
+
+ cdef int result = ArrowSchemaViewInit(&schema_view, self._ptr, &error)
+ if result != NANOARROW_OK:
+ raise ValueError(ArrowErrorMessage(&error))
+
+ out = {
+ 'name': self._ptr.name.decode('UTF-8') if self._ptr.name else None,
+ 'type': ArrowTypeString(schema_view.type).decode('UTF-8'),
+ 'storage_type': ArrowTypeString(schema_view.storage_type).decode('UTF-8')
+ }
+
+ if schema_view.storage_type in (NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY):
+ out['fixed_size'] = schema_view.fixed_size
+
+ if schema_view.storage_type in (NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256):
+ out['decimal_bitwidth'] = schema_view.decimal_bitwidth
+ out['decimal_precision'] = schema_view.decimal_precision
+ out['decimal_scale'] = schema_view.decimal_scale
+
+ return out
+
+cdef class CArray:
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef CSchema _schema
+
+ @staticmethod
+ def Empty(CSchema schema):
+ base = CArrayHolder()
+ return CArray(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, CSchema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def children(self):
+ return CArrayChildren(self)
+
+ def validate(self):
+ cdef CArrayViewHolder holder = CArrayViewHolder()
+
+ cdef ArrowError error
+ cdef int result = ArrowArrayViewInitFromSchema(&holder.c_array_view,
+ self._schema._ptr, &error)
+ if result != NANOARROW_OK:
+ raise ValueError(ArrowErrorMessage(&error))
+
+ result = ArrowArrayViewSetArray(&holder.c_array_view, self._ptr, &error)
+ if result != NANOARROW_OK:
+ raise ValueError(ArrowErrorMessage(&error))
+
+ return CArrayView(holder, holder._addr(), self)
Review Comment:
It sounds a bit strange that "validate" returns a array view object.
(I would personally also add all properties of CArrayView like length, buffers, etc to the CArray class itself)
##########
python/tests/test_nanoarrow.py:
##########
@@ -1,27 +1,155 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
import numpy as np
import pyarrow as pa
-import nanoarrow
+import nanoarrow as na
import pytest
+def test_version():
+ assert(na.version() == "0.2.0-SNAPSHOT")
def test_as_numpy_array():
-
+
arr = pa.array([1, 2, 3])
- result = nanoarrow.as_numpy_array(arr)
+ result = na.as_numpy_array(arr)
expected = arr.to_numpy()
np.testing.assert_array_equal(result, expected)
arr = pa.array([1, 2, 3], pa.uint8())
- result = nanoarrow.as_numpy_array(arr)
+ result = na.as_numpy_array(arr)
expected = arr.to_numpy()
np.testing.assert_array_equal(result, expected)
arr = pa.array([1, 2, None])
with pytest.raises(ValueError, match="Cannot convert array with nulls"):
- nanoarrow.as_numpy_array(arr)
+ na.as_numpy_array(arr)
arr = pa.array([[1], [2, 3]])
with pytest.raises(TypeError, match="Cannot convert a non-primitive array"):
- nanoarrow.as_numpy_array(arr)
+ na.as_numpy_array(arr)
+
+def test_schema_basic():
+ # Blank invalid schema
+ schema = na.CSchema.Empty()
+ assert(schema.is_valid() is False)
Review Comment:
```suggestion
assert schema.is_valid() is False
```
Styling nitpick: those brackets are not needed
(we should add black as pre-commit hook like in adbc, and that should also take care of this, I think: https://github.com/apache/arrow-adbc/blob/main/.pre-commit-config.yaml#L65)
##########
python/tests/test_nanoarrow.py:
##########
@@ -1,27 +1,155 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
import numpy as np
import pyarrow as pa
-import nanoarrow
+import nanoarrow as na
import pytest
+def test_version():
+ assert(na.version() == "0.2.0-SNAPSHOT")
def test_as_numpy_array():
-
+
arr = pa.array([1, 2, 3])
- result = nanoarrow.as_numpy_array(arr)
+ result = na.as_numpy_array(arr)
expected = arr.to_numpy()
np.testing.assert_array_equal(result, expected)
arr = pa.array([1, 2, 3], pa.uint8())
- result = nanoarrow.as_numpy_array(arr)
+ result = na.as_numpy_array(arr)
expected = arr.to_numpy()
np.testing.assert_array_equal(result, expected)
arr = pa.array([1, 2, None])
with pytest.raises(ValueError, match="Cannot convert array with nulls"):
- nanoarrow.as_numpy_array(arr)
+ na.as_numpy_array(arr)
arr = pa.array([[1], [2, 3]])
with pytest.raises(TypeError, match="Cannot convert a non-primitive array"):
- nanoarrow.as_numpy_array(arr)
+ na.as_numpy_array(arr)
+
+def test_schema_basic():
+ # Blank invalid schema
+ schema = na.CSchema.Empty()
+ assert(schema.is_valid() is False)
+ assert(repr(schema) == "[invalid: schema is released]")
+
+ pa_schema = pa.schema([pa.field("some_name", pa.int32())])
+ pa_schema._export_to_c(schema._addr())
+
+ assert(schema.format == "+s")
+ assert(schema.flags == 0)
+ assert(len(schema.children) == 1)
+ assert(schema.children[0].format == "i")
+ assert(schema.children[0].name == "some_name")
+ assert(repr(schema.children[0]) == "int32")
+
+ with pytest.raises(IndexError):
+ schema.children[1]
+
+def test_schema_parse():
+ schema = na.CSchema.Empty()
+ with pytest.raises(RuntimeError):
+ schema.parse()
+
+ pa.schema([pa.field("col1", pa.int32())])._export_to_c(schema._addr())
+
+ info = schema.parse()
+ assert(info['type'] == 'struct')
+ assert(info['storage_type'] == 'struct')
+ assert(info['name'] == '')
+
+ # Check on the child
+ child = schema.children[0]
+ child_info = child.parse()
+ assert(child_info['type'] == 'int32')
+ assert(child_info['storage_type'] == 'int32')
+ assert(child_info['name'] == 'col1')
+
+def test_schema_info_params():
+ schema = na.CSchema.Empty()
+ pa.binary(12)._export_to_c(schema._addr())
Review Comment:
It might be nice to add a `from_pyarrow`-like helper to avoid this constant pattern of creating an empty and calling export_to_c (this could also be just an internal helper for the tests if we don't want to expose that publicly (yet))
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1132719069
##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -84,3 +85,329 @@ def as_numpy_array(arr):
# TODO set base
return result
+
+
+def version():
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class CSchemaHolder:
+ cdef ArrowSchema c_schema
+
+ def __init__(self):
+ self.c_schema.release = NULL
+
+ def __del__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class CArrayHolder:
+ cdef ArrowArray c_array
+
+ def __init__(self):
+ self.c_array.release = NULL
+
+ def __del__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class CArrayViewHolder:
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __del__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+cdef class CSchema:
Review Comment:
You're totally right..it's all just `Schema`, `Array`, etc. now. There probably should never be anything higher level in nanoarrrow anyway!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1230307162
##########
.github/workflows/python.yaml:
##########
@@ -57,3 +57,31 @@ jobs:
- name: Run tests
run: |
pytest python/tests -v -s
+
+ - name: Run doctests
+ if: success() && matrix.python-version == '3.10'
+ run: |
+ # Needs editable install to run --doctest-cython
+ pip install pytest-cython
+ pip install -e python
Review Comment:
Locally, yes. I've found it to be rather finicky (but useful enough to play along with for now 🤷 ).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1229775507
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("schema is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def metadata(self):
+ self._assert_valid()
+ if self._ptr.metadata != NULL:
+ return SchemaMetadata(self, <uintptr_t>self._ptr.metadata)
+ else:
+ return None
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return SchemaChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Schema(self, <uintptr_t>self._ptr.dictionary)
+ else:
+ return None
+
+ def view(self):
+ self._assert_valid()
+ schema_view = SchemaView()
+ cdef Error error = Error()
+ cdef int result = ArrowSchemaViewInit(&schema_view._schema_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowSchemaViewInit()", result)
+
+ return schema_view
+
+
+cdef class SchemaView:
+ """ArrowSchemaView wrapper
+
+ The ArrowSchemaView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowSchema (e.g., parameter
+ values for parameterized types). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.decimal128(10, 3))
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'decimal128'
+ >>> schema_view.decimal_bitwidth
+ 128
+ >>> schema_view.decimal_precision
+ 10
+ >>> schema_view.decimal_scale
+ 3
+ """
+ cdef ArrowSchemaView _schema_view
+
+ _fixed_size_types = (
+ NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY
+ )
+
+ _decimal_types = (
+ NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256
+ )
+
+ _time_unit_types = (
+ NANOARROW_TYPE_TIME32,
+ NANOARROW_TYPE_TIME64,
+ NANOARROW_TYPE_DURATION,
+ NANOARROW_TYPE_TIMESTAMP
+ )
+
+ _union_types = (
+ NANOARROW_TYPE_DENSE_UNION,
+ NANOARROW_TYPE_SPARSE_UNION
+ )
+
+ def __init__(self):
+ self._schema_view.type = NANOARROW_TYPE_UNINITIALIZED
+ self._schema_view.storage_type = NANOARROW_TYPE_UNINITIALIZED
+
+ @property
+ def type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def storage_type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.storage_type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def fixed_size(self):
+ if self._schema_view.type in SchemaView._fixed_size_types:
+ return self._schema_view.fixed_size
+
+ @property
+ def decimal_bitwidth(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_bitwidth
+
+ @property
+ def decimal_precision(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_precision
+
+ @property
+ def decimal_scale(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_scale
+
+ @property
+ def time_unit(self):
+ if self._schema_view.type in SchemaView._time_unit_types:
+ return ArrowTimeUnitString(self._schema_view.time_unit).decode('UTF-8')
+
+ @property
+ def timezone(self):
+ if self._schema_view.type == NANOARROW_TYPE_TIMESTAMP:
+ return self._schema_view.timezone.decode('UTF_8')
+
+ @property
+ def union_type_ids(self):
+ if self._schema_view.type in SchemaView._union_types:
+ type_ids_str = self._schema_view.union_type_ids.decode('UTF-8').split(',')
+ return (int(type_id) for type_id in type_ids_str)
+
+ @property
+ def extension_name(self):
+ if self._schema_view.extension_name.data != NULL:
+ name_bytes = PyBytes_FromStringAndSize(
+ self._schema_view.extension_name.data,
+ self._schema_view.extension_name.size_bytes
+ )
+ return name_bytes.decode('UTF-8')
+
+ @property
+ def extension_metadata(self):
+ if self._schema_view.extension_name.data != NULL:
+ return PyBytes_FromStringAndSize(
+ self._schema_view.extension_metadata.data,
+ self._schema_view.extension_metadata.size_bytes
+ )
+
+cdef class Array:
+ """ArrowArray wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArray as defined in the Arrow C Data interface, holding an
+ optional reference to a Schema that can be used to safely deserialize
+ the content. These objects are usually created using `nanoarrow.array()`.
+ This Python wrapper allows access to array fields but does not
+ automatically deserialize their content: use `.view()` to validate and
+ deserialize the content into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array = na.array(pa.array(["one", "two", "three", None]))
+ >>> array.length
+ 4
+ >>> array.null_count
+ 1
+ >>> array_view = array.view()
+ """
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef Schema _schema
+
+ @staticmethod
+ def allocate(Schema schema):
+ base = ArrayHolder()
+ return Array(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, Schema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("Array is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def length(self):
+ self._assert_valid()
+ return self._ptr.length
+
+ @property
+ def offset(self):
+ self._assert_valid()
+ return self._ptr.offset
+
+ @property
+ def null_count(self):
+ return self._ptr.null_count
+
+ @property
+ def buffers(self):
+ return tuple(<uintptr_t>self._ptr.buffers[i] for i in range(self._ptr.n_buffers))
Review Comment:
Possibly just philosophically aligned with what the `ArrowArray` actually provides. If you're trying to debug something this would tell you if one of your pointers is NULL...to get the `view().buffers` the array would have to be validated against a `Schema`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot merged pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] jorisvandenbossche commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1132442022
##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -84,3 +85,329 @@ def as_numpy_array(arr):
# TODO set base
return result
+
+
+def version():
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class CSchemaHolder:
+ cdef ArrowSchema c_schema
+
+ def __init__(self):
+ self.c_schema.release = NULL
+
+ def __del__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class CArrayHolder:
+ cdef ArrowArray c_array
+
+ def __init__(self):
+ self.c_array.release = NULL
+
+ def __del__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class CArrayViewHolder:
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __del__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+cdef class CSchema:
Review Comment:
In pyarrow we have the pattern of calling the exposed C(++) names with a "C" prefix, and then the corresponding cython classes without.
So the equivalent here would be to use CArrowSchema (instead of ArrowSchema) for nanoarrow.c one (and so for the declarations in the pxd file), and then ArrowSchema for the python class here.
I assume that what you describe (CSchema and Schema) can be done in a single class as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1231694271
##########
python/nanoarrow/_lib.pyx:
##########
@@ -526,23 +546,28 @@ cdef class ArrayView:
@property
def dictionary(self):
- return ArrayView(self, <uintptr_t>self._ptr.dictionary, self._array.dictionary)
-
- @property
- def array(self):
- return self._array
+ if self._ptr.dictionary == NULL:
+ return None
+ else:
+ return ArrayView(
+ self,
+ <uintptr_t>self._ptr.dictionary,
+ self._schema.dictionary,
+ None
Review Comment:
The buffer shelter is None here because the child/dictionary's reference to `self` is sufficient to keep the buffers valid.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1231696846
##########
.github/workflows/python.yaml:
##########
@@ -57,3 +57,31 @@ jobs:
- name: Run tests
run: |
pytest python/tests -v -s
+
+ - name: Run doctests
+ if: success() && matrix.python-version == '3.10'
+ run: |
+ # Needs editable install to run --doctest-cython
+ pip install pytest-cython
+ pip install -e python
Review Comment:
I'll update this...there are many things that could have been wrong with my local setup!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1229747521
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("schema is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def metadata(self):
+ self._assert_valid()
+ if self._ptr.metadata != NULL:
+ return SchemaMetadata(self, <uintptr_t>self._ptr.metadata)
+ else:
+ return None
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return SchemaChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Schema(self, <uintptr_t>self._ptr.dictionary)
+ else:
+ return None
+
+ def view(self):
+ self._assert_valid()
+ schema_view = SchemaView()
+ cdef Error error = Error()
+ cdef int result = ArrowSchemaViewInit(&schema_view._schema_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowSchemaViewInit()", result)
+
+ return schema_view
+
+
+cdef class SchemaView:
+ """ArrowSchemaView wrapper
+
+ The ArrowSchemaView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowSchema (e.g., parameter
+ values for parameterized types). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.decimal128(10, 3))
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'decimal128'
+ >>> schema_view.decimal_bitwidth
+ 128
+ >>> schema_view.decimal_precision
+ 10
+ >>> schema_view.decimal_scale
+ 3
+ """
+ cdef ArrowSchemaView _schema_view
+
+ _fixed_size_types = (
+ NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY
+ )
+
+ _decimal_types = (
+ NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256
+ )
+
+ _time_unit_types = (
+ NANOARROW_TYPE_TIME32,
+ NANOARROW_TYPE_TIME64,
+ NANOARROW_TYPE_DURATION,
+ NANOARROW_TYPE_TIMESTAMP
+ )
+
+ _union_types = (
+ NANOARROW_TYPE_DENSE_UNION,
+ NANOARROW_TYPE_SPARSE_UNION
+ )
+
+ def __init__(self):
+ self._schema_view.type = NANOARROW_TYPE_UNINITIALIZED
+ self._schema_view.storage_type = NANOARROW_TYPE_UNINITIALIZED
+
+ @property
+ def type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def storage_type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.storage_type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def fixed_size(self):
+ if self._schema_view.type in SchemaView._fixed_size_types:
+ return self._schema_view.fixed_size
+
+ @property
+ def decimal_bitwidth(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_bitwidth
+
+ @property
+ def decimal_precision(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_precision
+
+ @property
+ def decimal_scale(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_scale
+
+ @property
+ def time_unit(self):
+ if self._schema_view.type in SchemaView._time_unit_types:
+ return ArrowTimeUnitString(self._schema_view.time_unit).decode('UTF-8')
+
+ @property
+ def timezone(self):
+ if self._schema_view.type == NANOARROW_TYPE_TIMESTAMP:
+ return self._schema_view.timezone.decode('UTF_8')
+
+ @property
+ def union_type_ids(self):
+ if self._schema_view.type in SchemaView._union_types:
+ type_ids_str = self._schema_view.union_type_ids.decode('UTF-8').split(',')
+ return (int(type_id) for type_id in type_ids_str)
+
+ @property
+ def extension_name(self):
+ if self._schema_view.extension_name.data != NULL:
+ name_bytes = PyBytes_FromStringAndSize(
+ self._schema_view.extension_name.data,
+ self._schema_view.extension_name.size_bytes
+ )
+ return name_bytes.decode('UTF-8')
+
+ @property
+ def extension_metadata(self):
+ if self._schema_view.extension_name.data != NULL:
+ return PyBytes_FromStringAndSize(
+ self._schema_view.extension_metadata.data,
+ self._schema_view.extension_metadata.size_bytes
+ )
+
+cdef class Array:
+ """ArrowArray wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArray as defined in the Arrow C Data interface, holding an
+ optional reference to a Schema that can be used to safely deserialize
+ the content. These objects are usually created using `nanoarrow.array()`.
+ This Python wrapper allows access to array fields but does not
+ automatically deserialize their content: use `.view()` to validate and
+ deserialize the content into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array = na.array(pa.array(["one", "two", "three", None]))
+ >>> array.length
+ 4
+ >>> array.null_count
+ 1
+ >>> array_view = array.view()
+ """
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef Schema _schema
+
+ @staticmethod
+ def allocate(Schema schema):
+ base = ArrayHolder()
+ return Array(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, Schema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("Array is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def length(self):
+ self._assert_valid()
+ return self._ptr.length
+
+ @property
+ def offset(self):
+ self._assert_valid()
+ return self._ptr.offset
+
+ @property
+ def null_count(self):
+ return self._ptr.null_count
+
+ @property
+ def buffers(self):
+ return tuple(<uintptr_t>self._ptr.buffers[i] for i in range(self._ptr.n_buffers))
+
+ @property
+ def children(self):
+ return ArrayChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Array(self, <uintptr_t>self._ptr.dictionary, self._schema.dictionary)
+ else:
+ return None
+
+ def view(self):
+ cdef ArrayViewHolder holder = ArrayViewHolder()
+
+ cdef Error error = Error()
+ cdef int result = ArrowArrayViewInitFromSchema(&holder.c_array_view,
+ self._schema._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewInitFromSchema()", result)
+
+ result = ArrowArrayViewSetArray(&holder.c_array_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewSetArray()", result)
+
+ return ArrayView(holder, holder._addr(), self)
+
+
+cdef class ArrayView:
+ """ArrowArrayView wrapper
+
+ The ArrowArrayView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowArray (e.g., buffer types,
+ lengths, and content). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array_view = na.array(pa.array(["one", "two", "three", None])).view()
+ >>> np.array(array_view.buffers[1])
+ array([ 0, 3, 6, 11, 11], dtype=int32)
+ >>> np.array(array_view.buffers[2])
+ array([b'o', b'n', b'e', b't', b'w', b'o', b't', b'h', b'r', b'e', b'e'],
+ dtype='|S1')
+ """
+ cdef object _base
+ cdef ArrowArrayView* _ptr
+ cdef Array _array
+
+ def __init__(self, object base, uintptr_t addr, Array array):
+ self._base = base,
+ self._ptr = <ArrowArrayView*>addr
+ self._array = array
+
+ @property
+ def children(self):
+ return ArrayViewChildren(self)
+
+ @property
+ def buffers(self):
+ return ArrayViewBuffers(self)
+
+ @property
+ def dictionary(self):
+ return ArrayView(self, <uintptr_t>self._ptr.dictionary, self._array.dictionary)
+
+ @property
+ def array(self):
+ return self._array
+
+ @property
+ def schema(self):
+ return self._array._schema
+
+cdef class SchemaChildren:
+ """Wrapper for a lazily-resolved list of Schema children
+ """
+ cdef Schema _parent
+ cdef int64_t _length
+
+ def __init__(self, Schema parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+
+ return Schema(self._parent, self._child_addr(k))
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowSchema** children = self._parent._ptr.children
+ cdef ArrowSchema* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class SchemaMetadata:
+ """Wrapper for a lazily-parsed Schema.metadata string
+ """
+
+ cdef object _parent
+ cdef const char* _metadata
+ cdef ArrowMetadataReader _reader
+
+ def __init__(self, object parent, uintptr_t ptr):
+ self._parent = parent
+ self._metadata = <const char*>ptr
+
+ def _init_reader(self):
+ cdef int result = ArrowMetadataReaderInit(&self._reader, self._metadata)
+ if result != NANOARROW_OK:
+ Error.raise_error("ArrowMetadataReaderInit()", result)
+
+ def __len__(self):
+ self._init_reader()
+ return self._reader.remaining_keys
+
+ def __iter__(self):
+ cdef ArrowStringView key
+ cdef ArrowStringView value
+ self._init_reader()
+ while self._reader.remaining_keys > 0:
+ ArrowMetadataReaderRead(&self._reader, &key, &value)
+ key_obj = PyBytes_FromStringAndSize(key.data, key.size_bytes).decode('UTF-8')
+ value_obj = PyBytes_FromStringAndSize(value.data, value.size_bytes)
+ yield key_obj, value_obj
+
+
+cdef class ArrayChildren:
+ """Wrapper for a lazily-resolved list of Array children
+ """
+ cdef Array _parent
+ cdef int64_t _length
+
+ def __init__(self, Array parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return Array(self._parent, self._child_addr(k), self._parent.schema.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArray** children = self._parent._ptr.children
+ cdef ArrowArray* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class ArrayViewChildren:
Review Comment:
It's slightly different: `ArrowArrayViewSetArray()` is recursive, so `[x for x in array_view.children]` involves fewer calculations than `[x.view() for x in array.children]`. I don't know if that matters (maybe faster if you had many many columns).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1230308709
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("schema is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def metadata(self):
+ self._assert_valid()
+ if self._ptr.metadata != NULL:
+ return SchemaMetadata(self, <uintptr_t>self._ptr.metadata)
+ else:
+ return None
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return SchemaChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Schema(self, <uintptr_t>self._ptr.dictionary)
+ else:
+ return None
+
+ def view(self):
+ self._assert_valid()
+ schema_view = SchemaView()
+ cdef Error error = Error()
+ cdef int result = ArrowSchemaViewInit(&schema_view._schema_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowSchemaViewInit()", result)
+
+ return schema_view
+
+
+cdef class SchemaView:
+ """ArrowSchemaView wrapper
+
+ The ArrowSchemaView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowSchema (e.g., parameter
+ values for parameterized types). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.decimal128(10, 3))
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'decimal128'
+ >>> schema_view.decimal_bitwidth
+ 128
+ >>> schema_view.decimal_precision
+ 10
+ >>> schema_view.decimal_scale
+ 3
+ """
+ cdef ArrowSchemaView _schema_view
+
+ _fixed_size_types = (
+ NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY
+ )
+
+ _decimal_types = (
+ NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256
+ )
+
+ _time_unit_types = (
+ NANOARROW_TYPE_TIME32,
+ NANOARROW_TYPE_TIME64,
+ NANOARROW_TYPE_DURATION,
+ NANOARROW_TYPE_TIMESTAMP
+ )
+
+ _union_types = (
+ NANOARROW_TYPE_DENSE_UNION,
+ NANOARROW_TYPE_SPARSE_UNION
+ )
+
+ def __init__(self):
+ self._schema_view.type = NANOARROW_TYPE_UNINITIALIZED
+ self._schema_view.storage_type = NANOARROW_TYPE_UNINITIALIZED
+
+ @property
+ def type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def storage_type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.storage_type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def fixed_size(self):
+ if self._schema_view.type in SchemaView._fixed_size_types:
+ return self._schema_view.fixed_size
+
+ @property
+ def decimal_bitwidth(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_bitwidth
+
+ @property
+ def decimal_precision(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_precision
+
+ @property
+ def decimal_scale(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_scale
+
+ @property
+ def time_unit(self):
+ if self._schema_view.type in SchemaView._time_unit_types:
+ return ArrowTimeUnitString(self._schema_view.time_unit).decode('UTF-8')
+
+ @property
+ def timezone(self):
+ if self._schema_view.type == NANOARROW_TYPE_TIMESTAMP:
+ return self._schema_view.timezone.decode('UTF_8')
+
+ @property
+ def union_type_ids(self):
+ if self._schema_view.type in SchemaView._union_types:
+ type_ids_str = self._schema_view.union_type_ids.decode('UTF-8').split(',')
+ return (int(type_id) for type_id in type_ids_str)
+
+ @property
+ def extension_name(self):
+ if self._schema_view.extension_name.data != NULL:
+ name_bytes = PyBytes_FromStringAndSize(
+ self._schema_view.extension_name.data,
+ self._schema_view.extension_name.size_bytes
+ )
+ return name_bytes.decode('UTF-8')
+
+ @property
+ def extension_metadata(self):
+ if self._schema_view.extension_name.data != NULL:
+ return PyBytes_FromStringAndSize(
+ self._schema_view.extension_metadata.data,
+ self._schema_view.extension_metadata.size_bytes
+ )
+
+cdef class Array:
+ """ArrowArray wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArray as defined in the Arrow C Data interface, holding an
+ optional reference to a Schema that can be used to safely deserialize
+ the content. These objects are usually created using `nanoarrow.array()`.
+ This Python wrapper allows access to array fields but does not
+ automatically deserialize their content: use `.view()` to validate and
+ deserialize the content into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array = na.array(pa.array(["one", "two", "three", None]))
+ >>> array.length
+ 4
+ >>> array.null_count
+ 1
+ >>> array_view = array.view()
+ """
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef Schema _schema
+
+ @staticmethod
+ def allocate(Schema schema):
+ base = ArrayHolder()
+ return Array(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, Schema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("Array is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def length(self):
+ self._assert_valid()
+ return self._ptr.length
+
+ @property
+ def offset(self):
+ self._assert_valid()
+ return self._ptr.offset
+
+ @property
+ def null_count(self):
+ return self._ptr.null_count
+
+ @property
+ def buffers(self):
+ return tuple(<uintptr_t>self._ptr.buffers[i] for i in range(self._ptr.n_buffers))
+
+ @property
+ def children(self):
+ return ArrayChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Array(self, <uintptr_t>self._ptr.dictionary, self._schema.dictionary)
+ else:
+ return None
+
+ def view(self):
+ cdef ArrayViewHolder holder = ArrayViewHolder()
+
+ cdef Error error = Error()
+ cdef int result = ArrowArrayViewInitFromSchema(&holder.c_array_view,
+ self._schema._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewInitFromSchema()", result)
+
+ result = ArrowArrayViewSetArray(&holder.c_array_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewSetArray()", result)
+
+ return ArrayView(holder, holder._addr(), self)
+
+
+cdef class ArrayView:
+ """ArrowArrayView wrapper
+
+ The ArrowArrayView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowArray (e.g., buffer types,
+ lengths, and content). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array_view = na.array(pa.array(["one", "two", "three", None])).view()
+ >>> np.array(array_view.buffers[1])
+ array([ 0, 3, 6, 11, 11], dtype=int32)
+ >>> np.array(array_view.buffers[2])
+ array([b'o', b'n', b'e', b't', b'w', b'o', b't', b'h', b'r', b'e', b'e'],
+ dtype='|S1')
+ """
+ cdef object _base
+ cdef ArrowArrayView* _ptr
+ cdef Array _array
+
+ def __init__(self, object base, uintptr_t addr, Array array):
+ self._base = base,
+ self._ptr = <ArrowArrayView*>addr
+ self._array = array
+
+ @property
+ def children(self):
+ return ArrayViewChildren(self)
+
+ @property
+ def buffers(self):
+ return ArrayViewBuffers(self)
+
+ @property
+ def dictionary(self):
+ return ArrayView(self, <uintptr_t>self._ptr.dictionary, self._array.dictionary)
+
+ @property
+ def array(self):
+ return self._array
+
+ @property
+ def schema(self):
+ return self._array._schema
+
+cdef class SchemaChildren:
+ """Wrapper for a lazily-resolved list of Schema children
+ """
+ cdef Schema _parent
+ cdef int64_t _length
+
+ def __init__(self, Schema parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+
+ return Schema(self._parent, self._child_addr(k))
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowSchema** children = self._parent._ptr.children
+ cdef ArrowSchema* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class SchemaMetadata:
+ """Wrapper for a lazily-parsed Schema.metadata string
+ """
+
+ cdef object _parent
+ cdef const char* _metadata
+ cdef ArrowMetadataReader _reader
+
+ def __init__(self, object parent, uintptr_t ptr):
+ self._parent = parent
+ self._metadata = <const char*>ptr
+
+ def _init_reader(self):
+ cdef int result = ArrowMetadataReaderInit(&self._reader, self._metadata)
+ if result != NANOARROW_OK:
+ Error.raise_error("ArrowMetadataReaderInit()", result)
+
+ def __len__(self):
+ self._init_reader()
+ return self._reader.remaining_keys
+
+ def __iter__(self):
+ cdef ArrowStringView key
+ cdef ArrowStringView value
+ self._init_reader()
+ while self._reader.remaining_keys > 0:
+ ArrowMetadataReaderRead(&self._reader, &key, &value)
+ key_obj = PyBytes_FromStringAndSize(key.data, key.size_bytes).decode('UTF-8')
+ value_obj = PyBytes_FromStringAndSize(value.data, value.size_bytes)
+ yield key_obj, value_obj
+
+
+cdef class ArrayChildren:
+ """Wrapper for a lazily-resolved list of Array children
+ """
+ cdef Array _parent
+ cdef int64_t _length
+
+ def __init__(self, Array parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return Array(self._parent, self._child_addr(k), self._parent.schema.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArray** children = self._parent._ptr.children
+ cdef ArrowArray* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class ArrayViewChildren:
+ """Wrapper for a lazily-resolved list of ArrayView children
+ """
+ cdef ArrayView _parent
+ cdef int64_t _length
+
+ def __init__(self, ArrayView parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return ArrayView(self._parent, self._child_addr(k), self._parent._array.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArrayView** children = self._parent._ptr.children
+ cdef ArrowArrayView* child = children[i]
+ return <uintptr_t>child
+
+cdef class BufferView:
+ """Wrapper for Array buffer content
+
+ This object is a Python wrapper around a buffer held by an Array.
+ It implements the Python buffer protocol and is best accessed through
+ another implementor (e.g., `np.array(array_view.buffers[1])`)). Note that
+ this buffer content does not apply any parent offset.
+ """
+ cdef object _base
+ cdef ArrowBufferView* _ptr
+ cdef ArrowBufferType _buffer_type
+ cdef ArrowType _buffer_data_type
+ cdef Py_ssize_t _element_size_bits
+ cdef Py_ssize_t _shape
+ cdef Py_ssize_t _strides
+
+ def __init__(self, object base, uintptr_t addr,
+ ArrowBufferType buffer_type, ArrowType buffer_data_type,
+ Py_ssize_t element_size_bits):
+ self._base = base
+ self._ptr = <ArrowBufferView*>addr
+ self._buffer_type = buffer_type
+ self._buffer_data_type = buffer_data_type
+ self._element_size_bits = element_size_bits
+ self._strides = self._item_size()
+ self._shape = self._ptr.size_bytes // self._strides
+
+
+ cdef Py_ssize_t _item_size(self):
+ if self._buffer_data_type == NANOARROW_TYPE_BOOL:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_BINARY:
+ return 1
+ else:
+ return self._element_size_bits // 8
+
+ cdef const char* _get_format(self):
+ if self._buffer_data_type == NANOARROW_TYPE_INT8:
+ return "b"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT8:
+ return "B"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT16:
+ return "h"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT16:
+ return "H"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT32:
+ return "i"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT32:
+ return "I"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT64:
+ return "l"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT64:
+ return "L"
+ elif self._buffer_data_type == NANOARROW_TYPE_FLOAT:
+ return "f"
+ elif self._buffer_data_type == NANOARROW_TYPE_DOUBLE:
+ return "d"
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return "c"
+ else:
+ return "B"
+
+ def __getbuffer__(self, Py_buffer *buffer, int flags):
+ buffer.buf = self._ptr.data.data
+ buffer.format = self._get_format()
+ buffer.internal = NULL
+ buffer.itemsize = self._strides
+ buffer.len = self._ptr.size_bytes
+ buffer.ndim = 1
+ buffer.obj = self
+ buffer.readonly = 1
+ buffer.shape = &self._shape
+ buffer.strides = &self._strides
+ buffer.suboffsets = NULL
+
+ def __releasebuffer__(self, Py_buffer *buffer):
+ pass
+
+
+cdef class ArrayViewBuffers:
+ """A lazily-resolved list of ArrayView buffers
+ """
+ cdef ArrayView _array_view
+ cdef int64_t _length
+
+ def __init__(self, ArrayView array_view):
+ self._array_view = array_view
+ self._length = array_view._array._ptr.n_buffers
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ cdef ArrowBufferView* buffer_view = &(self._array_view._ptr.buffer_views[k])
+ if buffer_view.data.data == NULL:
+ return None
+
+ return BufferView(
+ self._array_view,
+ <uintptr_t>buffer_view,
+ self._array_view._ptr.layout.buffer_type[k],
+ self._array_view._ptr.layout.buffer_data_type[k],
+ self._array_view._ptr.layout.element_size_bits[k]
+ )
+
+
+cdef class ArrayStream:
+ """ArrowArrayStream wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArrayStream as defined in the Arrow C Stream interface.
+ These objects are usually created using `nanoarrow.array_stream()`.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> pa_column = pa.array([1, 2, 3], pa.int32())
+ >>> pa_batch = pa.record_batch([pa_column], names=["col1"])
+ >>> pa_reader = pa.RecordBatchReader.from_batches(pa_batch.schema, [pa_batch])
+ >>> array_stream = na.array_stream(pa_reader)
+ >>> array_stream.get_schema()
+ struct<col1: int32>
+ >>> array_stream.get_next().length
+ 3
+ >>> array_stream.get_next() is None
+ True
+ """
+ cdef object _base
+ cdef ArrowArrayStream* _ptr
+ cdef object _cached_schema
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowArrayStream*>addr
+ self._cached_schema = None
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("array stream pointer is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("array stream is released")
+
+ def _get_schema(self, Schema schema):
+ self._assert_valid()
+ cdef int code = self._ptr.get_schema(self._ptr, schema._ptr)
+ cdef const char* message = NULL
+ if code != NANOARROW_OK:
+ message = self._ptr.get_last_error(self._ptr)
+ if message != NULL:
+ raise NanoarrowException(
+ "ArrowArrayStream::get_schema()",
+ code,
+ message.decode("UTF-8")
+ )
+ else:
+ Error.raise_error("ArrowArrayStream::get_schema()", code)
+
+ self._cached_schema = schema
+
+ def get_schema(self):
+ """Get the schema associated with this stream
+ """
+ # Update the cached copy of the schema as an independent object
+ self._cached_schema = Schema.allocate()
+ self._get_schema(self._cached_schema)
+
+ # Return an independent copy
Review Comment:
I think I found middle ground here...calling `.get_next()` always uses a cached schema but `.get_schema()` always gets you a fresh one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1132422895
##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -84,3 +85,329 @@ def as_numpy_array(arr):
# TODO set base
return result
+
+
+def version():
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class CSchemaHolder:
Review Comment:
Yes...it could be a capsule or this object but is also usefully the parent array (so that you can do `array.children[0]` and re-use the `CSchema` class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1132389167
##########
python/.gitignore:
##########
@@ -18,7 +18,8 @@
src/nanoarrow/nanoarrow.c
src/nanoarrow/nanoarrow.h
-src/nanoarrow/*.cpp
+src/nanoarrow/nanoarrow_c.pxd
Review Comment:
It is a pretty useful file to have on hand...nanoarrow.c/h are built for vendoring and there's no reason another Python package couldn't use the same .pxd (provided it's generated from the correct nanoarrow.h).
Maybe it could get checked into the dist/ folder next to the bundled nanoarrow/h?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1229740544
##########
.github/workflows/python.yaml:
##########
@@ -57,3 +57,31 @@ jobs:
- name: Run tests
run: |
pytest python/tests -v -s
+
+ - name: Run doctests
+ if: success() && matrix.python-version == '3.10'
+ run: |
+ # Needs editable install to run --doctest-cython
+ pip install pytest-cython
+ pip install -e python
Review Comment:
I think it's installed without `-e` above (maybe I'll just add it there, and maybe also add `pytest-cython` to pyproject.toml while I'm at it).
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
Review Comment:
Definitely not!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1229845689
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("schema is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def metadata(self):
+ self._assert_valid()
+ if self._ptr.metadata != NULL:
+ return SchemaMetadata(self, <uintptr_t>self._ptr.metadata)
+ else:
+ return None
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return SchemaChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Schema(self, <uintptr_t>self._ptr.dictionary)
+ else:
+ return None
+
+ def view(self):
+ self._assert_valid()
+ schema_view = SchemaView()
+ cdef Error error = Error()
+ cdef int result = ArrowSchemaViewInit(&schema_view._schema_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowSchemaViewInit()", result)
+
+ return schema_view
+
+
+cdef class SchemaView:
+ """ArrowSchemaView wrapper
+
+ The ArrowSchemaView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowSchema (e.g., parameter
+ values for parameterized types). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.decimal128(10, 3))
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'decimal128'
+ >>> schema_view.decimal_bitwidth
+ 128
+ >>> schema_view.decimal_precision
+ 10
+ >>> schema_view.decimal_scale
+ 3
+ """
+ cdef ArrowSchemaView _schema_view
+
+ _fixed_size_types = (
+ NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY
+ )
+
+ _decimal_types = (
+ NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256
+ )
+
+ _time_unit_types = (
+ NANOARROW_TYPE_TIME32,
+ NANOARROW_TYPE_TIME64,
+ NANOARROW_TYPE_DURATION,
+ NANOARROW_TYPE_TIMESTAMP
+ )
+
+ _union_types = (
+ NANOARROW_TYPE_DENSE_UNION,
+ NANOARROW_TYPE_SPARSE_UNION
+ )
+
+ def __init__(self):
+ self._schema_view.type = NANOARROW_TYPE_UNINITIALIZED
+ self._schema_view.storage_type = NANOARROW_TYPE_UNINITIALIZED
+
+ @property
+ def type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def storage_type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.storage_type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def fixed_size(self):
+ if self._schema_view.type in SchemaView._fixed_size_types:
+ return self._schema_view.fixed_size
+
+ @property
+ def decimal_bitwidth(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_bitwidth
+
+ @property
+ def decimal_precision(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_precision
+
+ @property
+ def decimal_scale(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_scale
+
+ @property
+ def time_unit(self):
+ if self._schema_view.type in SchemaView._time_unit_types:
+ return ArrowTimeUnitString(self._schema_view.time_unit).decode('UTF-8')
+
+ @property
+ def timezone(self):
+ if self._schema_view.type == NANOARROW_TYPE_TIMESTAMP:
+ return self._schema_view.timezone.decode('UTF_8')
+
+ @property
+ def union_type_ids(self):
+ if self._schema_view.type in SchemaView._union_types:
+ type_ids_str = self._schema_view.union_type_ids.decode('UTF-8').split(',')
+ return (int(type_id) for type_id in type_ids_str)
+
+ @property
+ def extension_name(self):
+ if self._schema_view.extension_name.data != NULL:
+ name_bytes = PyBytes_FromStringAndSize(
+ self._schema_view.extension_name.data,
+ self._schema_view.extension_name.size_bytes
+ )
+ return name_bytes.decode('UTF-8')
+
+ @property
+ def extension_metadata(self):
+ if self._schema_view.extension_name.data != NULL:
+ return PyBytes_FromStringAndSize(
+ self._schema_view.extension_metadata.data,
+ self._schema_view.extension_metadata.size_bytes
+ )
+
+cdef class Array:
+ """ArrowArray wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArray as defined in the Arrow C Data interface, holding an
+ optional reference to a Schema that can be used to safely deserialize
+ the content. These objects are usually created using `nanoarrow.array()`.
+ This Python wrapper allows access to array fields but does not
+ automatically deserialize their content: use `.view()` to validate and
+ deserialize the content into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array = na.array(pa.array(["one", "two", "three", None]))
+ >>> array.length
+ 4
+ >>> array.null_count
+ 1
+ >>> array_view = array.view()
+ """
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef Schema _schema
+
+ @staticmethod
+ def allocate(Schema schema):
+ base = ArrayHolder()
+ return Array(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, Schema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("Array is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def length(self):
+ self._assert_valid()
+ return self._ptr.length
+
+ @property
+ def offset(self):
+ self._assert_valid()
+ return self._ptr.offset
+
+ @property
+ def null_count(self):
+ return self._ptr.null_count
+
+ @property
+ def buffers(self):
+ return tuple(<uintptr_t>self._ptr.buffers[i] for i in range(self._ptr.n_buffers))
+
+ @property
+ def children(self):
+ return ArrayChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Array(self, <uintptr_t>self._ptr.dictionary, self._schema.dictionary)
+ else:
+ return None
+
+ def view(self):
+ cdef ArrayViewHolder holder = ArrayViewHolder()
+
+ cdef Error error = Error()
+ cdef int result = ArrowArrayViewInitFromSchema(&holder.c_array_view,
+ self._schema._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewInitFromSchema()", result)
+
+ result = ArrowArrayViewSetArray(&holder.c_array_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewSetArray()", result)
+
+ return ArrayView(holder, holder._addr(), self)
+
+
+cdef class ArrayView:
+ """ArrowArrayView wrapper
+
+ The ArrowArrayView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowArray (e.g., buffer types,
+ lengths, and content). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array_view = na.array(pa.array(["one", "two", "three", None])).view()
+ >>> np.array(array_view.buffers[1])
+ array([ 0, 3, 6, 11, 11], dtype=int32)
+ >>> np.array(array_view.buffers[2])
+ array([b'o', b'n', b'e', b't', b'w', b'o', b't', b'h', b'r', b'e', b'e'],
+ dtype='|S1')
+ """
+ cdef object _base
+ cdef ArrowArrayView* _ptr
+ cdef Array _array
+
+ def __init__(self, object base, uintptr_t addr, Array array):
+ self._base = base,
+ self._ptr = <ArrowArrayView*>addr
+ self._array = array
+
+ @property
+ def children(self):
+ return ArrayViewChildren(self)
+
+ @property
+ def buffers(self):
+ return ArrayViewBuffers(self)
+
+ @property
+ def dictionary(self):
+ return ArrayView(self, <uintptr_t>self._ptr.dictionary, self._array.dictionary)
+
+ @property
+ def array(self):
+ return self._array
+
+ @property
+ def schema(self):
+ return self._array._schema
+
+cdef class SchemaChildren:
+ """Wrapper for a lazily-resolved list of Schema children
+ """
+ cdef Schema _parent
+ cdef int64_t _length
+
+ def __init__(self, Schema parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+
+ return Schema(self._parent, self._child_addr(k))
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowSchema** children = self._parent._ptr.children
+ cdef ArrowSchema* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class SchemaMetadata:
+ """Wrapper for a lazily-parsed Schema.metadata string
+ """
+
+ cdef object _parent
+ cdef const char* _metadata
+ cdef ArrowMetadataReader _reader
+
+ def __init__(self, object parent, uintptr_t ptr):
+ self._parent = parent
+ self._metadata = <const char*>ptr
+
+ def _init_reader(self):
+ cdef int result = ArrowMetadataReaderInit(&self._reader, self._metadata)
+ if result != NANOARROW_OK:
+ Error.raise_error("ArrowMetadataReaderInit()", result)
+
+ def __len__(self):
+ self._init_reader()
+ return self._reader.remaining_keys
+
+ def __iter__(self):
+ cdef ArrowStringView key
+ cdef ArrowStringView value
+ self._init_reader()
+ while self._reader.remaining_keys > 0:
+ ArrowMetadataReaderRead(&self._reader, &key, &value)
+ key_obj = PyBytes_FromStringAndSize(key.data, key.size_bytes).decode('UTF-8')
+ value_obj = PyBytes_FromStringAndSize(value.data, value.size_bytes)
+ yield key_obj, value_obj
+
+
+cdef class ArrayChildren:
+ """Wrapper for a lazily-resolved list of Array children
+ """
+ cdef Array _parent
+ cdef int64_t _length
+
+ def __init__(self, Array parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return Array(self._parent, self._child_addr(k), self._parent.schema.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArray** children = self._parent._ptr.children
+ cdef ArrowArray* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class ArrayViewChildren:
+ """Wrapper for a lazily-resolved list of ArrayView children
+ """
+ cdef ArrayView _parent
+ cdef int64_t _length
+
+ def __init__(self, ArrayView parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return ArrayView(self._parent, self._child_addr(k), self._parent._array.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArrayView** children = self._parent._ptr.children
+ cdef ArrowArrayView* child = children[i]
+ return <uintptr_t>child
+
+cdef class BufferView:
+ """Wrapper for Array buffer content
+
+ This object is a Python wrapper around a buffer held by an Array.
+ It implements the Python buffer protocol and is best accessed through
+ another implementor (e.g., `np.array(array_view.buffers[1])`)). Note that
+ this buffer content does not apply any parent offset.
+ """
+ cdef object _base
+ cdef ArrowBufferView* _ptr
+ cdef ArrowBufferType _buffer_type
+ cdef ArrowType _buffer_data_type
+ cdef Py_ssize_t _element_size_bits
+ cdef Py_ssize_t _shape
+ cdef Py_ssize_t _strides
+
+ def __init__(self, object base, uintptr_t addr,
+ ArrowBufferType buffer_type, ArrowType buffer_data_type,
+ Py_ssize_t element_size_bits):
+ self._base = base
+ self._ptr = <ArrowBufferView*>addr
+ self._buffer_type = buffer_type
+ self._buffer_data_type = buffer_data_type
+ self._element_size_bits = element_size_bits
+ self._strides = self._item_size()
+ self._shape = self._ptr.size_bytes // self._strides
+
+
+ cdef Py_ssize_t _item_size(self):
+ if self._buffer_data_type == NANOARROW_TYPE_BOOL:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_BINARY:
+ return 1
+ else:
+ return self._element_size_bits // 8
+
+ cdef const char* _get_format(self):
+ if self._buffer_data_type == NANOARROW_TYPE_INT8:
+ return "b"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT8:
+ return "B"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT16:
+ return "h"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT16:
+ return "H"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT32:
+ return "i"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT32:
+ return "I"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT64:
+ return "l"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT64:
+ return "L"
+ elif self._buffer_data_type == NANOARROW_TYPE_FLOAT:
+ return "f"
+ elif self._buffer_data_type == NANOARROW_TYPE_DOUBLE:
+ return "d"
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return "c"
+ else:
+ return "B"
+
+ def __getbuffer__(self, Py_buffer *buffer, int flags):
+ buffer.buf = self._ptr.data.data
+ buffer.format = self._get_format()
+ buffer.internal = NULL
+ buffer.itemsize = self._strides
+ buffer.len = self._ptr.size_bytes
+ buffer.ndim = 1
+ buffer.obj = self
+ buffer.readonly = 1
+ buffer.shape = &self._shape
+ buffer.strides = &self._strides
+ buffer.suboffsets = NULL
+
+ def __releasebuffer__(self, Py_buffer *buffer):
+ pass
+
+
+cdef class ArrayViewBuffers:
+ """A lazily-resolved list of ArrayView buffers
+ """
+ cdef ArrayView _array_view
+ cdef int64_t _length
+
+ def __init__(self, ArrayView array_view):
+ self._array_view = array_view
+ self._length = array_view._array._ptr.n_buffers
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ cdef ArrowBufferView* buffer_view = &(self._array_view._ptr.buffer_views[k])
+ if buffer_view.data.data == NULL:
+ return None
+
+ return BufferView(
+ self._array_view,
+ <uintptr_t>buffer_view,
+ self._array_view._ptr.layout.buffer_type[k],
+ self._array_view._ptr.layout.buffer_data_type[k],
+ self._array_view._ptr.layout.element_size_bits[k]
+ )
+
+
+cdef class ArrayStream:
+ """ArrowArrayStream wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArrayStream as defined in the Arrow C Stream interface.
+ These objects are usually created using `nanoarrow.array_stream()`.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> pa_column = pa.array([1, 2, 3], pa.int32())
+ >>> pa_batch = pa.record_batch([pa_column], names=["col1"])
+ >>> pa_reader = pa.RecordBatchReader.from_batches(pa_batch.schema, [pa_batch])
+ >>> array_stream = na.array_stream(pa_reader)
+ >>> array_stream.get_schema()
+ struct<col1: int32>
+ >>> array_stream.get_next().length
+ 3
+ >>> array_stream.get_next() is None
+ True
+ """
+ cdef object _base
+ cdef ArrowArrayStream* _ptr
+ cdef object _cached_schema
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowArrayStream*>addr
+ self._cached_schema = None
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("array stream pointer is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("array stream is released")
+
+ def _get_schema(self, Schema schema):
+ self._assert_valid()
+ cdef int code = self._ptr.get_schema(self._ptr, schema._ptr)
+ cdef const char* message = NULL
+ if code != NANOARROW_OK:
+ message = self._ptr.get_last_error(self._ptr)
+ if message != NULL:
+ raise NanoarrowException(
+ "ArrowArrayStream::get_schema()",
+ code,
+ message.decode("UTF-8")
+ )
+ else:
+ Error.raise_error("ArrowArrayStream::get_schema()", code)
+
+ self._cached_schema = schema
+
+ def get_schema(self):
+ """Get the schema associated with this stream
+ """
+ # Update the cached copy of the schema as an independent object
+ self._cached_schema = Schema.allocate()
+ self._get_schema(self._cached_schema)
+
+ # Return an independent copy
+ out = Schema.allocate()
+ self._get_schema(out)
+ return out
+
+ def get_next(self):
+ """Get the next Array from this stream
+
+ Returns None when there are no more arrays in this stream.
+ """
+ self._assert_valid()
+
+ if self._cached_schema is None:
+ self._cached_schema = Schema.allocate()
+ self._get_schema(self._cached_schema)
+
+ cdef Array array = Array.allocate(self._cached_schema)
+ cdef int code = self._ptr.get_next(self._ptr, array._ptr)
+ cdef const char* message = NULL
+ if code != NANOARROW_OK:
+ message = self._ptr.get_last_error(self._ptr)
+ if message != NULL:
+ raise NanoarrowException(
+ "ArrowArrayStream::get_next()",
+ code,
+ message.decode("UTF-8")
+ )
+ else:
+ Error.raise_error("ArrowArrayStream::get_next()", code)
+
+ if not array.is_valid():
+ return None
Review Comment:
I think doing whatever `read_next_batch()` does is worth doing here. The R record batch reader returns NULL which is how I got here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1229843874
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("schema is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def metadata(self):
+ self._assert_valid()
+ if self._ptr.metadata != NULL:
+ return SchemaMetadata(self, <uintptr_t>self._ptr.metadata)
+ else:
+ return None
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return SchemaChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Schema(self, <uintptr_t>self._ptr.dictionary)
+ else:
+ return None
+
+ def view(self):
+ self._assert_valid()
+ schema_view = SchemaView()
+ cdef Error error = Error()
+ cdef int result = ArrowSchemaViewInit(&schema_view._schema_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowSchemaViewInit()", result)
+
+ return schema_view
+
+
+cdef class SchemaView:
+ """ArrowSchemaView wrapper
+
+ The ArrowSchemaView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowSchema (e.g., parameter
+ values for parameterized types). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.decimal128(10, 3))
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'decimal128'
+ >>> schema_view.decimal_bitwidth
+ 128
+ >>> schema_view.decimal_precision
+ 10
+ >>> schema_view.decimal_scale
+ 3
+ """
+ cdef ArrowSchemaView _schema_view
+
+ _fixed_size_types = (
+ NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY
+ )
+
+ _decimal_types = (
+ NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256
+ )
+
+ _time_unit_types = (
+ NANOARROW_TYPE_TIME32,
+ NANOARROW_TYPE_TIME64,
+ NANOARROW_TYPE_DURATION,
+ NANOARROW_TYPE_TIMESTAMP
+ )
+
+ _union_types = (
+ NANOARROW_TYPE_DENSE_UNION,
+ NANOARROW_TYPE_SPARSE_UNION
+ )
+
+ def __init__(self):
+ self._schema_view.type = NANOARROW_TYPE_UNINITIALIZED
+ self._schema_view.storage_type = NANOARROW_TYPE_UNINITIALIZED
+
+ @property
+ def type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def storage_type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.storage_type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def fixed_size(self):
+ if self._schema_view.type in SchemaView._fixed_size_types:
+ return self._schema_view.fixed_size
+
+ @property
+ def decimal_bitwidth(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_bitwidth
+
+ @property
+ def decimal_precision(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_precision
+
+ @property
+ def decimal_scale(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_scale
+
+ @property
+ def time_unit(self):
+ if self._schema_view.type in SchemaView._time_unit_types:
+ return ArrowTimeUnitString(self._schema_view.time_unit).decode('UTF-8')
+
+ @property
+ def timezone(self):
+ if self._schema_view.type == NANOARROW_TYPE_TIMESTAMP:
+ return self._schema_view.timezone.decode('UTF_8')
+
+ @property
+ def union_type_ids(self):
+ if self._schema_view.type in SchemaView._union_types:
+ type_ids_str = self._schema_view.union_type_ids.decode('UTF-8').split(',')
+ return (int(type_id) for type_id in type_ids_str)
+
+ @property
+ def extension_name(self):
+ if self._schema_view.extension_name.data != NULL:
+ name_bytes = PyBytes_FromStringAndSize(
+ self._schema_view.extension_name.data,
+ self._schema_view.extension_name.size_bytes
+ )
+ return name_bytes.decode('UTF-8')
+
+ @property
+ def extension_metadata(self):
+ if self._schema_view.extension_name.data != NULL:
+ return PyBytes_FromStringAndSize(
+ self._schema_view.extension_metadata.data,
+ self._schema_view.extension_metadata.size_bytes
+ )
+
+cdef class Array:
+ """ArrowArray wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArray as defined in the Arrow C Data interface, holding an
+ optional reference to a Schema that can be used to safely deserialize
+ the content. These objects are usually created using `nanoarrow.array()`.
+ This Python wrapper allows access to array fields but does not
+ automatically deserialize their content: use `.view()` to validate and
+ deserialize the content into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array = na.array(pa.array(["one", "two", "three", None]))
+ >>> array.length
+ 4
+ >>> array.null_count
+ 1
+ >>> array_view = array.view()
+ """
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef Schema _schema
+
+ @staticmethod
+ def allocate(Schema schema):
+ base = ArrayHolder()
+ return Array(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, Schema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("Array is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def length(self):
+ self._assert_valid()
+ return self._ptr.length
+
+ @property
+ def offset(self):
+ self._assert_valid()
+ return self._ptr.offset
+
+ @property
+ def null_count(self):
+ return self._ptr.null_count
+
+ @property
+ def buffers(self):
+ return tuple(<uintptr_t>self._ptr.buffers[i] for i in range(self._ptr.n_buffers))
+
+ @property
+ def children(self):
+ return ArrayChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Array(self, <uintptr_t>self._ptr.dictionary, self._schema.dictionary)
+ else:
+ return None
+
+ def view(self):
+ cdef ArrayViewHolder holder = ArrayViewHolder()
+
+ cdef Error error = Error()
+ cdef int result = ArrowArrayViewInitFromSchema(&holder.c_array_view,
+ self._schema._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewInitFromSchema()", result)
+
+ result = ArrowArrayViewSetArray(&holder.c_array_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewSetArray()", result)
+
+ return ArrayView(holder, holder._addr(), self)
+
+
+cdef class ArrayView:
+ """ArrowArrayView wrapper
+
+ The ArrowArrayView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowArray (e.g., buffer types,
+ lengths, and content). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array_view = na.array(pa.array(["one", "two", "three", None])).view()
+ >>> np.array(array_view.buffers[1])
+ array([ 0, 3, 6, 11, 11], dtype=int32)
+ >>> np.array(array_view.buffers[2])
+ array([b'o', b'n', b'e', b't', b'w', b'o', b't', b'h', b'r', b'e', b'e'],
+ dtype='|S1')
+ """
+ cdef object _base
+ cdef ArrowArrayView* _ptr
+ cdef Array _array
+
+ def __init__(self, object base, uintptr_t addr, Array array):
+ self._base = base,
+ self._ptr = <ArrowArrayView*>addr
+ self._array = array
+
+ @property
+ def children(self):
+ return ArrayViewChildren(self)
+
+ @property
+ def buffers(self):
+ return ArrayViewBuffers(self)
+
+ @property
+ def dictionary(self):
+ return ArrayView(self, <uintptr_t>self._ptr.dictionary, self._array.dictionary)
+
+ @property
+ def array(self):
+ return self._array
+
+ @property
+ def schema(self):
+ return self._array._schema
+
+cdef class SchemaChildren:
+ """Wrapper for a lazily-resolved list of Schema children
+ """
+ cdef Schema _parent
+ cdef int64_t _length
+
+ def __init__(self, Schema parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+
+ return Schema(self._parent, self._child_addr(k))
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowSchema** children = self._parent._ptr.children
+ cdef ArrowSchema* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class SchemaMetadata:
+ """Wrapper for a lazily-parsed Schema.metadata string
+ """
+
+ cdef object _parent
+ cdef const char* _metadata
+ cdef ArrowMetadataReader _reader
+
+ def __init__(self, object parent, uintptr_t ptr):
+ self._parent = parent
+ self._metadata = <const char*>ptr
+
+ def _init_reader(self):
+ cdef int result = ArrowMetadataReaderInit(&self._reader, self._metadata)
+ if result != NANOARROW_OK:
+ Error.raise_error("ArrowMetadataReaderInit()", result)
+
+ def __len__(self):
+ self._init_reader()
+ return self._reader.remaining_keys
+
+ def __iter__(self):
+ cdef ArrowStringView key
+ cdef ArrowStringView value
+ self._init_reader()
+ while self._reader.remaining_keys > 0:
+ ArrowMetadataReaderRead(&self._reader, &key, &value)
+ key_obj = PyBytes_FromStringAndSize(key.data, key.size_bytes).decode('UTF-8')
+ value_obj = PyBytes_FromStringAndSize(value.data, value.size_bytes)
+ yield key_obj, value_obj
+
+
+cdef class ArrayChildren:
+ """Wrapper for a lazily-resolved list of Array children
+ """
+ cdef Array _parent
+ cdef int64_t _length
+
+ def __init__(self, Array parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return Array(self._parent, self._child_addr(k), self._parent.schema.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArray** children = self._parent._ptr.children
+ cdef ArrowArray* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class ArrayViewChildren:
+ """Wrapper for a lazily-resolved list of ArrayView children
+ """
+ cdef ArrayView _parent
+ cdef int64_t _length
+
+ def __init__(self, ArrayView parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return ArrayView(self._parent, self._child_addr(k), self._parent._array.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArrayView** children = self._parent._ptr.children
+ cdef ArrowArrayView* child = children[i]
+ return <uintptr_t>child
+
+cdef class BufferView:
+ """Wrapper for Array buffer content
+
+ This object is a Python wrapper around a buffer held by an Array.
+ It implements the Python buffer protocol and is best accessed through
+ another implementor (e.g., `np.array(array_view.buffers[1])`)). Note that
+ this buffer content does not apply any parent offset.
+ """
+ cdef object _base
+ cdef ArrowBufferView* _ptr
+ cdef ArrowBufferType _buffer_type
+ cdef ArrowType _buffer_data_type
+ cdef Py_ssize_t _element_size_bits
+ cdef Py_ssize_t _shape
+ cdef Py_ssize_t _strides
+
+ def __init__(self, object base, uintptr_t addr,
+ ArrowBufferType buffer_type, ArrowType buffer_data_type,
+ Py_ssize_t element_size_bits):
+ self._base = base
+ self._ptr = <ArrowBufferView*>addr
+ self._buffer_type = buffer_type
+ self._buffer_data_type = buffer_data_type
+ self._element_size_bits = element_size_bits
+ self._strides = self._item_size()
+ self._shape = self._ptr.size_bytes // self._strides
+
+
+ cdef Py_ssize_t _item_size(self):
+ if self._buffer_data_type == NANOARROW_TYPE_BOOL:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_BINARY:
+ return 1
+ else:
+ return self._element_size_bits // 8
+
+ cdef const char* _get_format(self):
+ if self._buffer_data_type == NANOARROW_TYPE_INT8:
+ return "b"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT8:
+ return "B"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT16:
+ return "h"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT16:
+ return "H"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT32:
+ return "i"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT32:
+ return "I"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT64:
+ return "l"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT64:
+ return "L"
+ elif self._buffer_data_type == NANOARROW_TYPE_FLOAT:
+ return "f"
+ elif self._buffer_data_type == NANOARROW_TYPE_DOUBLE:
+ return "d"
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return "c"
+ else:
+ return "B"
+
+ def __getbuffer__(self, Py_buffer *buffer, int flags):
+ buffer.buf = self._ptr.data.data
+ buffer.format = self._get_format()
+ buffer.internal = NULL
+ buffer.itemsize = self._strides
+ buffer.len = self._ptr.size_bytes
+ buffer.ndim = 1
+ buffer.obj = self
+ buffer.readonly = 1
+ buffer.shape = &self._shape
+ buffer.strides = &self._strides
+ buffer.suboffsets = NULL
+
+ def __releasebuffer__(self, Py_buffer *buffer):
+ pass
+
+
+cdef class ArrayViewBuffers:
+ """A lazily-resolved list of ArrayView buffers
+ """
+ cdef ArrayView _array_view
+ cdef int64_t _length
+
+ def __init__(self, ArrayView array_view):
+ self._array_view = array_view
+ self._length = array_view._array._ptr.n_buffers
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ cdef ArrowBufferView* buffer_view = &(self._array_view._ptr.buffer_views[k])
+ if buffer_view.data.data == NULL:
+ return None
+
+ return BufferView(
+ self._array_view,
+ <uintptr_t>buffer_view,
+ self._array_view._ptr.layout.buffer_type[k],
+ self._array_view._ptr.layout.buffer_data_type[k],
+ self._array_view._ptr.layout.element_size_bits[k]
+ )
+
+
+cdef class ArrayStream:
+ """ArrowArrayStream wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArrayStream as defined in the Arrow C Stream interface.
+ These objects are usually created using `nanoarrow.array_stream()`.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> pa_column = pa.array([1, 2, 3], pa.int32())
+ >>> pa_batch = pa.record_batch([pa_column], names=["col1"])
+ >>> pa_reader = pa.RecordBatchReader.from_batches(pa_batch.schema, [pa_batch])
+ >>> array_stream = na.array_stream(pa_reader)
+ >>> array_stream.get_schema()
+ struct<col1: int32>
+ >>> array_stream.get_next().length
+ 3
+ >>> array_stream.get_next() is None
+ True
+ """
+ cdef object _base
+ cdef ArrowArrayStream* _ptr
+ cdef object _cached_schema
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowArrayStream*>addr
+ self._cached_schema = None
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("array stream pointer is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("array stream is released")
+
+ def _get_schema(self, Schema schema):
+ self._assert_valid()
+ cdef int code = self._ptr.get_schema(self._ptr, schema._ptr)
+ cdef const char* message = NULL
+ if code != NANOARROW_OK:
+ message = self._ptr.get_last_error(self._ptr)
+ if message != NULL:
+ raise NanoarrowException(
+ "ArrowArrayStream::get_schema()",
+ code,
+ message.decode("UTF-8")
+ )
+ else:
+ Error.raise_error("ArrowArrayStream::get_schema()", code)
+
+ self._cached_schema = schema
+
+ def get_schema(self):
+ """Get the schema associated with this stream
+ """
+ # Update the cached copy of the schema as an independent object
+ self._cached_schema = Schema.allocate()
+ self._get_schema(self._cached_schema)
+
+ # Return an independent copy
Review Comment:
I think I wanted to make sure that calling `.get_schema()` on the Python object definitely called `.get_schema()` on the C object, but also that calling `.get_next()` didn't call `.get_schema()` every single time. I don't know that the first use-case is ever going to come up, or perhaps I'm overemphasizing the "test your C code from Python" use case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] jorisvandenbossche commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1229348924
##########
.github/workflows/python.yaml:
##########
@@ -57,3 +57,31 @@ jobs:
- name: Run tests
run: |
pytest python/tests -v -s
+
+ - name: Run doctests
+ if: success() && matrix.python-version == '3.10'
+ run: |
+ # Needs editable install to run --doctest-cython
+ pip install pytest-cython
+ pip install -e python
Review Comment:
Is this needed? (it was already installed in two steps above?)
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
Review Comment:
```suggestion
whose memory it is responsible for. When this object is deleted,
```
?
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("schema is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def metadata(self):
+ self._assert_valid()
+ if self._ptr.metadata != NULL:
+ return SchemaMetadata(self, <uintptr_t>self._ptr.metadata)
+ else:
+ return None
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return SchemaChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Schema(self, <uintptr_t>self._ptr.dictionary)
+ else:
+ return None
+
+ def view(self):
+ self._assert_valid()
+ schema_view = SchemaView()
+ cdef Error error = Error()
+ cdef int result = ArrowSchemaViewInit(&schema_view._schema_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowSchemaViewInit()", result)
+
+ return schema_view
+
+
+cdef class SchemaView:
+ """ArrowSchemaView wrapper
+
+ The ArrowSchemaView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowSchema (e.g., parameter
+ values for parameterized types). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.decimal128(10, 3))
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'decimal128'
+ >>> schema_view.decimal_bitwidth
+ 128
+ >>> schema_view.decimal_precision
+ 10
+ >>> schema_view.decimal_scale
+ 3
+ """
+ cdef ArrowSchemaView _schema_view
+
+ _fixed_size_types = (
+ NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY
+ )
+
+ _decimal_types = (
+ NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256
+ )
+
+ _time_unit_types = (
+ NANOARROW_TYPE_TIME32,
+ NANOARROW_TYPE_TIME64,
+ NANOARROW_TYPE_DURATION,
+ NANOARROW_TYPE_TIMESTAMP
+ )
+
+ _union_types = (
+ NANOARROW_TYPE_DENSE_UNION,
+ NANOARROW_TYPE_SPARSE_UNION
+ )
+
+ def __init__(self):
+ self._schema_view.type = NANOARROW_TYPE_UNINITIALIZED
+ self._schema_view.storage_type = NANOARROW_TYPE_UNINITIALIZED
+
+ @property
+ def type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def storage_type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.storage_type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def fixed_size(self):
+ if self._schema_view.type in SchemaView._fixed_size_types:
+ return self._schema_view.fixed_size
+
+ @property
+ def decimal_bitwidth(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_bitwidth
+
+ @property
+ def decimal_precision(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_precision
+
+ @property
+ def decimal_scale(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_scale
+
+ @property
+ def time_unit(self):
+ if self._schema_view.type in SchemaView._time_unit_types:
+ return ArrowTimeUnitString(self._schema_view.time_unit).decode('UTF-8')
+
+ @property
+ def timezone(self):
+ if self._schema_view.type == NANOARROW_TYPE_TIMESTAMP:
+ return self._schema_view.timezone.decode('UTF_8')
+
+ @property
+ def union_type_ids(self):
+ if self._schema_view.type in SchemaView._union_types:
+ type_ids_str = self._schema_view.union_type_ids.decode('UTF-8').split(',')
+ return (int(type_id) for type_id in type_ids_str)
+
+ @property
+ def extension_name(self):
+ if self._schema_view.extension_name.data != NULL:
+ name_bytes = PyBytes_FromStringAndSize(
+ self._schema_view.extension_name.data,
+ self._schema_view.extension_name.size_bytes
+ )
+ return name_bytes.decode('UTF-8')
+
+ @property
+ def extension_metadata(self):
+ if self._schema_view.extension_name.data != NULL:
+ return PyBytes_FromStringAndSize(
+ self._schema_view.extension_metadata.data,
+ self._schema_view.extension_metadata.size_bytes
+ )
+
+cdef class Array:
+ """ArrowArray wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArray as defined in the Arrow C Data interface, holding an
+ optional reference to a Schema that can be used to safely deserialize
+ the content. These objects are usually created using `nanoarrow.array()`.
+ This Python wrapper allows access to array fields but does not
+ automatically deserialize their content: use `.view()` to validate and
+ deserialize the content into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array = na.array(pa.array(["one", "two", "three", None]))
+ >>> array.length
+ 4
+ >>> array.null_count
+ 1
+ >>> array_view = array.view()
+ """
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef Schema _schema
+
+ @staticmethod
+ def allocate(Schema schema):
+ base = ArrayHolder()
+ return Array(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, Schema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("Array is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def length(self):
+ self._assert_valid()
+ return self._ptr.length
+
+ @property
+ def offset(self):
+ self._assert_valid()
+ return self._ptr.offset
+
+ @property
+ def null_count(self):
+ return self._ptr.null_count
+
+ @property
+ def buffers(self):
+ return tuple(<uintptr_t>self._ptr.buffers[i] for i in range(self._ptr.n_buffers))
+
+ @property
+ def children(self):
+ return ArrayChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Array(self, <uintptr_t>self._ptr.dictionary, self._schema.dictionary)
+ else:
+ return None
+
+ def view(self):
+ cdef ArrayViewHolder holder = ArrayViewHolder()
+
+ cdef Error error = Error()
+ cdef int result = ArrowArrayViewInitFromSchema(&holder.c_array_view,
+ self._schema._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewInitFromSchema()", result)
+
+ result = ArrowArrayViewSetArray(&holder.c_array_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewSetArray()", result)
+
+ return ArrayView(holder, holder._addr(), self)
+
+
+cdef class ArrayView:
+ """ArrowArrayView wrapper
+
+ The ArrowArrayView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowArray (e.g., buffer types,
+ lengths, and content). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array_view = na.array(pa.array(["one", "two", "three", None])).view()
+ >>> np.array(array_view.buffers[1])
+ array([ 0, 3, 6, 11, 11], dtype=int32)
+ >>> np.array(array_view.buffers[2])
+ array([b'o', b'n', b'e', b't', b'w', b'o', b't', b'h', b'r', b'e', b'e'],
+ dtype='|S1')
+ """
+ cdef object _base
+ cdef ArrowArrayView* _ptr
+ cdef Array _array
+
+ def __init__(self, object base, uintptr_t addr, Array array):
+ self._base = base,
+ self._ptr = <ArrowArrayView*>addr
+ self._array = array
+
+ @property
+ def children(self):
+ return ArrayViewChildren(self)
+
+ @property
+ def buffers(self):
+ return ArrayViewBuffers(self)
+
+ @property
+ def dictionary(self):
+ return ArrayView(self, <uintptr_t>self._ptr.dictionary, self._array.dictionary)
+
+ @property
+ def array(self):
+ return self._array
+
+ @property
+ def schema(self):
+ return self._array._schema
+
+cdef class SchemaChildren:
+ """Wrapper for a lazily-resolved list of Schema children
+ """
+ cdef Schema _parent
+ cdef int64_t _length
+
+ def __init__(self, Schema parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+
+ return Schema(self._parent, self._child_addr(k))
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowSchema** children = self._parent._ptr.children
+ cdef ArrowSchema* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class SchemaMetadata:
+ """Wrapper for a lazily-parsed Schema.metadata string
+ """
+
+ cdef object _parent
+ cdef const char* _metadata
+ cdef ArrowMetadataReader _reader
+
+ def __init__(self, object parent, uintptr_t ptr):
+ self._parent = parent
+ self._metadata = <const char*>ptr
+
+ def _init_reader(self):
+ cdef int result = ArrowMetadataReaderInit(&self._reader, self._metadata)
+ if result != NANOARROW_OK:
+ Error.raise_error("ArrowMetadataReaderInit()", result)
+
+ def __len__(self):
+ self._init_reader()
+ return self._reader.remaining_keys
+
+ def __iter__(self):
+ cdef ArrowStringView key
+ cdef ArrowStringView value
+ self._init_reader()
+ while self._reader.remaining_keys > 0:
+ ArrowMetadataReaderRead(&self._reader, &key, &value)
+ key_obj = PyBytes_FromStringAndSize(key.data, key.size_bytes).decode('UTF-8')
+ value_obj = PyBytes_FromStringAndSize(value.data, value.size_bytes)
+ yield key_obj, value_obj
+
+
+cdef class ArrayChildren:
+ """Wrapper for a lazily-resolved list of Array children
+ """
+ cdef Array _parent
+ cdef int64_t _length
+
+ def __init__(self, Array parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return Array(self._parent, self._child_addr(k), self._parent.schema.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArray** children = self._parent._ptr.children
+ cdef ArrowArray* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class ArrayViewChildren:
+ """Wrapper for a lazily-resolved list of ArrayView children
+ """
+ cdef ArrayView _parent
+ cdef int64_t _length
+
+ def __init__(self, ArrayView parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return ArrayView(self._parent, self._child_addr(k), self._parent._array.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArrayView** children = self._parent._ptr.children
+ cdef ArrowArrayView* child = children[i]
+ return <uintptr_t>child
+
+cdef class BufferView:
+ """Wrapper for Array buffer content
+
+ This object is a Python wrapper around a buffer held by an Array.
+ It implements the Python buffer protocol and is best accessed through
+ another implementor (e.g., `np.array(array_view.buffers[1])`)). Note that
+ this buffer content does not apply any parent offset.
+ """
+ cdef object _base
+ cdef ArrowBufferView* _ptr
+ cdef ArrowBufferType _buffer_type
+ cdef ArrowType _buffer_data_type
+ cdef Py_ssize_t _element_size_bits
+ cdef Py_ssize_t _shape
+ cdef Py_ssize_t _strides
+
+ def __init__(self, object base, uintptr_t addr,
+ ArrowBufferType buffer_type, ArrowType buffer_data_type,
+ Py_ssize_t element_size_bits):
+ self._base = base
+ self._ptr = <ArrowBufferView*>addr
+ self._buffer_type = buffer_type
+ self._buffer_data_type = buffer_data_type
+ self._element_size_bits = element_size_bits
+ self._strides = self._item_size()
+ self._shape = self._ptr.size_bytes // self._strides
+
+
+ cdef Py_ssize_t _item_size(self):
+ if self._buffer_data_type == NANOARROW_TYPE_BOOL:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_BINARY:
+ return 1
+ else:
+ return self._element_size_bits // 8
+
+ cdef const char* _get_format(self):
+ if self._buffer_data_type == NANOARROW_TYPE_INT8:
+ return "b"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT8:
+ return "B"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT16:
+ return "h"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT16:
+ return "H"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT32:
+ return "i"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT32:
+ return "I"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT64:
+ return "l"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT64:
+ return "L"
+ elif self._buffer_data_type == NANOARROW_TYPE_FLOAT:
+ return "f"
+ elif self._buffer_data_type == NANOARROW_TYPE_DOUBLE:
+ return "d"
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return "c"
+ else:
+ return "B"
+
+ def __getbuffer__(self, Py_buffer *buffer, int flags):
+ buffer.buf = self._ptr.data.data
+ buffer.format = self._get_format()
+ buffer.internal = NULL
+ buffer.itemsize = self._strides
+ buffer.len = self._ptr.size_bytes
+ buffer.ndim = 1
+ buffer.obj = self
+ buffer.readonly = 1
+ buffer.shape = &self._shape
+ buffer.strides = &self._strides
+ buffer.suboffsets = NULL
+
+ def __releasebuffer__(self, Py_buffer *buffer):
+ pass
+
+
+cdef class ArrayViewBuffers:
+ """A lazily-resolved list of ArrayView buffers
+ """
+ cdef ArrayView _array_view
+ cdef int64_t _length
+
+ def __init__(self, ArrayView array_view):
+ self._array_view = array_view
+ self._length = array_view._array._ptr.n_buffers
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ cdef ArrowBufferView* buffer_view = &(self._array_view._ptr.buffer_views[k])
+ if buffer_view.data.data == NULL:
+ return None
+
+ return BufferView(
+ self._array_view,
+ <uintptr_t>buffer_view,
+ self._array_view._ptr.layout.buffer_type[k],
+ self._array_view._ptr.layout.buffer_data_type[k],
+ self._array_view._ptr.layout.element_size_bits[k]
+ )
+
+
+cdef class ArrayStream:
+ """ArrowArrayStream wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArrayStream as defined in the Arrow C Stream interface.
+ These objects are usually created using `nanoarrow.array_stream()`.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> pa_column = pa.array([1, 2, 3], pa.int32())
+ >>> pa_batch = pa.record_batch([pa_column], names=["col1"])
+ >>> pa_reader = pa.RecordBatchReader.from_batches(pa_batch.schema, [pa_batch])
+ >>> array_stream = na.array_stream(pa_reader)
+ >>> array_stream.get_schema()
+ struct<col1: int32>
+ >>> array_stream.get_next().length
+ 3
+ >>> array_stream.get_next() is None
+ True
+ """
+ cdef object _base
+ cdef ArrowArrayStream* _ptr
+ cdef object _cached_schema
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowArrayStream*>addr
+ self._cached_schema = None
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("array stream pointer is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("array stream is released")
+
+ def _get_schema(self, Schema schema):
+ self._assert_valid()
+ cdef int code = self._ptr.get_schema(self._ptr, schema._ptr)
+ cdef const char* message = NULL
+ if code != NANOARROW_OK:
+ message = self._ptr.get_last_error(self._ptr)
+ if message != NULL:
+ raise NanoarrowException(
+ "ArrowArrayStream::get_schema()",
+ code,
+ message.decode("UTF-8")
+ )
+ else:
+ Error.raise_error("ArrowArrayStream::get_schema()", code)
+
+ self._cached_schema = schema
+
+ def get_schema(self):
+ """Get the schema associated with this stream
+ """
+ # Update the cached copy of the schema as an independent object
+ self._cached_schema = Schema.allocate()
+ self._get_schema(self._cached_schema)
+
+ # Return an independent copy
Review Comment:
This could also call the stream's get_schema once, and return here the cached version? (I would expect that get_schema return value can never change over time?)
Or is this mostly to mimic the behaviour of the ArrowArrayStream.get_schema?
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("schema is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def metadata(self):
+ self._assert_valid()
+ if self._ptr.metadata != NULL:
+ return SchemaMetadata(self, <uintptr_t>self._ptr.metadata)
+ else:
+ return None
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return SchemaChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Schema(self, <uintptr_t>self._ptr.dictionary)
+ else:
+ return None
+
+ def view(self):
+ self._assert_valid()
+ schema_view = SchemaView()
+ cdef Error error = Error()
+ cdef int result = ArrowSchemaViewInit(&schema_view._schema_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowSchemaViewInit()", result)
+
+ return schema_view
+
+
+cdef class SchemaView:
+ """ArrowSchemaView wrapper
+
+ The ArrowSchemaView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowSchema (e.g., parameter
+ values for parameterized types). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.decimal128(10, 3))
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'decimal128'
+ >>> schema_view.decimal_bitwidth
+ 128
+ >>> schema_view.decimal_precision
+ 10
+ >>> schema_view.decimal_scale
+ 3
+ """
+ cdef ArrowSchemaView _schema_view
+
+ _fixed_size_types = (
+ NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY
+ )
+
+ _decimal_types = (
+ NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256
+ )
+
+ _time_unit_types = (
+ NANOARROW_TYPE_TIME32,
+ NANOARROW_TYPE_TIME64,
+ NANOARROW_TYPE_DURATION,
+ NANOARROW_TYPE_TIMESTAMP
+ )
+
+ _union_types = (
+ NANOARROW_TYPE_DENSE_UNION,
+ NANOARROW_TYPE_SPARSE_UNION
+ )
+
+ def __init__(self):
+ self._schema_view.type = NANOARROW_TYPE_UNINITIALIZED
+ self._schema_view.storage_type = NANOARROW_TYPE_UNINITIALIZED
+
+ @property
+ def type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def storage_type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.storage_type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def fixed_size(self):
+ if self._schema_view.type in SchemaView._fixed_size_types:
+ return self._schema_view.fixed_size
+
+ @property
+ def decimal_bitwidth(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_bitwidth
+
+ @property
+ def decimal_precision(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_precision
+
+ @property
+ def decimal_scale(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_scale
+
+ @property
+ def time_unit(self):
+ if self._schema_view.type in SchemaView._time_unit_types:
+ return ArrowTimeUnitString(self._schema_view.time_unit).decode('UTF-8')
+
+ @property
+ def timezone(self):
+ if self._schema_view.type == NANOARROW_TYPE_TIMESTAMP:
+ return self._schema_view.timezone.decode('UTF_8')
+
+ @property
+ def union_type_ids(self):
+ if self._schema_view.type in SchemaView._union_types:
+ type_ids_str = self._schema_view.union_type_ids.decode('UTF-8').split(',')
+ return (int(type_id) for type_id in type_ids_str)
+
+ @property
+ def extension_name(self):
+ if self._schema_view.extension_name.data != NULL:
+ name_bytes = PyBytes_FromStringAndSize(
+ self._schema_view.extension_name.data,
+ self._schema_view.extension_name.size_bytes
+ )
+ return name_bytes.decode('UTF-8')
+
+ @property
+ def extension_metadata(self):
+ if self._schema_view.extension_name.data != NULL:
+ return PyBytes_FromStringAndSize(
+ self._schema_view.extension_metadata.data,
+ self._schema_view.extension_metadata.size_bytes
+ )
+
+cdef class Array:
+ """ArrowArray wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArray as defined in the Arrow C Data interface, holding an
+ optional reference to a Schema that can be used to safely deserialize
+ the content. These objects are usually created using `nanoarrow.array()`.
+ This Python wrapper allows access to array fields but does not
+ automatically deserialize their content: use `.view()` to validate and
+ deserialize the content into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array = na.array(pa.array(["one", "two", "three", None]))
+ >>> array.length
+ 4
+ >>> array.null_count
+ 1
+ >>> array_view = array.view()
+ """
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef Schema _schema
+
+ @staticmethod
+ def allocate(Schema schema):
+ base = ArrayHolder()
+ return Array(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, Schema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("Array is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def length(self):
+ self._assert_valid()
+ return self._ptr.length
+
+ @property
+ def offset(self):
+ self._assert_valid()
+ return self._ptr.offset
+
+ @property
+ def null_count(self):
+ return self._ptr.null_count
+
+ @property
+ def buffers(self):
+ return tuple(<uintptr_t>self._ptr.buffers[i] for i in range(self._ptr.n_buffers))
+
+ @property
+ def children(self):
+ return ArrayChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Array(self, <uintptr_t>self._ptr.dictionary, self._schema.dictionary)
+ else:
+ return None
+
+ def view(self):
+ cdef ArrayViewHolder holder = ArrayViewHolder()
+
+ cdef Error error = Error()
+ cdef int result = ArrowArrayViewInitFromSchema(&holder.c_array_view,
+ self._schema._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewInitFromSchema()", result)
+
+ result = ArrowArrayViewSetArray(&holder.c_array_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewSetArray()", result)
+
+ return ArrayView(holder, holder._addr(), self)
+
+
+cdef class ArrayView:
+ """ArrowArrayView wrapper
+
+ The ArrowArrayView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowArray (e.g., buffer types,
+ lengths, and content). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array_view = na.array(pa.array(["one", "two", "three", None])).view()
+ >>> np.array(array_view.buffers[1])
+ array([ 0, 3, 6, 11, 11], dtype=int32)
+ >>> np.array(array_view.buffers[2])
+ array([b'o', b'n', b'e', b't', b'w', b'o', b't', b'h', b'r', b'e', b'e'],
+ dtype='|S1')
+ """
+ cdef object _base
+ cdef ArrowArrayView* _ptr
+ cdef Array _array
+
+ def __init__(self, object base, uintptr_t addr, Array array):
+ self._base = base,
+ self._ptr = <ArrowArrayView*>addr
+ self._array = array
+
+ @property
+ def children(self):
+ return ArrayViewChildren(self)
+
+ @property
+ def buffers(self):
+ return ArrayViewBuffers(self)
+
+ @property
+ def dictionary(self):
+ return ArrayView(self, <uintptr_t>self._ptr.dictionary, self._array.dictionary)
+
+ @property
+ def array(self):
+ return self._array
+
+ @property
+ def schema(self):
+ return self._array._schema
+
+cdef class SchemaChildren:
+ """Wrapper for a lazily-resolved list of Schema children
+ """
+ cdef Schema _parent
+ cdef int64_t _length
+
+ def __init__(self, Schema parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+
+ return Schema(self._parent, self._child_addr(k))
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowSchema** children = self._parent._ptr.children
+ cdef ArrowSchema* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class SchemaMetadata:
+ """Wrapper for a lazily-parsed Schema.metadata string
+ """
+
+ cdef object _parent
+ cdef const char* _metadata
+ cdef ArrowMetadataReader _reader
+
+ def __init__(self, object parent, uintptr_t ptr):
+ self._parent = parent
+ self._metadata = <const char*>ptr
+
+ def _init_reader(self):
+ cdef int result = ArrowMetadataReaderInit(&self._reader, self._metadata)
+ if result != NANOARROW_OK:
+ Error.raise_error("ArrowMetadataReaderInit()", result)
+
+ def __len__(self):
+ self._init_reader()
+ return self._reader.remaining_keys
+
+ def __iter__(self):
+ cdef ArrowStringView key
+ cdef ArrowStringView value
+ self._init_reader()
+ while self._reader.remaining_keys > 0:
+ ArrowMetadataReaderRead(&self._reader, &key, &value)
+ key_obj = PyBytes_FromStringAndSize(key.data, key.size_bytes).decode('UTF-8')
+ value_obj = PyBytes_FromStringAndSize(value.data, value.size_bytes)
+ yield key_obj, value_obj
+
+
+cdef class ArrayChildren:
+ """Wrapper for a lazily-resolved list of Array children
+ """
+ cdef Array _parent
+ cdef int64_t _length
+
+ def __init__(self, Array parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return Array(self._parent, self._child_addr(k), self._parent.schema.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArray** children = self._parent._ptr.children
+ cdef ArrowArray* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class ArrayViewChildren:
Review Comment:
Is there a need to have both ArrayChildren and ArrayViewChildren? If you want to look at the children, you could also do `arr.children[0].view()` instead of `arr.view().children[0]`? Or does those two ways give you different capabilities?
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("schema is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def metadata(self):
+ self._assert_valid()
+ if self._ptr.metadata != NULL:
+ return SchemaMetadata(self, <uintptr_t>self._ptr.metadata)
+ else:
+ return None
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return SchemaChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Schema(self, <uintptr_t>self._ptr.dictionary)
+ else:
+ return None
+
+ def view(self):
+ self._assert_valid()
+ schema_view = SchemaView()
+ cdef Error error = Error()
+ cdef int result = ArrowSchemaViewInit(&schema_view._schema_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowSchemaViewInit()", result)
+
+ return schema_view
+
+
+cdef class SchemaView:
+ """ArrowSchemaView wrapper
+
+ The ArrowSchemaView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowSchema (e.g., parameter
+ values for parameterized types). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.decimal128(10, 3))
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'decimal128'
+ >>> schema_view.decimal_bitwidth
+ 128
+ >>> schema_view.decimal_precision
+ 10
+ >>> schema_view.decimal_scale
+ 3
+ """
+ cdef ArrowSchemaView _schema_view
+
+ _fixed_size_types = (
+ NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY
+ )
+
+ _decimal_types = (
+ NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256
+ )
+
+ _time_unit_types = (
+ NANOARROW_TYPE_TIME32,
+ NANOARROW_TYPE_TIME64,
+ NANOARROW_TYPE_DURATION,
+ NANOARROW_TYPE_TIMESTAMP
+ )
+
+ _union_types = (
+ NANOARROW_TYPE_DENSE_UNION,
+ NANOARROW_TYPE_SPARSE_UNION
+ )
+
+ def __init__(self):
+ self._schema_view.type = NANOARROW_TYPE_UNINITIALIZED
+ self._schema_view.storage_type = NANOARROW_TYPE_UNINITIALIZED
+
+ @property
+ def type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def storage_type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.storage_type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def fixed_size(self):
+ if self._schema_view.type in SchemaView._fixed_size_types:
+ return self._schema_view.fixed_size
+
+ @property
+ def decimal_bitwidth(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_bitwidth
+
+ @property
+ def decimal_precision(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_precision
+
+ @property
+ def decimal_scale(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_scale
+
+ @property
+ def time_unit(self):
+ if self._schema_view.type in SchemaView._time_unit_types:
+ return ArrowTimeUnitString(self._schema_view.time_unit).decode('UTF-8')
+
+ @property
+ def timezone(self):
+ if self._schema_view.type == NANOARROW_TYPE_TIMESTAMP:
+ return self._schema_view.timezone.decode('UTF_8')
+
+ @property
+ def union_type_ids(self):
+ if self._schema_view.type in SchemaView._union_types:
+ type_ids_str = self._schema_view.union_type_ids.decode('UTF-8').split(',')
+ return (int(type_id) for type_id in type_ids_str)
+
+ @property
+ def extension_name(self):
+ if self._schema_view.extension_name.data != NULL:
+ name_bytes = PyBytes_FromStringAndSize(
+ self._schema_view.extension_name.data,
+ self._schema_view.extension_name.size_bytes
+ )
+ return name_bytes.decode('UTF-8')
+
+ @property
+ def extension_metadata(self):
+ if self._schema_view.extension_name.data != NULL:
+ return PyBytes_FromStringAndSize(
+ self._schema_view.extension_metadata.data,
+ self._schema_view.extension_metadata.size_bytes
+ )
+
+cdef class Array:
+ """ArrowArray wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArray as defined in the Arrow C Data interface, holding an
+ optional reference to a Schema that can be used to safely deserialize
+ the content. These objects are usually created using `nanoarrow.array()`.
+ This Python wrapper allows access to array fields but does not
+ automatically deserialize their content: use `.view()` to validate and
+ deserialize the content into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array = na.array(pa.array(["one", "two", "three", None]))
+ >>> array.length
+ 4
+ >>> array.null_count
+ 1
+ >>> array_view = array.view()
+ """
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef Schema _schema
+
+ @staticmethod
+ def allocate(Schema schema):
+ base = ArrayHolder()
+ return Array(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, Schema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("Array is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def length(self):
+ self._assert_valid()
+ return self._ptr.length
+
+ @property
+ def offset(self):
+ self._assert_valid()
+ return self._ptr.offset
+
+ @property
+ def null_count(self):
+ return self._ptr.null_count
+
+ @property
+ def buffers(self):
+ return tuple(<uintptr_t>self._ptr.buffers[i] for i in range(self._ptr.n_buffers))
Review Comment:
Is this a useful attribute in practice?
(I would expect it to return essentially what `self.view().buffers` returns)
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
Review Comment:
Is the trailing comma here on purpose? (that will turn it into a tuple)
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("schema is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def metadata(self):
+ self._assert_valid()
+ if self._ptr.metadata != NULL:
+ return SchemaMetadata(self, <uintptr_t>self._ptr.metadata)
+ else:
+ return None
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return SchemaChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Schema(self, <uintptr_t>self._ptr.dictionary)
+ else:
+ return None
+
+ def view(self):
+ self._assert_valid()
+ schema_view = SchemaView()
+ cdef Error error = Error()
+ cdef int result = ArrowSchemaViewInit(&schema_view._schema_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowSchemaViewInit()", result)
+
+ return schema_view
+
+
+cdef class SchemaView:
+ """ArrowSchemaView wrapper
+
+ The ArrowSchemaView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowSchema (e.g., parameter
+ values for parameterized types). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.decimal128(10, 3))
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'decimal128'
+ >>> schema_view.decimal_bitwidth
+ 128
+ >>> schema_view.decimal_precision
+ 10
+ >>> schema_view.decimal_scale
+ 3
+ """
+ cdef ArrowSchemaView _schema_view
+
+ _fixed_size_types = (
+ NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY
+ )
+
+ _decimal_types = (
+ NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256
+ )
+
+ _time_unit_types = (
+ NANOARROW_TYPE_TIME32,
+ NANOARROW_TYPE_TIME64,
+ NANOARROW_TYPE_DURATION,
+ NANOARROW_TYPE_TIMESTAMP
+ )
+
+ _union_types = (
+ NANOARROW_TYPE_DENSE_UNION,
+ NANOARROW_TYPE_SPARSE_UNION
+ )
+
+ def __init__(self):
+ self._schema_view.type = NANOARROW_TYPE_UNINITIALIZED
+ self._schema_view.storage_type = NANOARROW_TYPE_UNINITIALIZED
+
+ @property
+ def type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def storage_type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.storage_type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def fixed_size(self):
+ if self._schema_view.type in SchemaView._fixed_size_types:
+ return self._schema_view.fixed_size
+
+ @property
+ def decimal_bitwidth(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_bitwidth
+
+ @property
+ def decimal_precision(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_precision
+
+ @property
+ def decimal_scale(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_scale
+
+ @property
+ def time_unit(self):
+ if self._schema_view.type in SchemaView._time_unit_types:
+ return ArrowTimeUnitString(self._schema_view.time_unit).decode('UTF-8')
+
+ @property
+ def timezone(self):
+ if self._schema_view.type == NANOARROW_TYPE_TIMESTAMP:
+ return self._schema_view.timezone.decode('UTF_8')
+
+ @property
+ def union_type_ids(self):
+ if self._schema_view.type in SchemaView._union_types:
+ type_ids_str = self._schema_view.union_type_ids.decode('UTF-8').split(',')
+ return (int(type_id) for type_id in type_ids_str)
+
+ @property
+ def extension_name(self):
+ if self._schema_view.extension_name.data != NULL:
+ name_bytes = PyBytes_FromStringAndSize(
+ self._schema_view.extension_name.data,
+ self._schema_view.extension_name.size_bytes
+ )
+ return name_bytes.decode('UTF-8')
+
+ @property
+ def extension_metadata(self):
+ if self._schema_view.extension_name.data != NULL:
+ return PyBytes_FromStringAndSize(
+ self._schema_view.extension_metadata.data,
+ self._schema_view.extension_metadata.size_bytes
+ )
+
+cdef class Array:
+ """ArrowArray wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArray as defined in the Arrow C Data interface, holding an
+ optional reference to a Schema that can be used to safely deserialize
+ the content. These objects are usually created using `nanoarrow.array()`.
+ This Python wrapper allows access to array fields but does not
+ automatically deserialize their content: use `.view()` to validate and
+ deserialize the content into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array = na.array(pa.array(["one", "two", "three", None]))
+ >>> array.length
+ 4
+ >>> array.null_count
+ 1
+ >>> array_view = array.view()
+ """
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef Schema _schema
+
+ @staticmethod
+ def allocate(Schema schema):
+ base = ArrayHolder()
+ return Array(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, Schema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("Array is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def length(self):
+ self._assert_valid()
+ return self._ptr.length
+
+ @property
+ def offset(self):
+ self._assert_valid()
+ return self._ptr.offset
+
+ @property
+ def null_count(self):
+ return self._ptr.null_count
+
+ @property
+ def buffers(self):
+ return tuple(<uintptr_t>self._ptr.buffers[i] for i in range(self._ptr.n_buffers))
+
+ @property
+ def children(self):
+ return ArrayChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Array(self, <uintptr_t>self._ptr.dictionary, self._schema.dictionary)
+ else:
+ return None
+
+ def view(self):
+ cdef ArrayViewHolder holder = ArrayViewHolder()
+
+ cdef Error error = Error()
+ cdef int result = ArrowArrayViewInitFromSchema(&holder.c_array_view,
+ self._schema._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewInitFromSchema()", result)
+
+ result = ArrowArrayViewSetArray(&holder.c_array_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewSetArray()", result)
+
+ return ArrayView(holder, holder._addr(), self)
+
+
+cdef class ArrayView:
+ """ArrowArrayView wrapper
+
+ The ArrowArrayView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowArray (e.g., buffer types,
+ lengths, and content). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array_view = na.array(pa.array(["one", "two", "three", None])).view()
+ >>> np.array(array_view.buffers[1])
+ array([ 0, 3, 6, 11, 11], dtype=int32)
+ >>> np.array(array_view.buffers[2])
+ array([b'o', b'n', b'e', b't', b'w', b'o', b't', b'h', b'r', b'e', b'e'],
+ dtype='|S1')
+ """
+ cdef object _base
+ cdef ArrowArrayView* _ptr
+ cdef Array _array
+
+ def __init__(self, object base, uintptr_t addr, Array array):
+ self._base = base,
+ self._ptr = <ArrowArrayView*>addr
+ self._array = array
+
+ @property
+ def children(self):
+ return ArrayViewChildren(self)
+
+ @property
+ def buffers(self):
+ return ArrayViewBuffers(self)
+
+ @property
+ def dictionary(self):
+ return ArrayView(self, <uintptr_t>self._ptr.dictionary, self._array.dictionary)
+
+ @property
+ def array(self):
+ return self._array
+
+ @property
+ def schema(self):
+ return self._array._schema
+
+cdef class SchemaChildren:
+ """Wrapper for a lazily-resolved list of Schema children
+ """
+ cdef Schema _parent
+ cdef int64_t _length
+
+ def __init__(self, Schema parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+
+ return Schema(self._parent, self._child_addr(k))
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowSchema** children = self._parent._ptr.children
+ cdef ArrowSchema* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class SchemaMetadata:
+ """Wrapper for a lazily-parsed Schema.metadata string
+ """
+
+ cdef object _parent
+ cdef const char* _metadata
+ cdef ArrowMetadataReader _reader
+
+ def __init__(self, object parent, uintptr_t ptr):
+ self._parent = parent
+ self._metadata = <const char*>ptr
+
+ def _init_reader(self):
+ cdef int result = ArrowMetadataReaderInit(&self._reader, self._metadata)
+ if result != NANOARROW_OK:
+ Error.raise_error("ArrowMetadataReaderInit()", result)
+
+ def __len__(self):
+ self._init_reader()
+ return self._reader.remaining_keys
+
+ def __iter__(self):
+ cdef ArrowStringView key
+ cdef ArrowStringView value
+ self._init_reader()
+ while self._reader.remaining_keys > 0:
+ ArrowMetadataReaderRead(&self._reader, &key, &value)
+ key_obj = PyBytes_FromStringAndSize(key.data, key.size_bytes).decode('UTF-8')
+ value_obj = PyBytes_FromStringAndSize(value.data, value.size_bytes)
+ yield key_obj, value_obj
+
+
+cdef class ArrayChildren:
+ """Wrapper for a lazily-resolved list of Array children
+ """
+ cdef Array _parent
+ cdef int64_t _length
+
+ def __init__(self, Array parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return Array(self._parent, self._child_addr(k), self._parent.schema.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArray** children = self._parent._ptr.children
+ cdef ArrowArray* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class ArrayViewChildren:
+ """Wrapper for a lazily-resolved list of ArrayView children
+ """
+ cdef ArrayView _parent
+ cdef int64_t _length
+
+ def __init__(self, ArrayView parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return ArrayView(self._parent, self._child_addr(k), self._parent._array.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArrayView** children = self._parent._ptr.children
+ cdef ArrowArrayView* child = children[i]
+ return <uintptr_t>child
+
+cdef class BufferView:
+ """Wrapper for Array buffer content
+
+ This object is a Python wrapper around a buffer held by an Array.
+ It implements the Python buffer protocol and is best accessed through
+ another implementor (e.g., `np.array(array_view.buffers[1])`)). Note that
+ this buffer content does not apply any parent offset.
+ """
+ cdef object _base
+ cdef ArrowBufferView* _ptr
+ cdef ArrowBufferType _buffer_type
+ cdef ArrowType _buffer_data_type
+ cdef Py_ssize_t _element_size_bits
+ cdef Py_ssize_t _shape
+ cdef Py_ssize_t _strides
+
+ def __init__(self, object base, uintptr_t addr,
+ ArrowBufferType buffer_type, ArrowType buffer_data_type,
+ Py_ssize_t element_size_bits):
+ self._base = base
+ self._ptr = <ArrowBufferView*>addr
+ self._buffer_type = buffer_type
+ self._buffer_data_type = buffer_data_type
+ self._element_size_bits = element_size_bits
+ self._strides = self._item_size()
+ self._shape = self._ptr.size_bytes // self._strides
+
+
+ cdef Py_ssize_t _item_size(self):
+ if self._buffer_data_type == NANOARROW_TYPE_BOOL:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_BINARY:
+ return 1
+ else:
+ return self._element_size_bits // 8
+
+ cdef const char* _get_format(self):
+ if self._buffer_data_type == NANOARROW_TYPE_INT8:
+ return "b"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT8:
+ return "B"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT16:
+ return "h"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT16:
+ return "H"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT32:
+ return "i"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT32:
+ return "I"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT64:
+ return "l"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT64:
+ return "L"
+ elif self._buffer_data_type == NANOARROW_TYPE_FLOAT:
+ return "f"
+ elif self._buffer_data_type == NANOARROW_TYPE_DOUBLE:
+ return "d"
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return "c"
+ else:
+ return "B"
+
+ def __getbuffer__(self, Py_buffer *buffer, int flags):
+ buffer.buf = self._ptr.data.data
+ buffer.format = self._get_format()
+ buffer.internal = NULL
+ buffer.itemsize = self._strides
+ buffer.len = self._ptr.size_bytes
+ buffer.ndim = 1
+ buffer.obj = self
+ buffer.readonly = 1
+ buffer.shape = &self._shape
+ buffer.strides = &self._strides
+ buffer.suboffsets = NULL
+
+ def __releasebuffer__(self, Py_buffer *buffer):
+ pass
+
+
+cdef class ArrayViewBuffers:
+ """A lazily-resolved list of ArrayView buffers
+ """
+ cdef ArrayView _array_view
+ cdef int64_t _length
+
+ def __init__(self, ArrayView array_view):
+ self._array_view = array_view
+ self._length = array_view._array._ptr.n_buffers
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ cdef ArrowBufferView* buffer_view = &(self._array_view._ptr.buffer_views[k])
+ if buffer_view.data.data == NULL:
+ return None
+
+ return BufferView(
+ self._array_view,
+ <uintptr_t>buffer_view,
+ self._array_view._ptr.layout.buffer_type[k],
+ self._array_view._ptr.layout.buffer_data_type[k],
+ self._array_view._ptr.layout.element_size_bits[k]
+ )
+
+
+cdef class ArrayStream:
+ """ArrowArrayStream wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArrayStream as defined in the Arrow C Stream interface.
+ These objects are usually created using `nanoarrow.array_stream()`.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> pa_column = pa.array([1, 2, 3], pa.int32())
+ >>> pa_batch = pa.record_batch([pa_column], names=["col1"])
+ >>> pa_reader = pa.RecordBatchReader.from_batches(pa_batch.schema, [pa_batch])
+ >>> array_stream = na.array_stream(pa_reader)
+ >>> array_stream.get_schema()
+ struct<col1: int32>
+ >>> array_stream.get_next().length
+ 3
+ >>> array_stream.get_next() is None
+ True
+ """
+ cdef object _base
+ cdef ArrowArrayStream* _ptr
+ cdef object _cached_schema
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowArrayStream*>addr
+ self._cached_schema = None
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("array stream pointer is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("array stream is released")
+
+ def _get_schema(self, Schema schema):
+ self._assert_valid()
+ cdef int code = self._ptr.get_schema(self._ptr, schema._ptr)
+ cdef const char* message = NULL
+ if code != NANOARROW_OK:
+ message = self._ptr.get_last_error(self._ptr)
+ if message != NULL:
+ raise NanoarrowException(
+ "ArrowArrayStream::get_schema()",
+ code,
+ message.decode("UTF-8")
+ )
+ else:
+ Error.raise_error("ArrowArrayStream::get_schema()", code)
Review Comment:
Could also use the `raise NanoarrowException(..)` pattern here, just like in the `if` branch above? (or is there a reason to have both patterns?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] jorisvandenbossche commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1229623958
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("schema is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def metadata(self):
+ self._assert_valid()
+ if self._ptr.metadata != NULL:
+ return SchemaMetadata(self, <uintptr_t>self._ptr.metadata)
+ else:
+ return None
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return SchemaChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Schema(self, <uintptr_t>self._ptr.dictionary)
+ else:
+ return None
+
+ def view(self):
+ self._assert_valid()
+ schema_view = SchemaView()
+ cdef Error error = Error()
+ cdef int result = ArrowSchemaViewInit(&schema_view._schema_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowSchemaViewInit()", result)
+
+ return schema_view
+
+
+cdef class SchemaView:
+ """ArrowSchemaView wrapper
+
+ The ArrowSchemaView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowSchema (e.g., parameter
+ values for parameterized types). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.decimal128(10, 3))
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'decimal128'
+ >>> schema_view.decimal_bitwidth
+ 128
+ >>> schema_view.decimal_precision
+ 10
+ >>> schema_view.decimal_scale
+ 3
+ """
+ cdef ArrowSchemaView _schema_view
+
+ _fixed_size_types = (
+ NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY
+ )
+
+ _decimal_types = (
+ NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256
+ )
+
+ _time_unit_types = (
+ NANOARROW_TYPE_TIME32,
+ NANOARROW_TYPE_TIME64,
+ NANOARROW_TYPE_DURATION,
+ NANOARROW_TYPE_TIMESTAMP
+ )
+
+ _union_types = (
+ NANOARROW_TYPE_DENSE_UNION,
+ NANOARROW_TYPE_SPARSE_UNION
+ )
+
+ def __init__(self):
+ self._schema_view.type = NANOARROW_TYPE_UNINITIALIZED
+ self._schema_view.storage_type = NANOARROW_TYPE_UNINITIALIZED
+
+ @property
+ def type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def storage_type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.storage_type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def fixed_size(self):
+ if self._schema_view.type in SchemaView._fixed_size_types:
+ return self._schema_view.fixed_size
+
+ @property
+ def decimal_bitwidth(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_bitwidth
+
+ @property
+ def decimal_precision(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_precision
+
+ @property
+ def decimal_scale(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_scale
+
+ @property
+ def time_unit(self):
+ if self._schema_view.type in SchemaView._time_unit_types:
+ return ArrowTimeUnitString(self._schema_view.time_unit).decode('UTF-8')
+
+ @property
+ def timezone(self):
+ if self._schema_view.type == NANOARROW_TYPE_TIMESTAMP:
+ return self._schema_view.timezone.decode('UTF_8')
+
+ @property
+ def union_type_ids(self):
+ if self._schema_view.type in SchemaView._union_types:
+ type_ids_str = self._schema_view.union_type_ids.decode('UTF-8').split(',')
+ return (int(type_id) for type_id in type_ids_str)
+
+ @property
+ def extension_name(self):
+ if self._schema_view.extension_name.data != NULL:
+ name_bytes = PyBytes_FromStringAndSize(
+ self._schema_view.extension_name.data,
+ self._schema_view.extension_name.size_bytes
+ )
+ return name_bytes.decode('UTF-8')
+
+ @property
+ def extension_metadata(self):
+ if self._schema_view.extension_name.data != NULL:
+ return PyBytes_FromStringAndSize(
+ self._schema_view.extension_metadata.data,
+ self._schema_view.extension_metadata.size_bytes
+ )
+
+cdef class Array:
+ """ArrowArray wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArray as defined in the Arrow C Data interface, holding an
+ optional reference to a Schema that can be used to safely deserialize
+ the content. These objects are usually created using `nanoarrow.array()`.
+ This Python wrapper allows access to array fields but does not
+ automatically deserialize their content: use `.view()` to validate and
+ deserialize the content into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array = na.array(pa.array(["one", "two", "three", None]))
+ >>> array.length
+ 4
+ >>> array.null_count
+ 1
+ >>> array_view = array.view()
+ """
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef Schema _schema
+
+ @staticmethod
+ def allocate(Schema schema):
+ base = ArrayHolder()
+ return Array(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, Schema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("Array is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def length(self):
+ self._assert_valid()
+ return self._ptr.length
+
+ @property
+ def offset(self):
+ self._assert_valid()
+ return self._ptr.offset
+
+ @property
+ def null_count(self):
+ return self._ptr.null_count
+
+ @property
+ def buffers(self):
+ return tuple(<uintptr_t>self._ptr.buffers[i] for i in range(self._ptr.n_buffers))
+
+ @property
+ def children(self):
+ return ArrayChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Array(self, <uintptr_t>self._ptr.dictionary, self._schema.dictionary)
+ else:
+ return None
+
+ def view(self):
+ cdef ArrayViewHolder holder = ArrayViewHolder()
+
+ cdef Error error = Error()
+ cdef int result = ArrowArrayViewInitFromSchema(&holder.c_array_view,
+ self._schema._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewInitFromSchema()", result)
+
+ result = ArrowArrayViewSetArray(&holder.c_array_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewSetArray()", result)
+
+ return ArrayView(holder, holder._addr(), self)
+
+
+cdef class ArrayView:
+ """ArrowArrayView wrapper
+
+ The ArrowArrayView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowArray (e.g., buffer types,
+ lengths, and content). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array_view = na.array(pa.array(["one", "two", "three", None])).view()
+ >>> np.array(array_view.buffers[1])
+ array([ 0, 3, 6, 11, 11], dtype=int32)
+ >>> np.array(array_view.buffers[2])
+ array([b'o', b'n', b'e', b't', b'w', b'o', b't', b'h', b'r', b'e', b'e'],
+ dtype='|S1')
+ """
+ cdef object _base
+ cdef ArrowArrayView* _ptr
+ cdef Array _array
+
+ def __init__(self, object base, uintptr_t addr, Array array):
+ self._base = base,
+ self._ptr = <ArrowArrayView*>addr
+ self._array = array
+
+ @property
+ def children(self):
+ return ArrayViewChildren(self)
+
+ @property
+ def buffers(self):
+ return ArrayViewBuffers(self)
+
+ @property
+ def dictionary(self):
+ return ArrayView(self, <uintptr_t>self._ptr.dictionary, self._array.dictionary)
+
+ @property
+ def array(self):
+ return self._array
+
+ @property
+ def schema(self):
+ return self._array._schema
+
+cdef class SchemaChildren:
+ """Wrapper for a lazily-resolved list of Schema children
+ """
+ cdef Schema _parent
+ cdef int64_t _length
+
+ def __init__(self, Schema parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+
+ return Schema(self._parent, self._child_addr(k))
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowSchema** children = self._parent._ptr.children
+ cdef ArrowSchema* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class SchemaMetadata:
+ """Wrapper for a lazily-parsed Schema.metadata string
+ """
+
+ cdef object _parent
+ cdef const char* _metadata
+ cdef ArrowMetadataReader _reader
+
+ def __init__(self, object parent, uintptr_t ptr):
+ self._parent = parent
+ self._metadata = <const char*>ptr
+
+ def _init_reader(self):
+ cdef int result = ArrowMetadataReaderInit(&self._reader, self._metadata)
+ if result != NANOARROW_OK:
+ Error.raise_error("ArrowMetadataReaderInit()", result)
+
+ def __len__(self):
+ self._init_reader()
+ return self._reader.remaining_keys
+
+ def __iter__(self):
+ cdef ArrowStringView key
+ cdef ArrowStringView value
+ self._init_reader()
+ while self._reader.remaining_keys > 0:
+ ArrowMetadataReaderRead(&self._reader, &key, &value)
+ key_obj = PyBytes_FromStringAndSize(key.data, key.size_bytes).decode('UTF-8')
+ value_obj = PyBytes_FromStringAndSize(value.data, value.size_bytes)
+ yield key_obj, value_obj
+
+
+cdef class ArrayChildren:
+ """Wrapper for a lazily-resolved list of Array children
+ """
+ cdef Array _parent
+ cdef int64_t _length
+
+ def __init__(self, Array parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return Array(self._parent, self._child_addr(k), self._parent.schema.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArray** children = self._parent._ptr.children
+ cdef ArrowArray* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class ArrayViewChildren:
+ """Wrapper for a lazily-resolved list of ArrayView children
+ """
+ cdef ArrayView _parent
+ cdef int64_t _length
+
+ def __init__(self, ArrayView parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return ArrayView(self._parent, self._child_addr(k), self._parent._array.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArrayView** children = self._parent._ptr.children
+ cdef ArrowArrayView* child = children[i]
+ return <uintptr_t>child
+
+cdef class BufferView:
+ """Wrapper for Array buffer content
+
+ This object is a Python wrapper around a buffer held by an Array.
+ It implements the Python buffer protocol and is best accessed through
+ another implementor (e.g., `np.array(array_view.buffers[1])`)). Note that
+ this buffer content does not apply any parent offset.
+ """
+ cdef object _base
+ cdef ArrowBufferView* _ptr
+ cdef ArrowBufferType _buffer_type
+ cdef ArrowType _buffer_data_type
+ cdef Py_ssize_t _element_size_bits
+ cdef Py_ssize_t _shape
+ cdef Py_ssize_t _strides
+
+ def __init__(self, object base, uintptr_t addr,
+ ArrowBufferType buffer_type, ArrowType buffer_data_type,
+ Py_ssize_t element_size_bits):
+ self._base = base
+ self._ptr = <ArrowBufferView*>addr
+ self._buffer_type = buffer_type
+ self._buffer_data_type = buffer_data_type
+ self._element_size_bits = element_size_bits
+ self._strides = self._item_size()
+ self._shape = self._ptr.size_bytes // self._strides
+
+
+ cdef Py_ssize_t _item_size(self):
+ if self._buffer_data_type == NANOARROW_TYPE_BOOL:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_BINARY:
+ return 1
+ else:
+ return self._element_size_bits // 8
+
+ cdef const char* _get_format(self):
+ if self._buffer_data_type == NANOARROW_TYPE_INT8:
+ return "b"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT8:
+ return "B"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT16:
+ return "h"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT16:
+ return "H"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT32:
+ return "i"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT32:
+ return "I"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT64:
+ return "l"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT64:
+ return "L"
+ elif self._buffer_data_type == NANOARROW_TYPE_FLOAT:
+ return "f"
+ elif self._buffer_data_type == NANOARROW_TYPE_DOUBLE:
+ return "d"
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return "c"
+ else:
+ return "B"
+
+ def __getbuffer__(self, Py_buffer *buffer, int flags):
+ buffer.buf = self._ptr.data.data
+ buffer.format = self._get_format()
+ buffer.internal = NULL
+ buffer.itemsize = self._strides
+ buffer.len = self._ptr.size_bytes
+ buffer.ndim = 1
+ buffer.obj = self
+ buffer.readonly = 1
+ buffer.shape = &self._shape
+ buffer.strides = &self._strides
+ buffer.suboffsets = NULL
+
+ def __releasebuffer__(self, Py_buffer *buffer):
+ pass
+
+
+cdef class ArrayViewBuffers:
+ """A lazily-resolved list of ArrayView buffers
+ """
+ cdef ArrayView _array_view
+ cdef int64_t _length
+
+ def __init__(self, ArrayView array_view):
+ self._array_view = array_view
+ self._length = array_view._array._ptr.n_buffers
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ cdef ArrowBufferView* buffer_view = &(self._array_view._ptr.buffer_views[k])
+ if buffer_view.data.data == NULL:
+ return None
+
+ return BufferView(
+ self._array_view,
+ <uintptr_t>buffer_view,
+ self._array_view._ptr.layout.buffer_type[k],
+ self._array_view._ptr.layout.buffer_data_type[k],
+ self._array_view._ptr.layout.element_size_bits[k]
+ )
+
+
+cdef class ArrayStream:
+ """ArrowArrayStream wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArrayStream as defined in the Arrow C Stream interface.
+ These objects are usually created using `nanoarrow.array_stream()`.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> pa_column = pa.array([1, 2, 3], pa.int32())
+ >>> pa_batch = pa.record_batch([pa_column], names=["col1"])
+ >>> pa_reader = pa.RecordBatchReader.from_batches(pa_batch.schema, [pa_batch])
+ >>> array_stream = na.array_stream(pa_reader)
+ >>> array_stream.get_schema()
+ struct<col1: int32>
+ >>> array_stream.get_next().length
+ 3
+ >>> array_stream.get_next() is None
+ True
+ """
+ cdef object _base
+ cdef ArrowArrayStream* _ptr
+ cdef object _cached_schema
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowArrayStream*>addr
+ self._cached_schema = None
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("array stream pointer is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("array stream is released")
+
+ def _get_schema(self, Schema schema):
+ self._assert_valid()
+ cdef int code = self._ptr.get_schema(self._ptr, schema._ptr)
+ cdef const char* message = NULL
+ if code != NANOARROW_OK:
+ message = self._ptr.get_last_error(self._ptr)
+ if message != NULL:
+ raise NanoarrowException(
+ "ArrowArrayStream::get_schema()",
+ code,
+ message.decode("UTF-8")
+ )
+ else:
+ Error.raise_error("ArrowArrayStream::get_schema()", code)
+
+ self._cached_schema = schema
+
+ def get_schema(self):
+ """Get the schema associated with this stream
+ """
+ # Update the cached copy of the schema as an independent object
+ self._cached_schema = Schema.allocate()
+ self._get_schema(self._cached_schema)
+
+ # Return an independent copy
+ out = Schema.allocate()
+ self._get_schema(out)
+ return out
+
+ def get_next(self):
+ """Get the next Array from this stream
+
+ Returns None when there are no more arrays in this stream.
+ """
+ self._assert_valid()
+
+ if self._cached_schema is None:
+ self._cached_schema = Schema.allocate()
+ self._get_schema(self._cached_schema)
+
+ cdef Array array = Array.allocate(self._cached_schema)
+ cdef int code = self._ptr.get_next(self._ptr, array._ptr)
+ cdef const char* message = NULL
+ if code != NANOARROW_OK:
+ message = self._ptr.get_last_error(self._ptr)
+ if message != NULL:
+ raise NanoarrowException(
+ "ArrowArrayStream::get_next()",
+ code,
+ message.decode("UTF-8")
+ )
+ else:
+ Error.raise_error("ArrowArrayStream::get_next()", code)
+
+ if not array.is_valid():
+ return None
Review Comment:
Another option could be to raise a StopIteration exception when the stream is finished.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#issuecomment-1592191344
> I am not sure what the benefit is of the splits between the base class and the View class (eg Array and ArrayView).
I'm happy to workshop this more in subsequent PRs...it's a tiny bit unfortunate that there a number of overlapping names; the `Array`, `Schema`, and `ArrayStream` in nanoarrow are 1:1 aligned with the C data interface names (in the C library and in the R package, too). The `Schema` and `Array` in particular are poor analogues for the function of schemas and arrays elsewhere (notably the pyarrow or Arrow C++ concept of an Array). Here, it's "just" the thinnest possible wrapper around an `ArrowArray*`. The nanoarrow C library's `ArrowArrayView` functions more like an `Array`, as does the `ArrayView` here.
Notably, the `ArrayView` doesn't care where its buffer contents came from. The most logical way to populate its buffers is from an `Array` but you could also do it from serialized IPC (via tha IPC extension) or Python buffers or whatever. You can also modify the `offset` and `length` since it's not pointing to any particular `ArrowArray` that might be pointed to from elsewhere. Earlier versions of nanoarrow and of this PR didn't make that distinction evident and I've tried to do that here (e.g., by dropping every reference from the ArrayView class into the Array class).
There was also a note about using the `src/` directory...I really did try to make it work with code coverage. The Cython coverage plugin is very finicky but given that most of this package is currently implemented in Cython, I think it's an important development tool to keep until I can figure out the proper `pytest --cov` invocation with the `src/` directory version (or maybe somebody else can figure it out more quickly than I can).
There are definitely unfinished points here, but I do want to merge this because it (and its limited test suite) get a lot of the boilerplate out of the way so that the discussion can focus on more substantial topics when it does arise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] jorisvandenbossche commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1231429193
##########
.github/workflows/python.yaml:
##########
@@ -57,3 +57,31 @@ jobs:
- name: Run tests
run: |
pytest python/tests -v -s
+
+ - name: Run doctests
+ if: success() && matrix.python-version == '3.10'
+ run: |
+ # Needs editable install to run --doctest-cython
+ pip install pytest-cython
+ pip install -e python
Review Comment:
FWIW, we use this package in Arrow CI with a normal install of pyarrow (non-editable), so I think it should certainly work. What you might have noticed locally is that a test didn't pass after updating it if you didn't re-install it? (that's the reason for locally using editable, so you don't have to reinstall after every tiny change. But on CI that doesn't matter)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] codecov-commenter commented on pull request #117: feat(python): Python schema wrapper
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#issuecomment-1461185447
# [Codecov](https://codecov.io/gh/apache/arrow-nanoarrow/pull/117?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#117](https://codecov.io/gh/apache/arrow-nanoarrow/pull/117?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c11fc4b) into [main](https://codecov.io/gh/apache/arrow-nanoarrow/commit/9128f21da165a717768e7a6be1ae25132f454c35?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9128f21) will **not change** coverage.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## main #117 +/- ##
=======================================
Coverage 93.28% 93.28%
=======================================
Files 7 7
Lines 1638 1638
Branches 53 53
=======================================
Hits 1528 1528
Misses 78 78
Partials 32 32
```
| [Impacted Files](https://codecov.io/gh/apache/arrow-nanoarrow/pull/117?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [src/nanoarrow/nanoarrow\_types.h](https://codecov.io/gh/apache/arrow-nanoarrow/pull/117?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c3JjL25hbm9hcnJvdy9uYW5vYXJyb3dfdHlwZXMuaA==) | `92.30% <ø> (ø)` | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1229854237
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("schema is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def metadata(self):
+ self._assert_valid()
+ if self._ptr.metadata != NULL:
+ return SchemaMetadata(self, <uintptr_t>self._ptr.metadata)
+ else:
+ return None
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return SchemaChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Schema(self, <uintptr_t>self._ptr.dictionary)
+ else:
+ return None
+
+ def view(self):
+ self._assert_valid()
+ schema_view = SchemaView()
+ cdef Error error = Error()
+ cdef int result = ArrowSchemaViewInit(&schema_view._schema_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowSchemaViewInit()", result)
+
+ return schema_view
+
+
+cdef class SchemaView:
+ """ArrowSchemaView wrapper
+
+ The ArrowSchemaView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowSchema (e.g., parameter
+ values for parameterized types). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.decimal128(10, 3))
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'decimal128'
+ >>> schema_view.decimal_bitwidth
+ 128
+ >>> schema_view.decimal_precision
+ 10
+ >>> schema_view.decimal_scale
+ 3
+ """
+ cdef ArrowSchemaView _schema_view
+
+ _fixed_size_types = (
+ NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY
+ )
+
+ _decimal_types = (
+ NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256
+ )
+
+ _time_unit_types = (
+ NANOARROW_TYPE_TIME32,
+ NANOARROW_TYPE_TIME64,
+ NANOARROW_TYPE_DURATION,
+ NANOARROW_TYPE_TIMESTAMP
+ )
+
+ _union_types = (
+ NANOARROW_TYPE_DENSE_UNION,
+ NANOARROW_TYPE_SPARSE_UNION
+ )
+
+ def __init__(self):
+ self._schema_view.type = NANOARROW_TYPE_UNINITIALIZED
+ self._schema_view.storage_type = NANOARROW_TYPE_UNINITIALIZED
+
+ @property
+ def type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def storage_type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.storage_type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def fixed_size(self):
+ if self._schema_view.type in SchemaView._fixed_size_types:
+ return self._schema_view.fixed_size
+
+ @property
+ def decimal_bitwidth(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_bitwidth
+
+ @property
+ def decimal_precision(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_precision
+
+ @property
+ def decimal_scale(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_scale
+
+ @property
+ def time_unit(self):
+ if self._schema_view.type in SchemaView._time_unit_types:
+ return ArrowTimeUnitString(self._schema_view.time_unit).decode('UTF-8')
+
+ @property
+ def timezone(self):
+ if self._schema_view.type == NANOARROW_TYPE_TIMESTAMP:
+ return self._schema_view.timezone.decode('UTF_8')
+
+ @property
+ def union_type_ids(self):
+ if self._schema_view.type in SchemaView._union_types:
+ type_ids_str = self._schema_view.union_type_ids.decode('UTF-8').split(',')
+ return (int(type_id) for type_id in type_ids_str)
+
+ @property
+ def extension_name(self):
+ if self._schema_view.extension_name.data != NULL:
+ name_bytes = PyBytes_FromStringAndSize(
+ self._schema_view.extension_name.data,
+ self._schema_view.extension_name.size_bytes
+ )
+ return name_bytes.decode('UTF-8')
+
+ @property
+ def extension_metadata(self):
+ if self._schema_view.extension_name.data != NULL:
+ return PyBytes_FromStringAndSize(
+ self._schema_view.extension_metadata.data,
+ self._schema_view.extension_metadata.size_bytes
+ )
+
+cdef class Array:
+ """ArrowArray wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArray as defined in the Arrow C Data interface, holding an
+ optional reference to a Schema that can be used to safely deserialize
+ the content. These objects are usually created using `nanoarrow.array()`.
+ This Python wrapper allows access to array fields but does not
+ automatically deserialize their content: use `.view()` to validate and
+ deserialize the content into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array = na.array(pa.array(["one", "two", "three", None]))
+ >>> array.length
+ 4
+ >>> array.null_count
+ 1
+ >>> array_view = array.view()
+ """
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef Schema _schema
+
+ @staticmethod
+ def allocate(Schema schema):
+ base = ArrayHolder()
+ return Array(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, Schema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("Array is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def length(self):
+ self._assert_valid()
+ return self._ptr.length
+
+ @property
+ def offset(self):
+ self._assert_valid()
+ return self._ptr.offset
+
+ @property
+ def null_count(self):
+ return self._ptr.null_count
+
+ @property
+ def buffers(self):
+ return tuple(<uintptr_t>self._ptr.buffers[i] for i in range(self._ptr.n_buffers))
+
+ @property
+ def children(self):
+ return ArrayChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Array(self, <uintptr_t>self._ptr.dictionary, self._schema.dictionary)
+ else:
+ return None
+
+ def view(self):
+ cdef ArrayViewHolder holder = ArrayViewHolder()
+
+ cdef Error error = Error()
+ cdef int result = ArrowArrayViewInitFromSchema(&holder.c_array_view,
+ self._schema._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewInitFromSchema()", result)
+
+ result = ArrowArrayViewSetArray(&holder.c_array_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewSetArray()", result)
+
+ return ArrayView(holder, holder._addr(), self)
+
+
+cdef class ArrayView:
+ """ArrowArrayView wrapper
+
+ The ArrowArrayView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowArray (e.g., buffer types,
+ lengths, and content). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array_view = na.array(pa.array(["one", "two", "three", None])).view()
+ >>> np.array(array_view.buffers[1])
+ array([ 0, 3, 6, 11, 11], dtype=int32)
+ >>> np.array(array_view.buffers[2])
+ array([b'o', b'n', b'e', b't', b'w', b'o', b't', b'h', b'r', b'e', b'e'],
+ dtype='|S1')
+ """
+ cdef object _base
+ cdef ArrowArrayView* _ptr
+ cdef Array _array
+
+ def __init__(self, object base, uintptr_t addr, Array array):
+ self._base = base,
+ self._ptr = <ArrowArrayView*>addr
+ self._array = array
+
+ @property
+ def children(self):
+ return ArrayViewChildren(self)
+
+ @property
+ def buffers(self):
+ return ArrayViewBuffers(self)
+
+ @property
+ def dictionary(self):
+ return ArrayView(self, <uintptr_t>self._ptr.dictionary, self._array.dictionary)
+
+ @property
+ def array(self):
+ return self._array
+
+ @property
+ def schema(self):
+ return self._array._schema
+
+cdef class SchemaChildren:
+ """Wrapper for a lazily-resolved list of Schema children
+ """
+ cdef Schema _parent
+ cdef int64_t _length
+
+ def __init__(self, Schema parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+
+ return Schema(self._parent, self._child_addr(k))
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowSchema** children = self._parent._ptr.children
+ cdef ArrowSchema* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class SchemaMetadata:
+ """Wrapper for a lazily-parsed Schema.metadata string
+ """
+
+ cdef object _parent
+ cdef const char* _metadata
+ cdef ArrowMetadataReader _reader
+
+ def __init__(self, object parent, uintptr_t ptr):
+ self._parent = parent
+ self._metadata = <const char*>ptr
+
+ def _init_reader(self):
+ cdef int result = ArrowMetadataReaderInit(&self._reader, self._metadata)
+ if result != NANOARROW_OK:
+ Error.raise_error("ArrowMetadataReaderInit()", result)
+
+ def __len__(self):
+ self._init_reader()
+ return self._reader.remaining_keys
+
+ def __iter__(self):
+ cdef ArrowStringView key
+ cdef ArrowStringView value
+ self._init_reader()
+ while self._reader.remaining_keys > 0:
+ ArrowMetadataReaderRead(&self._reader, &key, &value)
+ key_obj = PyBytes_FromStringAndSize(key.data, key.size_bytes).decode('UTF-8')
+ value_obj = PyBytes_FromStringAndSize(value.data, value.size_bytes)
+ yield key_obj, value_obj
+
+
+cdef class ArrayChildren:
+ """Wrapper for a lazily-resolved list of Array children
+ """
+ cdef Array _parent
+ cdef int64_t _length
+
+ def __init__(self, Array parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return Array(self._parent, self._child_addr(k), self._parent.schema.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArray** children = self._parent._ptr.children
+ cdef ArrowArray* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class ArrayViewChildren:
+ """Wrapper for a lazily-resolved list of ArrayView children
+ """
+ cdef ArrayView _parent
+ cdef int64_t _length
+
+ def __init__(self, ArrayView parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return ArrayView(self._parent, self._child_addr(k), self._parent._array.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArrayView** children = self._parent._ptr.children
+ cdef ArrowArrayView* child = children[i]
+ return <uintptr_t>child
+
+cdef class BufferView:
+ """Wrapper for Array buffer content
+
+ This object is a Python wrapper around a buffer held by an Array.
+ It implements the Python buffer protocol and is best accessed through
+ another implementor (e.g., `np.array(array_view.buffers[1])`)). Note that
+ this buffer content does not apply any parent offset.
+ """
+ cdef object _base
+ cdef ArrowBufferView* _ptr
+ cdef ArrowBufferType _buffer_type
+ cdef ArrowType _buffer_data_type
+ cdef Py_ssize_t _element_size_bits
+ cdef Py_ssize_t _shape
+ cdef Py_ssize_t _strides
+
+ def __init__(self, object base, uintptr_t addr,
+ ArrowBufferType buffer_type, ArrowType buffer_data_type,
+ Py_ssize_t element_size_bits):
+ self._base = base
+ self._ptr = <ArrowBufferView*>addr
+ self._buffer_type = buffer_type
+ self._buffer_data_type = buffer_data_type
+ self._element_size_bits = element_size_bits
+ self._strides = self._item_size()
+ self._shape = self._ptr.size_bytes // self._strides
+
+
+ cdef Py_ssize_t _item_size(self):
+ if self._buffer_data_type == NANOARROW_TYPE_BOOL:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_BINARY:
+ return 1
+ else:
+ return self._element_size_bits // 8
+
+ cdef const char* _get_format(self):
+ if self._buffer_data_type == NANOARROW_TYPE_INT8:
+ return "b"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT8:
+ return "B"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT16:
+ return "h"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT16:
+ return "H"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT32:
+ return "i"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT32:
+ return "I"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT64:
+ return "l"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT64:
+ return "L"
+ elif self._buffer_data_type == NANOARROW_TYPE_FLOAT:
+ return "f"
+ elif self._buffer_data_type == NANOARROW_TYPE_DOUBLE:
+ return "d"
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return "c"
+ else:
+ return "B"
+
+ def __getbuffer__(self, Py_buffer *buffer, int flags):
+ buffer.buf = self._ptr.data.data
+ buffer.format = self._get_format()
+ buffer.internal = NULL
+ buffer.itemsize = self._strides
+ buffer.len = self._ptr.size_bytes
+ buffer.ndim = 1
+ buffer.obj = self
+ buffer.readonly = 1
+ buffer.shape = &self._shape
+ buffer.strides = &self._strides
+ buffer.suboffsets = NULL
+
+ def __releasebuffer__(self, Py_buffer *buffer):
+ pass
+
+
+cdef class ArrayViewBuffers:
+ """A lazily-resolved list of ArrayView buffers
+ """
+ cdef ArrayView _array_view
+ cdef int64_t _length
+
+ def __init__(self, ArrayView array_view):
+ self._array_view = array_view
+ self._length = array_view._array._ptr.n_buffers
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ cdef ArrowBufferView* buffer_view = &(self._array_view._ptr.buffer_views[k])
+ if buffer_view.data.data == NULL:
+ return None
+
+ return BufferView(
+ self._array_view,
+ <uintptr_t>buffer_view,
+ self._array_view._ptr.layout.buffer_type[k],
+ self._array_view._ptr.layout.buffer_data_type[k],
+ self._array_view._ptr.layout.element_size_bits[k]
+ )
+
+
+cdef class ArrayStream:
+ """ArrowArrayStream wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArrayStream as defined in the Arrow C Stream interface.
+ These objects are usually created using `nanoarrow.array_stream()`.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> pa_column = pa.array([1, 2, 3], pa.int32())
+ >>> pa_batch = pa.record_batch([pa_column], names=["col1"])
+ >>> pa_reader = pa.RecordBatchReader.from_batches(pa_batch.schema, [pa_batch])
+ >>> array_stream = na.array_stream(pa_reader)
+ >>> array_stream.get_schema()
+ struct<col1: int32>
+ >>> array_stream.get_next().length
+ 3
+ >>> array_stream.get_next() is None
+ True
+ """
+ cdef object _base
+ cdef ArrowArrayStream* _ptr
+ cdef object _cached_schema
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowArrayStream*>addr
+ self._cached_schema = None
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("array stream pointer is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("array stream is released")
+
+ def _get_schema(self, Schema schema):
+ self._assert_valid()
+ cdef int code = self._ptr.get_schema(self._ptr, schema._ptr)
+ cdef const char* message = NULL
+ if code != NANOARROW_OK:
+ message = self._ptr.get_last_error(self._ptr)
+ if message != NULL:
+ raise NanoarrowException(
+ "ArrowArrayStream::get_schema()",
+ code,
+ message.decode("UTF-8")
+ )
+ else:
+ Error.raise_error("ArrowArrayStream::get_schema()", code)
Review Comment:
They should definitely be the same!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] jorisvandenbossche commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1231433082
##########
python/nanoarrow/_lib.pyx:
##########
@@ -526,23 +546,28 @@ cdef class ArrayView:
@property
def dictionary(self):
- return ArrayView(self, <uintptr_t>self._ptr.dictionary, self._array.dictionary)
-
- @property
- def array(self):
- return self._array
+ if self._ptr.dictionary == NULL:
+ return None
+ else:
+ return ArrayView(
+ self,
+ <uintptr_t>self._ptr.dictionary,
+ self._schema.dictionary,
+ None
Review Comment:
Why is this None in this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] jorisvandenbossche commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1229812366
##########
.github/workflows/python.yaml:
##########
@@ -57,3 +57,31 @@ jobs:
- name: Run tests
run: |
pytest python/tests -v -s
+
+ - name: Run doctests
+ if: success() && matrix.python-version == '3.10'
+ run: |
+ # Needs editable install to run --doctest-cython
+ pip install pytest-cython
+ pip install -e python
Review Comment:
Does the pytest-cython need the package to be installed in editable mode? (I would expect it to work with the installed package as well)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] jorisvandenbossche commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1229623958
##########
python/nanoarrow/_lib.pyx:
##########
@@ -0,0 +1,869 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+# cython: language_level = 3
+# cython: linetrace=True
+
+"""Low-level nanoarrow Python bindings
+
+This Cython extension provides low-level Python wrappers around the
+Arrow C Data and Arrow C Stream interface structs. In general, there
+is one wrapper per C struct and pointer validity is managed by keeping
+strong references to Python objects. These wrappers are intended to
+be literal and stay close to the structure definitions.
+"""
+
+from libc.stdint cimport uintptr_t, int64_t
+from cpython.mem cimport PyMem_Malloc, PyMem_Free
+from cpython.bytes cimport PyBytes_FromStringAndSize
+from cpython cimport Py_buffer
+from nanoarrow_c cimport *
+
+def c_version():
+ """Return the nanoarrow C library version string
+ """
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class SchemaHolder:
+ """Memory holder for an ArrowSchema
+
+ This class is responsible for the lifecycle of the ArrowSchema
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowSchema c_schema
+
+ def __cinit__(self):
+ self.c_schema.release = NULL
+
+ def __dealloc__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class ArrayHolder:
+ """Memory holder for an ArrowArray
+
+ This class is responsible for the lifecycle of the ArrowArray
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArray c_array
+
+ def __cinit__(self):
+ self.c_array.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class ArrayStreamHolder:
+ """Memory holder for an ArrowArrayStream
+
+ This class is responsible for the lifecycle of the ArrowArrayStream
+ whose memory it is responsible. When this object is deleted,
+ a non-NULL release callback is invoked.
+ """
+ cdef ArrowArrayStream c_array_stream
+
+ def __cinit__(self):
+ self.c_array_stream.release = NULL
+
+ def __dealloc__(self):
+ if self.c_array_stream.release != NULL:
+ self.c_array_stream.release(&self.c_array_stream)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_stream
+
+cdef class ArrayViewHolder:
+ """Memory holder for an ArrowArrayView
+
+ This class is responsible for the lifecycle of the ArrowArrayView
+ whose memory it is responsible. When this object is deleted,
+ ArrowArrayViewReset() is called on the contents.
+ """
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __dealloc__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+
+class NanoarrowException(RuntimeError):
+ """An error resulting from a call to the nanoarrow C library
+
+ Calls to the nanoarrow C library and/or the Arrow C Stream interface
+ callbacks return an errno error code and sometimes a message with extra
+ detail. This exception wraps a RuntimeError to format a suitable message
+ and store the components of the original error.
+ """
+
+ def __init__(self, what, code, message):
+ self.what = what
+ self.code = code
+ self.message = message
+
+ if self.message == "":
+ super().__init__(f"{self.what} failed ({self.code})")
+ else:
+ super().__init__(f"{self.what} failed ({self.code}): {self.message}")
+
+
+cdef class Error:
+ """Memory holder for an ArrowError
+
+ ArrowError is the C struct that is optionally passed to nanoarrow functions
+ when a detailed error message might be returned. This class holds a C
+ reference to the object and provides helpers for raising exceptions based
+ on the contained message.
+ """
+ cdef ArrowError c_error
+
+ def __cinit__(self):
+ self.c_error.message[0] = 0
+
+ def raise_message(self, what, code):
+ """Raise a NanoarrowException from this message
+ """
+ raise NanoarrowException(what, code, self.c_error.message.decode("UTF-8"))
+
+ @staticmethod
+ def raise_error(what, code):
+ """Raise a NanoarrowException without a message
+ """
+ raise NanoarrowException(what, code, "")
+
+
+cdef class Schema:
+ """ArrowSchema wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowSchema as defined in the Arrow C Data interface. These objects
+ are usually created using `nanoarrow.schema()`. This Python wrapper
+ allows access to schema fields but does not automatically deserialize
+ their content: use `.view()` to validate and deserialize the content
+ into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.int32())
+ >>> schema.is_valid()
+ True
+ >>> schema.format
+ 'i'
+ >>> schema.name
+ ''
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'int32'
+ """
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def allocate():
+ base = SchemaHolder()
+ return Schema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("schema is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def metadata(self):
+ self._assert_valid()
+ if self._ptr.metadata != NULL:
+ return SchemaMetadata(self, <uintptr_t>self._ptr.metadata)
+ else:
+ return None
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return SchemaChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Schema(self, <uintptr_t>self._ptr.dictionary)
+ else:
+ return None
+
+ def view(self):
+ self._assert_valid()
+ schema_view = SchemaView()
+ cdef Error error = Error()
+ cdef int result = ArrowSchemaViewInit(&schema_view._schema_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowSchemaViewInit()", result)
+
+ return schema_view
+
+
+cdef class SchemaView:
+ """ArrowSchemaView wrapper
+
+ The ArrowSchemaView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowSchema (e.g., parameter
+ values for parameterized types). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> schema = na.schema(pa.decimal128(10, 3))
+ >>> schema_view = schema.view()
+ >>> schema_view.type
+ 'decimal128'
+ >>> schema_view.decimal_bitwidth
+ 128
+ >>> schema_view.decimal_precision
+ 10
+ >>> schema_view.decimal_scale
+ 3
+ """
+ cdef ArrowSchemaView _schema_view
+
+ _fixed_size_types = (
+ NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY
+ )
+
+ _decimal_types = (
+ NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256
+ )
+
+ _time_unit_types = (
+ NANOARROW_TYPE_TIME32,
+ NANOARROW_TYPE_TIME64,
+ NANOARROW_TYPE_DURATION,
+ NANOARROW_TYPE_TIMESTAMP
+ )
+
+ _union_types = (
+ NANOARROW_TYPE_DENSE_UNION,
+ NANOARROW_TYPE_SPARSE_UNION
+ )
+
+ def __init__(self):
+ self._schema_view.type = NANOARROW_TYPE_UNINITIALIZED
+ self._schema_view.storage_type = NANOARROW_TYPE_UNINITIALIZED
+
+ @property
+ def type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def storage_type(self):
+ cdef const char* type_str = ArrowTypeString(self._schema_view.storage_type)
+ if type_str != NULL:
+ return type_str.decode('UTF-8')
+
+ @property
+ def fixed_size(self):
+ if self._schema_view.type in SchemaView._fixed_size_types:
+ return self._schema_view.fixed_size
+
+ @property
+ def decimal_bitwidth(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_bitwidth
+
+ @property
+ def decimal_precision(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_precision
+
+ @property
+ def decimal_scale(self):
+ if self._schema_view.type in SchemaView._decimal_types:
+ return self._schema_view.decimal_scale
+
+ @property
+ def time_unit(self):
+ if self._schema_view.type in SchemaView._time_unit_types:
+ return ArrowTimeUnitString(self._schema_view.time_unit).decode('UTF-8')
+
+ @property
+ def timezone(self):
+ if self._schema_view.type == NANOARROW_TYPE_TIMESTAMP:
+ return self._schema_view.timezone.decode('UTF_8')
+
+ @property
+ def union_type_ids(self):
+ if self._schema_view.type in SchemaView._union_types:
+ type_ids_str = self._schema_view.union_type_ids.decode('UTF-8').split(',')
+ return (int(type_id) for type_id in type_ids_str)
+
+ @property
+ def extension_name(self):
+ if self._schema_view.extension_name.data != NULL:
+ name_bytes = PyBytes_FromStringAndSize(
+ self._schema_view.extension_name.data,
+ self._schema_view.extension_name.size_bytes
+ )
+ return name_bytes.decode('UTF-8')
+
+ @property
+ def extension_metadata(self):
+ if self._schema_view.extension_name.data != NULL:
+ return PyBytes_FromStringAndSize(
+ self._schema_view.extension_metadata.data,
+ self._schema_view.extension_metadata.size_bytes
+ )
+
+cdef class Array:
+ """ArrowArray wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArray as defined in the Arrow C Data interface, holding an
+ optional reference to a Schema that can be used to safely deserialize
+ the content. These objects are usually created using `nanoarrow.array()`.
+ This Python wrapper allows access to array fields but does not
+ automatically deserialize their content: use `.view()` to validate and
+ deserialize the content into a more easily inspectable object.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array = na.array(pa.array(["one", "two", "three", None]))
+ >>> array.length
+ 4
+ >>> array.null_count
+ 1
+ >>> array_view = array.view()
+ """
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef Schema _schema
+
+ @staticmethod
+ def allocate(Schema schema):
+ base = ArrayHolder()
+ return Array(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, Schema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("Array is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def length(self):
+ self._assert_valid()
+ return self._ptr.length
+
+ @property
+ def offset(self):
+ self._assert_valid()
+ return self._ptr.offset
+
+ @property
+ def null_count(self):
+ return self._ptr.null_count
+
+ @property
+ def buffers(self):
+ return tuple(<uintptr_t>self._ptr.buffers[i] for i in range(self._ptr.n_buffers))
+
+ @property
+ def children(self):
+ return ArrayChildren(self)
+
+ @property
+ def dictionary(self):
+ self._assert_valid()
+ if self._ptr.dictionary != NULL:
+ return Array(self, <uintptr_t>self._ptr.dictionary, self._schema.dictionary)
+ else:
+ return None
+
+ def view(self):
+ cdef ArrayViewHolder holder = ArrayViewHolder()
+
+ cdef Error error = Error()
+ cdef int result = ArrowArrayViewInitFromSchema(&holder.c_array_view,
+ self._schema._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewInitFromSchema()", result)
+
+ result = ArrowArrayViewSetArray(&holder.c_array_view, self._ptr, &error.c_error)
+ if result != NANOARROW_OK:
+ error.raise_message("ArrowArrayViewSetArray()", result)
+
+ return ArrayView(holder, holder._addr(), self)
+
+
+cdef class ArrayView:
+ """ArrowArrayView wrapper
+
+ The ArrowArrayView is a nanoarrow C library structure that facilitates
+ access to the deserialized content of an ArrowArray (e.g., buffer types,
+ lengths, and content). This wrapper extends that facility to Python.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import numpy as np
+ >>> import nanoarrow as na
+ >>> array_view = na.array(pa.array(["one", "two", "three", None])).view()
+ >>> np.array(array_view.buffers[1])
+ array([ 0, 3, 6, 11, 11], dtype=int32)
+ >>> np.array(array_view.buffers[2])
+ array([b'o', b'n', b'e', b't', b'w', b'o', b't', b'h', b'r', b'e', b'e'],
+ dtype='|S1')
+ """
+ cdef object _base
+ cdef ArrowArrayView* _ptr
+ cdef Array _array
+
+ def __init__(self, object base, uintptr_t addr, Array array):
+ self._base = base,
+ self._ptr = <ArrowArrayView*>addr
+ self._array = array
+
+ @property
+ def children(self):
+ return ArrayViewChildren(self)
+
+ @property
+ def buffers(self):
+ return ArrayViewBuffers(self)
+
+ @property
+ def dictionary(self):
+ return ArrayView(self, <uintptr_t>self._ptr.dictionary, self._array.dictionary)
+
+ @property
+ def array(self):
+ return self._array
+
+ @property
+ def schema(self):
+ return self._array._schema
+
+cdef class SchemaChildren:
+ """Wrapper for a lazily-resolved list of Schema children
+ """
+ cdef Schema _parent
+ cdef int64_t _length
+
+ def __init__(self, Schema parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+
+ return Schema(self._parent, self._child_addr(k))
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowSchema** children = self._parent._ptr.children
+ cdef ArrowSchema* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class SchemaMetadata:
+ """Wrapper for a lazily-parsed Schema.metadata string
+ """
+
+ cdef object _parent
+ cdef const char* _metadata
+ cdef ArrowMetadataReader _reader
+
+ def __init__(self, object parent, uintptr_t ptr):
+ self._parent = parent
+ self._metadata = <const char*>ptr
+
+ def _init_reader(self):
+ cdef int result = ArrowMetadataReaderInit(&self._reader, self._metadata)
+ if result != NANOARROW_OK:
+ Error.raise_error("ArrowMetadataReaderInit()", result)
+
+ def __len__(self):
+ self._init_reader()
+ return self._reader.remaining_keys
+
+ def __iter__(self):
+ cdef ArrowStringView key
+ cdef ArrowStringView value
+ self._init_reader()
+ while self._reader.remaining_keys > 0:
+ ArrowMetadataReaderRead(&self._reader, &key, &value)
+ key_obj = PyBytes_FromStringAndSize(key.data, key.size_bytes).decode('UTF-8')
+ value_obj = PyBytes_FromStringAndSize(value.data, value.size_bytes)
+ yield key_obj, value_obj
+
+
+cdef class ArrayChildren:
+ """Wrapper for a lazily-resolved list of Array children
+ """
+ cdef Array _parent
+ cdef int64_t _length
+
+ def __init__(self, Array parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return Array(self._parent, self._child_addr(k), self._parent.schema.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArray** children = self._parent._ptr.children
+ cdef ArrowArray* child = children[i]
+ return <uintptr_t>child
+
+
+cdef class ArrayViewChildren:
+ """Wrapper for a lazily-resolved list of ArrayView children
+ """
+ cdef ArrayView _parent
+ cdef int64_t _length
+
+ def __init__(self, ArrayView parent):
+ self._parent = parent
+ self._length = parent._ptr.n_children
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ return ArrayView(self._parent, self._child_addr(k), self._parent._array.children[k])
+
+ cdef _child_addr(self, int64_t i):
+ cdef ArrowArrayView** children = self._parent._ptr.children
+ cdef ArrowArrayView* child = children[i]
+ return <uintptr_t>child
+
+cdef class BufferView:
+ """Wrapper for Array buffer content
+
+ This object is a Python wrapper around a buffer held by an Array.
+ It implements the Python buffer protocol and is best accessed through
+ another implementor (e.g., `np.array(array_view.buffers[1])`)). Note that
+ this buffer content does not apply any parent offset.
+ """
+ cdef object _base
+ cdef ArrowBufferView* _ptr
+ cdef ArrowBufferType _buffer_type
+ cdef ArrowType _buffer_data_type
+ cdef Py_ssize_t _element_size_bits
+ cdef Py_ssize_t _shape
+ cdef Py_ssize_t _strides
+
+ def __init__(self, object base, uintptr_t addr,
+ ArrowBufferType buffer_type, ArrowType buffer_data_type,
+ Py_ssize_t element_size_bits):
+ self._base = base
+ self._ptr = <ArrowBufferView*>addr
+ self._buffer_type = buffer_type
+ self._buffer_data_type = buffer_data_type
+ self._element_size_bits = element_size_bits
+ self._strides = self._item_size()
+ self._shape = self._ptr.size_bytes // self._strides
+
+
+ cdef Py_ssize_t _item_size(self):
+ if self._buffer_data_type == NANOARROW_TYPE_BOOL:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return 1
+ elif self._buffer_data_type == NANOARROW_TYPE_BINARY:
+ return 1
+ else:
+ return self._element_size_bits // 8
+
+ cdef const char* _get_format(self):
+ if self._buffer_data_type == NANOARROW_TYPE_INT8:
+ return "b"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT8:
+ return "B"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT16:
+ return "h"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT16:
+ return "H"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT32:
+ return "i"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT32:
+ return "I"
+ elif self._buffer_data_type == NANOARROW_TYPE_INT64:
+ return "l"
+ elif self._buffer_data_type == NANOARROW_TYPE_UINT64:
+ return "L"
+ elif self._buffer_data_type == NANOARROW_TYPE_FLOAT:
+ return "f"
+ elif self._buffer_data_type == NANOARROW_TYPE_DOUBLE:
+ return "d"
+ elif self._buffer_data_type == NANOARROW_TYPE_STRING:
+ return "c"
+ else:
+ return "B"
+
+ def __getbuffer__(self, Py_buffer *buffer, int flags):
+ buffer.buf = self._ptr.data.data
+ buffer.format = self._get_format()
+ buffer.internal = NULL
+ buffer.itemsize = self._strides
+ buffer.len = self._ptr.size_bytes
+ buffer.ndim = 1
+ buffer.obj = self
+ buffer.readonly = 1
+ buffer.shape = &self._shape
+ buffer.strides = &self._strides
+ buffer.suboffsets = NULL
+
+ def __releasebuffer__(self, Py_buffer *buffer):
+ pass
+
+
+cdef class ArrayViewBuffers:
+ """A lazily-resolved list of ArrayView buffers
+ """
+ cdef ArrayView _array_view
+ cdef int64_t _length
+
+ def __init__(self, ArrayView array_view):
+ self._array_view = array_view
+ self._length = array_view._array._ptr.n_buffers
+
+ def __len__(self):
+ return self._length
+
+ def __getitem__(self, k):
+ k = int(k)
+ if k < 0 or k >= self._length:
+ raise IndexError(f"{k} out of range [0, {self._length})")
+ cdef ArrowBufferView* buffer_view = &(self._array_view._ptr.buffer_views[k])
+ if buffer_view.data.data == NULL:
+ return None
+
+ return BufferView(
+ self._array_view,
+ <uintptr_t>buffer_view,
+ self._array_view._ptr.layout.buffer_type[k],
+ self._array_view._ptr.layout.buffer_data_type[k],
+ self._array_view._ptr.layout.element_size_bits[k]
+ )
+
+
+cdef class ArrayStream:
+ """ArrowArrayStream wrapper
+
+ This class provides a user-facing interface to access the fields of
+ an ArrowArrayStream as defined in the Arrow C Stream interface.
+ These objects are usually created using `nanoarrow.array_stream()`.
+
+ Examples
+ --------
+
+ >>> import pyarrow as pa
+ >>> import nanoarrow as na
+ >>> pa_column = pa.array([1, 2, 3], pa.int32())
+ >>> pa_batch = pa.record_batch([pa_column], names=["col1"])
+ >>> pa_reader = pa.RecordBatchReader.from_batches(pa_batch.schema, [pa_batch])
+ >>> array_stream = na.array_stream(pa_reader)
+ >>> array_stream.get_schema()
+ struct<col1: int32>
+ >>> array_stream.get_next().length
+ 3
+ >>> array_stream.get_next() is None
+ True
+ """
+ cdef object _base
+ cdef ArrowArrayStream* _ptr
+ cdef object _cached_schema
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowArrayStream*>addr
+ self._cached_schema = None
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr != NULL and self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr == NULL:
+ raise RuntimeError("array stream pointer is NULL")
+ if self._ptr.release == NULL:
+ raise RuntimeError("array stream is released")
+
+ def _get_schema(self, Schema schema):
+ self._assert_valid()
+ cdef int code = self._ptr.get_schema(self._ptr, schema._ptr)
+ cdef const char* message = NULL
+ if code != NANOARROW_OK:
+ message = self._ptr.get_last_error(self._ptr)
+ if message != NULL:
+ raise NanoarrowException(
+ "ArrowArrayStream::get_schema()",
+ code,
+ message.decode("UTF-8")
+ )
+ else:
+ Error.raise_error("ArrowArrayStream::get_schema()", code)
+
+ self._cached_schema = schema
+
+ def get_schema(self):
+ """Get the schema associated with this stream
+ """
+ # Update the cached copy of the schema as an independent object
+ self._cached_schema = Schema.allocate()
+ self._get_schema(self._cached_schema)
+
+ # Return an independent copy
+ out = Schema.allocate()
+ self._get_schema(out)
+ return out
+
+ def get_next(self):
+ """Get the next Array from this stream
+
+ Returns None when there are no more arrays in this stream.
+ """
+ self._assert_valid()
+
+ if self._cached_schema is None:
+ self._cached_schema = Schema.allocate()
+ self._get_schema(self._cached_schema)
+
+ cdef Array array = Array.allocate(self._cached_schema)
+ cdef int code = self._ptr.get_next(self._ptr, array._ptr)
+ cdef const char* message = NULL
+ if code != NANOARROW_OK:
+ message = self._ptr.get_last_error(self._ptr)
+ if message != NULL:
+ raise NanoarrowException(
+ "ArrowArrayStream::get_next()",
+ code,
+ message.decode("UTF-8")
+ )
+ else:
+ Error.raise_error("ArrowArrayStream::get_next()", code)
+
+ if not array.is_valid():
+ return None
Review Comment:
Another option could be to raise a StopIteration exception when the stream is finished (not sure what is best; this is what you would do in a `__next__` dunder method). I think in pyarrow for example RecordBatchReader.read_next_batch does that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1132715658
##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -84,3 +85,329 @@ def as_numpy_array(arr):
# TODO set base
return result
+
+
+def version():
+ return ArrowNanoarrowVersion().decode("UTF-8")
+
+cdef class CSchemaHolder:
+ cdef ArrowSchema c_schema
+
+ def __init__(self):
+ self.c_schema.release = NULL
+
+ def __del__(self):
+ if self.c_schema.release != NULL:
+ self.c_schema.release(&self.c_schema)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_schema
+
+cdef class CArrayHolder:
+ cdef ArrowArray c_array
+
+ def __init__(self):
+ self.c_array.release = NULL
+
+ def __del__(self):
+ if self.c_array.release != NULL:
+ self.c_array.release(&self.c_array)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array
+
+cdef class CArrayViewHolder:
+ cdef ArrowArrayView c_array_view
+
+ def __init__(self):
+ ArrowArrayViewInitFromType(&self.c_array_view, NANOARROW_TYPE_UNINITIALIZED)
+
+ def __del__(self):
+ ArrowArrayViewReset(&self.c_array_view)
+
+ def _addr(self):
+ return <uintptr_t>&self.c_array_view
+
+cdef class CSchema:
+ cdef object _base
+ cdef ArrowSchema* _ptr
+
+ @staticmethod
+ def Empty():
+ base = CSchemaHolder()
+ return CSchema(base, base._addr())
+
+ def __init__(self, object base, uintptr_t addr):
+ self._base = base,
+ self._ptr = <ArrowSchema*>addr
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr.release == NULL:
+ raise RuntimeError("schema is released")
+
+ def __repr__(self):
+ cdef int64_t n_chars = ArrowSchemaToString(self._ptr, NULL, 0, True)
+ cdef char* out = <char*>PyMem_Malloc(n_chars + 1)
+ if not out:
+ raise MemoryError()
+
+ ArrowSchemaToString(self._ptr, out, n_chars + 1, True)
+ out_str = out.decode("UTF-8")
+ PyMem_Free(out)
+
+ return out_str
+
+ @property
+ def format(self):
+ self._assert_valid()
+ if self._ptr.format != NULL:
+ return self._ptr.format.decode("UTF-8")
+
+ @property
+ def name(self):
+ self._assert_valid()
+ if self._ptr.name != NULL:
+ return self._ptr.name.decode("UTF-8")
+ else:
+ return None
+
+ @property
+ def flags(self):
+ return self._ptr.flags
+
+ @property
+ def children(self):
+ self._assert_valid()
+ return CSchemaChildren(self)
+
+ def parse(self):
+ self._assert_valid()
+
+ cdef ArrowError error
+ cdef ArrowSchemaView schema_view
+
+ cdef int result = ArrowSchemaViewInit(&schema_view, self._ptr, &error)
+ if result != NANOARROW_OK:
+ raise ValueError(ArrowErrorMessage(&error))
+
+ out = {
+ 'name': self._ptr.name.decode('UTF-8') if self._ptr.name else None,
+ 'type': ArrowTypeString(schema_view.type).decode('UTF-8'),
+ 'storage_type': ArrowTypeString(schema_view.storage_type).decode('UTF-8')
+ }
+
+ if schema_view.storage_type in (NANOARROW_TYPE_FIXED_SIZE_LIST,
+ NANOARROW_TYPE_FIXED_SIZE_BINARY):
+ out['fixed_size'] = schema_view.fixed_size
+
+ if schema_view.storage_type in (NANOARROW_TYPE_DECIMAL128,
+ NANOARROW_TYPE_DECIMAL256):
+ out['decimal_bitwidth'] = schema_view.decimal_bitwidth
+ out['decimal_precision'] = schema_view.decimal_precision
+ out['decimal_scale'] = schema_view.decimal_scale
+
+ return out
+
+cdef class CArray:
+ cdef object _base
+ cdef ArrowArray* _ptr
+ cdef CSchema _schema
+
+ @staticmethod
+ def Empty(CSchema schema):
+ base = CArrayHolder()
+ return CArray(base, base._addr(), schema)
+
+ def __init__(self, object base, uintptr_t addr, CSchema schema):
+ self._base = base,
+ self._ptr = <ArrowArray*>addr
+ self._schema = schema
+
+ def _addr(self):
+ return <uintptr_t>self._ptr
+
+ def is_valid(self):
+ return self._ptr.release != NULL
+
+ def _assert_valid(self):
+ if self._ptr.release == NULL:
+ raise RuntimeError("Array is released")
+
+ @property
+ def schema(self):
+ return self._schema
+
+ @property
+ def children(self):
+ return CArrayChildren(self)
+
+ def validate(self):
+ cdef CArrayViewHolder holder = CArrayViewHolder()
+
+ cdef ArrowError error
+ cdef int result = ArrowArrayViewInitFromSchema(&holder.c_array_view,
+ self._schema._ptr, &error)
+ if result != NANOARROW_OK:
+ raise ValueError(ArrowErrorMessage(&error))
+
+ result = ArrowArrayViewSetArray(&holder.c_array_view, self._ptr, &error)
+ if result != NANOARROW_OK:
+ raise ValueError(ArrowErrorMessage(&error))
+
+ return CArrayView(holder, holder._addr(), self)
Review Comment:
This was a bit of a rabbit hole but you're totally right. Now you can do `array.view()` and `schema.view()`, which is a bit more intuitive since you get a `SchemaView` and `ArrayView` back. I also completed the element accessors on the array and schema classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] paleolimbot commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1132717996
##########
python/tests/test_nanoarrow.py:
##########
@@ -1,27 +1,155 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
import numpy as np
import pyarrow as pa
-import nanoarrow
+import nanoarrow as na
import pytest
+def test_version():
+ assert(na.version() == "0.2.0-SNAPSHOT")
def test_as_numpy_array():
-
+
arr = pa.array([1, 2, 3])
- result = nanoarrow.as_numpy_array(arr)
+ result = na.as_numpy_array(arr)
expected = arr.to_numpy()
np.testing.assert_array_equal(result, expected)
arr = pa.array([1, 2, 3], pa.uint8())
- result = nanoarrow.as_numpy_array(arr)
+ result = na.as_numpy_array(arr)
expected = arr.to_numpy()
np.testing.assert_array_equal(result, expected)
arr = pa.array([1, 2, None])
with pytest.raises(ValueError, match="Cannot convert array with nulls"):
- nanoarrow.as_numpy_array(arr)
+ na.as_numpy_array(arr)
arr = pa.array([[1], [2, 3]])
with pytest.raises(TypeError, match="Cannot convert a non-primitive array"):
- nanoarrow.as_numpy_array(arr)
+ na.as_numpy_array(arr)
+
+def test_schema_basic():
+ # Blank invalid schema
+ schema = na.CSchema.Empty()
+ assert(schema.is_valid() is False)
Review Comment:
I updated all the asserts...I might punt on setting up pre-commit-config (although it definitely should be done for more than just that!). I quite like how ADBC did it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-nanoarrow] jorisvandenbossche commented on a diff in pull request #117: feat(python): Python schema, array, and array view skeleton
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #117:
URL: https://github.com/apache/arrow-nanoarrow/pull/117#discussion_r1132414520
##########
python/.gitignore:
##########
@@ -18,7 +18,8 @@
src/nanoarrow/nanoarrow.c
src/nanoarrow/nanoarrow.h
-src/nanoarrow/*.cpp
+src/nanoarrow/nanoarrow_c.pxd
Review Comment:
Ah, that's also a good idea, if we are OK with having a python specific file in that directory. Then the automated bot could generate it together with the .c and .h file?
For local development, doesn't have that the same potential drawback of not using the correct one? (if you edited the c sources) Although the same is true currently for the .c and .h files, so I suppose the idea is to still keep the cmake build step to ensure you use the latest for local development?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org