You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2024/05/04 03:14:17 UTC
[PR] feat(python): Add visitor pattern + builders for column sequences [arrow-nanoarrow]
paleolimbot opened a new pull request, #454:
URL: https://github.com/apache/arrow-nanoarrow/pull/454
Assembling columns from chunked things is rather difficult to do and is a valid thing that somebody might want to assemble from Arrow data. This PR adds some helpers to assemble concatenated buffers that somebody could pass to numpy (or some other library).
Still working on the dispatch of supported types and null handling...
```python
import nanoarrow as na
import pyarrow as pa
import pandas as pd
from nanoarrow import visitor
url = "https://github.com/apache/arrow-experiments/raw/main/data/arrow-commits/arrow-commits.arrows"
array = na.ArrayStream.from_url(url).read_all()
{k: type(v) for k, v, in visitor.to_columns(array).items()}
#> {'commit': list, 'time': list, 'files': list, 'merge': list, 'message': list}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Add visitor pattern + builders for column sequences [arrow-nanoarrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #454:
URL: https://github.com/apache/arrow-nanoarrow/pull/454#discussion_r1599993840
##########
python/src/nanoarrow/visitor.py:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, List, Sequence, Tuple, Union
+
+from nanoarrow._lib import CArrayView
+from nanoarrow.c_array_stream import c_array_stream
+from nanoarrow.iterator import ArrayViewBaseIterator, PyIterator
+from nanoarrow.schema import Type
+
+
+def to_pylist(obj, schema=None) -> List:
+ """Convert ``obj`` to a ``list()` of Python objects
+
+ Computes an identical value to ``list(iterator.iter_py())`` but is several
+ times faster.
Review Comment:
What is actually the reason that it is multiple times faster? As I would expect it uses the same iteration code under the hood?
##########
python/src/nanoarrow/array.py:
##########
@@ -344,6 +345,42 @@ def iter_chunk_views(self) -> Iterable[CArrayView]:
"""
return iter_array_views(self)
+ def to_pylist(self) -> List:
+ """Convert this Array to a ``list()` of Python objects
+
+ Computes an identical value to list(:meth:`iter_py`) but can be several
+ times faster.
+
+ Examples
+ --------
+
+ >>> import nanoarrow as na
+ >>> array = na.Array([1, 2, 3], na.int32())
+ >>> array.to_pylist()
+ [1, 2, 3]
+ """
+ return to_pylist(self)
+
+ def to_columns(self) -> Tuple[str, Sequence]:
+ """Convert this Array to a ``list()` of sequences
+
+ Converts a stream of struct arrays into its column-wise representation
+ such that each column is either a contiguous buffer or a ``list()``.
Review Comment:
I don't fully understand the "or" here. When would it be a contiguous buffer and when a list? (my interpretation is that it is always a python list?)
And would it make more sense to not yet convert to a list? (the user can always call `to_pylist` on the resulting column values if they want a list) Because right now you cannot use this method if you don't want the python list for the actual values, but still want a list of columns?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Add visitor pattern + builders for column sequences [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #454:
URL: https://github.com/apache/arrow-nanoarrow/pull/454
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Add visitor pattern + builders for column sequences [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #454:
URL: https://github.com/apache/arrow-nanoarrow/pull/454#discussion_r1600010007
##########
python/src/nanoarrow/array.py:
##########
@@ -344,6 +345,42 @@ def iter_chunk_views(self) -> Iterable[CArrayView]:
"""
return iter_array_views(self)
+ def to_pylist(self) -> List:
+ """Convert this Array to a ``list()` of Python objects
+
+ Computes an identical value to list(:meth:`iter_py`) but can be several
+ times faster.
+
+ Examples
+ --------
+
+ >>> import nanoarrow as na
+ >>> array = na.Array([1, 2, 3], na.int32())
+ >>> array.to_pylist()
+ [1, 2, 3]
+ """
+ return to_pylist(self)
+
+ def to_columns(self) -> Tuple[str, Sequence]:
+ """Convert this Array to a ``list()` of sequences
+
+ Converts a stream of struct arrays into its column-wise representation
+ such that each column is either a contiguous buffer or a ``list()``.
Review Comment:
See https://github.com/apache/arrow-nanoarrow/pull/464 ! (I think I copied the help text from a previous PR where I'd implemented both before splitting it up 😬 )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Add visitor pattern + builders for column sequences [arrow-nanoarrow]
Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #454:
URL: https://github.com/apache/arrow-nanoarrow/pull/454#discussion_r1600337166
##########
python/src/nanoarrow/visitor.py:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, List, Sequence, Tuple, Union
+
+from nanoarrow._lib import CArrayView
+from nanoarrow.c_array_stream import c_array_stream
+from nanoarrow.iterator import ArrayViewBaseIterator, PyIterator
+from nanoarrow.schema import Type
+
+
+def to_pylist(obj, schema=None) -> List:
+ """Convert ``obj`` to a ``list()` of Python objects
+
+ Computes an identical value to ``list(iterator.iter_py())`` but is several
+ times faster.
Review Comment:
Looking at the code, the main difference seems to be that the faster one reuses the `array_view` argument to the iterator constructor? (although that shouldn't make any difference for a case with one chunk like the above?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Add visitor pattern + builders for column sequences [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #454:
URL: https://github.com/apache/arrow-nanoarrow/pull/454#discussion_r1598621234
##########
python/src/nanoarrow/visitor.py:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, List, Sequence, Tuple, Union
+
+from nanoarrow._lib import CArrayView
+from nanoarrow.c_array_stream import c_array_stream
+from nanoarrow.iterator import ArrayViewBaseIterator, PyIterator
+from nanoarrow.schema import Type
+
+
+def to_pylist(obj, schema=None) -> List:
Review Comment:
Probably all the visitors and iterators would be nice in an abstract base class shared between the `Array` and `ArrayStream`! I added the user-facing bit here to ensure that `na.Array(...).to_pylist()` is what users type and if there comes a time we have to start adding documentation in multiple places (very possible with `to_columns()`) we can revisit the base class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Add visitor pattern + builders for column sequences [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #454:
URL: https://github.com/apache/arrow-nanoarrow/pull/454#discussion_r1598498344
##########
python/tests/test_visitor.py:
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import pytest
+
+import nanoarrow as na
+from nanoarrow import visitor
+
+
+def test_to_pylist():
+ assert visitor.to_pylist([1, 2, 3], na.int32()) == [1, 2, 3]
Review Comment:
Good call!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Add visitor pattern + builders for column sequences [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #454:
URL: https://github.com/apache/arrow-nanoarrow/pull/454#discussion_r1600021908
##########
python/src/nanoarrow/visitor.py:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, List, Sequence, Tuple, Union
+
+from nanoarrow._lib import CArrayView
+from nanoarrow.c_array_stream import c_array_stream
+from nanoarrow.iterator import ArrayViewBaseIterator, PyIterator
+from nanoarrow.schema import Type
+
+
+def to_pylist(obj, schema=None) -> List:
+ """Convert ``obj`` to a ``list()` of Python objects
+
+ Computes an identical value to ``list(iterator.iter_py())`` but is several
+ times faster.
Review Comment:
I have no idea! "Several times" is probably too strong here. The iteration code is slightly different, though (`out_list.extend(array_iter) in a loop` vs `list(yield from array_iter)`.
```python
import nanoarrow as na
import numpy as np
big = na.Array(np.random.random(int(1e6)))
%timeit big.to_pylist()
#> 19.8 ms ± 586 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit list(big.iter_py())
#> 50.6 ms ± 325 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Add visitor pattern + builders for column sequences [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #454:
URL: https://github.com/apache/arrow-nanoarrow/pull/454#discussion_r1598520308
##########
python/src/nanoarrow/visitor.py:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, List, Sequence, Tuple, Union
+
+from nanoarrow._lib import CArrayView
+from nanoarrow.c_array_stream import c_array_stream
+from nanoarrow.iterator import ArrayViewBaseIterator, PyIterator
+from nanoarrow.schema import Type
+
+
+def to_pylist(obj, schema=None) -> List:
+ """Convert ``obj`` to a ``list()` of Python objects
+
+ Computes an identical value to ``list(iterator.iter_py())`` but is several
+ times faster.
+
+ Paramters
+ ---------
+ obj : array stream-like
+ An array-like or array stream-like object as sanitized by
+ :func:`c_array_stream`.
+ schema : schema-like, optional
+ An optional schema, passed to :func:`c_array_stream`.
+
+ Examples
+ --------
+
+ >>> import nanoarrow as na
+ >>> from nanoarrow import visitor
+ >>> array = na.c_array([1, 2, 3], na.int32())
+ >>> visitor.to_pylist(array)
+ [1, 2, 3]
+ """
+ return ListBuilder.visit(obj, schema)
+
+
+def to_columns(obj, schema=None) -> Tuple[List[str], List[Sequence]]:
+ """Convert ``obj`` to a ``list()` of sequences
+
+ Converts a stream of struct arrays into its column-wise representation
+ such that each column is either a contiguous buffer or a ``list()``.
+
+ Paramters
+ ---------
+ obj : array stream-like
+ An array-like or array stream-like object as sanitized by
+ :func:`c_array_stream`.
+ schema : schema-like, optional
+ An optional schema, passed to :func:`c_array_stream`.
+
+ Examples
+ --------
+
+ >>> import nanoarrow as na
+ >>> from nanoarrow import visitor
+ >>> import pyarrow as pa
+ >>> array = pa.record_batch([pa.array([1, 2, 3])], names=["col1"])
+ >>> names, columns = visitor.to_columns(array)
+ >>> names
+ ['col1']
+ >>> columns
+ [[1, 2, 3]]
+ """
+ return ColumnsBuilder.visit(obj, schema)
+
+
+class ArrayStreamVisitor(ArrayViewBaseIterator):
+ """Compute a value from one or more arrays in an ArrowArrayStream
+
+ This class supports a (currently internal) pattern for building
+ output from a zero or more arrays in a stream.
+
+ """
+
+ @classmethod
+ def visit(cls, obj, schema=None, total_elements=None, **kwargs):
+ """Visit all chunks in ``obj`` as a :func:`c_array_stream`."""
+
+ if total_elements is None and hasattr(obj, "__len__"):
+ total_elements = len(obj)
+
+ with c_array_stream(obj, schema=schema) as stream:
+ visitor = cls(stream._get_cached_schema(), **kwargs)
+ visitor.begin(total_elements)
+
+ visitor_set_array = visitor._set_array
+ visit_chunk_view = visitor.visit_chunk_view
+ array_view = visitor._array_view
+
+ for array in stream:
+ visitor_set_array(array)
+ visit_chunk_view(array_view)
+
+ return visitor.finish()
+
+ def begin(self, total_elements: Union[int, None] = None):
+ """Called after the schema has been resolved but before any
+ chunks have been visited. If the total number of elements
+ (i.e., the sum of all chunk lengths) is known, it is provided here.
+ """
+ pass
Review Comment:
It is rare, but valid, that an implementation of a visitor does not need to do anything at one or more of `begin`, `visit`, and `finish()` (e.g., print chunks, time the consumption process). If/when this is public it's definitely worth revisiting how best to communicate this (e.g., maybe force implementations to explicitly do nothing for clarity).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Add visitor pattern + builders for column sequences [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #454:
URL: https://github.com/apache/arrow-nanoarrow/pull/454#discussion_r1598502717
##########
python/src/nanoarrow/visitor.py:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, List, Sequence, Tuple, Union
+
+from nanoarrow._lib import CArrayView
+from nanoarrow.c_array_stream import c_array_stream
+from nanoarrow.iterator import ArrayViewBaseIterator, PyIterator
+from nanoarrow.schema import Type
+
+
+def to_pylist(obj, schema=None) -> List:
+ """Convert ``obj`` to a ``list()` of Python objects
+
+ Computes an identical value to ``list(iterator.iter_py())`` but is several
+ times faster.
+
+ Paramters
+ ---------
+ obj : array stream-like
+ An array-like or array stream-like object as sanitized by
+ :func:`c_array_stream`.
+ schema : schema-like, optional
+ An optional schema, passed to :func:`c_array_stream`.
+
+ Examples
+ --------
+
+ >>> import nanoarrow as na
+ >>> from nanoarrow import visitor
+ >>> array = na.c_array([1, 2, 3], na.int32())
+ >>> visitor.to_pylist(array)
+ [1, 2, 3]
+ """
+ return ListBuilder.visit(obj, schema)
+
+
+def to_columns(obj, schema=None) -> Tuple[List[str], List[Sequence]]:
+ """Convert ``obj`` to a ``list()` of sequences
+
+ Converts a stream of struct arrays into its column-wise representation
+ such that each column is either a contiguous buffer or a ``list()``.
+
+ Paramters
+ ---------
+ obj : array stream-like
+ An array-like or array stream-like object as sanitized by
+ :func:`c_array_stream`.
+ schema : schema-like, optional
+ An optional schema, passed to :func:`c_array_stream`.
+
+ Examples
+ --------
+
+ >>> import nanoarrow as na
+ >>> from nanoarrow import visitor
+ >>> import pyarrow as pa
+ >>> array = pa.record_batch([pa.array([1, 2, 3])], names=["col1"])
+ >>> names, columns = visitor.to_columns(array)
+ >>> names
+ ['col1']
+ >>> columns
+ [[1, 2, 3]]
+ """
+ return ColumnsBuilder.visit(obj, schema)
+
+
+class ArrayStreamVisitor(ArrayViewBaseIterator):
+ """Compute a value from one or more arrays in an ArrowArrayStream
+
+ This class supports a (currently internal) pattern for building
+ output from a zero or more arrays in a stream.
+
+ """
+
+ @classmethod
+ def visit(cls, obj, schema=None, total_elements=None, **kwargs):
+ """Visit all chunks in ``obj`` as a :func:`c_array_stream`."""
+
+ if total_elements is None and hasattr(obj, "__len__"):
+ total_elements = len(obj)
+
+ with c_array_stream(obj, schema=schema) as stream:
+ visitor = cls(stream._get_cached_schema(), **kwargs)
+ visitor.begin(total_elements)
+
+ visitor_set_array = visitor._set_array
+ visit_chunk_view = visitor.visit_chunk_view
+ array_view = visitor._array_view
+
+ for array in stream:
+ visitor_set_array(array)
+ visit_chunk_view(array_view)
+
+ return visitor.finish()
+
+ def begin(self, total_elements: Union[int, None] = None):
+ """Called after the schema has been resolved but before any
+ chunks have been visited. If the total number of elements
+ (i.e., the sum of all chunk lengths) is known, it is provided here.
+ """
+ pass
+
+ def visit_chunk_view(self, array_view: CArrayView) -> None:
+ """Called exactly one for each chunk seen."""
+ pass
+
+ def finish(self) -> Any:
+ """Called exactly once after all chunks have been visited."""
+ return None
+
+
+class ListBuilder(ArrayStreamVisitor):
+ def __init__(self, schema, *, iterator_cls=PyIterator, _array_view=None):
Review Comment:
Good call! I'd used `_array_view` because it's sort of internal; however, the whole API is currently internal. If/when it's made public there should perhaps be a better system for instantiating/efficiently reusing child array views that does not stick out like this does.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Add visitor pattern + builders for column sequences [arrow-nanoarrow]
Posted by "danepitkin (via GitHub)" <gi...@apache.org>.
danepitkin commented on code in PR #454:
URL: https://github.com/apache/arrow-nanoarrow/pull/454#discussion_r1597196922
##########
python/tests/test_visitor.py:
##########
@@ -0,0 +1,46 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import pytest
+
+import nanoarrow as na
+from nanoarrow import visitor
+
+
+def test_to_pylist():
+ assert visitor.to_pylist([1, 2, 3], na.int32()) == [1, 2, 3]
Review Comment:
This feels like an odd test because the input and output are technically both pylists. Would it be better to build an array and use that?
##########
python/src/nanoarrow/visitor.py:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, List, Sequence, Tuple, Union
+
+from nanoarrow._lib import CArrayView
+from nanoarrow.c_array_stream import c_array_stream
+from nanoarrow.iterator import ArrayViewBaseIterator, PyIterator
+from nanoarrow.schema import Type
+
+
+def to_pylist(obj, schema=None) -> List:
+ """Convert ``obj`` to a ``list()` of Python objects
+
+ Computes an identical value to ``list(iterator.iter_py())`` but is several
+ times faster.
+
+ Paramters
+ ---------
+ obj : array stream-like
+ An array-like or array stream-like object as sanitized by
+ :func:`c_array_stream`.
+ schema : schema-like, optional
+ An optional schema, passed to :func:`c_array_stream`.
+
+ Examples
+ --------
+
+ >>> import nanoarrow as na
+ >>> from nanoarrow import visitor
+ >>> array = na.c_array([1, 2, 3], na.int32())
+ >>> visitor.to_pylist(array)
+ [1, 2, 3]
+ """
+ return ListBuilder.visit(obj, schema)
+
+
+def to_columns(obj, schema=None) -> Tuple[List[str], List[Sequence]]:
+ """Convert ``obj`` to a ``list()` of sequences
+
+ Converts a stream of struct arrays into its column-wise representation
+ such that each column is either a contiguous buffer or a ``list()``.
+
+ Paramters
+ ---------
+ obj : array stream-like
+ An array-like or array stream-like object as sanitized by
+ :func:`c_array_stream`.
+ schema : schema-like, optional
+ An optional schema, passed to :func:`c_array_stream`.
+
+ Examples
+ --------
+
+ >>> import nanoarrow as na
+ >>> from nanoarrow import visitor
+ >>> import pyarrow as pa
+ >>> array = pa.record_batch([pa.array([1, 2, 3])], names=["col1"])
+ >>> names, columns = visitor.to_columns(array)
+ >>> names
+ ['col1']
+ >>> columns
+ [[1, 2, 3]]
+ """
+ return ColumnsBuilder.visit(obj, schema)
+
+
+class ArrayStreamVisitor(ArrayViewBaseIterator):
+ """Compute a value from one or more arrays in an ArrowArrayStream
+
+ This class supports a (currently internal) pattern for building
+ output from a zero or more arrays in a stream.
+
+ """
+
+ @classmethod
+ def visit(cls, obj, schema=None, total_elements=None, **kwargs):
+ """Visit all chunks in ``obj`` as a :func:`c_array_stream`."""
+
+ if total_elements is None and hasattr(obj, "__len__"):
+ total_elements = len(obj)
+
+ with c_array_stream(obj, schema=schema) as stream:
+ visitor = cls(stream._get_cached_schema(), **kwargs)
+ visitor.begin(total_elements)
+
+ visitor_set_array = visitor._set_array
+ visit_chunk_view = visitor.visit_chunk_view
+ array_view = visitor._array_view
+
+ for array in stream:
+ visitor_set_array(array)
+ visit_chunk_view(array_view)
+
+ return visitor.finish()
+
+ def begin(self, total_elements: Union[int, None] = None):
+ """Called after the schema has been resolved but before any
+ chunks have been visited. If the total number of elements
+ (i.e., the sum of all chunk lengths) is known, it is provided here.
+ """
+ pass
Review Comment:
Maybe raise `NotImplementedError()` here? Alternatively, you could also look into https://docs.python.org/3/library/abc.html and abstract methods.
##########
python/src/nanoarrow/iterator.py:
##########
@@ -199,7 +175,41 @@ class PyIterator(ArrayViewBaseIterator):
Intended for internal use.
"""
+ @classmethod
+ def get_iterator(cls, obj, schema=None):
+ with c_array_stream(obj, schema=schema) as stream:
+ iterator = cls(stream._get_cached_schema())
+ for array in stream:
+ iterator._set_array(array)
+ yield from iterator
+
+ def __init__(self, schema, *, _array_view=None):
Review Comment:
nit: `_array_view` -> `array_view` as a param name (also applies to the base class)
##########
python/src/nanoarrow/visitor.py:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, List, Sequence, Tuple, Union
+
+from nanoarrow._lib import CArrayView
+from nanoarrow.c_array_stream import c_array_stream
+from nanoarrow.iterator import ArrayViewBaseIterator, PyIterator
+from nanoarrow.schema import Type
+
+
+def to_pylist(obj, schema=None) -> List:
Review Comment:
I think my personal preference would be to have `to_pylist` and `to_columns` be class APIs instead of helper functions. e.g.
```
>>> import nanoarrow as na
>>> array = na.c_array([1, 2, 3], na.int32())
>>> array.to_pylist()
```
You could put these two apis in an abstract base class and add the base class to other classes. WDYT?
##########
python/src/nanoarrow/visitor.py:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, List, Sequence, Tuple, Union
+
+from nanoarrow._lib import CArrayView
+from nanoarrow.c_array_stream import c_array_stream
+from nanoarrow.iterator import ArrayViewBaseIterator, PyIterator
+from nanoarrow.schema import Type
+
+
+def to_pylist(obj, schema=None) -> List:
+ """Convert ``obj`` to a ``list()` of Python objects
+
+ Computes an identical value to ``list(iterator.iter_py())`` but is several
+ times faster.
+
+ Paramters
+ ---------
+ obj : array stream-like
+ An array-like or array stream-like object as sanitized by
+ :func:`c_array_stream`.
+ schema : schema-like, optional
+ An optional schema, passed to :func:`c_array_stream`.
+
+ Examples
+ --------
+
+ >>> import nanoarrow as na
+ >>> from nanoarrow import visitor
+ >>> array = na.c_array([1, 2, 3], na.int32())
+ >>> visitor.to_pylist(array)
+ [1, 2, 3]
+ """
+ return ListBuilder.visit(obj, schema)
+
+
+def to_columns(obj, schema=None) -> Tuple[List[str], List[Sequence]]:
+ """Convert ``obj`` to a ``list()` of sequences
+
+ Converts a stream of struct arrays into its column-wise representation
+ such that each column is either a contiguous buffer or a ``list()``.
+
+ Paramters
+ ---------
+ obj : array stream-like
+ An array-like or array stream-like object as sanitized by
+ :func:`c_array_stream`.
+ schema : schema-like, optional
+ An optional schema, passed to :func:`c_array_stream`.
+
+ Examples
+ --------
+
+ >>> import nanoarrow as na
+ >>> from nanoarrow import visitor
+ >>> import pyarrow as pa
+ >>> array = pa.record_batch([pa.array([1, 2, 3])], names=["col1"])
+ >>> names, columns = visitor.to_columns(array)
+ >>> names
+ ['col1']
+ >>> columns
+ [[1, 2, 3]]
+ """
+ return ColumnsBuilder.visit(obj, schema)
+
+
+class ArrayStreamVisitor(ArrayViewBaseIterator):
+ """Compute a value from one or more arrays in an ArrowArrayStream
+
+ This class supports a (currently internal) pattern for building
+ output from a zero or more arrays in a stream.
+
+ """
+
+ @classmethod
+ def visit(cls, obj, schema=None, total_elements=None, **kwargs):
+ """Visit all chunks in ``obj`` as a :func:`c_array_stream`."""
+
+ if total_elements is None and hasattr(obj, "__len__"):
+ total_elements = len(obj)
+
+ with c_array_stream(obj, schema=schema) as stream:
+ visitor = cls(stream._get_cached_schema(), **kwargs)
+ visitor.begin(total_elements)
+
+ visitor_set_array = visitor._set_array
+ visit_chunk_view = visitor.visit_chunk_view
+ array_view = visitor._array_view
+
+ for array in stream:
+ visitor_set_array(array)
+ visit_chunk_view(array_view)
+
+ return visitor.finish()
+
+ def begin(self, total_elements: Union[int, None] = None):
+ """Called after the schema has been resolved but before any
+ chunks have been visited. If the total number of elements
+ (i.e., the sum of all chunk lengths) is known, it is provided here.
+ """
+ pass
+
+ def visit_chunk_view(self, array_view: CArrayView) -> None:
+ """Called exactly one for each chunk seen."""
+ pass
+
+ def finish(self) -> Any:
+ """Called exactly once after all chunks have been visited."""
+ return None
+
+
+class ListBuilder(ArrayStreamVisitor):
+ def __init__(self, schema, *, iterator_cls=PyIterator, _array_view=None):
Review Comment:
nit: `_array_view` -> `array_view` in ListBuilder/ColumnsBuilder
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feat(python): Add visitor pattern + builders for column sequences [arrow-nanoarrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #454:
URL: https://github.com/apache/arrow-nanoarrow/pull/454#discussion_r1600353033
##########
python/src/nanoarrow/visitor.py:
##########
@@ -0,0 +1,179 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied. See the License for the
+# specific language governing permissions and limitations
+# under the License.
+
+from typing import Any, List, Sequence, Tuple, Union
+
+from nanoarrow._lib import CArrayView
+from nanoarrow.c_array_stream import c_array_stream
+from nanoarrow.iterator import ArrayViewBaseIterator, PyIterator
+from nanoarrow.schema import Type
+
+
+def to_pylist(obj, schema=None) -> List:
+ """Convert ``obj`` to a ``list()` of Python objects
+
+ Computes an identical value to ``list(iterator.iter_py())`` but is several
+ times faster.
Review Comment:
I believe that they both reuse the array_view...it seems to be `yield from` but I'm not an expert.
```python
import numpy as np
list_of_lists = []
for i in range(1000):
list_of_lists.append(list(np.random.random(1000)))
def iter_all():
for item in list_of_lists:
yield from item
def extend_all():
out = []
for item in list_of_lists:
out.extend(item)
return out
assert list(iter_all()) == extend_all()
%timeit list(iter_all())
#> 34.5 ms ± 522 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
%timeit extend_all()
#> 4.35 ms ± 517 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org