You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "paleolimbot (via GitHub)" <gi...@apache.org> on 2024/04/15 16:44:50 UTC

[PR] feat(python): Create string/binary arrays from iterables [arrow-nanoarrow]

paleolimbot opened a new pull request, #430:
URL: https://github.com/apache/arrow-nanoarrow/pull/430

   This PR adds support for building string and binary arrays via iterable:
   
   ```python
   import nanoarrow as na
   
   strings = ["pizza", "yogurt", "noodles", "peanut butter sandwiches"]
   
   na.Array(strings, na.string())
   #> nanoarrow.Array<string>[4]
   #> 'pizza'
   #> 'yogurt'
   #> 'noodles'
   #> 'peanut butter sandwiches'
   
   na.Array((s.encode() for s in strings), na.binary())
   #> nanoarrow.Array<binary>[4]
   #> b'pizza'
   #> b'yogurt'
   #> b'noodles'
   #> b'peanut butter sandwiches'
   ```
   
   The "build from iterable" code is now sufficiently complicated that it should be separated out. I did an initial attempt at that for this PR; however, it scrambles things up a bit and is complicated by the interdependence between the functions that sanitize arguments (e.g., `c_schema()`, `c_array()`) and the functions that build from iterable.
   
   Currently faster for strings and slightly slower for bytes than pyarrow.
   
   ```python
   from itertools import cycle, islice
   import nanoarrow as na
   import pyarrow as pa
   
   strings = ["pizza", "yogurt", "noodles", "peanut butter sandwiches"]
   binary = [s.encode() for s in strings]
   
   def many_strings():
       return islice(cycle(strings), int(1e6))
   
   def many_strings_with_nulls():
       return islice(cycle(strings + [None]), int(1e6))
   
   def many_bytes():
       return islice(cycle(binary), int(1e6))
   
   def many_bytes_with_nulls():
       return islice(cycle(binary + [None]), int(1e6))
   
   %timeit pa.array(many_strings(), pa.string())
   #> 23.4 ms ± 488 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
   %timeit na.c_array(many_strings(), na.string())
   #> 14.3 ms ± 112 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
   
   %timeit pa.array(many_strings_with_nulls(), pa.string())
   #> 21.4 ms ± 340 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
   %timeit na.c_array(many_strings_with_nulls(), na.string())
   #> 17.1 ms ± 340 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
   
   %timeit pa.array(many_bytes(), pa.binary())
   #> 19.7 ms ± 283 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
   %timeit na.c_array(many_bytes(), na.binary())
   #> 16.3 ms ± 136 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
   
   %timeit pa.array(many_bytes_with_nulls(), pa.binary())
   #> 17.6 ms ± 37.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
   %timeit na.c_array(many_bytes_with_nulls(), na.binary())
   #> 19 ms ± 378 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python): Create string/binary arrays from iterables [arrow-nanoarrow]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot merged PR #430:
URL: https://github.com/apache/arrow-nanoarrow/pull/430


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python): Create string/binary arrays from iterables [arrow-nanoarrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #430:
URL: https://github.com/apache/arrow-nanoarrow/pull/430#discussion_r1566774514


##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -2009,22 +2011,12 @@ cdef class CBufferBuilder:
         if self._buffer._data_type in (NANOARROW_TYPE_INTERVAL_DAY_TIME,
                                        NANOARROW_TYPE_INTERVAL_MONTH_DAY_NANO):
             for item in obj:
-                code = ArrowBufferReserve(self._buffer._ptr, bytes_per_element)
-                if code != NANOARROW_OK:
-                    Error.raise_error("ArrowBufferReserve()", code)
-
-                pack_into(self, self._buffer._ptr.size_bytes, *item)
-                self._buffer._ptr.size_bytes += bytes_per_element
+                write(pack(*item))
                 n_values += 1

Review Comment:
   Is there a reason you rolled this back? 
   (AFAIK this part was not the reason for the failures?)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python): Create string/binary arrays from iterables [arrow-nanoarrow]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #430:
URL: https://github.com/apache/arrow-nanoarrow/pull/430#discussion_r1566276378


##########
python/bootstrap.py:
##########
@@ -23,7 +23,7 @@
 import warnings
 
 
-# Generate the nanoarrow_c.pxd file used by the Cython extension
+# Generate the nanoarrow_c.pxd file used by the Cython extensions

Review Comment:
   This was basically just to force the wheels to build 😬 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python): Create string/binary arrays from iterables [arrow-nanoarrow]

Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #430:
URL: https://github.com/apache/arrow-nanoarrow/pull/430#discussion_r1567251754


##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -2009,22 +2011,12 @@ cdef class CBufferBuilder:
         if self._buffer._data_type in (NANOARROW_TYPE_INTERVAL_DAY_TIME,
                                        NANOARROW_TYPE_INTERVAL_MONTH_DAY_NANO):
             for item in obj:
-                code = ArrowBufferReserve(self._buffer._ptr, bytes_per_element)
-                if code != NANOARROW_OK:
-                    Error.raise_error("ArrowBufferReserve()", code)
-
-                pack_into(self, self._buffer._ptr.size_bytes, *item)
-                self._buffer._ptr.size_bytes += bytes_per_element
+                write(pack(*item))
                 n_values += 1

Review Comment:
   It wasn't a reason for the original failure but was exposed once that was fixed. Exactly what type of object is considered a buffer is different for (maybe just some versions of) PyPy, so `Struct.packinto()` threw an error. If we have time we can roll that back in (or figure out if there's some minimum version of PyPy we have to drop to make it work), but at least this gets us back to wheels that build!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] feat(python): Create string/binary arrays from iterables [arrow-nanoarrow]

Posted by "jorisvandenbossche (via GitHub)" <gi...@apache.org>.
jorisvandenbossche commented on code in PR #430:
URL: https://github.com/apache/arrow-nanoarrow/pull/430#discussion_r1567271235


##########
python/src/nanoarrow/_lib.pyx:
##########
@@ -2009,22 +2011,12 @@ cdef class CBufferBuilder:
         if self._buffer._data_type in (NANOARROW_TYPE_INTERVAL_DAY_TIME,
                                        NANOARROW_TYPE_INTERVAL_MONTH_DAY_NANO):
             for item in obj:
-                code = ArrowBufferReserve(self._buffer._ptr, bytes_per_element)
-                if code != NANOARROW_OK:
-                    Error.raise_error("ArrowBufferReserve()", code)
-
-                pack_into(self, self._buffer._ptr.size_bytes, *item)
-                self._buffer._ptr.size_bytes += bytes_per_element
+                write(pack(*item))
                 n_values += 1

Review Comment:
   Ah I see, sounds good!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org