[I] Run End Encoding not working with 16.1.0 [arrow]

["Tunneller (via GitHub)" <gi...@apache.org>]

Tunneller opened a new issue, #42159:
URL: https://github.com/apache/arrow/issues/42159

   ### Describe the bug, including details regarding any error messages, version, and platform.
   
   In Github launch for run-end encoding, the author cites that the following should not fail:
   
       import pyarrow as pa
       import pyarrow.compute as pc
       arr = pc.run_end_encode([1, 1, 2, 3, 3, 3, 6])
       arr.to_numpy()
   
   
   With PyArrow 16.1.0 I get 
   
       pyarrow.lib.ArrowInvalid: Need to dencode a RunEndEncodedArray, but only zero-copy conversions allowed
   
   This with python 3.10 and numpy 1.23.5
   
   Once run-end encoding is supported, will I be able to apply encoding to the output of Bigquery,  something like q.to_arrow(compression=run-end-encoding).to_pandas()  ?
   
   ### Component(s)
   
   Archery, Python


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Run End Encoding not working with 16.1.0 [arrow]

["rok (via GitHub)" <gi...@apache.org>]

rok commented on issue #42159:
URL: https://github.com/apache/arrow/issues/42159#issuecomment-2170464485

   Thanks for reporting this @Tunneller.
   Run-end encoding should work if you try: `arr.to_numpy(zero_copy_only=False)`. Implication being that if your original array was chunked data will be copied when creating the new numpy array.
   
   I'm not familiar with the bigquery interface. Could you provide more information about what `q` is?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Run End Encoding not working with 16.1.0 [arrow]

["Tunneller (via GitHub)" <gi...@apache.org>]

Tunneller commented on issue #42159:
URL: https://github.com/apache/arrow/issues/42159#issuecomment-2170523839

   Yes, that does work, thanks. 
   
   If query is an SQL string and bq_client is a BigQuery client (that has had credentials, etc, sorted) then:
   
       job_query = bq_client.query(query)
       q = job_query.result() 
   
   creates a temporary table on BigQuery which has got the results of the SQL call. The table is in column structure, largely similar to an Arrow file, I guess. The recommended way to get it off BigQuery is  py = q.to_arrow(). Followed by py.to_pandas() to be available for local analysis. Because BQ is column oriented, it internally takes advantage of encoding along the columns. And now Arrow files have run-end encoding that supports to_pandas().
   
   So what is missing from the chain is to pass encoded columns in BQ to encoded columns in Arrow without inflation in between. Passing data from BigQuery to Arrow is a bottleneck at the moment.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Run End Encoding not working with 16.1.0 [arrow]

["rok (via GitHub)" <gi...@apache.org>]

rok commented on issue #42159:
URL: https://github.com/apache/arrow/issues/42159#issuecomment-2170977900

   @Tunneller I'm not familiar with BigQuery internals. I'm guessing that BigQuery client returns Arrow so there shouldn't be need to change anything on BigQuery client->Arrow boundary. Arrow->numpy boundary will most likely always incur a copy.
   Perhaps if you can use pandas with Arrow backend you can avoid numpy altogether.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org