[GitHub] [arrow] edponce edited a comment on issue #11559: cython how to access / check / print values for each row

[GitBox <gi...@apache.org>]

edponce edited a comment on issue #11559:
URL: https://github.com/apache/arrow/issues/11559#issuecomment-954659954


   Recall that Arrow's format is column-oriented so accessing rows is not an efficient operation. I do not expect a meaningful difference between using the Python and Cython APIs for iterating across rows (you would need to measure). Here are some code snippets using both APIs. Also, keep in mind that Cython's API is smaller and thus limited compared to Python's.
   
   _Note_: The following examples assume there are no NULL values. You would need to add those checks, so consider them as templates. Also, the examples use generators to get the values from each row.
   
   **VERSION 1: Cython/Python**
   **Cython code**
   ```python
   from pyarrow.lib cimport *
   
   # Helper function to extract a Scalar object from a column (CChunkedArray)
   cdef shared_ptr[CScalar] get_scalar_from_chunked_array(shared_ptr[CChunkedArray] c_chunked_array, int index):
       cdef:
            shared_ptr[CArray] c_array
            CResult[shared_ptr[CScalar]] result
   
       # Iterate through chunks/rows until finding the corresponding index
       chunked_array = c_chunked_array.get()
       for ichunk in range(chunked_array.num_chunks()):
           c_array = chunked_array.chunk(ichunk)
           array = c_array.get()
   
           if index < array.length():
               result = array.GetScalar(index)
               # NOTE: GetResultValue is exposed to Cython directly from here
               # https://github.com/apache/arrow/blob/master/cpp/src/arrow/python/common.h#L63
               return GetResultValue(result)
   
           # Update index relative to next chunk
           index = index - array.length()
   
   
   def iterate_table(obj):
       cdef:
            shared_ptr[CTable] c_table = pyarrow_unwrap_table(obj)
            shared_ptr[CChunkedArray] c_chunked_array
            shared_ptr[CScalar] c_scalar
   
       table = c_table.get()
       if table == NULL:
           raise TypeError("not a table")
   
       # Arrow format is column-oriented, so iterate first on rows then columns
       for irow in range(table.num_rows()):
           for icol in range(table.num_columns()):
               c_chunked_array = table.column(icol)
               c_scalar = get_scalar_from_chunked_array(c_chunked_array, irow)
               yield pyarrow_wrap_scalar(c_scalar)
   ```
   **Python code**
   ```python
   import pyarrow as pa
   import pandas as pd
   import example
   
   df = pd.DataFrame({
       'date': pd.date_range(start='2020-01-01 00:00:00', periods=3, freq='1min'),
       'name': ['jack', 'tim', 'frank'],
       'age': [32, 25, 65],
       'weight': [66.46, 84.11, 71.52]
   })
   
   table = pa.Table.from_pandas(df)
   
   # Print entire Table
   print(table)
   print()
   
   # Print Table row-by-row
   for i, scalar in enumerate(example.iterate_table(table)):
       if i % table.num_columns == 0:
           print()
       print(scalar, end=', ')
   print()
   ```
   
   **VERSION 2: Python API**
   ```python
   import pyarrow as pa
   import pandas as pd
   
   df = pd.DataFrame({
       'date': pd.date_range(start='2020-01-01 00:00:00', periods=3, freq='1min'),
       'name': ['jack', 'tim', 'frank'],
       'age': [32, 25, 65],
       'weight': [66.46, 84.11, 71.52]
   })
   
   table = pa.Table.from_pandas(df)
   
   # Print entire Table
   print(table)
   print()
   
   # Print Table row-by-row
   for irow in range(table.num_rows):
       for chunked_array in table.itercolumns():
           scalar = chunked_array[irow]
           print(scalar, end=', ')
       print()
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org