Re: [I] [Python] Conversion of pyArrow table to pandas fails with error: ArrowException: Unknown error: Wrapping Q� failed [arrow]

["Gobi2511 (via GitHub)" <gi...@apache.org>]

Gobi2511 commented on issue #34239:
URL: https://github.com/apache/arrow/issues/34239#issuecomment-2630835633

   I'm also getting the same error but I'm getting this error when reading the data from databricks which has some utf-8 encoded data. Sharring my exception details below:
   
   **Code**
   
   ``` 
   sql_query = f""" SELECT Column FROM table LIMIT 10 """
   
   host = os.getenv("DATABRICKS_HOST")
   http_path = os.getenv("DATABRICKS_HTTP_PATH")
   
   connection = sql.connect(server_hostname=host, http_path=http_path)
   with conn.cursor() as cursor:
           cursor.execute(sql_query)
           response = cursor.fetchmany(1000)
   ```
   
   **Error**
   
   ```
                  ^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/my_agent/app_helper/db_helper.py", line 35, in fetchmany
       query_results = cursor.fetchmany(fetch_size)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/my_agent_connectors/dq_databricks/dependencies/databricks/sql/client.py", line 976, in fetchmany
       return self.active_result_set.fetchmany(size)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/my_agent_connectors/dq_databricks/dependencies/databricks/sql/client.py", line 1235, in fetchmany
       return self._convert_arrow_table(self.fetchmany_arrow(size))
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/my_agent_connectors/dq_databricks/dependencies/databricks/sql/client.py", line 1161, in _convert_arrow_table
       df = table_renamed.to_pandas(
            ^^^^^^^^^^^^^^^^^^^^^^^^
     File "pyarrow/array.pxi", line 884, in pyarrow.lib._PandasConvertible.to_pandas
     File "pyarrow/table.pxi", line 4192, in pyarrow.lib.Table._to_pandas
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 776, in table_to_dataframe
       blocks = _table_to_blocks(options, table, categories, ext_columns_dtypes)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 1131, in _table_to_blocks
       return [_reconstruct_block(item, columns, extension_columns)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 1131, in <listcomp>
       return [_reconstruct_block(item, columns, extension_columns)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/pyarrow/pandas_compat.py", line 736, in _reconstruct_block
       pd_ext_arr = pandas_dtype.__from_arrow__(arr)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/airflow/.local/lib/python3.11/site-packages/pandas/core/arrays/string_.py", line 230, in __from_arrow__
       arr = arr.to_numpy(zero_copy_only=False)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "pyarrow/array.pxi", line 1581, in pyarrow.lib.Array.to_numpy
     File "pyarrow/error.pxi", line 91, in pyarrow.lib.check_status
   pyarrow.lib.ArrowException: Unknown error: Wrapping Ruta 204 � Zapote failed
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org