[jira] [Resolved] (ARROW-13480) [C++] [R] [Python] C-interface error propagation

["Antoine Pitrou (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/ARROW-13480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Antoine Pitrou resolved ARROW-13480.
------------------------------------
    Fix Version/s: 6.0.0
       Resolution: Fixed

Issue resolved by pull request 10993
[https://github.com/apache/arrow/pull/10993]

> [C++] [R] [Python] C-interface error propagation 
> -------------------------------------------------
>
>                 Key: ARROW-13480
>                 URL: https://issues.apache.org/jira/browse/ARROW-13480
>             Project: Apache Arrow
>          Issue Type: Improvement
>          Components: C++, Python, R
>            Reporter: Jonathan Keane
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: 6.0.0
>
>          Time Spent: 1h
>  Remaining Estimate: 0h
>
> Working on integration with DuckDB, we ran into an issue where it looks like errors are not being propagated fully/correctly with record batch readers using the C-interface. The DuckDB issue where this came up is https://github.com/duckdb/duckdb/issues/2055
> In the example I'm passing a dataset with either one or two files from R to python. I've specifically mis-specified the schema to get an error The one file version works like I expect percolating the error up:
> {code:r}
> > library("arrow")
> > 
> > venv <- try(reticulate::virtualenv_create("arrow-test"))
> virtualenv: arrow-test
> > install_pyarrow("arrow-test", nightly = TRUE)
> [output from installing pyarrow ...]
> > reticulate::use_virtualenv("arrow-test")
> > 
> > file <- "arrow/r/inst/v0.7.1.parquet"
> > arrow_table <- arrow::open_dataset(rep(file, 1), schema(x=arrow::null()))
> > 
> > scan <- Scanner$create(arrow_table)
> > reader <- scan$ToRecordBatchReader()
> > pyreader <- reticulate::r_to_py(reader)
> > pytab <- pyreader$read_all()
> Error in py_call_impl(callable, dots$args, dots$keywords) : 
>   OSError: NotImplemented: Unsupported cast from double to null using function cast_null
> Detailed traceback:
>   File "pyarrow/ipc.pxi", line 563, in pyarrow.lib.RecordBatchReader.read_all
>   File "pyarrow/error.pxi", line 114, in pyarrow.lib.check_status
> {code}
> But when having 2 (or more) files, the process hangs reading all of the batches:
> {code:r}
> > library("arrow")
> > 
> > venv <- try(reticulate::virtualenv_create("arrow-test"))
> virtualenv: arrow-test
> > install_pyarrow("arrow-test", nightly = TRUE)
> [output from installing pyarrow ...]
> > reticulate::use_virtualenv("arrow-test")
> > 
> > file <- "arrow/r/inst/v0.7.1.parquet"
> > arrow_table <- arrow::open_dataset(rep(file, 2), schema(x=arrow::null()))
> > 
> > scan <- Scanner$create(arrow_table)
> > reader <- scan$ToRecordBatchReader()
> > pyreader <- reticulate::r_to_py(reader)
> > pytab <- pyreader$read_all()
> {hangs forever here}
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)