You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by Naveen Michaud-Agrawal <na...@gmail.com> on 2020/12/12 18:43:47 UTC

CDataInterface object from IPC format buffer

Is there any util code in the project (C++ or C) that can generate the
appropriate CDataInterface struct from an Arrow buffer in the IPC format?
As a motivating use case, I would like to run the webassembly version of
DuckDb in a browser, but feed it data from an external process that can
generate arrow output. DuckDB natively supports arrow data but only using
the CDataInterface.

Thanks,
Naveen Michaud-Agrawal

Re: CDataInterface object from IPC format buffer

Posted by Naveen Michaud-Agrawal <na...@gmail.com>.

Hi Antoine,

Thanks for the information, that should work for my purposes.

Regards,
Naveen

On Sat, Dec 12, 2020 at 1:49 PM Antoine Pitrou <an...@python.org> wrote:

>
> Hello Naveen,
>
> Le 12/12/2020 à 19:43, Naveen Michaud-Agrawal a écrit :
> > Is there any util code in the project (C++ or C) that can generate the
> > appropriate CDataInterface struct from an Arrow buffer in the IPC format?
>
> Not directly, but you can deserialize the IPC buffer(s) to regular Arrow
> record batches and then export those record batches to the C data
> interface format, using the regular C++ APIs.
>
> - use RecordBatchStreamReader to get a stream of RecordBatch objects
>   (see arrow/ipc/reader.h for details)
> - use ExportRecordBatch to export each RecordBatch to the C data interface
>   (see arrow/c/bridge.h for details)
>
> Regards
>
> Antoine.
>
>
> > As a motivating use case, I would like to run the webassembly version of
> > DuckDb in a browser, but feed it data from an external process that can
> > generate arrow output. DuckDB natively supports arrow data but only using
> > the CDataInterface.
> >
> > Thanks,
> > Naveen Michaud-Agrawal
> >
>


-- 
-----------------------------------
Naveen Michaud-Agrawal

Re: CDataInterface object from IPC format buffer

Posted by Antoine Pitrou <an...@python.org>.

Hello Naveen,

Le 12/12/2020 à 19:43, Naveen Michaud-Agrawal a écrit :
> Is there any util code in the project (C++ or C) that can generate the
> appropriate CDataInterface struct from an Arrow buffer in the IPC format?

Not directly, but you can deserialize the IPC buffer(s) to regular Arrow
record batches and then export those record batches to the C data
interface format, using the regular C++ APIs.

- use RecordBatchStreamReader to get a stream of RecordBatch objects
  (see arrow/ipc/reader.h for details)
- use ExportRecordBatch to export each RecordBatch to the C data interface
  (see arrow/c/bridge.h for details)

Regards

Antoine.


> As a motivating use case, I would like to run the webassembly version of
> DuckDb in a browser, but feed it data from an external process that can
> generate arrow output. DuckDB natively supports arrow data but only using
> the CDataInterface.
> 
> Thanks,
> Naveen Michaud-Agrawal
>