C++: Code that read parquet into Arrow Arrays?

[Li Jin <ic...@gmail.com>]

Hi,

I am recently investigating a null/nan issue with Parquet and Arrow and
wonder if someone can give me a pointer to the code that decodes Parquet
row group into Arrow float/double arrays?

Thanks,
Li

Re: C++: Code that read parquet into Arrow Arrays?

[Li Jin <ic...@gmail.com>]

Thank you!

On Fri, Nov 17, 2023 at 5:40 PM Bryce Mecum <br...@gmail.com> wrote:

> Hi Li, I think what you're after is ColumnReaderImpl::NextBatch [1]
> which looks like it eventually calls TransferZeroCopy [2] in the case
> of primitive types like float/double (amongst others).
>
> [1]
> https://github.com/apache/arrow/blob/main/cpp/src/parquet/arrow/reader.cc#L107
> [2]
> https://github.com/apache/arrow/blob/main/cpp/src/parquet/arrow/reader_internal.cc#L345
>
> On Fri, Nov 17, 2023 at 12:27 PM Li Jin <ic...@gmail.com> wrote:
> >
> > Hi,
> >
> > I am recently investigating a null/nan issue with Parquet and Arrow and
> > wonder if someone can give me a pointer to the code that decodes Parquet
> > row group into Arrow float/double arrays?
> >
> > Thanks,
> > Li
>

Re: C++: Code that read parquet into Arrow Arrays?

[Bryce Mecum <br...@gmail.com>]

Hi Li, I think what you're after is ColumnReaderImpl::NextBatch [1]
which looks like it eventually calls TransferZeroCopy [2] in the case
of primitive types like float/double (amongst others).

[1] https://github.com/apache/arrow/blob/main/cpp/src/parquet/arrow/reader.cc#L107
[2] https://github.com/apache/arrow/blob/main/cpp/src/parquet/arrow/reader_internal.cc#L345

On Fri, Nov 17, 2023 at 12:27 PM Li Jin <ic...@gmail.com> wrote:
>
> Hi,
>
> I am recently investigating a null/nan issue with Parquet and Arrow and
> wonder if someone can give me a pointer to the code that decodes Parquet
> row group into Arrow float/double arrays?
>
> Thanks,
> Li

Re: C++: Code that read parquet into Arrow Arrays?

[Li Jin <ic...@gmail.com>]

Ty!

On Fri, Nov 17, 2023 at 9:12 PM wish maple <ma...@gmail.com> wrote:

> Hi,
>
> The parquet is divided into arrow and parquet part.
>
> 1. The parquet part lowest position is parquet decoder, in [1].
>     The float point might choosing PLAIN, RLE_DCIT or BYTE_STREAM_SPLIT
>     encoding.
> 2. parquet::ColumnReader is applied beyond decoder, each row-group might
> have
>     one or two ( if choosing dictionary encoding and fall-back to plain,
> there're
>     two encoding in a RowGroup for a column). This is in [2]
>
> Other modules are mentioned by Bryce.
>
> Best,
> Xuwei Fu
>
> [1] https://github.com/apache/arrow/blob/main/cpp/src/parquet/encoding.cc
> [2]
> https://github.com/apache/arrow/blob/main/cpp/src/parquet/column_reader.cc
>
> Li Jin <ic...@gmail.com> 于2023年11月18日周六 05:27写道：
>
> > Hi,
> >
> > I am recently investigating a null/nan issue with Parquet and Arrow and
> > wonder if someone can give me a pointer to the code that decodes Parquet
> > row group into Arrow float/double arrays?
> >
> > Thanks,
> > Li
> >
>

Re: C++: Code that read parquet into Arrow Arrays?

[wish maple <ma...@gmail.com>]

Hi,

The parquet is divided into arrow and parquet part.

1. The parquet part lowest position is parquet decoder, in [1].
    The float point might choosing PLAIN, RLE_DCIT or BYTE_STREAM_SPLIT
    encoding.
2. parquet::ColumnReader is applied beyond decoder, each row-group might
have
    one or two ( if choosing dictionary encoding and fall-back to plain,
there're
    two encoding in a RowGroup for a column). This is in [2]

Other modules are mentioned by Bryce.

Best,
Xuwei Fu

[1] https://github.com/apache/arrow/blob/main/cpp/src/parquet/encoding.cc
[2]
https://github.com/apache/arrow/blob/main/cpp/src/parquet/column_reader.cc

Li Jin <ic...@gmail.com> 于2023年11月18日周六 05:27写道：

> Hi,
>
> I am recently investigating a null/nan issue with Parquet and Arrow and
> wonder if someone can give me a pointer to the code that decodes Parquet
> row group into Arrow float/double arrays?
>
> Thanks,
> Li
>