Implementation of Arrow table to Parquet File Writer

[saurabh pratap singh <sa...@gmail.com>]

Hi

I wanted to know whether there is a support/library available for
writing arrow tables as parquet files.
Meanwhile I tried writing my own converter where I am using SchemaConverter
provided by arrow (to convert arrow schema to parquet ) Then Converting
Arrow table to Group(ParquetExample Group reader/writer as a reference from
parquet-mr) and dump as parquet .This works for primitive types without any
issues but for nested types it will be little complicated so wanted to know
if anything like this already exists or planned in near future .

Please let me know if some other information is required from my side.

Thanks in advance.

Re: Implementation of Arrow table to Parquet File Writer

[Ryan Blue <rb...@netflix.com.INVALID>]

Right now, the implementation in Iceberg is focused on building out a read
path. The write path still uses a row-oriented interface. If you have Arrow
data and you need to write it to Parquet, you should be able to use a
combination of Spark’s Arrow support and Iceberg. Spark has a ColumnarBatch
implementation that can read Arrow as InternalRow, then Iceberg has
utilities to write InternalRow to Parquet. Here’s the Iceberg side
<https://github.com/apache/incubator-iceberg/blob/master/spark/src/test/java/org/apache/iceberg/spark/data/TestSparkParquetWriter.java#L79-L84>
:

    try (FileAppender<InternalRow> writer =
Parquet.write(Files.localOutput(testFile))
        .schema(schema)
        .createWriterFunc(msgType ->
SparkParquetWriters.buildWriter(schema, msgType))
        .build()) {
      writer.addAll(records);
    }


On Mon, Jan 6, 2020 at 10:17 AM Wes McKinney <we...@gmail.com> wrote:

> There may be some work around this happening in Apache Iceberg
> (incubating). I don't know of other independent fully open source
> Arrow<->Parquet implementations in Java
>
> On Mon, Jan 6, 2020 at 11:54 AM saurabh pratap singh
> <sa...@gmail.com> wrote:
> >
> > forgot to mention using JAVA
> >
> > On Thu, Jan 2, 2020 at 3:42 PM saurabh pratap singh <
> saurabh.cse16@gmail.com>
> > wrote:
> >
> > > Hi
> > >
> > > I wanted to know whether there is a support/library available for
> > > writing arrow tables as parquet files.
> > > Meanwhile I tried writing my own converter where I am using
> > > SchemaConverter provided by arrow (to convert arrow schema to parquet )
> > > Then Converting Arrow table to Group(ParquetExample Group
> reader/writer as
> > > a reference from parquet-mr) and dump as parquet .This works for
> > > primitive types without any issues but for nested types it will be
> little
> > > complicated so wanted to know if anything like this already exists or
> > > planned in near future .
> > >
> > > Please let me know if some other information is required from my side.
> > >
> > > Thanks in advance.
> > >
> > >
>


-- 
Ryan Blue
Software Engineer
Netflix

Re: Implementation of Arrow table to Parquet File Writer

[Wes McKinney <we...@gmail.com>]

There may be some work around this happening in Apache Iceberg
(incubating). I don't know of other independent fully open source
Arrow<->Parquet implementations in Java

On Mon, Jan 6, 2020 at 11:54 AM saurabh pratap singh
<sa...@gmail.com> wrote:
>
> forgot to mention using JAVA
>
> On Thu, Jan 2, 2020 at 3:42 PM saurabh pratap singh <sa...@gmail.com>
> wrote:
>
> > Hi
> >
> > I wanted to know whether there is a support/library available for
> > writing arrow tables as parquet files.
> > Meanwhile I tried writing my own converter where I am using
> > SchemaConverter provided by arrow (to convert arrow schema to parquet )
> > Then Converting Arrow table to Group(ParquetExample Group reader/writer as
> > a reference from parquet-mr) and dump as parquet .This works for
> > primitive types without any issues but for nested types it will be little
> > complicated so wanted to know if anything like this already exists or
> > planned in near future .
> >
> > Please let me know if some other information is required from my side.
> >
> > Thanks in advance.
> >
> >

Re: Implementation of Arrow table to Parquet File Writer

[saurabh pratap singh <sa...@gmail.com>]

forgot to mention using JAVA

On Thu, Jan 2, 2020 at 3:42 PM saurabh pratap singh <sa...@gmail.com>
wrote:

> Hi
>
> I wanted to know whether there is a support/library available for
> writing arrow tables as parquet files.
> Meanwhile I tried writing my own converter where I am using
> SchemaConverter provided by arrow (to convert arrow schema to parquet )
> Then Converting Arrow table to Group(ParquetExample Group reader/writer as
> a reference from parquet-mr) and dump as parquet .This works for
> primitive types without any issues but for nested types it will be little
> complicated so wanted to know if anything like this already exists or
> planned in near future .
>
> Please let me know if some other information is required from my side.
>
> Thanks in advance.
>
>