You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@arrow.apache.org by Wenbo Zhao <We...@twosigma.com> on 2018/03/06 19:27:34 UTC

Parquet to arrow java converter

Hi,

Sorry that if someone may have asked the same question before. We are interested in providing a java convertor from Parquet to Arrow. Should I implement this converter in Parquet-mr/Parquet-arrow or under the Arrow project? I have the feeling that putting the implementation in Parquet-mr/Parquet-arrow would be preferable https://www.mail-archive.com/dev@arrow.apache.org/msg02606.html?

Thanks,

Wenbo

Re: Parquet to arrow java converter

Posted by Li Jin <ic...@gmail.com>.

This definitely sounds like a useful tool. It seems like Julien started
some of work in Parquet-arrow a while back.

Julien, I am wondering what's your thoughts on whether such code should
live in parquet-mr or arrow codebase?

On Tue, Mar 6, 2018 at 2:27 PM, Wenbo Zhao <We...@twosigma.com> wrote:

> Hi,
>
> Sorry that if someone may have asked the same question before. We are
> interested in providing a java convertor from Parquet to Arrow. Should I
> implement this converter in Parquet-mr/Parquet-arrow or under the Arrow
> project? I have the feeling that putting the implementation in
> Parquet-mr/Parquet-arrow would be preferable https://www.mail-archive.com/
> dev@arrow.apache.org/msg02606.html?
>
> Thanks,
>
> Wenbo
>

RE: Parquet to arrow java converter

Posted by Wenbo Zhao <We...@twosigma.com>.

Thanks Julien and Wes. There is an ongoing PR https://github.com/apache/parquet-mr/pull/443 (update Arrow version to 0.8.0) which I may be depending on. Should I wait for this? 

Wenbo 

-----Original Message-----
From: Julien Le Dem [mailto:julien.ledem@gmail.com] 
Sent: Tuesday, March 6, 2018 5:27 PM
To: dev@arrow.apache.org
Subject: Re: Parquet to arrow java converter

I would put in the parquet-mr codebase. I have contributed the schéma conversion code there. I’m happy to provide feedback on PRs in this area. 

Julien

> On Mar 6, 2018, at 12:18, Wes McKinney <we...@gmail.com> wrote:
> 
> When it had been discussed in the past, the thinking had been to 
> implement it in the Parquet Java codebase. I'd be interested in 
> others' opinions about this (since I'm not an expert on Java matters)
> 
> - Wes
> 
>> On Tue, Mar 6, 2018 at 2:27 PM, Wenbo Zhao <We...@twosigma.com> wrote:
>> Hi,
>> 
>> Sorry that if someone may have asked the same question before. We are interested in providing a java convertor from Parquet to Arrow. Should I implement this converter in Parquet-mr/Parquet-arrow or under the Arrow project? I have the feeling that putting the implementation in Parquet-mr/Parquet-arrow would be preferable https://www.mail-archive.com/dev@arrow.apache.org/msg02606.html?
>> 
>> Thanks,
>> 
>> Wenbo

Re: Parquet to arrow java converter

Posted by Julien Le Dem <ju...@gmail.com>.

I would put in the parquet-mr codebase. I have contributed the schéma conversion code there. I’m happy to provide feedback on PRs in this area. 

Julien

> On Mar 6, 2018, at 12:18, Wes McKinney <we...@gmail.com> wrote:
> 
> When it had been discussed in the past, the thinking had been to
> implement it in the Parquet Java codebase. I'd be interested in
> others' opinions about this (since I'm not an expert on Java matters)
> 
> - Wes
> 
>> On Tue, Mar 6, 2018 at 2:27 PM, Wenbo Zhao <We...@twosigma.com> wrote:
>> Hi,
>> 
>> Sorry that if someone may have asked the same question before. We are interested in providing a java convertor from Parquet to Arrow. Should I implement this converter in Parquet-mr/Parquet-arrow or under the Arrow project? I have the feeling that putting the implementation in Parquet-mr/Parquet-arrow would be preferable https://www.mail-archive.com/dev@arrow.apache.org/msg02606.html?
>> 
>> Thanks,
>> 
>> Wenbo

Re: Parquet to arrow java converter

Posted by Wes McKinney <we...@gmail.com>.

When it had been discussed in the past, the thinking had been to
implement it in the Parquet Java codebase. I'd be interested in
others' opinions about this (since I'm not an expert on Java matters)

- Wes

On Tue, Mar 6, 2018 at 2:27 PM, Wenbo Zhao <We...@twosigma.com> wrote:
> Hi,
>
> Sorry that if someone may have asked the same question before. We are interested in providing a java convertor from Parquet to Arrow. Should I implement this converter in Parquet-mr/Parquet-arrow or under the Arrow project? I have the feeling that putting the implementation in Parquet-mr/Parquet-arrow would be preferable https://www.mail-archive.com/dev@arrow.apache.org/msg02606.html?
>
> Thanks,
>
> Wenbo