You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Dong, Xin" <xi...@intel.com> on 2021/06/03 02:18:17 UTC

Decouple parquet-mr compression API from hadoop compression API

Hi, All,
Currently parquet-mr compression logic is using Hadoop compression API which makes parquet-mr compression highly coupled with Hadoop. Does community have any plan to decouple those two APIs? To make the things easier, maybe we can just using api similar to Hadoop compression APIs but belongs to parquet-mr namespace. And simply change current codec to implements the new parquet-mr API. Any thoughts?
Thanks,
Xin Dong


Re: Decouple parquet-mr compression API from hadoop compression API

Posted by Xinli shang <sh...@uber.com.INVALID>.
Thanks for working on this! Overall this is a good initiative to move away
from Hadoop. I just left some comments on the doc.

On Tue, Jun 29, 2021 at 2:02 AM Xu, Cheng A <ch...@intel.com> wrote:

>
> https://docs.google.com/document/d/1Ki14QAj1TP8u-SXk-PiKsdVskDAH-IVGWulYTiK54SM/edit
> Doc is uploaded on Google Doc FYI.
>
> Thanks
> Cheng Xu
>
> -----Original Message-----
> From: Gabor Szadovszky <ga...@cloudera.com.INVALID>
> Sent: Friday, June 25, 2021 4:05 PM
> To: Parquet Dev <de...@parquet.apache.org>
> Subject: Re: Decouple parquet-mr compression API from hadoop compression
> API
>
> Hi XinDong,
>
> Could you please put it up to google docs so anyone in the community can
> comment in it?
>
> Cheers,
> Gabor
>
> On Fri, Jun 25, 2021 at 9:47 AM Dong, Xin <xi...@intel.com> wrote:
>
> >  Hi, Gabor and Xinli,
> >
> >  We worked out a proposal for Parquet-mr pluggable compression framework.
> > It includes the work to decouple parquet-mr compression API from
> > Hadoop compression API. Please take a look and any comments are welcome.
> >
> > Thanks,
> > XinDong
> >
> > -----Original Message-----
> > From: Dong, Xin
> > Sent: Thursday, June 3, 2021 4:02 PM
> > To: dev@parquet.apache.org
> > Subject: RE: Decouple parquet-mr compression API from hadoop
> > compression API
> >
> > Hi Gabor,
> >
> > Thank you very much for the quick reply. We are interested in
> > separating parquet-mr compression API from Hadoop and will try to make
> > a proposal for the new API. Will let you know when we are ready and
> > your comments will be appreciated.
> >
> > Thanks,
> > Xin Dong
> >
> > -----Original Message-----
> > From: Gabor Szadovszky <ga...@apache.org>
> > Sent: Thursday, June 3, 2021 3:49 PM
> > To: Parquet Dev <de...@parquet.apache.org>
> > Subject: Re: Decouple parquet-mr compression API from hadoop
> > compression API
> >
> > Hi Xin Dong,
> >
> > There are a couple of open jiras related to this. Like PARQUET-1812 <
> > https://issues.apache.org/jira/browse/PARQUET-1812> about using the
> > airlift implementation of the codecs or your own jiras about the
> > provider-aware codecs. I strongly agree on having compression codecs
> > that are independent from Hadoop. It would also be required to have
> > our own compression codecs shipped if we want to achieve the core
> > features support (
> > PARQUET-1950 <https://issues.apache.org/jira/browse/PARQUET-1950>).
> > Feel free to create a jira about this. Also, if you want to invest on
> > it I'm happy to review the related PRs.
> >
> > Cheers,
> > Gabor
> >
> > On Thu, Jun 3, 2021 at 4:18 AM Dong, Xin <xi...@intel.com> wrote:
> >
> > > Hi, All,
> > > Currently parquet-mr compression logic is using Hadoop compression
> > > API which makes parquet-mr compression highly coupled with Hadoop.
> > > Does community have any plan to decouple those two APIs? To make the
> > > things easier, maybe we can just using api similar to Hadoop
> > > compression APIs but belongs to parquet-mr namespace. And simply
> > > change current codec to implements the new parquet-mr API. Any
> thoughts?
> > > Thanks,
> > > Xin Dong
> > >
> > >
> >
>


-- 
Xinli Shang

RE: Decouple parquet-mr compression API from hadoop compression API

Posted by "Xu, Cheng A" <ch...@intel.com>.
https://docs.google.com/document/d/1Ki14QAj1TP8u-SXk-PiKsdVskDAH-IVGWulYTiK54SM/edit Doc is uploaded on Google Doc FYI.

Thanks
Cheng Xu

-----Original Message-----
From: Gabor Szadovszky <ga...@cloudera.com.INVALID> 
Sent: Friday, June 25, 2021 4:05 PM
To: Parquet Dev <de...@parquet.apache.org>
Subject: Re: Decouple parquet-mr compression API from hadoop compression API

Hi XinDong,

Could you please put it up to google docs so anyone in the community can comment in it?

Cheers,
Gabor

On Fri, Jun 25, 2021 at 9:47 AM Dong, Xin <xi...@intel.com> wrote:

>  Hi, Gabor and Xinli,
>
>  We worked out a proposal for Parquet-mr pluggable compression framework.
> It includes the work to decouple parquet-mr compression API from 
> Hadoop compression API. Please take a look and any comments are welcome.
>
> Thanks,
> XinDong
>
> -----Original Message-----
> From: Dong, Xin
> Sent: Thursday, June 3, 2021 4:02 PM
> To: dev@parquet.apache.org
> Subject: RE: Decouple parquet-mr compression API from hadoop 
> compression API
>
> Hi Gabor,
>
> Thank you very much for the quick reply. We are interested in  
> separating parquet-mr compression API from Hadoop and will try to make 
> a proposal for the new API. Will let you know when we are ready and 
> your comments will be appreciated.
>
> Thanks,
> Xin Dong
>
> -----Original Message-----
> From: Gabor Szadovszky <ga...@apache.org>
> Sent: Thursday, June 3, 2021 3:49 PM
> To: Parquet Dev <de...@parquet.apache.org>
> Subject: Re: Decouple parquet-mr compression API from hadoop 
> compression API
>
> Hi Xin Dong,
>
> There are a couple of open jiras related to this. Like PARQUET-1812 < 
> https://issues.apache.org/jira/browse/PARQUET-1812> about using the 
> airlift implementation of the codecs or your own jiras about the 
> provider-aware codecs. I strongly agree on having compression codecs 
> that are independent from Hadoop. It would also be required to have 
> our own compression codecs shipped if we want to achieve the core 
> features support (
> PARQUET-1950 <https://issues.apache.org/jira/browse/PARQUET-1950>).
> Feel free to create a jira about this. Also, if you want to invest on 
> it I'm happy to review the related PRs.
>
> Cheers,
> Gabor
>
> On Thu, Jun 3, 2021 at 4:18 AM Dong, Xin <xi...@intel.com> wrote:
>
> > Hi, All,
> > Currently parquet-mr compression logic is using Hadoop compression 
> > API which makes parquet-mr compression highly coupled with Hadoop. 
> > Does community have any plan to decouple those two APIs? To make the 
> > things easier, maybe we can just using api similar to Hadoop 
> > compression APIs but belongs to parquet-mr namespace. And simply 
> > change current codec to implements the new parquet-mr API. Any thoughts?
> > Thanks,
> > Xin Dong
> >
> >
>

Re: Decouple parquet-mr compression API from hadoop compression API

Posted by Gabor Szadovszky <ga...@cloudera.com.INVALID>.
Hi XinDong,

Could you please put it up to google docs so anyone in the community can
comment in it?

Cheers,
Gabor

On Fri, Jun 25, 2021 at 9:47 AM Dong, Xin <xi...@intel.com> wrote:

>  Hi, Gabor and Xinli,
>
>  We worked out a proposal for Parquet-mr pluggable compression framework.
> It includes the work to decouple parquet-mr compression API from Hadoop
> compression API. Please take a look and any comments are welcome.
>
> Thanks,
> XinDong
>
> -----Original Message-----
> From: Dong, Xin
> Sent: Thursday, June 3, 2021 4:02 PM
> To: dev@parquet.apache.org
> Subject: RE: Decouple parquet-mr compression API from hadoop compression
> API
>
> Hi Gabor,
>
> Thank you very much for the quick reply. We are interested in  separating
> parquet-mr compression API from Hadoop and will try to make a proposal for
> the new API. Will let you know when we are ready and your comments will be
> appreciated.
>
> Thanks,
> Xin Dong
>
> -----Original Message-----
> From: Gabor Szadovszky <ga...@apache.org>
> Sent: Thursday, June 3, 2021 3:49 PM
> To: Parquet Dev <de...@parquet.apache.org>
> Subject: Re: Decouple parquet-mr compression API from hadoop compression
> API
>
> Hi Xin Dong,
>
> There are a couple of open jiras related to this. Like PARQUET-1812 <
> https://issues.apache.org/jira/browse/PARQUET-1812> about using the
> airlift implementation of the codecs or your own jiras about the
> provider-aware codecs. I strongly agree on having compression codecs that
> are independent from Hadoop. It would also be required to have our own
> compression codecs shipped if we want to achieve the core features support (
> PARQUET-1950 <https://issues.apache.org/jira/browse/PARQUET-1950>).
> Feel free to create a jira about this. Also, if you want to invest on it
> I'm happy to review the related PRs.
>
> Cheers,
> Gabor
>
> On Thu, Jun 3, 2021 at 4:18 AM Dong, Xin <xi...@intel.com> wrote:
>
> > Hi, All,
> > Currently parquet-mr compression logic is using Hadoop compression API
> > which makes parquet-mr compression highly coupled with Hadoop. Does
> > community have any plan to decouple those two APIs? To make the things
> > easier, maybe we can just using api similar to Hadoop compression APIs
> > but belongs to parquet-mr namespace. And simply change current codec
> > to implements the new parquet-mr API. Any thoughts?
> > Thanks,
> > Xin Dong
> >
> >
>

RE: Decouple parquet-mr compression API from hadoop compression API

Posted by "Dong, Xin" <xi...@intel.com>.
 Hi, Gabor and Xinli,

 We worked out a proposal for Parquet-mr pluggable compression framework. It includes the work to decouple parquet-mr compression API from Hadoop compression API. Please take a look and any comments are welcome.

Thanks,
XinDong

-----Original Message-----
From: Dong, Xin 
Sent: Thursday, June 3, 2021 4:02 PM
To: dev@parquet.apache.org
Subject: RE: Decouple parquet-mr compression API from hadoop compression API

Hi Gabor,

Thank you very much for the quick reply. We are interested in  separating parquet-mr compression API from Hadoop and will try to make a proposal for the new API. Will let you know when we are ready and your comments will be appreciated.

Thanks,
Xin Dong

-----Original Message-----
From: Gabor Szadovszky <ga...@apache.org>
Sent: Thursday, June 3, 2021 3:49 PM
To: Parquet Dev <de...@parquet.apache.org>
Subject: Re: Decouple parquet-mr compression API from hadoop compression API

Hi Xin Dong,

There are a couple of open jiras related to this. Like PARQUET-1812 <https://issues.apache.org/jira/browse/PARQUET-1812> about using the airlift implementation of the codecs or your own jiras about the provider-aware codecs. I strongly agree on having compression codecs that are independent from Hadoop. It would also be required to have our own compression codecs shipped if we want to achieve the core features support (
PARQUET-1950 <https://issues.apache.org/jira/browse/PARQUET-1950>).
Feel free to create a jira about this. Also, if you want to invest on it I'm happy to review the related PRs.

Cheers,
Gabor

On Thu, Jun 3, 2021 at 4:18 AM Dong, Xin <xi...@intel.com> wrote:

> Hi, All,
> Currently parquet-mr compression logic is using Hadoop compression API 
> which makes parquet-mr compression highly coupled with Hadoop. Does 
> community have any plan to decouple those two APIs? To make the things 
> easier, maybe we can just using api similar to Hadoop compression APIs 
> but belongs to parquet-mr namespace. And simply change current codec 
> to implements the new parquet-mr API. Any thoughts?
> Thanks,
> Xin Dong
>
>

RE: Decouple parquet-mr compression API from hadoop compression API

Posted by "Dong, Xin" <xi...@intel.com>.
Hi Gabor,

Thank you very much for the quick reply. We are interested in  separating parquet-mr compression API from Hadoop and will try to make a proposal for the new API. Will let you know when we are ready and your comments will be appreciated.

Thanks,
Xin Dong

-----Original Message-----
From: Gabor Szadovszky <ga...@apache.org> 
Sent: Thursday, June 3, 2021 3:49 PM
To: Parquet Dev <de...@parquet.apache.org>
Subject: Re: Decouple parquet-mr compression API from hadoop compression API

Hi Xin Dong,

There are a couple of open jiras related to this. Like PARQUET-1812 <https://issues.apache.org/jira/browse/PARQUET-1812> about using the airlift implementation of the codecs or your own jiras about the provider-aware codecs. I strongly agree on having compression codecs that are independent from Hadoop. It would also be required to have our own compression codecs shipped if we want to achieve the core features support (
PARQUET-1950 <https://issues.apache.org/jira/browse/PARQUET-1950>).
Feel free to create a jira about this. Also, if you want to invest on it I'm happy to review the related PRs.

Cheers,
Gabor

On Thu, Jun 3, 2021 at 4:18 AM Dong, Xin <xi...@intel.com> wrote:

> Hi, All,
> Currently parquet-mr compression logic is using Hadoop compression API 
> which makes parquet-mr compression highly coupled with Hadoop. Does 
> community have any plan to decouple those two APIs? To make the things 
> easier, maybe we can just using api similar to Hadoop compression APIs 
> but belongs to parquet-mr namespace. And simply change current codec 
> to implements the new parquet-mr API. Any thoughts?
> Thanks,
> Xin Dong
>
>

Re: Decouple parquet-mr compression API from hadoop compression API

Posted by Gabor Szadovszky <ga...@apache.org>.
Hi Xin Dong,

There are a couple of open jiras related to this. Like PARQUET-1812
<https://issues.apache.org/jira/browse/PARQUET-1812> about using the
airlift implementation of the codecs or your own jiras about the
provider-aware codecs. I strongly agree on having compression codecs that
are independent from Hadoop. It would also be required to have our own
compression codecs shipped if we want to achieve the core features support (
PARQUET-1950 <https://issues.apache.org/jira/browse/PARQUET-1950>).
Feel free to create a jira about this. Also, if you want to invest on it
I'm happy to review the related PRs.

Cheers,
Gabor

On Thu, Jun 3, 2021 at 4:18 AM Dong, Xin <xi...@intel.com> wrote:

> Hi, All,
> Currently parquet-mr compression logic is using Hadoop compression API
> which makes parquet-mr compression highly coupled with Hadoop. Does
> community have any plan to decouple those two APIs? To make the things
> easier, maybe we can just using api similar to Hadoop compression APIs but
> belongs to parquet-mr namespace. And simply change current codec to
> implements the new parquet-mr API. Any thoughts?
> Thanks,
> Xin Dong
>
>