You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by Julien Le Dem <ju...@twitter.com.INVALID> on 2015/04/03 01:14:28 UTC
Re: About the hadoop-2 profile
1. Parquet works with hadoop 1 and 2, we work around the Hadoop
incompatibilities.
See:
https://github.com/apache/incubator-parquet-mr/blob/master/parquet-hadoop/src/main/java/parquet/hadoop/util/ContextUtil.java
Both profiles are run in travis-ci:
https://github.com/apache/incubator-parquet-mr/blob/master/.travis.yml#L26
2. You do not need to recompile parquet to the version of hadoop you use.
For the same reason described above.
On Sun, Mar 29, 2015 at 4:46 AM, Cheng Lian <li...@gmail.com> wrote:
> Hi developers,
>
> I have two questions about the Hadoop version Parquet uses. In the Maven
> POM of
> Parquet, the default Hadoop version is 1.1.0, while with `hadoop-2` profile
> enabled, the version is 2.3.0. So my questions are:
>
> 1. What's the Hadoop version the official parquet-hadoop Maven artifact
> (take
> 1.6.0rc7 as an example) depends on?
>
> Namely, when compiled and published to Maven repo, is `hadoop-2` profile
> enabled? (I must confess that I'm not quite familiar with Maven release
> management process, apologize in advance if this is a dumb question.)
>
> 2. When building an application which depends on a specific Hadoop
> version, do I
> have to compile my own `parquet-hadoop` library if the Hadoop version is
> inconsistent with the `parquet-hadoop` Maven artifact?
>
> Thanks!
> Cheng
>
Re: About the hadoop-2 profile
Posted by Cheng Lian <li...@gmail.com>.
Thanks for the super clear explanation :)
On 4/3/15 10:43 PM, Julien Le Dem wrote:
> the goal is to make sure that the parquet libraries compiled against hadoop
> 1 will also work with hadoop 2
>
> On Fri, Apr 3, 2015 at 4:12 AM, Cheng Lian <li...@gmail.com> wrote:
>
>> Thanks Julien! This is very helpful. If I understand this correctly, the
>> hadoop-2 profile is only used together with Travis CI for testing purposes?
>>
>> Cheng
>>
>>
>> On 4/3/15 7:14 AM, Julien Le Dem wrote:
>>
>>> 1. Parquet works with hadoop 1 and 2, we work around the Hadoop
>>> incompatibilities.
>>> See:
>>> https://github.com/apache/incubator-parquet-mr/blob/
>>> master/parquet-hadoop/src/main/java/parquet/hadoop/util/ContextUtil.java
>>> Both profiles are run in travis-ci:
>>> https://github.com/apache/incubator-parquet-mr/blob/
>>> master/.travis.yml#L26
>>>
>>> 2. You do not need to recompile parquet to the version of hadoop you use.
>>> For the same reason described above.
>>>
>>>
>>>
>>>
>>> On Sun, Mar 29, 2015 at 4:46 AM, Cheng Lian <li...@gmail.com>
>>> wrote:
>>>
>>> Hi developers,
>>>> I have two questions about the Hadoop version Parquet uses. In the Maven
>>>> POM of
>>>> Parquet, the default Hadoop version is 1.1.0, while with `hadoop-2`
>>>> profile
>>>> enabled, the version is 2.3.0. So my questions are:
>>>>
>>>> 1. What's the Hadoop version the official parquet-hadoop Maven artifact
>>>> (take
>>>> 1.6.0rc7 as an example) depends on?
>>>>
>>>> Namely, when compiled and published to Maven repo, is `hadoop-2`
>>>> profile
>>>> enabled? (I must confess that I'm not quite familiar with Maven
>>>> release
>>>> management process, apologize in advance if this is a dumb question.)
>>>>
>>>> 2. When building an application which depends on a specific Hadoop
>>>> version, do I
>>>> have to compile my own `parquet-hadoop` library if the Hadoop
>>>> version is
>>>> inconsistent with the `parquet-hadoop` Maven artifact?
>>>>
>>>> Thanks!
>>>> Cheng
>>>>
>>>>
Re: About the hadoop-2 profile
Posted by Julien Le Dem <ju...@twitter.com.INVALID>.
the goal is to make sure that the parquet libraries compiled against hadoop
1 will also work with hadoop 2
On Fri, Apr 3, 2015 at 4:12 AM, Cheng Lian <li...@gmail.com> wrote:
> Thanks Julien! This is very helpful. If I understand this correctly, the
> hadoop-2 profile is only used together with Travis CI for testing purposes?
>
> Cheng
>
>
> On 4/3/15 7:14 AM, Julien Le Dem wrote:
>
>> 1. Parquet works with hadoop 1 and 2, we work around the Hadoop
>> incompatibilities.
>> See:
>> https://github.com/apache/incubator-parquet-mr/blob/
>> master/parquet-hadoop/src/main/java/parquet/hadoop/util/ContextUtil.java
>> Both profiles are run in travis-ci:
>> https://github.com/apache/incubator-parquet-mr/blob/
>> master/.travis.yml#L26
>>
>> 2. You do not need to recompile parquet to the version of hadoop you use.
>> For the same reason described above.
>>
>>
>>
>>
>> On Sun, Mar 29, 2015 at 4:46 AM, Cheng Lian <li...@gmail.com>
>> wrote:
>>
>> Hi developers,
>>>
>>> I have two questions about the Hadoop version Parquet uses. In the Maven
>>> POM of
>>> Parquet, the default Hadoop version is 1.1.0, while with `hadoop-2`
>>> profile
>>> enabled, the version is 2.3.0. So my questions are:
>>>
>>> 1. What's the Hadoop version the official parquet-hadoop Maven artifact
>>> (take
>>> 1.6.0rc7 as an example) depends on?
>>>
>>> Namely, when compiled and published to Maven repo, is `hadoop-2`
>>> profile
>>> enabled? (I must confess that I'm not quite familiar with Maven
>>> release
>>> management process, apologize in advance if this is a dumb question.)
>>>
>>> 2. When building an application which depends on a specific Hadoop
>>> version, do I
>>> have to compile my own `parquet-hadoop` library if the Hadoop
>>> version is
>>> inconsistent with the `parquet-hadoop` Maven artifact?
>>>
>>> Thanks!
>>> Cheng
>>>
>>>
>
Re: About the hadoop-2 profile
Posted by Cheng Lian <li...@gmail.com>.
Thanks Julien! This is very helpful. If I understand this correctly, the
hadoop-2 profile is only used together with Travis CI for testing purposes?
Cheng
On 4/3/15 7:14 AM, Julien Le Dem wrote:
> 1. Parquet works with hadoop 1 and 2, we work around the Hadoop
> incompatibilities.
> See:
> https://github.com/apache/incubator-parquet-mr/blob/master/parquet-hadoop/src/main/java/parquet/hadoop/util/ContextUtil.java
> Both profiles are run in travis-ci:
> https://github.com/apache/incubator-parquet-mr/blob/master/.travis.yml#L26
>
> 2. You do not need to recompile parquet to the version of hadoop you use.
> For the same reason described above.
>
>
>
>
> On Sun, Mar 29, 2015 at 4:46 AM, Cheng Lian <li...@gmail.com> wrote:
>
>> Hi developers,
>>
>> I have two questions about the Hadoop version Parquet uses. In the Maven
>> POM of
>> Parquet, the default Hadoop version is 1.1.0, while with `hadoop-2` profile
>> enabled, the version is 2.3.0. So my questions are:
>>
>> 1. What's the Hadoop version the official parquet-hadoop Maven artifact
>> (take
>> 1.6.0rc7 as an example) depends on?
>>
>> Namely, when compiled and published to Maven repo, is `hadoop-2` profile
>> enabled? (I must confess that I'm not quite familiar with Maven release
>> management process, apologize in advance if this is a dumb question.)
>>
>> 2. When building an application which depends on a specific Hadoop
>> version, do I
>> have to compile my own `parquet-hadoop` library if the Hadoop version is
>> inconsistent with the `parquet-hadoop` Maven artifact?
>>
>> Thanks!
>> Cheng
>>