You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Roshan Naik <ro...@hortonworks.com> on 2015/04/01 22:04:02 UTC
Is 1.6 a good time to switch to hadoop2 ?
Don't recall if this has been discussed before. Its been sometime since hadoop2 has been out.
Sooner or later Flume will switch to hadoop2 based builds (hbase98 profile?) as the default. Not sure if 1.6 is the time or worth waiting longer.
-roshan
Re: Is 1.6 a good time to switch to hadoop2 ?
Posted by Roshan Naik <ro...@hortonworks.com>.
Seems like hbase 1.0.0 has now dropped hadoop1 support.
-roshan
On 4/1/15 4:19 PM, "Jarek Jarcec Cecho" <ja...@apache.org> wrote:
>Sqoop2 method is very simple - we have different profiles for Hadoop 1
>and 2 and simply use classifiers to mark the appropriate hadoop
>"profile". No need to generate different pom.xml file. Check it out here:
>
>https://github.com/apache/sqoop/blob/branch-1.99.4/execution/mapreduce/pom
>.xml#L65
>
>Jarcec
>
>> On Apr 1, 2015, at 2:56 PM, Hari Shreedharan
>><hs...@cloudera.com> wrote:
>>
>> We should choose which ever is the easiest to do I guess (Sqoop method
>>or HBase method).
>>
>>
>>
>>
>> Thanks, Hari
>>
>> On Wed, Apr 1, 2015 at 1:33 PM, Roshan Naik <ro...@hortonworks.com>
>> wrote:
>>
>>> FWIWŠ From what I recollect, Hbase did this when they started
>>>supporting
>>> hadoop1&2. Not sure if its still the same method they use..
>>> - their default pom had the artifacts named for hadoop1 binaries
>>> - they had a shell script to produce a modified pom ..with -hadoop2
>>>suffix
>>> for the artifact names
>>> - they built and published the hadoop2 binaries using the modified pom,
>>> and hadoop1 binaries form the original pom
>>> -roshan
>>> On 4/1/15 1:24 PM, "Hari Shreedharan" <hs...@cloudera.com>
>>>wrote:
>>>> I know sqoop2 does it. Maybe Jarcec can help?
>>>>
>>>>
>>>>
>>>>
>>>> Thanks, Hari
>>>>
>>>> On Wed, Apr 1, 2015 at 1:13 PM, Roshan Naik <ro...@hortonworks.com>
>>>> wrote:
>>>>
>>>>> To push artifacts for both hadoop2 and hadoop1 will need to name the
>>>>> hadoop2 and hadoop1 artifacts differently. Tricky to do that with one
>>>>> pom
>>>>> I think.
>>>>> -roshan
>>>>> On 4/1/15 1:05 PM, "Hari Shreedharan" <hs...@cloudera.com>
>>>>>wrote:
>>>>>> I am a +1 for doing this, since this is the last JDK 6 release.
>>>>>>Maybe we
>>>>>> should push artifacts for both Hadoop-1 and hbase-98 profiles this
>>>>>>time
>>>>>> and switch out to Hadoop 2 exclusively from 1.7?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> Thanks, Hari
>>>>>>
>>>>>> On Wed, Apr 1, 2015 at 1:04 PM, Roshan Naik <ro...@hortonworks.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Don't recall if this has been discussed before. Its been sometime
>>>>>>> since
>>>>>>> hadoop2 has been out.
>>>>>>> Sooner or later Flume will switch to hadoop2 based builds (hbase98
>>>>>>> profile?) as the default. Not sure if 1.6 is the time or worth
>>>>>>>waiting
>>>>>>> longer.
>>>>>>> -roshan
>
Re: Is 1.6 a good time to switch to hadoop2 ?
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
Sqoop2 method is very simple - we have different profiles for Hadoop 1 and 2 and simply use classifiers to mark the appropriate hadoop “profile”. No need to generate different pom.xml file. Check it out here:
https://github.com/apache/sqoop/blob/branch-1.99.4/execution/mapreduce/pom.xml#L65
Jarcec
> On Apr 1, 2015, at 2:56 PM, Hari Shreedharan <hs...@cloudera.com> wrote:
>
> We should choose which ever is the easiest to do I guess (Sqoop method or HBase method).
>
>
>
>
> Thanks, Hari
>
> On Wed, Apr 1, 2015 at 1:33 PM, Roshan Naik <ro...@hortonworks.com>
> wrote:
>
>> FWIWŠ From what I recollect, Hbase did this when they started supporting
>> hadoop1&2. Not sure if its still the same method they use..
>> - their default pom had the artifacts named for hadoop1 binaries
>> - they had a shell script to produce a modified pom ..with -hadoop2 suffix
>> for the artifact names
>> - they built and published the hadoop2 binaries using the modified pom,
>> and hadoop1 binaries form the original pom
>> -roshan
>> On 4/1/15 1:24 PM, "Hari Shreedharan" <hs...@cloudera.com> wrote:
>>> I know sqoop2 does it. Maybe Jarcec can help?
>>>
>>>
>>>
>>>
>>> Thanks, Hari
>>>
>>> On Wed, Apr 1, 2015 at 1:13 PM, Roshan Naik <ro...@hortonworks.com>
>>> wrote:
>>>
>>>> To push artifacts for both hadoop2 and hadoop1 will need to name the
>>>> hadoop2 and hadoop1 artifacts differently. Tricky to do that with one
>>>> pom
>>>> I think.
>>>> -roshan
>>>> On 4/1/15 1:05 PM, "Hari Shreedharan" <hs...@cloudera.com> wrote:
>>>>> I am a +1 for doing this, since this is the last JDK 6 release. Maybe we
>>>>> should push artifacts for both Hadoop-1 and hbase-98 profiles this time
>>>>> and switch out to Hadoop 2 exclusively from 1.7?
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Thanks, Hari
>>>>>
>>>>> On Wed, Apr 1, 2015 at 1:04 PM, Roshan Naik <ro...@hortonworks.com>
>>>>> wrote:
>>>>>
>>>>>> Don't recall if this has been discussed before. Its been sometime
>>>>>> since
>>>>>> hadoop2 has been out.
>>>>>> Sooner or later Flume will switch to hadoop2 based builds (hbase98
>>>>>> profile?) as the default. Not sure if 1.6 is the time or worth waiting
>>>>>> longer.
>>>>>> -roshan
Re: Is 1.6 a good time to switch to hadoop2 ?
Posted by Hari Shreedharan <hs...@cloudera.com>.
We should choose which ever is the easiest to do I guess (Sqoop method or HBase method).
Thanks, Hari
On Wed, Apr 1, 2015 at 1:33 PM, Roshan Naik <ro...@hortonworks.com>
wrote:
> FWIWŠ From what I recollect, Hbase did this when they started supporting
> hadoop1&2. Not sure if its still the same method they use..
> - their default pom had the artifacts named for hadoop1 binaries
> - they had a shell script to produce a modified pom ..with -hadoop2 suffix
> for the artifact names
> - they built and published the hadoop2 binaries using the modified pom,
> and hadoop1 binaries form the original pom
> -roshan
> On 4/1/15 1:24 PM, "Hari Shreedharan" <hs...@cloudera.com> wrote:
>>I know sqoop2 does it. Maybe Jarcec can help?
>>
>>
>>
>>
>>Thanks, Hari
>>
>>On Wed, Apr 1, 2015 at 1:13 PM, Roshan Naik <ro...@hortonworks.com>
>>wrote:
>>
>>> To push artifacts for both hadoop2 and hadoop1 will need to name the
>>> hadoop2 and hadoop1 artifacts differently. Tricky to do that with one
>>>pom
>>> I think.
>>> -roshan
>>> On 4/1/15 1:05 PM, "Hari Shreedharan" <hs...@cloudera.com> wrote:
>>>>I am a +1 for doing this, since this is the last JDK 6 release. Maybe we
>>>>should push artifacts for both Hadoop-1 and hbase-98 profiles this time
>>>>and switch out to Hadoop 2 exclusively from 1.7?
>>>>
>>>>
>>>>
>>>>
>>>>Thanks, Hari
>>>>
>>>>On Wed, Apr 1, 2015 at 1:04 PM, Roshan Naik <ro...@hortonworks.com>
>>>>wrote:
>>>>
>>>>> Don't recall if this has been discussed before. Its been sometime
>>>>>since
>>>>>hadoop2 has been out.
>>>>> Sooner or later Flume will switch to hadoop2 based builds (hbase98
>>>>>profile?) as the default. Not sure if 1.6 is the time or worth waiting
>>>>>longer.
>>>>> -roshan
Re: Is 1.6 a good time to switch to hadoop2 ?
Posted by Roshan Naik <ro...@hortonworks.com>.
FWIWŠ From what I recollect, Hbase did this when they started supporting
hadoop1&2. Not sure if its still the same method they use..
- their default pom had the artifacts named for hadoop1 binaries
- they had a shell script to produce a modified pom ..with -hadoop2 suffix
for the artifact names
- they built and published the hadoop2 binaries using the modified pom,
and hadoop1 binaries form the original pom
-roshan
On 4/1/15 1:24 PM, "Hari Shreedharan" <hs...@cloudera.com> wrote:
>I know sqoop2 does it. Maybe Jarcec can help?
>
>
>
>
>Thanks, Hari
>
>On Wed, Apr 1, 2015 at 1:13 PM, Roshan Naik <ro...@hortonworks.com>
>wrote:
>
>> To push artifacts for both hadoop2 and hadoop1 will need to name the
>> hadoop2 and hadoop1 artifacts differently. Tricky to do that with one
>>pom
>> I think.
>> -roshan
>> On 4/1/15 1:05 PM, "Hari Shreedharan" <hs...@cloudera.com> wrote:
>>>I am a +1 for doing this, since this is the last JDK 6 release. Maybe we
>>>should push artifacts for both Hadoop-1 and hbase-98 profiles this time
>>>and switch out to Hadoop 2 exclusively from 1.7?
>>>
>>>
>>>
>>>
>>>Thanks, Hari
>>>
>>>On Wed, Apr 1, 2015 at 1:04 PM, Roshan Naik <ro...@hortonworks.com>
>>>wrote:
>>>
>>>> Don't recall if this has been discussed before. Its been sometime
>>>>since
>>>>hadoop2 has been out.
>>>> Sooner or later Flume will switch to hadoop2 based builds (hbase98
>>>>profile?) as the default. Not sure if 1.6 is the time or worth waiting
>>>>longer.
>>>> -roshan
Re: Is 1.6 a good time to switch to hadoop2 ?
Posted by Hari Shreedharan <hs...@cloudera.com>.
I know sqoop2 does it. Maybe Jarcec can help?
Thanks, Hari
On Wed, Apr 1, 2015 at 1:13 PM, Roshan Naik <ro...@hortonworks.com>
wrote:
> To push artifacts for both hadoop2 and hadoop1 will need to name the
> hadoop2 and hadoop1 artifacts differently. Tricky to do that with one pom
> I think.
> -roshan
> On 4/1/15 1:05 PM, "Hari Shreedharan" <hs...@cloudera.com> wrote:
>>I am a +1 for doing this, since this is the last JDK 6 release. Maybe we
>>should push artifacts for both Hadoop-1 and hbase-98 profiles this time
>>and switch out to Hadoop 2 exclusively from 1.7?
>>
>>
>>
>>
>>Thanks, Hari
>>
>>On Wed, Apr 1, 2015 at 1:04 PM, Roshan Naik <ro...@hortonworks.com>
>>wrote:
>>
>>> Don't recall if this has been discussed before. Its been sometime since
>>>hadoop2 has been out.
>>> Sooner or later Flume will switch to hadoop2 based builds (hbase98
>>>profile?) as the default. Not sure if 1.6 is the time or worth waiting
>>>longer.
>>> -roshan
Re: Is 1.6 a good time to switch to hadoop2 ?
Posted by Roshan Naik <ro...@hortonworks.com>.
To push artifacts for both hadoop2 and hadoop1 will need to name the
hadoop2 and hadoop1 artifacts differently. Tricky to do that with one pom
I think.
-roshan
On 4/1/15 1:05 PM, "Hari Shreedharan" <hs...@cloudera.com> wrote:
>I am a +1 for doing this, since this is the last JDK 6 release. Maybe we
>should push artifacts for both Hadoop-1 and hbase-98 profiles this time
>and switch out to Hadoop 2 exclusively from 1.7?
>
>
>
>
>Thanks, Hari
>
>On Wed, Apr 1, 2015 at 1:04 PM, Roshan Naik <ro...@hortonworks.com>
>wrote:
>
>> Don't recall if this has been discussed before. Its been sometime since
>>hadoop2 has been out.
>> Sooner or later Flume will switch to hadoop2 based builds (hbase98
>>profile?) as the default. Not sure if 1.6 is the time or worth waiting
>>longer.
>> -roshan
Re: Is 1.6 a good time to switch to hadoop2 ?
Posted by Hari Shreedharan <hs...@cloudera.com>.
I am a +1 for doing this, since this is the last JDK 6 release. Maybe we should push artifacts for both Hadoop-1 and hbase-98 profiles this time and switch out to Hadoop 2 exclusively from 1.7?
Thanks, Hari
On Wed, Apr 1, 2015 at 1:04 PM, Roshan Naik <ro...@hortonworks.com>
wrote:
> Don't recall if this has been discussed before. Its been sometime since hadoop2 has been out.
> Sooner or later Flume will switch to hadoop2 based builds (hbase98 profile?) as the default. Not sure if 1.6 is the time or worth waiting longer.
> -roshan