You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@sqoop.apache.org by Venkat Ranganathan <vr...@hortonworks.com> on 2016/05/18 18:31:44 UTC
Sqoop branch 1.5.x
Proposal for Sqoop 1.5
We have Sqoop 1.4.x going on which is the production version of Sqoop, with support for ancient versions for Hadoop (from 0.20), Hive 0.7+ and HBase 0.94 among others.
There is a good amount of interest in contribution to Sqoop 1 as it is the current production version. But Sqoop has a few issues that make Hadoop 1.x is causing issues in bringing new features easily into Sqoop 1.x (for example getting Phoenix changes into Sqoop and potentially others waiting in the wings)
Also, we have been using Ant/Ivy based project, which is causing issues with component version management. We can potentially use a Maven profile based configuration to easily allow multiple component versions to have more flexibility in builds and packaging and how we publish artifacts
To that end here is what I propose (had a brief discussion with Jarcec last week) in the order of priority
Create a new Sqoop 1.5 branch where we
1. Deprecate support for Hadoop 1 and older versions of HBase (only support 1.0+) and Hive (only support 1.0+)
2. Mavenize the project
3. Clean up the package jumble in the code – only have org.apache.sqoop packages
4. Bring in all the new features that otherwise are difficult to bring in with older
What should we do with 1.4.x branch? My initial thought is that we do a 1.4.7 release with what is available and have 1.5.x as the branch to make further changes.
Thoughts?
Thanks
Venkat
Re: Sqoop branch 1.5.x
Posted by Venkat Ranganathan <vr...@hortonworks.com>.
Thanks for taking this up Sowmya.
Anybody interested in mentoring Sowmya as a community RM that we have been doing for a few releases. I can volunteer if needed
Thanks
Venkat
On 8/23/16, 1:57 PM, "Sowmya Ramesh" <sr...@hortonworks.com> wrote:
Hi All,
I got busy with other stuff and couldn¹t focus on this.
I would like to resume Sqoop release task.
As a first step I would like to work on 1.4.7 release with what is
available.
Please let me know if you have any concerns.
Thanks!
On 5/18/16, 12:05 PM, "Attila Szabo" <as...@cloudera.com> wrote:
>Hi,
>
>Until we have 1.5.x I would still advise to apply my proposed ivy changes
>on the trunk (maybe on some branches too).
>https://reviews.apache.org/r/47110/diff/1#index_header
>
>And I would also provide my help to Sowmya, especially around mavenizing.
>(I'm working in GMT+1 timezone, maybe we could leverage from the different
>timezones).
>
>Cheers,
>Attila
>
>
>On Wed, May 18, 2016 at 8:40 PM, Venkat Ranganathan <
>vranganathan@hortonworks.com> wrote:
>
>> Thanks Jarcec.
>>
>> Agreed, let us remove support for older versions of Hadoop 1, Hbase and
>> Hive
>>
>>
>> Sowmya, thanks for volunteering
>>
>> Venkat
>>
>>
>>
>> On 5/18/16, 11:38 AM, "Jarek Jarcec Cecho" <jarcec@gmail.com on behalf
>>of
>> jarcec@apache.org> wrote:
>>
>> >I like the proposal and I would second it.
>> >
>> >> 1. Deprecate support for Hadoop 1 and older versions of HBase
>>(only
>> support 1.0+) and Hive (only support 1.0+)
>> >
>> >I would even suggest to be more extreme and rather then ³deprecating² I
>> would directly remove that support.
>> >
>> >Jarcec
>> >
>> >> On May 18, 2016, at 11:31 AM, Venkat Ranganathan <
>> vranganathan@hortonworks.com> wrote:
>> >>
>> >> Proposal for Sqoop 1.5
>> >>
>> >> We have Sqoop 1.4.x going on which is the production version of
>>Sqoop,
>> with support for ancient versions for Hadoop (from 0.20), Hive 0.7+ and
>> HBase 0.94 among others.
>> >>
>> >> There is a good amount of interest in contribution to Sqoop 1 as it
>> is the current production version. But Sqoop has a few issues that make
>> Hadoop 1.x is causing issues in bringing new features easily into Sqoop
>> 1.x (for example getting Phoenix changes into Sqoop and potentially
>> others waiting in the wings)
>> >>
>> >> Also, we have been using Ant/Ivy based project, which is causing
>>issues
>> with component version management. We can potentially use a Maven
>>profile
>> based configuration to easily allow multiple component versions to have
>> more flexibility in builds and packaging and how we publish artifacts
>> >>
>> >> To that end here is what I propose (had a brief discussion with
>>Jarcec
>> last week) in the order of priority
>> >>
>> >> Create a new Sqoop 1.5 branch where we
>> >>
>> >>
>> >> 1. Deprecate support for Hadoop 1 and older versions of HBase
>>(only
>> support 1.0+) and Hive (only support 1.0+)
>> >>
>> >> 2. Mavenize the project
>> >>
>> >> 3. Clean up the package jumble in the code only have
>> org.apache.sqoop packages
>> >>
>> >> 4. Bring in all the new features that otherwise are difficult to
>> bring in with older
>> >>
>> >> What should we do with 1.4.x branch? My initial thought is that we
>>do
>> a 1.4.7 release with what is available and have 1.5.x as the branch to
>>make
>> further changes.
>> >>
>> >> Thoughts?
>> >>
>> >> Thanks
>> >>
>> >> Venkat
>> >
>> >
>>
>
>
>
>--
>Best regards,
>
>Attila Szabo
>Sotware Engineer
>
><http://www.cloudera.com>
Re: Sqoop branch 1.5.x
Posted by Sowmya Ramesh <sr...@hortonworks.com>.
Hi All,
I got busy with other stuff and couldn¹t focus on this.
I would like to resume Sqoop release task.
As a first step I would like to work on 1.4.7 release with what is
available.
Please let me know if you have any concerns.
Thanks!
On 5/18/16, 12:05 PM, "Attila Szabo" <as...@cloudera.com> wrote:
>Hi,
>
>Until we have 1.5.x I would still advise to apply my proposed ivy changes
>on the trunk (maybe on some branches too).
>https://reviews.apache.org/r/47110/diff/1#index_header
>
>And I would also provide my help to Sowmya, especially around mavenizing.
>(I'm working in GMT+1 timezone, maybe we could leverage from the different
>timezones).
>
>Cheers,
>Attila
>
>
>On Wed, May 18, 2016 at 8:40 PM, Venkat Ranganathan <
>vranganathan@hortonworks.com> wrote:
>
>> Thanks Jarcec.
>>
>> Agreed, let us remove support for older versions of Hadoop 1, Hbase and
>> Hive
>>
>>
>> Sowmya, thanks for volunteering
>>
>> Venkat
>>
>>
>>
>> On 5/18/16, 11:38 AM, "Jarek Jarcec Cecho" <jarcec@gmail.com on behalf
>>of
>> jarcec@apache.org> wrote:
>>
>> >I like the proposal and I would second it.
>> >
>> >> 1. Deprecate support for Hadoop 1 and older versions of HBase
>>(only
>> support 1.0+) and Hive (only support 1.0+)
>> >
>> >I would even suggest to be more extreme and rather then ³deprecating² I
>> would directly remove that support.
>> >
>> >Jarcec
>> >
>> >> On May 18, 2016, at 11:31 AM, Venkat Ranganathan <
>> vranganathan@hortonworks.com> wrote:
>> >>
>> >> Proposal for Sqoop 1.5
>> >>
>> >> We have Sqoop 1.4.x going on which is the production version of
>>Sqoop,
>> with support for ancient versions for Hadoop (from 0.20), Hive 0.7+ and
>> HBase 0.94 among others.
>> >>
>> >> There is a good amount of interest in contribution to Sqoop 1 as it
>> is the current production version. But Sqoop has a few issues that make
>> Hadoop 1.x is causing issues in bringing new features easily into Sqoop
>> 1.x (for example getting Phoenix changes into Sqoop and potentially
>> others waiting in the wings)
>> >>
>> >> Also, we have been using Ant/Ivy based project, which is causing
>>issues
>> with component version management. We can potentially use a Maven
>>profile
>> based configuration to easily allow multiple component versions to have
>> more flexibility in builds and packaging and how we publish artifacts
>> >>
>> >> To that end here is what I propose (had a brief discussion with
>>Jarcec
>> last week) in the order of priority
>> >>
>> >> Create a new Sqoop 1.5 branch where we
>> >>
>> >>
>> >> 1. Deprecate support for Hadoop 1 and older versions of HBase
>>(only
>> support 1.0+) and Hive (only support 1.0+)
>> >>
>> >> 2. Mavenize the project
>> >>
>> >> 3. Clean up the package jumble in the code only have
>> org.apache.sqoop packages
>> >>
>> >> 4. Bring in all the new features that otherwise are difficult to
>> bring in with older
>> >>
>> >> What should we do with 1.4.x branch? My initial thought is that we
>>do
>> a 1.4.7 release with what is available and have 1.5.x as the branch to
>>make
>> further changes.
>> >>
>> >> Thoughts?
>> >>
>> >> Thanks
>> >>
>> >> Venkat
>> >
>> >
>>
>
>
>
>--
>Best regards,
>
>Attila Szabo
>Sotware Engineer
>
><http://www.cloudera.com>
Re: Sqoop branch 1.5.x
Posted by Attila Szabo <as...@cloudera.com>.
Hi,
Until we have 1.5.x I would still advise to apply my proposed ivy changes
on the trunk (maybe on some branches too).
https://reviews.apache.org/r/47110/diff/1#index_header
And I would also provide my help to Sowmya, especially around mavenizing.
(I'm working in GMT+1 timezone, maybe we could leverage from the different
timezones).
Cheers,
Attila
On Wed, May 18, 2016 at 8:40 PM, Venkat Ranganathan <
vranganathan@hortonworks.com> wrote:
> Thanks Jarcec.
>
> Agreed, let us remove support for older versions of Hadoop 1, Hbase and
> Hive
>
>
> Sowmya, thanks for volunteering
>
> Venkat
>
>
>
> On 5/18/16, 11:38 AM, "Jarek Jarcec Cecho" <jarcec@gmail.com on behalf of
> jarcec@apache.org> wrote:
>
> >I like the proposal and I would second it.
> >
> >> 1. Deprecate support for Hadoop 1 and older versions of HBase (only
> support 1.0+) and Hive (only support 1.0+)
> >
> >I would even suggest to be more extreme and rather then “deprecating” I
> would directly remove that support.
> >
> >Jarcec
> >
> >> On May 18, 2016, at 11:31 AM, Venkat Ranganathan <
> vranganathan@hortonworks.com> wrote:
> >>
> >> Proposal for Sqoop 1.5
> >>
> >> We have Sqoop 1.4.x going on which is the production version of Sqoop,
> with support for ancient versions for Hadoop (from 0.20), Hive 0.7+ and
> HBase 0.94 among others.
> >>
> >> There is a good amount of interest in contribution to Sqoop 1 as it
> is the current production version. But Sqoop has a few issues that make
> Hadoop 1.x is causing issues in bringing new features easily into Sqoop
> 1.x (for example getting Phoenix changes into Sqoop and potentially
> others waiting in the wings)
> >>
> >> Also, we have been using Ant/Ivy based project, which is causing issues
> with component version management. We can potentially use a Maven profile
> based configuration to easily allow multiple component versions to have
> more flexibility in builds and packaging and how we publish artifacts
> >>
> >> To that end here is what I propose (had a brief discussion with Jarcec
> last week) in the order of priority
> >>
> >> Create a new Sqoop 1.5 branch where we
> >>
> >>
> >> 1. Deprecate support for Hadoop 1 and older versions of HBase (only
> support 1.0+) and Hive (only support 1.0+)
> >>
> >> 2. Mavenize the project
> >>
> >> 3. Clean up the package jumble in the code – only have
> org.apache.sqoop packages
> >>
> >> 4. Bring in all the new features that otherwise are difficult to
> bring in with older
> >>
> >> What should we do with 1.4.x branch? My initial thought is that we do
> a 1.4.7 release with what is available and have 1.5.x as the branch to make
> further changes.
> >>
> >> Thoughts?
> >>
> >> Thanks
> >>
> >> Venkat
> >
> >
>
--
Best regards,
Attila Szabo
Sotware Engineer
<http://www.cloudera.com>
Re: Sqoop branch 1.5.x
Posted by Venkat Ranganathan <vr...@hortonworks.com>.
Thanks Jarcec.
Agreed, let us remove support for older versions of Hadoop 1, Hbase and Hive
Sowmya, thanks for volunteering
Venkat
On 5/18/16, 11:38 AM, "Jarek Jarcec Cecho" <jarcec@gmail.com on behalf of jarcec@apache.org> wrote:
>I like the proposal and I would second it.
>
>> 1. Deprecate support for Hadoop 1 and older versions of HBase (only support 1.0+) and Hive (only support 1.0+)
>
>I would even suggest to be more extreme and rather then “deprecating” I would directly remove that support.
>
>Jarcec
>
>> On May 18, 2016, at 11:31 AM, Venkat Ranganathan <vr...@hortonworks.com> wrote:
>>
>> Proposal for Sqoop 1.5
>>
>> We have Sqoop 1.4.x going on which is the production version of Sqoop, with support for ancient versions for Hadoop (from 0.20), Hive 0.7+ and HBase 0.94 among others.
>>
>> There is a good amount of interest in contribution to Sqoop 1 as it is the current production version. But Sqoop has a few issues that make Hadoop 1.x is causing issues in bringing new features easily into Sqoop 1.x (for example getting Phoenix changes into Sqoop and potentially others waiting in the wings)
>>
>> Also, we have been using Ant/Ivy based project, which is causing issues with component version management. We can potentially use a Maven profile based configuration to easily allow multiple component versions to have more flexibility in builds and packaging and how we publish artifacts
>>
>> To that end here is what I propose (had a brief discussion with Jarcec last week) in the order of priority
>>
>> Create a new Sqoop 1.5 branch where we
>>
>>
>> 1. Deprecate support for Hadoop 1 and older versions of HBase (only support 1.0+) and Hive (only support 1.0+)
>>
>> 2. Mavenize the project
>>
>> 3. Clean up the package jumble in the code – only have org.apache.sqoop packages
>>
>> 4. Bring in all the new features that otherwise are difficult to bring in with older
>>
>> What should we do with 1.4.x branch? My initial thought is that we do a 1.4.7 release with what is available and have 1.5.x as the branch to make further changes.
>>
>> Thoughts?
>>
>> Thanks
>>
>> Venkat
>
>
Re: Sqoop branch 1.5.x
Posted by Jarek Jarcec Cecho <ja...@apache.org>.
I like the proposal and I would second it.
> 1. Deprecate support for Hadoop 1 and older versions of HBase (only support 1.0+) and Hive (only support 1.0+)
I would even suggest to be more extreme and rather then “deprecating” I would directly remove that support.
Jarcec
> On May 18, 2016, at 11:31 AM, Venkat Ranganathan <vr...@hortonworks.com> wrote:
>
> Proposal for Sqoop 1.5
>
> We have Sqoop 1.4.x going on which is the production version of Sqoop, with support for ancient versions for Hadoop (from 0.20), Hive 0.7+ and HBase 0.94 among others.
>
> There is a good amount of interest in contribution to Sqoop 1 as it is the current production version. But Sqoop has a few issues that make Hadoop 1.x is causing issues in bringing new features easily into Sqoop 1.x (for example getting Phoenix changes into Sqoop and potentially others waiting in the wings)
>
> Also, we have been using Ant/Ivy based project, which is causing issues with component version management. We can potentially use a Maven profile based configuration to easily allow multiple component versions to have more flexibility in builds and packaging and how we publish artifacts
>
> To that end here is what I propose (had a brief discussion with Jarcec last week) in the order of priority
>
> Create a new Sqoop 1.5 branch where we
>
>
> 1. Deprecate support for Hadoop 1 and older versions of HBase (only support 1.0+) and Hive (only support 1.0+)
>
> 2. Mavenize the project
>
> 3. Clean up the package jumble in the code – only have org.apache.sqoop packages
>
> 4. Bring in all the new features that otherwise are difficult to bring in with older
>
> What should we do with 1.4.x branch? My initial thought is that we do a 1.4.7 release with what is available and have 1.5.x as the branch to make further changes.
>
> Thoughts?
>
> Thanks
>
> Venkat
Re: Sqoop branch 1.5.x
Posted by Sowmya Ramesh <sr...@hortonworks.com>.
I would like to take up this task.
I will create the sub tasks after the discussion is closed.
Thanks,
Sowmya
On 5/18/16, 11:31 AM, "Venkat Ranganathan" <vr...@hortonworks.com>
wrote:
>Proposal for Sqoop 1.5
>
>We have Sqoop 1.4.x going on which is the production version of Sqoop,
>with support for ancient versions for Hadoop (from 0.20), Hive 0.7+ and
>HBase 0.94 among others.
>
>There is a good amount of interest in contribution to Sqoop 1 as it is
>the current production version. But Sqoop has a few issues that make
>Hadoop 1.x is causing issues in bringing new features easily into Sqoop
>1.x (for example getting Phoenix changes into Sqoop and potentially
>others waiting in the wings)
>
>Also, we have been using Ant/Ivy based project, which is causing issues
>with component version management. We can potentially use a Maven
>profile based configuration to easily allow multiple component versions
>to have more flexibility in builds and packaging and how we publish
>artifacts
>
>To that end here is what I propose (had a brief discussion with Jarcec
>last week) in the order of priority
>
>Create a new Sqoop 1.5 branch where we
>
>
>1. Deprecate support for Hadoop 1 and older versions of HBase (only
>support 1.0+) and Hive (only support 1.0+)
>
>2. Mavenize the project
>
>3. Clean up the package jumble in the code only have
>org.apache.sqoop packages
>
>4. Bring in all the new features that otherwise are difficult to
>bring in with older
>
>What should we do with 1.4.x branch? My initial thought is that we do a
>1.4.7 release with what is available and have 1.5.x as the branch to make
>further changes.
>
>Thoughts?
>
>Thanks
>
>Venkat