You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Tim St Clair <ts...@redhat.com> on 2013/06/11 17:54:45 UTC
Bringing Hadoop to Fedora
Greetings Hadoop Development Community,
As some of you are aware, we've been looking to bring the upstream 2.X series into the Fedora channels. We believe that there are several benefits that this can bring to the community:
- Updated & streamlined rpm packaging
- Deeper level of system integration
- System managed dependencies (security + defect tracking)
- Proving ground for OpenJDK7
- Greater exposure for the community (both adoption and defect tracking)
...
During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop), we've come across some build issues as they relate to Fedora packaging guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these issues (listed below) are very minor dependency changes, and we were wondering if there are folks who would be willing to review.
fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623 (uber deprecated)
There are other patches we have in the queue but they still need to be flushed out, and in an effort to follow the KISS principles we figure this would be a good start.
Cheers,
Tim
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Thanks Arun!
Cheers,
Tim
----- Original Message -----
> From: "Arun C Murthy" <ac...@hortonworks.com>
> To: yarn-dev@hadoop.apache.org
> Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> Sent: Tuesday, June 11, 2013 12:54:11 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> Tim,
>
> This is great! I'll take a look at some of these patches, welcome!
>
> Arun
>
> On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
>
> > Greetings Hadoop Development Community,
> >
> > As some of you are aware, we've been looking to bring the upstream 2.X
> > series into the Fedora channels. We believe that there are several
> > benefits that this can bring to the community:
> >
> > - Updated & streamlined rpm packaging
> > - Deeper level of system integration
> > - System managed dependencies (security + defect tracking)
> > - Proving ground for OpenJDK7
> > - Greater exposure for the community (both adoption and defect tracking)
> > ...
> >
> > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop),
> > we've come across some build issues as they relate to Fedora packaging
> > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these
> > issues (listed below) are very minor dependency changes, and we were
> > wondering if there are folks who would be willing to review.
> >
> > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > (uber deprecated)
> >
> > There are other patches we have in the queue but they still need to be
> > flushed out, and in an effort to follow the KISS principles we figure this
> > would be a good start.
> >
> > Cheers,
> > Tim
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Thanks Arun!
Cheers,
Tim
----- Original Message -----
> From: "Arun C Murthy" <ac...@hortonworks.com>
> To: yarn-dev@hadoop.apache.org
> Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> Sent: Tuesday, June 11, 2013 12:54:11 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> Tim,
>
> This is great! I'll take a look at some of these patches, welcome!
>
> Arun
>
> On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
>
> > Greetings Hadoop Development Community,
> >
> > As some of you are aware, we've been looking to bring the upstream 2.X
> > series into the Fedora channels. We believe that there are several
> > benefits that this can bring to the community:
> >
> > - Updated & streamlined rpm packaging
> > - Deeper level of system integration
> > - System managed dependencies (security + defect tracking)
> > - Proving ground for OpenJDK7
> > - Greater exposure for the community (both adoption and defect tracking)
> > ...
> >
> > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop),
> > we've come across some build issues as they relate to Fedora packaging
> > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these
> > issues (listed below) are very minor dependency changes, and we were
> > wondering if there are folks who would be willing to review.
> >
> > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > (uber deprecated)
> >
> > There are other patches we have in the queue but they still need to be
> > flushed out, and in an effort to follow the KISS principles we figure this
> > would be a good start.
> >
> > Cheers,
> > Tim
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Thanks Arun!
Cheers,
Tim
----- Original Message -----
> From: "Arun C Murthy" <ac...@hortonworks.com>
> To: yarn-dev@hadoop.apache.org
> Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> Sent: Tuesday, June 11, 2013 12:54:11 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> Tim,
>
> This is great! I'll take a look at some of these patches, welcome!
>
> Arun
>
> On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
>
> > Greetings Hadoop Development Community,
> >
> > As some of you are aware, we've been looking to bring the upstream 2.X
> > series into the Fedora channels. We believe that there are several
> > benefits that this can bring to the community:
> >
> > - Updated & streamlined rpm packaging
> > - Deeper level of system integration
> > - System managed dependencies (security + defect tracking)
> > - Proving ground for OpenJDK7
> > - Greater exposure for the community (both adoption and defect tracking)
> > ...
> >
> > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop),
> > we've come across some build issues as they relate to Fedora packaging
> > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these
> > issues (listed below) are very minor dependency changes, and we were
> > wondering if there are folks who would be willing to review.
> >
> > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > (uber deprecated)
> >
> > There are other patches we have in the queue but they still need to be
> > flushed out, and in an effort to follow the KISS principles we figure this
> > would be a good start.
> >
> > Cheers,
> > Tim
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Thanks Alejandro.
comments inline below
----- Original Message -----
> From: "Alejandro Abdelnur" <tu...@cloudera.com>
> To: common-dev@hadoop.apache.org
> Sent: Monday, July 8, 2013 4:06:37 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> [moving bigtop to bcc]
>
> Tim,
>
> Except from HADOOP-9680 which has significant code changes and some false
> changes (which I did not go thru), all other changes seem OK.
>
> * Have you had a change to run ALL Hadoop testcases with them applied to
> make sure there are not regression?
>
Yes for everything, except (https://issues.apache.org/jira/browse/HADOOP-9680 / 9623).
> * Have you look at the output of running 'mvn dependency:tree' without/with
> the patches to verify unwanted dependencies are not sneaking in?
>
We've been tracking the dep-graph with a pretty close eye, but it would always be good to have another set of eyes gander at the delta.
> On HADOOP-9650 (not included in the above list):
Done on purpose at this point because the shift in Fedora 19 was just recently absorbed, and we are running through
tests and use cases to verify we didn't break the world ;-).
>
> Moving from Jetty6 to Jetty9 should be OK if there are not regressions. I
> was chatting with ToddL about the issues we had before in and according to
> Jetty guys they've been fixed in newer Jetty versions. Also, we are not
> using Jetty for shuffle anymore (we use Netty), so Jetty is not 'stress'
> that much anymore as it is used for the web UIs and for NN-SNN
> checkpointing.
>
> From the patch some code changes are required, once that is take care we
> should repeat the bullet item above for this patch.
>
> In addition, we have to see how this will play with other projects like
> HBase that are using HttpServer from hadoop-common. I'll forward this part
> of the email to their dev@ so they can watch/jump-in if necessary in the
> JIRA.
Agreed, which is part of the reason why we wanted to include BIGTOP on this thread, as they would likely
know more about the stack dependencies then we would.
>
> Thanks.
>
>
> On Mon, Jul 8, 2013 at 11:28 AM, Tim St Clair <ts...@redhat.com> wrote:
>
> > Arun,
> >
> > I realize you're probably pretty busy (or on vacation), but I figured I
> > would re-ping this thread to inquire about the status of the patch set
> > listed below.?.?
> >
> > https://issues.apache.org/jira/browse/HADOOP-9594
> > https://issues.apache.org/jira/browse/HADOOP-9610
> > https://issues.apache.org/jira/browse/HADOOP-9611
> > https://issues.apache.org/jira/browse/HADOOP-9613
> > https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
> >
> > Cheers,
> > Tim
> >
> > ----- Original Message -----
> > > From: "Arun C Murthy" <ac...@hortonworks.com>
> > > To: yarn-dev@hadoop.apache.org
> > > Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org,
> > hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> > > Sent: Tuesday, June 11, 2013 12:54:11 PM
> > > Subject: Re: Bringing Hadoop to Fedora
> > >
> > > Tim,
> > >
> > > This is great! I'll take a look at some of these patches, welcome!
> > >
> > > Arun
> > >
> > > On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> > >
> > > > Greetings Hadoop Development Community,
> > > >
> > > > As some of you are aware, we've been looking to bring the upstream 2.X
> > > > series into the Fedora channels. We believe that there are several
> > > > benefits that this can bring to the community:
> > > >
> > > > - Updated & streamlined rpm packaging
> > > > - Deeper level of system integration
> > > > - System managed dependencies (security + defect tracking)
> > > > - Proving ground for OpenJDK7
> > > > - Greater exposure for the community (both adoption and defect
> > tracking)
> > > > ...
> > > >
> > > > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop
> > ),
> > > > we've come across some build issues as they relate to Fedora packaging
> > > > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of
> > these
> > > > issues (listed below) are very minor dependency changes, and we were
> > > > wondering if there are folks who would be willing to review.
> > > >
> > > > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > > > fedora-patch-collections:
> > https://issues.apache.org/jira/browse/HADOOP-9610
> > > > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > > > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > > > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > > > (uber deprecated)
> > > >
> > > > There are other patches we have in the queue but they still need to be
> > > > flushed out, and in an effort to follow the KISS principles we figure
> > this
> > > > would be a good start.
> > > >
> > > > Cheers,
> > > > Tim
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > >
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Bringing Hadoop to Fedora
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
suresh,
thanks for jumping in from webhdfs side.
apparently, the issues that were affecting us with jetty 6 have been address with newer jetty versions.
my take would be we move forward with the version change and we make sure webhdfs is not affected, do you have a test that exercises webhdfs under load? that would definitely help
thx
Alejandro
(phone typing)
On Jul 8, 2013, at 17:43, Suresh Srinivas <su...@hortonworks.com> wrote:
> Isn't Jetty used by WebHDFS? Given that, Jetty performance is still
> important.
>
>
> On Mon, Jul 8, 2013 at 2:06 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> [moving bigtop to bcc]
>>
>> Tim,
>>
>> Except from HADOOP-9680 which has significant code changes and some false
>> changes (which I did not go thru), all other changes seem OK.
>>
>> * Have you had a change to run ALL Hadoop testcases with them applied to
>> make sure there are not regression?
>>
>> * Have you look at the output of running 'mvn dependency:tree' without/with
>> the patches to verify unwanted dependencies are not sneaking in?
>>
>> On HADOOP-9650 (not included in the above list):
>>
>> Moving from Jetty6 to Jetty9 should be OK if there are not regressions. I
>> was chatting with ToddL about the issues we had before in and according to
>> Jetty guys they've been fixed in newer Jetty versions. Also, we are not
>> using Jetty for shuffle anymore (we use Netty), so Jetty is not 'stress'
>> that much anymore as it is used for the web UIs and for NN-SNN
>> checkpointing.
>>
>> From the patch some code changes are required, once that is take care we
>> should repeat the bullet item above for this patch.
>>
>> In addition, we have to see how this will play with other projects like
>> HBase that are using HttpServer from hadoop-common. I'll forward this part
>> of the email to their dev@ so they can watch/jump-in if necessary in the
>> JIRA.
>>
>> Thanks.
>>
>>
>> On Mon, Jul 8, 2013 at 11:28 AM, Tim St Clair <ts...@redhat.com> wrote:
>>
>>> Arun,
>>>
>>> I realize you're probably pretty busy (or on vacation), but I figured I
>>> would re-ping this thread to inquire about the status of the patch set
>>> listed below.?.?
>>>
>>> https://issues.apache.org/jira/browse/HADOOP-9594
>>> https://issues.apache.org/jira/browse/HADOOP-9610
>>> https://issues.apache.org/jira/browse/HADOOP-9611
>>> https://issues.apache.org/jira/browse/HADOOP-9613
>>> https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
>>>
>>> Cheers,
>>> Tim
>>>
>>> ----- Original Message -----
>>>> From: "Arun C Murthy" <ac...@hortonworks.com>
>>>> To: yarn-dev@hadoop.apache.org
>>>> Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org,
>>> hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
>>>> Sent: Tuesday, June 11, 2013 12:54:11 PM
>>>> Subject: Re: Bringing Hadoop to Fedora
>>>>
>>>> Tim,
>>>>
>>>> This is great! I'll take a look at some of these patches, welcome!
>>>>
>>>> Arun
>>>>
>>>> On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
>>>>
>>>>> Greetings Hadoop Development Community,
>>>>>
>>>>> As some of you are aware, we've been looking to bring the upstream
>> 2.X
>>>>> series into the Fedora channels. We believe that there are several
>>>>> benefits that this can bring to the community:
>>>>>
>>>>> - Updated & streamlined rpm packaging
>>>>> - Deeper level of system integration
>>>>> - System managed dependencies (security + defect tracking)
>>>>> - Proving ground for OpenJDK7
>>>>> - Greater exposure for the community (both adoption and defect
>>> tracking)
>>>>> ...
>>>>>
>>>>> During our evaluation (
>> https://fedoraproject.org/wiki/Features/Hadoop
>>> ),
>>>>> we've come across some build issues as they relate to Fedora
>> packaging
>>>>> guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of
>>> these
>>>>> issues (listed below) are very minor dependency changes, and we were
>>>>> wondering if there are folks who would be willing to review.
>>>>>
>>>>> fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
>>>>> fedora-patch-collections:
>>> https://issues.apache.org/jira/browse/HADOOP-9610
>>>>> fedora-patch-cglib:
>> https://issues.apache.org/jira/browse/HADOOP-9611
>>>>> fedora-patch-jersey:
>> https://issues.apache.org/jira/browse/HADOOP-9613
>>>>> fedora-patch-jets3t:
>> https://issues.apache.org/jira/browse/HADOOP-9623
>>>>> (uber deprecated)
>>>>>
>>>>> There are other patches we have in the queue but they still need to
>> be
>>>>> flushed out, and in an effort to follow the KISS principles we figure
>>> this
>>>>> would be a good start.
>>>>>
>>>>> Cheers,
>>>>> Tim
>>>>
>>>> --
>>>> Arun C. Murthy
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>
>>
>>
>> --
>> Alejandro
>
>
>
> --
> http://hortonworks.com/download/
Re: Bringing Hadoop to Fedora
Posted by Suresh Srinivas <su...@hortonworks.com>.
Isn't Jetty used by WebHDFS? Given that, Jetty performance is still
important.
On Mon, Jul 8, 2013 at 2:06 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> [moving bigtop to bcc]
>
> Tim,
>
> Except from HADOOP-9680 which has significant code changes and some false
> changes (which I did not go thru), all other changes seem OK.
>
> * Have you had a change to run ALL Hadoop testcases with them applied to
> make sure there are not regression?
>
> * Have you look at the output of running 'mvn dependency:tree' without/with
> the patches to verify unwanted dependencies are not sneaking in?
>
> On HADOOP-9650 (not included in the above list):
>
> Moving from Jetty6 to Jetty9 should be OK if there are not regressions. I
> was chatting with ToddL about the issues we had before in and according to
> Jetty guys they've been fixed in newer Jetty versions. Also, we are not
> using Jetty for shuffle anymore (we use Netty), so Jetty is not 'stress'
> that much anymore as it is used for the web UIs and for NN-SNN
> checkpointing.
>
> From the patch some code changes are required, once that is take care we
> should repeat the bullet item above for this patch.
>
> In addition, we have to see how this will play with other projects like
> HBase that are using HttpServer from hadoop-common. I'll forward this part
> of the email to their dev@ so they can watch/jump-in if necessary in the
> JIRA.
>
> Thanks.
>
>
> On Mon, Jul 8, 2013 at 11:28 AM, Tim St Clair <ts...@redhat.com> wrote:
>
> > Arun,
> >
> > I realize you're probably pretty busy (or on vacation), but I figured I
> > would re-ping this thread to inquire about the status of the patch set
> > listed below.?.?
> >
> > https://issues.apache.org/jira/browse/HADOOP-9594
> > https://issues.apache.org/jira/browse/HADOOP-9610
> > https://issues.apache.org/jira/browse/HADOOP-9611
> > https://issues.apache.org/jira/browse/HADOOP-9613
> > https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
> >
> > Cheers,
> > Tim
> >
> > ----- Original Message -----
> > > From: "Arun C Murthy" <ac...@hortonworks.com>
> > > To: yarn-dev@hadoop.apache.org
> > > Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org,
> > hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> > > Sent: Tuesday, June 11, 2013 12:54:11 PM
> > > Subject: Re: Bringing Hadoop to Fedora
> > >
> > > Tim,
> > >
> > > This is great! I'll take a look at some of these patches, welcome!
> > >
> > > Arun
> > >
> > > On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> > >
> > > > Greetings Hadoop Development Community,
> > > >
> > > > As some of you are aware, we've been looking to bring the upstream
> 2.X
> > > > series into the Fedora channels. We believe that there are several
> > > > benefits that this can bring to the community:
> > > >
> > > > - Updated & streamlined rpm packaging
> > > > - Deeper level of system integration
> > > > - System managed dependencies (security + defect tracking)
> > > > - Proving ground for OpenJDK7
> > > > - Greater exposure for the community (both adoption and defect
> > tracking)
> > > > ...
> > > >
> > > > During our evaluation (
> https://fedoraproject.org/wiki/Features/Hadoop
> > ),
> > > > we've come across some build issues as they relate to Fedora
> packaging
> > > > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of
> > these
> > > > issues (listed below) are very minor dependency changes, and we were
> > > > wondering if there are folks who would be willing to review.
> > > >
> > > > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > > > fedora-patch-collections:
> > https://issues.apache.org/jira/browse/HADOOP-9610
> > > > fedora-patch-cglib:
> https://issues.apache.org/jira/browse/HADOOP-9611
> > > > fedora-patch-jersey:
> https://issues.apache.org/jira/browse/HADOOP-9613
> > > > fedora-patch-jets3t:
> https://issues.apache.org/jira/browse/HADOOP-9623
> > > > (uber deprecated)
> > > >
> > > > There are other patches we have in the queue but they still need to
> be
> > > > flushed out, and in an effort to follow the KISS principles we figure
> > this
> > > > would be a good start.
> > > >
> > > > Cheers,
> > > > Tim
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > >
> > >
> >
>
>
>
> --
> Alejandro
>
--
http://hortonworks.com/download/
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Thanks Alejandro.
comments inline below
----- Original Message -----
> From: "Alejandro Abdelnur" <tu...@cloudera.com>
> To: common-dev@hadoop.apache.org
> Sent: Monday, July 8, 2013 4:06:37 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> [moving bigtop to bcc]
>
> Tim,
>
> Except from HADOOP-9680 which has significant code changes and some false
> changes (which I did not go thru), all other changes seem OK.
>
> * Have you had a change to run ALL Hadoop testcases with them applied to
> make sure there are not regression?
>
Yes for everything, except (https://issues.apache.org/jira/browse/HADOOP-9680 / 9623).
> * Have you look at the output of running 'mvn dependency:tree' without/with
> the patches to verify unwanted dependencies are not sneaking in?
>
We've been tracking the dep-graph with a pretty close eye, but it would always be good to have another set of eyes gander at the delta.
> On HADOOP-9650 (not included in the above list):
Done on purpose at this point because the shift in Fedora 19 was just recently absorbed, and we are running through
tests and use cases to verify we didn't break the world ;-).
>
> Moving from Jetty6 to Jetty9 should be OK if there are not regressions. I
> was chatting with ToddL about the issues we had before in and according to
> Jetty guys they've been fixed in newer Jetty versions. Also, we are not
> using Jetty for shuffle anymore (we use Netty), so Jetty is not 'stress'
> that much anymore as it is used for the web UIs and for NN-SNN
> checkpointing.
>
> From the patch some code changes are required, once that is take care we
> should repeat the bullet item above for this patch.
>
> In addition, we have to see how this will play with other projects like
> HBase that are using HttpServer from hadoop-common. I'll forward this part
> of the email to their dev@ so they can watch/jump-in if necessary in the
> JIRA.
Agreed, which is part of the reason why we wanted to include BIGTOP on this thread, as they would likely
know more about the stack dependencies then we would.
>
> Thanks.
>
>
> On Mon, Jul 8, 2013 at 11:28 AM, Tim St Clair <ts...@redhat.com> wrote:
>
> > Arun,
> >
> > I realize you're probably pretty busy (or on vacation), but I figured I
> > would re-ping this thread to inquire about the status of the patch set
> > listed below.?.?
> >
> > https://issues.apache.org/jira/browse/HADOOP-9594
> > https://issues.apache.org/jira/browse/HADOOP-9610
> > https://issues.apache.org/jira/browse/HADOOP-9611
> > https://issues.apache.org/jira/browse/HADOOP-9613
> > https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
> >
> > Cheers,
> > Tim
> >
> > ----- Original Message -----
> > > From: "Arun C Murthy" <ac...@hortonworks.com>
> > > To: yarn-dev@hadoop.apache.org
> > > Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org,
> > hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> > > Sent: Tuesday, June 11, 2013 12:54:11 PM
> > > Subject: Re: Bringing Hadoop to Fedora
> > >
> > > Tim,
> > >
> > > This is great! I'll take a look at some of these patches, welcome!
> > >
> > > Arun
> > >
> > > On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> > >
> > > > Greetings Hadoop Development Community,
> > > >
> > > > As some of you are aware, we've been looking to bring the upstream 2.X
> > > > series into the Fedora channels. We believe that there are several
> > > > benefits that this can bring to the community:
> > > >
> > > > - Updated & streamlined rpm packaging
> > > > - Deeper level of system integration
> > > > - System managed dependencies (security + defect tracking)
> > > > - Proving ground for OpenJDK7
> > > > - Greater exposure for the community (both adoption and defect
> > tracking)
> > > > ...
> > > >
> > > > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop
> > ),
> > > > we've come across some build issues as they relate to Fedora packaging
> > > > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of
> > these
> > > > issues (listed below) are very minor dependency changes, and we were
> > > > wondering if there are folks who would be willing to review.
> > > >
> > > > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > > > fedora-patch-collections:
> > https://issues.apache.org/jira/browse/HADOOP-9610
> > > > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > > > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > > > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > > > (uber deprecated)
> > > >
> > > > There are other patches we have in the queue but they still need to be
> > > > flushed out, and in an effort to follow the KISS principles we figure
> > this
> > > > would be a good start.
> > > >
> > > > Cheers,
> > > > Tim
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > >
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Bringing Hadoop to Fedora
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
[moving bigtop to bcc]
Tim,
Except from HADOOP-9680 which has significant code changes and some false
changes (which I did not go thru), all other changes seem OK.
* Have you had a change to run ALL Hadoop testcases with them applied to
make sure there are not regression?
* Have you look at the output of running 'mvn dependency:tree' without/with
the patches to verify unwanted dependencies are not sneaking in?
On HADOOP-9650 (not included in the above list):
Moving from Jetty6 to Jetty9 should be OK if there are not regressions. I
was chatting with ToddL about the issues we had before in and according to
Jetty guys they've been fixed in newer Jetty versions. Also, we are not
using Jetty for shuffle anymore (we use Netty), so Jetty is not 'stress'
that much anymore as it is used for the web UIs and for NN-SNN
checkpointing.
>From the patch some code changes are required, once that is take care we
should repeat the bullet item above for this patch.
In addition, we have to see how this will play with other projects like
HBase that are using HttpServer from hadoop-common. I'll forward this part
of the email to their dev@ so they can watch/jump-in if necessary in the
JIRA.
Thanks.
On Mon, Jul 8, 2013 at 11:28 AM, Tim St Clair <ts...@redhat.com> wrote:
> Arun,
>
> I realize you're probably pretty busy (or on vacation), but I figured I
> would re-ping this thread to inquire about the status of the patch set
> listed below.?.?
>
> https://issues.apache.org/jira/browse/HADOOP-9594
> https://issues.apache.org/jira/browse/HADOOP-9610
> https://issues.apache.org/jira/browse/HADOOP-9611
> https://issues.apache.org/jira/browse/HADOOP-9613
> https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
>
> Cheers,
> Tim
>
> ----- Original Message -----
> > From: "Arun C Murthy" <ac...@hortonworks.com>
> > To: yarn-dev@hadoop.apache.org
> > Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org,
> hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> > Sent: Tuesday, June 11, 2013 12:54:11 PM
> > Subject: Re: Bringing Hadoop to Fedora
> >
> > Tim,
> >
> > This is great! I'll take a look at some of these patches, welcome!
> >
> > Arun
> >
> > On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> >
> > > Greetings Hadoop Development Community,
> > >
> > > As some of you are aware, we've been looking to bring the upstream 2.X
> > > series into the Fedora channels. We believe that there are several
> > > benefits that this can bring to the community:
> > >
> > > - Updated & streamlined rpm packaging
> > > - Deeper level of system integration
> > > - System managed dependencies (security + defect tracking)
> > > - Proving ground for OpenJDK7
> > > - Greater exposure for the community (both adoption and defect
> tracking)
> > > ...
> > >
> > > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop
> ),
> > > we've come across some build issues as they relate to Fedora packaging
> > > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of
> these
> > > issues (listed below) are very minor dependency changes, and we were
> > > wondering if there are folks who would be willing to review.
> > >
> > > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > > fedora-patch-collections:
> https://issues.apache.org/jira/browse/HADOOP-9610
> > > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > > (uber deprecated)
> > >
> > > There are other patches we have in the queue but they still need to be
> > > flushed out, and in an effort to follow the KISS principles we figure
> this
> > > would be a good start.
> > >
> > > Cheers,
> > > Tim
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
> >
>
--
Alejandro
Re: Bringing Hadoop to Fedora
Posted by Steve Loughran <st...@hortonworks.com>.
On 8 July 2013 19:28, Tim St Clair <ts...@redhat.com> wrote:
> Arun,
>
>
> https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
>
>
fixing S3 is something that is niggling me as something I want to sit down
and do once the Swift stuff is in -and once we've clarified some quirks w/
the FS API, and got some more FS contract tests.
There's been about 3 move to JetS3t 9 JIRAs, they'll need to be merged in,
along with a test that tries to do a many GB file upload; the kind of test
you'd only do in-EC2.
I'm certainly not going to go near this until august; if other people can
play with it too that'd be great -but it does need some good stressing.
Re: Bringing Hadoop to Fedora
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
[moving bigtop to bcc]
Tim,
Except from HADOOP-9680 which has significant code changes and some false
changes (which I did not go thru), all other changes seem OK.
* Have you had a change to run ALL Hadoop testcases with them applied to
make sure there are not regression?
* Have you look at the output of running 'mvn dependency:tree' without/with
the patches to verify unwanted dependencies are not sneaking in?
On HADOOP-9650 (not included in the above list):
Moving from Jetty6 to Jetty9 should be OK if there are not regressions. I
was chatting with ToddL about the issues we had before in and according to
Jetty guys they've been fixed in newer Jetty versions. Also, we are not
using Jetty for shuffle anymore (we use Netty), so Jetty is not 'stress'
that much anymore as it is used for the web UIs and for NN-SNN
checkpointing.
>From the patch some code changes are required, once that is take care we
should repeat the bullet item above for this patch.
In addition, we have to see how this will play with other projects like
HBase that are using HttpServer from hadoop-common. I'll forward this part
of the email to their dev@ so they can watch/jump-in if necessary in the
JIRA.
Thanks.
On Mon, Jul 8, 2013 at 11:28 AM, Tim St Clair <ts...@redhat.com> wrote:
> Arun,
>
> I realize you're probably pretty busy (or on vacation), but I figured I
> would re-ping this thread to inquire about the status of the patch set
> listed below.?.?
>
> https://issues.apache.org/jira/browse/HADOOP-9594
> https://issues.apache.org/jira/browse/HADOOP-9610
> https://issues.apache.org/jira/browse/HADOOP-9611
> https://issues.apache.org/jira/browse/HADOOP-9613
> https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
>
> Cheers,
> Tim
>
> ----- Original Message -----
> > From: "Arun C Murthy" <ac...@hortonworks.com>
> > To: yarn-dev@hadoop.apache.org
> > Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org,
> hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> > Sent: Tuesday, June 11, 2013 12:54:11 PM
> > Subject: Re: Bringing Hadoop to Fedora
> >
> > Tim,
> >
> > This is great! I'll take a look at some of these patches, welcome!
> >
> > Arun
> >
> > On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> >
> > > Greetings Hadoop Development Community,
> > >
> > > As some of you are aware, we've been looking to bring the upstream 2.X
> > > series into the Fedora channels. We believe that there are several
> > > benefits that this can bring to the community:
> > >
> > > - Updated & streamlined rpm packaging
> > > - Deeper level of system integration
> > > - System managed dependencies (security + defect tracking)
> > > - Proving ground for OpenJDK7
> > > - Greater exposure for the community (both adoption and defect
> tracking)
> > > ...
> > >
> > > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop
> ),
> > > we've come across some build issues as they relate to Fedora packaging
> > > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of
> these
> > > issues (listed below) are very minor dependency changes, and we were
> > > wondering if there are folks who would be willing to review.
> > >
> > > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > > fedora-patch-collections:
> https://issues.apache.org/jira/browse/HADOOP-9610
> > > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > > (uber deprecated)
> > >
> > > There are other patches we have in the queue but they still need to be
> > > flushed out, and in an effort to follow the KISS principles we figure
> this
> > > would be a good start.
> > >
> > > Cheers,
> > > Tim
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
> >
>
--
Alejandro
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Arun,
I realize you're probably pretty busy (or on vacation), but I figured I would re-ping this thread to inquire about the status of the patch set listed below.?.?
https://issues.apache.org/jira/browse/HADOOP-9594
https://issues.apache.org/jira/browse/HADOOP-9610
https://issues.apache.org/jira/browse/HADOOP-9611
https://issues.apache.org/jira/browse/HADOOP-9613
https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
Cheers,
Tim
----- Original Message -----
> From: "Arun C Murthy" <ac...@hortonworks.com>
> To: yarn-dev@hadoop.apache.org
> Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> Sent: Tuesday, June 11, 2013 12:54:11 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> Tim,
>
> This is great! I'll take a look at some of these patches, welcome!
>
> Arun
>
> On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
>
> > Greetings Hadoop Development Community,
> >
> > As some of you are aware, we've been looking to bring the upstream 2.X
> > series into the Fedora channels. We believe that there are several
> > benefits that this can bring to the community:
> >
> > - Updated & streamlined rpm packaging
> > - Deeper level of system integration
> > - System managed dependencies (security + defect tracking)
> > - Proving ground for OpenJDK7
> > - Greater exposure for the community (both adoption and defect tracking)
> > ...
> >
> > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop),
> > we've come across some build issues as they relate to Fedora packaging
> > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these
> > issues (listed below) are very minor dependency changes, and we were
> > wondering if there are folks who would be willing to review.
> >
> > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > (uber deprecated)
> >
> > There are other patches we have in the queue but they still need to be
> > flushed out, and in an effort to follow the KISS principles we figure this
> > would be a good start.
> >
> > Cheers,
> > Tim
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Arun,
I realize you're probably pretty busy (or on vacation), but I figured I would re-ping this thread to inquire about the status of the patch set listed below.?.?
https://issues.apache.org/jira/browse/HADOOP-9594
https://issues.apache.org/jira/browse/HADOOP-9610
https://issues.apache.org/jira/browse/HADOOP-9611
https://issues.apache.org/jira/browse/HADOOP-9613
https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
Cheers,
Tim
----- Original Message -----
> From: "Arun C Murthy" <ac...@hortonworks.com>
> To: yarn-dev@hadoop.apache.org
> Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> Sent: Tuesday, June 11, 2013 12:54:11 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> Tim,
>
> This is great! I'll take a look at some of these patches, welcome!
>
> Arun
>
> On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
>
> > Greetings Hadoop Development Community,
> >
> > As some of you are aware, we've been looking to bring the upstream 2.X
> > series into the Fedora channels. We believe that there are several
> > benefits that this can bring to the community:
> >
> > - Updated & streamlined rpm packaging
> > - Deeper level of system integration
> > - System managed dependencies (security + defect tracking)
> > - Proving ground for OpenJDK7
> > - Greater exposure for the community (both adoption and defect tracking)
> > ...
> >
> > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop),
> > we've come across some build issues as they relate to Fedora packaging
> > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these
> > issues (listed below) are very minor dependency changes, and we were
> > wondering if there are folks who would be willing to review.
> >
> > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > (uber deprecated)
> >
> > There are other patches we have in the queue but they still need to be
> > flushed out, and in an effort to follow the KISS principles we figure this
> > would be a good start.
> >
> > Cheers,
> > Tim
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Re: Bringing Hadoop to Fedora
Posted by Arun C Murthy <ac...@hortonworks.com>.
Tim,
This is great! I'll take a look at some of these patches, welcome!
Arun
On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> Greetings Hadoop Development Community,
>
> As some of you are aware, we've been looking to bring the upstream 2.X series into the Fedora channels. We believe that there are several benefits that this can bring to the community:
>
> - Updated & streamlined rpm packaging
> - Deeper level of system integration
> - System managed dependencies (security + defect tracking)
> - Proving ground for OpenJDK7
> - Greater exposure for the community (both adoption and defect tracking)
> ...
>
> During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop), we've come across some build issues as they relate to Fedora packaging guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these issues (listed below) are very minor dependency changes, and we were wondering if there are folks who would be willing to review.
>
> fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623 (uber deprecated)
>
> There are other patches we have in the queue but they still need to be flushed out, and in an effort to follow the KISS principles we figure this would be a good start.
>
> Cheers,
> Tim
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Bringing Hadoop to Fedora
Posted by Arun C Murthy <ac...@hortonworks.com>.
Tim,
This is great! I'll take a look at some of these patches, welcome!
Arun
On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> Greetings Hadoop Development Community,
>
> As some of you are aware, we've been looking to bring the upstream 2.X series into the Fedora channels. We believe that there are several benefits that this can bring to the community:
>
> - Updated & streamlined rpm packaging
> - Deeper level of system integration
> - System managed dependencies (security + defect tracking)
> - Proving ground for OpenJDK7
> - Greater exposure for the community (both adoption and defect tracking)
> ...
>
> During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop), we've come across some build issues as they relate to Fedora packaging guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these issues (listed below) are very minor dependency changes, and we were wondering if there are folks who would be willing to review.
>
> fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623 (uber deprecated)
>
> There are other patches we have in the queue but they still need to be flushed out, and in an effort to follow the KISS principles we figure this would be a good start.
>
> Cheers,
> Tim
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Bringing Hadoop to Fedora
Posted by Arun C Murthy <ac...@hortonworks.com>.
Tim,
This is great! I'll take a look at some of these patches, welcome!
Arun
On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> Greetings Hadoop Development Community,
>
> As some of you are aware, we've been looking to bring the upstream 2.X series into the Fedora channels. We believe that there are several benefits that this can bring to the community:
>
> - Updated & streamlined rpm packaging
> - Deeper level of system integration
> - System managed dependencies (security + defect tracking)
> - Proving ground for OpenJDK7
> - Greater exposure for the community (both adoption and defect tracking)
> ...
>
> During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop), we've come across some build issues as they relate to Fedora packaging guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these issues (listed below) are very minor dependency changes, and we were wondering if there are folks who would be willing to review.
>
> fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623 (uber deprecated)
>
> There are other patches we have in the queue but they still need to be flushed out, and in an effort to follow the KISS principles we figure this would be a good start.
>
> Cheers,
> Tim
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Bringing Hadoop to Fedora
Posted by Arun C Murthy <ac...@hortonworks.com>.
Tim,
This is great! I'll take a look at some of these patches, welcome!
Arun
On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> Greetings Hadoop Development Community,
>
> As some of you are aware, we've been looking to bring the upstream 2.X series into the Fedora channels. We believe that there are several benefits that this can bring to the community:
>
> - Updated & streamlined rpm packaging
> - Deeper level of system integration
> - System managed dependencies (security + defect tracking)
> - Proving ground for OpenJDK7
> - Greater exposure for the community (both adoption and defect tracking)
> ...
>
> During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop), we've come across some build issues as they relate to Fedora packaging guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these issues (listed below) are very minor dependency changes, and we were wondering if there are folks who would be willing to review.
>
> fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623 (uber deprecated)
>
> There are other patches we have in the queue but they still need to be flushed out, and in an effort to follow the KISS principles we figure this would be a good start.
>
> Cheers,
> Tim
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Bringing Hadoop to Fedora
Posted by Arun C Murthy <ac...@hortonworks.com>.
Tim,
This is great! I'll take a look at some of these patches, welcome!
Arun
On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> Greetings Hadoop Development Community,
>
> As some of you are aware, we've been looking to bring the upstream 2.X series into the Fedora channels. We believe that there are several benefits that this can bring to the community:
>
> - Updated & streamlined rpm packaging
> - Deeper level of system integration
> - System managed dependencies (security + defect tracking)
> - Proving ground for OpenJDK7
> - Greater exposure for the community (both adoption and defect tracking)
> ...
>
> During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop), we've come across some build issues as they relate to Fedora packaging guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these issues (listed below) are very minor dependency changes, and we were wondering if there are folks who would be willing to review.
>
> fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623 (uber deprecated)
>
> There are other patches we have in the queue but they still need to be flushed out, and in an effort to follow the KISS principles we figure this would be a good start.
>
> Cheers,
> Tim
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Hi Marcos -
Currently we have a list of outstanding items that can be found here:
https://fedoraproject.org/wiki/Features/Hadoop
and are tentatively targeting the Fedora 20 time-frame.
Feel free to checkout https://fedoraproject.org/wiki/Big_data_SIG for contact details, and we welcome any folks who interested.
Cheers
Tim
----- Original Message -----
> From: "Marcos Luis Ortiz Valmaseda" <ma...@gmail.com>
> To: rvs@apache.org
> Cc: dev@bigtop.apache.org, tstclair@redhat.com
> Sent: Tuesday, June 11, 2013 6:44:50 PM
> Subject: Re: Bringing Hadoop to Fedora
> It would be nice to help on this development in the new Fedora release (19).
> Tim, Can we work together on this?
> 2013/6/11 Roman Shaposhnik < rvs@apache.org >
> > On Tue, Jun 11, 2013 at 8:54 AM, Tim St Clair < tstclair@redhat.com >
> > wrote:
>
> > > Greetings Hadoop Development Community,
>
> > >
>
> > > As some of you are aware, we've been looking to bring the upstream 2.X
> > > series into the Fedora channels.
>
> > That's pretty awesome! Now, here's a couple question I've got:
>
> > 1. is your focus exclusively on Hadoop or would you want to bring
>
> > Hadoop ecosystem projects into Fedora as well?
>
> > 2. since Hadoop is fundamentally a distributed application what
>
> > are you plans wrt. deployment mechanism? Do you guys envision
>
> > anything along the lines of juju?
>
> > 3. can you elaborate a little bit on how do you see this effort leveraging
>
> > Apache Bigtop?
>
> > Thanks,
>
> > Roman.
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
Re: Bringing Hadoop to Fedora
Posted by Marcos Luis Ortiz Valmaseda <ma...@gmail.com>.
It would be nice to help on this development in the new Fedora release
(19).
Tim, Can we work together on this?
2013/6/11 Roman Shaposhnik <rv...@apache.org>
> On Tue, Jun 11, 2013 at 8:54 AM, Tim St Clair <ts...@redhat.com> wrote:
> > Greetings Hadoop Development Community,
> >
> > As some of you are aware, we've been looking to bring the upstream 2.X
> series into the Fedora channels.
>
> That's pretty awesome! Now, here's a couple question I've got:
> 1. is your focus exclusively on Hadoop or would you want to bring
> Hadoop ecosystem projects into Fedora as well?
>
> 2. since Hadoop is fundamentally a distributed application what
> are you plans wrt. deployment mechanism? Do you guys envision
> anything along the lines of juju?
>
> 3. can you elaborate a little bit on how do you see this effort
> leveraging
> Apache Bigtop?
>
> Thanks,
> Roman.
>
--
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz
Re: Bringing Hadoop to Fedora
Posted by Konstantin Boudnik <co...@apache.org>.
It is definitely nice to see such a major group of open-source supporters
joining Bigtop effort! I am sure our project and community can greatly benefit
from your knowledge in RPMs and other RH/Fedora specific issues.
As you might know, Bigtop is distro agnostic stack framework, and we devide
out attention equally between Ubuntu, SUSE, and Fedora/CentOS. My guess is
that you would be most interested in the latter part of the distro set.
Hence, for your convenience here's the list of outstanding RPM specific issues
https://issues.apache.org/jira/issues/?jql=project%20%3D%20BIGTOP%20AND%20resolution%20%3D%20Unresolved%20AND%20component%20%3D%20RPM%20ORDER%20BY%20priority%20DESC
that you might consider worthy helping with! I am sure we all are looking
forward for the contributions and new patches!
Welcome aboard, guys!
Cos
On Tue, Jun 11, 2013 at 11:17PM, Tim St Clair wrote:
> Hi Roman -
>
> ----- Original Message -----
> > From: "Roman Shaposhnik" <rv...@apache.org>
> > To: dev@bigtop.apache.org, tstclair@redhat.com
> > Cc: general@hadoop.apache.org
> > Sent: Tuesday, June 11, 2013 3:02:41 PM
> > Subject: Re: Bringing Hadoop to Fedora
> >
> > On Tue, Jun 11, 2013 at 8:54 AM, Tim St Clair <ts...@redhat.com> wrote:
> > > Greetings Hadoop Development Community,
> > >
> > > As some of you are aware, we've been looking to bring the upstream 2.X
> > > series into the Fedora channels.
> >
> > That's pretty awesome! Now, here's a couple question I've got:
> > 1. is your focus exclusively on Hadoop or would you want to bring
> > Hadoop ecosystem projects into Fedora as well?
> >
>
> At this stage "I'm" primarily focused on Hadoop, and will defer to matt(cc'd) on ecosystem scope.
>
> > 2. since Hadoop is fundamentally a distributed application what
> > are you plans wrt. deployment mechanism? Do you guys envision
> > anything along the lines of juju?
>
> It's too early at this point for us to determine best distribution+deployment practices for hadoop. We've had plenty of experience dealing with other distributed applications at very large scales, but we would be all ears to hear the experiences of others.
>
> >
> > 3. can you elaborate a little bit on how do you see this effort leveraging
> > Apache Bigtop?
>
> We've been looking @ numerous rpms, including BigTop, and have steadily been making modifications to adhere to Fedora packaging guidelines, and systems integration. We would be happy to chat sometime and here your thoughts.
>
> >
> > Thanks,
> > Roman.
> >
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Hi Roman -
----- Original Message -----
> From: "Roman Shaposhnik" <rv...@apache.org>
> To: dev@bigtop.apache.org, tstclair@redhat.com
> Cc: general@hadoop.apache.org
> Sent: Tuesday, June 11, 2013 3:02:41 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> On Tue, Jun 11, 2013 at 8:54 AM, Tim St Clair <ts...@redhat.com> wrote:
> > Greetings Hadoop Development Community,
> >
> > As some of you are aware, we've been looking to bring the upstream 2.X
> > series into the Fedora channels.
>
> That's pretty awesome! Now, here's a couple question I've got:
> 1. is your focus exclusively on Hadoop or would you want to bring
> Hadoop ecosystem projects into Fedora as well?
>
At this stage "I'm" primarily focused on Hadoop, and will defer to matt(cc'd) on ecosystem scope.
> 2. since Hadoop is fundamentally a distributed application what
> are you plans wrt. deployment mechanism? Do you guys envision
> anything along the lines of juju?
It's too early at this point for us to determine best distribution+deployment practices for hadoop. We've had plenty of experience dealing with other distributed applications at very large scales, but we would be all ears to hear the experiences of others.
>
> 3. can you elaborate a little bit on how do you see this effort leveraging
> Apache Bigtop?
We've been looking @ numerous rpms, including BigTop, and have steadily been making modifications to adhere to Fedora packaging guidelines, and systems integration. We would be happy to chat sometime and here your thoughts.
>
> Thanks,
> Roman.
>
Re: Bringing Hadoop to Fedora
Posted by Arun C Murthy <ac...@hortonworks.com>.
Can we please keep these discussions on the *-dev@ lists? Thanks.
On Jun 11, 2013, at 2:23 PM, Marcos Luis Ortiz Valmaseda wrote:
> Well, Roman. I was reading yesterday about the new Big Data SIG, and
> this group is focused not just in Hadoop, but in several projects too:
>
> http://fedoraproject.org/wiki/Big_data_SIG
> http://www.samkottler.com/the-fedora-big-data-sig
>
> Best wishes
>
> 2013/6/11, Roman Shaposhnik <rv...@apache.org>:
>> On Tue, Jun 11, 2013 at 8:54 AM, Tim St Clair <ts...@redhat.com> wrote:
>>> Greetings Hadoop Development Community,
>>>
>>> As some of you are aware, we've been looking to bring the upstream 2.X
>>> series into the Fedora channels.
>>
>> That's pretty awesome! Now, here's a couple question I've got:
>> 1. is your focus exclusively on Hadoop or would you want to bring
>> Hadoop ecosystem projects into Fedora as well?
>>
>> 2. since Hadoop is fundamentally a distributed application what
>> are you plans wrt. deployment mechanism? Do you guys envision
>> anything along the lines of juju?
>>
>> 3. can you elaborate a little bit on how do you see this effort
>> leveraging
>> Apache Bigtop?
>>
>> Thanks,
>> Roman.
>>
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Bringing Hadoop to Fedora
Posted by Arun C Murthy <ac...@hortonworks.com>.
Can we please keep these discussions on the *-dev@ lists? Thanks.
On Jun 11, 2013, at 2:23 PM, Marcos Luis Ortiz Valmaseda wrote:
> Well, Roman. I was reading yesterday about the new Big Data SIG, and
> this group is focused not just in Hadoop, but in several projects too:
>
> http://fedoraproject.org/wiki/Big_data_SIG
> http://www.samkottler.com/the-fedora-big-data-sig
>
> Best wishes
>
> 2013/6/11, Roman Shaposhnik <rv...@apache.org>:
>> On Tue, Jun 11, 2013 at 8:54 AM, Tim St Clair <ts...@redhat.com> wrote:
>>> Greetings Hadoop Development Community,
>>>
>>> As some of you are aware, we've been looking to bring the upstream 2.X
>>> series into the Fedora channels.
>>
>> That's pretty awesome! Now, here's a couple question I've got:
>> 1. is your focus exclusively on Hadoop or would you want to bring
>> Hadoop ecosystem projects into Fedora as well?
>>
>> 2. since Hadoop is fundamentally a distributed application what
>> are you plans wrt. deployment mechanism? Do you guys envision
>> anything along the lines of juju?
>>
>> 3. can you elaborate a little bit on how do you see this effort
>> leveraging
>> Apache Bigtop?
>>
>> Thanks,
>> Roman.
>>
>
>
> --
> Marcos Ortiz Valmaseda
> Product Manager at PDVSA
> http://about.me/marcosortiz
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Bringing Hadoop to Fedora
Posted by Marcos Luis Ortiz Valmaseda <ma...@gmail.com>.
Well, Roman. I was reading yesterday about the new Big Data SIG, and
this group is focused not just in Hadoop, but in several projects too:
http://fedoraproject.org/wiki/Big_data_SIG
http://www.samkottler.com/the-fedora-big-data-sig
Best wishes
2013/6/11, Roman Shaposhnik <rv...@apache.org>:
> On Tue, Jun 11, 2013 at 8:54 AM, Tim St Clair <ts...@redhat.com> wrote:
>> Greetings Hadoop Development Community,
>>
>> As some of you are aware, we've been looking to bring the upstream 2.X
>> series into the Fedora channels.
>
> That's pretty awesome! Now, here's a couple question I've got:
> 1. is your focus exclusively on Hadoop or would you want to bring
> Hadoop ecosystem projects into Fedora as well?
>
> 2. since Hadoop is fundamentally a distributed application what
> are you plans wrt. deployment mechanism? Do you guys envision
> anything along the lines of juju?
>
> 3. can you elaborate a little bit on how do you see this effort
> leveraging
> Apache Bigtop?
>
> Thanks,
> Roman.
>
--
Marcos Ortiz Valmaseda
Product Manager at PDVSA
http://about.me/marcosortiz
Re: Bringing Hadoop to Fedora
Posted by Roman Shaposhnik <rv...@apache.org>.
On Tue, Jun 11, 2013 at 8:54 AM, Tim St Clair <ts...@redhat.com> wrote:
> Greetings Hadoop Development Community,
>
> As some of you are aware, we've been looking to bring the upstream 2.X series into the Fedora channels.
That's pretty awesome! Now, here's a couple question I've got:
1. is your focus exclusively on Hadoop or would you want to bring
Hadoop ecosystem projects into Fedora as well?
2. since Hadoop is fundamentally a distributed application what
are you plans wrt. deployment mechanism? Do you guys envision
anything along the lines of juju?
3. can you elaborate a little bit on how do you see this effort leveraging
Apache Bigtop?
Thanks,
Roman.
Re: Bringing Hadoop to Fedora
Posted by Roman Shaposhnik <rv...@apache.org>.
On Tue, Jun 11, 2013 at 8:54 AM, Tim St Clair <ts...@redhat.com> wrote:
> Greetings Hadoop Development Community,
>
> As some of you are aware, we've been looking to bring the upstream 2.X series into the Fedora channels.
That's pretty awesome! Now, here's a couple question I've got:
1. is your focus exclusively on Hadoop or would you want to bring
Hadoop ecosystem projects into Fedora as well?
2. since Hadoop is fundamentally a distributed application what
are you plans wrt. deployment mechanism? Do you guys envision
anything along the lines of juju?
3. can you elaborate a little bit on how do you see this effort leveraging
Apache Bigtop?
Thanks,
Roman.