You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Arun C Murthy <ac...@hortonworks.com> on 2013/06/11 19:54:11 UTC
Re: Bringing Hadoop to Fedora
Tim,
This is great! I'll take a look at some of these patches, welcome!
Arun
On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> Greetings Hadoop Development Community,
>
> As some of you are aware, we've been looking to bring the upstream 2.X series into the Fedora channels. We believe that there are several benefits that this can bring to the community:
>
> - Updated & streamlined rpm packaging
> - Deeper level of system integration
> - System managed dependencies (security + defect tracking)
> - Proving ground for OpenJDK7
> - Greater exposure for the community (both adoption and defect tracking)
> ...
>
> During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop), we've come across some build issues as they relate to Fedora packaging guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these issues (listed below) are very minor dependency changes, and we were wondering if there are folks who would be willing to review.
>
> fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623 (uber deprecated)
>
> There are other patches we have in the queue but they still need to be flushed out, and in an effort to follow the KISS principles we figure this would be a good start.
>
> Cheers,
> Tim
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Thanks Arun!
Cheers,
Tim
----- Original Message -----
> From: "Arun C Murthy" <ac...@hortonworks.com>
> To: yarn-dev@hadoop.apache.org
> Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> Sent: Tuesday, June 11, 2013 12:54:11 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> Tim,
>
> This is great! I'll take a look at some of these patches, welcome!
>
> Arun
>
> On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
>
> > Greetings Hadoop Development Community,
> >
> > As some of you are aware, we've been looking to bring the upstream 2.X
> > series into the Fedora channels. We believe that there are several
> > benefits that this can bring to the community:
> >
> > - Updated & streamlined rpm packaging
> > - Deeper level of system integration
> > - System managed dependencies (security + defect tracking)
> > - Proving ground for OpenJDK7
> > - Greater exposure for the community (both adoption and defect tracking)
> > ...
> >
> > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop),
> > we've come across some build issues as they relate to Fedora packaging
> > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these
> > issues (listed below) are very minor dependency changes, and we were
> > wondering if there are folks who would be willing to review.
> >
> > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > (uber deprecated)
> >
> > There are other patches we have in the queue but they still need to be
> > flushed out, and in an effort to follow the KISS principles we figure this
> > would be a good start.
> >
> > Cheers,
> > Tim
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Thanks Arun!
Cheers,
Tim
----- Original Message -----
> From: "Arun C Murthy" <ac...@hortonworks.com>
> To: yarn-dev@hadoop.apache.org
> Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> Sent: Tuesday, June 11, 2013 12:54:11 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> Tim,
>
> This is great! I'll take a look at some of these patches, welcome!
>
> Arun
>
> On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
>
> > Greetings Hadoop Development Community,
> >
> > As some of you are aware, we've been looking to bring the upstream 2.X
> > series into the Fedora channels. We believe that there are several
> > benefits that this can bring to the community:
> >
> > - Updated & streamlined rpm packaging
> > - Deeper level of system integration
> > - System managed dependencies (security + defect tracking)
> > - Proving ground for OpenJDK7
> > - Greater exposure for the community (both adoption and defect tracking)
> > ...
> >
> > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop),
> > we've come across some build issues as they relate to Fedora packaging
> > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these
> > issues (listed below) are very minor dependency changes, and we were
> > wondering if there are folks who would be willing to review.
> >
> > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > (uber deprecated)
> >
> > There are other patches we have in the queue but they still need to be
> > flushed out, and in an effort to follow the KISS principles we figure this
> > would be a good start.
> >
> > Cheers,
> > Tim
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Thanks Arun!
Cheers,
Tim
----- Original Message -----
> From: "Arun C Murthy" <ac...@hortonworks.com>
> To: yarn-dev@hadoop.apache.org
> Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> Sent: Tuesday, June 11, 2013 12:54:11 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> Tim,
>
> This is great! I'll take a look at some of these patches, welcome!
>
> Arun
>
> On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
>
> > Greetings Hadoop Development Community,
> >
> > As some of you are aware, we've been looking to bring the upstream 2.X
> > series into the Fedora channels. We believe that there are several
> > benefits that this can bring to the community:
> >
> > - Updated & streamlined rpm packaging
> > - Deeper level of system integration
> > - System managed dependencies (security + defect tracking)
> > - Proving ground for OpenJDK7
> > - Greater exposure for the community (both adoption and defect tracking)
> > ...
> >
> > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop),
> > we've come across some build issues as they relate to Fedora packaging
> > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these
> > issues (listed below) are very minor dependency changes, and we were
> > wondering if there are folks who would be willing to review.
> >
> > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > (uber deprecated)
> >
> > There are other patches we have in the queue but they still need to be
> > flushed out, and in an effort to follow the KISS principles we figure this
> > would be a good start.
> >
> > Cheers,
> > Tim
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Thanks Alejandro.
comments inline below
----- Original Message -----
> From: "Alejandro Abdelnur" <tu...@cloudera.com>
> To: common-dev@hadoop.apache.org
> Sent: Monday, July 8, 2013 4:06:37 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> [moving bigtop to bcc]
>
> Tim,
>
> Except from HADOOP-9680 which has significant code changes and some false
> changes (which I did not go thru), all other changes seem OK.
>
> * Have you had a change to run ALL Hadoop testcases with them applied to
> make sure there are not regression?
>
Yes for everything, except (https://issues.apache.org/jira/browse/HADOOP-9680 / 9623).
> * Have you look at the output of running 'mvn dependency:tree' without/with
> the patches to verify unwanted dependencies are not sneaking in?
>
We've been tracking the dep-graph with a pretty close eye, but it would always be good to have another set of eyes gander at the delta.
> On HADOOP-9650 (not included in the above list):
Done on purpose at this point because the shift in Fedora 19 was just recently absorbed, and we are running through
tests and use cases to verify we didn't break the world ;-).
>
> Moving from Jetty6 to Jetty9 should be OK if there are not regressions. I
> was chatting with ToddL about the issues we had before in and according to
> Jetty guys they've been fixed in newer Jetty versions. Also, we are not
> using Jetty for shuffle anymore (we use Netty), so Jetty is not 'stress'
> that much anymore as it is used for the web UIs and for NN-SNN
> checkpointing.
>
> From the patch some code changes are required, once that is take care we
> should repeat the bullet item above for this patch.
>
> In addition, we have to see how this will play with other projects like
> HBase that are using HttpServer from hadoop-common. I'll forward this part
> of the email to their dev@ so they can watch/jump-in if necessary in the
> JIRA.
Agreed, which is part of the reason why we wanted to include BIGTOP on this thread, as they would likely
know more about the stack dependencies then we would.
>
> Thanks.
>
>
> On Mon, Jul 8, 2013 at 11:28 AM, Tim St Clair <ts...@redhat.com> wrote:
>
> > Arun,
> >
> > I realize you're probably pretty busy (or on vacation), but I figured I
> > would re-ping this thread to inquire about the status of the patch set
> > listed below.?.?
> >
> > https://issues.apache.org/jira/browse/HADOOP-9594
> > https://issues.apache.org/jira/browse/HADOOP-9610
> > https://issues.apache.org/jira/browse/HADOOP-9611
> > https://issues.apache.org/jira/browse/HADOOP-9613
> > https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
> >
> > Cheers,
> > Tim
> >
> > ----- Original Message -----
> > > From: "Arun C Murthy" <ac...@hortonworks.com>
> > > To: yarn-dev@hadoop.apache.org
> > > Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org,
> > hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> > > Sent: Tuesday, June 11, 2013 12:54:11 PM
> > > Subject: Re: Bringing Hadoop to Fedora
> > >
> > > Tim,
> > >
> > > This is great! I'll take a look at some of these patches, welcome!
> > >
> > > Arun
> > >
> > > On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> > >
> > > > Greetings Hadoop Development Community,
> > > >
> > > > As some of you are aware, we've been looking to bring the upstream 2.X
> > > > series into the Fedora channels. We believe that there are several
> > > > benefits that this can bring to the community:
> > > >
> > > > - Updated & streamlined rpm packaging
> > > > - Deeper level of system integration
> > > > - System managed dependencies (security + defect tracking)
> > > > - Proving ground for OpenJDK7
> > > > - Greater exposure for the community (both adoption and defect
> > tracking)
> > > > ...
> > > >
> > > > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop
> > ),
> > > > we've come across some build issues as they relate to Fedora packaging
> > > > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of
> > these
> > > > issues (listed below) are very minor dependency changes, and we were
> > > > wondering if there are folks who would be willing to review.
> > > >
> > > > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > > > fedora-patch-collections:
> > https://issues.apache.org/jira/browse/HADOOP-9610
> > > > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > > > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > > > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > > > (uber deprecated)
> > > >
> > > > There are other patches we have in the queue but they still need to be
> > > > flushed out, and in an effort to follow the KISS principles we figure
> > this
> > > > would be a good start.
> > > >
> > > > Cheers,
> > > > Tim
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > >
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Bringing Hadoop to Fedora
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
suresh,
thanks for jumping in from webhdfs side.
apparently, the issues that were affecting us with jetty 6 have been address with newer jetty versions.
my take would be we move forward with the version change and we make sure webhdfs is not affected, do you have a test that exercises webhdfs under load? that would definitely help
thx
Alejandro
(phone typing)
On Jul 8, 2013, at 17:43, Suresh Srinivas <su...@hortonworks.com> wrote:
> Isn't Jetty used by WebHDFS? Given that, Jetty performance is still
> important.
>
>
> On Mon, Jul 8, 2013 at 2:06 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> [moving bigtop to bcc]
>>
>> Tim,
>>
>> Except from HADOOP-9680 which has significant code changes and some false
>> changes (which I did not go thru), all other changes seem OK.
>>
>> * Have you had a change to run ALL Hadoop testcases with them applied to
>> make sure there are not regression?
>>
>> * Have you look at the output of running 'mvn dependency:tree' without/with
>> the patches to verify unwanted dependencies are not sneaking in?
>>
>> On HADOOP-9650 (not included in the above list):
>>
>> Moving from Jetty6 to Jetty9 should be OK if there are not regressions. I
>> was chatting with ToddL about the issues we had before in and according to
>> Jetty guys they've been fixed in newer Jetty versions. Also, we are not
>> using Jetty for shuffle anymore (we use Netty), so Jetty is not 'stress'
>> that much anymore as it is used for the web UIs and for NN-SNN
>> checkpointing.
>>
>> From the patch some code changes are required, once that is take care we
>> should repeat the bullet item above for this patch.
>>
>> In addition, we have to see how this will play with other projects like
>> HBase that are using HttpServer from hadoop-common. I'll forward this part
>> of the email to their dev@ so they can watch/jump-in if necessary in the
>> JIRA.
>>
>> Thanks.
>>
>>
>> On Mon, Jul 8, 2013 at 11:28 AM, Tim St Clair <ts...@redhat.com> wrote:
>>
>>> Arun,
>>>
>>> I realize you're probably pretty busy (or on vacation), but I figured I
>>> would re-ping this thread to inquire about the status of the patch set
>>> listed below.?.?
>>>
>>> https://issues.apache.org/jira/browse/HADOOP-9594
>>> https://issues.apache.org/jira/browse/HADOOP-9610
>>> https://issues.apache.org/jira/browse/HADOOP-9611
>>> https://issues.apache.org/jira/browse/HADOOP-9613
>>> https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
>>>
>>> Cheers,
>>> Tim
>>>
>>> ----- Original Message -----
>>>> From: "Arun C Murthy" <ac...@hortonworks.com>
>>>> To: yarn-dev@hadoop.apache.org
>>>> Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org,
>>> hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
>>>> Sent: Tuesday, June 11, 2013 12:54:11 PM
>>>> Subject: Re: Bringing Hadoop to Fedora
>>>>
>>>> Tim,
>>>>
>>>> This is great! I'll take a look at some of these patches, welcome!
>>>>
>>>> Arun
>>>>
>>>> On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
>>>>
>>>>> Greetings Hadoop Development Community,
>>>>>
>>>>> As some of you are aware, we've been looking to bring the upstream
>> 2.X
>>>>> series into the Fedora channels. We believe that there are several
>>>>> benefits that this can bring to the community:
>>>>>
>>>>> - Updated & streamlined rpm packaging
>>>>> - Deeper level of system integration
>>>>> - System managed dependencies (security + defect tracking)
>>>>> - Proving ground for OpenJDK7
>>>>> - Greater exposure for the community (both adoption and defect
>>> tracking)
>>>>> ...
>>>>>
>>>>> During our evaluation (
>> https://fedoraproject.org/wiki/Features/Hadoop
>>> ),
>>>>> we've come across some build issues as they relate to Fedora
>> packaging
>>>>> guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of
>>> these
>>>>> issues (listed below) are very minor dependency changes, and we were
>>>>> wondering if there are folks who would be willing to review.
>>>>>
>>>>> fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
>>>>> fedora-patch-collections:
>>> https://issues.apache.org/jira/browse/HADOOP-9610
>>>>> fedora-patch-cglib:
>> https://issues.apache.org/jira/browse/HADOOP-9611
>>>>> fedora-patch-jersey:
>> https://issues.apache.org/jira/browse/HADOOP-9613
>>>>> fedora-patch-jets3t:
>> https://issues.apache.org/jira/browse/HADOOP-9623
>>>>> (uber deprecated)
>>>>>
>>>>> There are other patches we have in the queue but they still need to
>> be
>>>>> flushed out, and in an effort to follow the KISS principles we figure
>>> this
>>>>> would be a good start.
>>>>>
>>>>> Cheers,
>>>>> Tim
>>>>
>>>> --
>>>> Arun C. Murthy
>>>> Hortonworks Inc.
>>>> http://hortonworks.com/
>>
>>
>>
>> --
>> Alejandro
>
>
>
> --
> http://hortonworks.com/download/
Re: Bringing Hadoop to Fedora
Posted by Suresh Srinivas <su...@hortonworks.com>.
Isn't Jetty used by WebHDFS? Given that, Jetty performance is still
important.
On Mon, Jul 8, 2013 at 2:06 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> [moving bigtop to bcc]
>
> Tim,
>
> Except from HADOOP-9680 which has significant code changes and some false
> changes (which I did not go thru), all other changes seem OK.
>
> * Have you had a change to run ALL Hadoop testcases with them applied to
> make sure there are not regression?
>
> * Have you look at the output of running 'mvn dependency:tree' without/with
> the patches to verify unwanted dependencies are not sneaking in?
>
> On HADOOP-9650 (not included in the above list):
>
> Moving from Jetty6 to Jetty9 should be OK if there are not regressions. I
> was chatting with ToddL about the issues we had before in and according to
> Jetty guys they've been fixed in newer Jetty versions. Also, we are not
> using Jetty for shuffle anymore (we use Netty), so Jetty is not 'stress'
> that much anymore as it is used for the web UIs and for NN-SNN
> checkpointing.
>
> From the patch some code changes are required, once that is take care we
> should repeat the bullet item above for this patch.
>
> In addition, we have to see how this will play with other projects like
> HBase that are using HttpServer from hadoop-common. I'll forward this part
> of the email to their dev@ so they can watch/jump-in if necessary in the
> JIRA.
>
> Thanks.
>
>
> On Mon, Jul 8, 2013 at 11:28 AM, Tim St Clair <ts...@redhat.com> wrote:
>
> > Arun,
> >
> > I realize you're probably pretty busy (or on vacation), but I figured I
> > would re-ping this thread to inquire about the status of the patch set
> > listed below.?.?
> >
> > https://issues.apache.org/jira/browse/HADOOP-9594
> > https://issues.apache.org/jira/browse/HADOOP-9610
> > https://issues.apache.org/jira/browse/HADOOP-9611
> > https://issues.apache.org/jira/browse/HADOOP-9613
> > https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
> >
> > Cheers,
> > Tim
> >
> > ----- Original Message -----
> > > From: "Arun C Murthy" <ac...@hortonworks.com>
> > > To: yarn-dev@hadoop.apache.org
> > > Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org,
> > hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> > > Sent: Tuesday, June 11, 2013 12:54:11 PM
> > > Subject: Re: Bringing Hadoop to Fedora
> > >
> > > Tim,
> > >
> > > This is great! I'll take a look at some of these patches, welcome!
> > >
> > > Arun
> > >
> > > On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> > >
> > > > Greetings Hadoop Development Community,
> > > >
> > > > As some of you are aware, we've been looking to bring the upstream
> 2.X
> > > > series into the Fedora channels. We believe that there are several
> > > > benefits that this can bring to the community:
> > > >
> > > > - Updated & streamlined rpm packaging
> > > > - Deeper level of system integration
> > > > - System managed dependencies (security + defect tracking)
> > > > - Proving ground for OpenJDK7
> > > > - Greater exposure for the community (both adoption and defect
> > tracking)
> > > > ...
> > > >
> > > > During our evaluation (
> https://fedoraproject.org/wiki/Features/Hadoop
> > ),
> > > > we've come across some build issues as they relate to Fedora
> packaging
> > > > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of
> > these
> > > > issues (listed below) are very minor dependency changes, and we were
> > > > wondering if there are folks who would be willing to review.
> > > >
> > > > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > > > fedora-patch-collections:
> > https://issues.apache.org/jira/browse/HADOOP-9610
> > > > fedora-patch-cglib:
> https://issues.apache.org/jira/browse/HADOOP-9611
> > > > fedora-patch-jersey:
> https://issues.apache.org/jira/browse/HADOOP-9613
> > > > fedora-patch-jets3t:
> https://issues.apache.org/jira/browse/HADOOP-9623
> > > > (uber deprecated)
> > > >
> > > > There are other patches we have in the queue but they still need to
> be
> > > > flushed out, and in an effort to follow the KISS principles we figure
> > this
> > > > would be a good start.
> > > >
> > > > Cheers,
> > > > Tim
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > >
> > >
> >
>
>
>
> --
> Alejandro
>
--
http://hortonworks.com/download/
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Thanks Alejandro.
comments inline below
----- Original Message -----
> From: "Alejandro Abdelnur" <tu...@cloudera.com>
> To: common-dev@hadoop.apache.org
> Sent: Monday, July 8, 2013 4:06:37 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> [moving bigtop to bcc]
>
> Tim,
>
> Except from HADOOP-9680 which has significant code changes and some false
> changes (which I did not go thru), all other changes seem OK.
>
> * Have you had a change to run ALL Hadoop testcases with them applied to
> make sure there are not regression?
>
Yes for everything, except (https://issues.apache.org/jira/browse/HADOOP-9680 / 9623).
> * Have you look at the output of running 'mvn dependency:tree' without/with
> the patches to verify unwanted dependencies are not sneaking in?
>
We've been tracking the dep-graph with a pretty close eye, but it would always be good to have another set of eyes gander at the delta.
> On HADOOP-9650 (not included in the above list):
Done on purpose at this point because the shift in Fedora 19 was just recently absorbed, and we are running through
tests and use cases to verify we didn't break the world ;-).
>
> Moving from Jetty6 to Jetty9 should be OK if there are not regressions. I
> was chatting with ToddL about the issues we had before in and according to
> Jetty guys they've been fixed in newer Jetty versions. Also, we are not
> using Jetty for shuffle anymore (we use Netty), so Jetty is not 'stress'
> that much anymore as it is used for the web UIs and for NN-SNN
> checkpointing.
>
> From the patch some code changes are required, once that is take care we
> should repeat the bullet item above for this patch.
>
> In addition, we have to see how this will play with other projects like
> HBase that are using HttpServer from hadoop-common. I'll forward this part
> of the email to their dev@ so they can watch/jump-in if necessary in the
> JIRA.
Agreed, which is part of the reason why we wanted to include BIGTOP on this thread, as they would likely
know more about the stack dependencies then we would.
>
> Thanks.
>
>
> On Mon, Jul 8, 2013 at 11:28 AM, Tim St Clair <ts...@redhat.com> wrote:
>
> > Arun,
> >
> > I realize you're probably pretty busy (or on vacation), but I figured I
> > would re-ping this thread to inquire about the status of the patch set
> > listed below.?.?
> >
> > https://issues.apache.org/jira/browse/HADOOP-9594
> > https://issues.apache.org/jira/browse/HADOOP-9610
> > https://issues.apache.org/jira/browse/HADOOP-9611
> > https://issues.apache.org/jira/browse/HADOOP-9613
> > https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
> >
> > Cheers,
> > Tim
> >
> > ----- Original Message -----
> > > From: "Arun C Murthy" <ac...@hortonworks.com>
> > > To: yarn-dev@hadoop.apache.org
> > > Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org,
> > hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> > > Sent: Tuesday, June 11, 2013 12:54:11 PM
> > > Subject: Re: Bringing Hadoop to Fedora
> > >
> > > Tim,
> > >
> > > This is great! I'll take a look at some of these patches, welcome!
> > >
> > > Arun
> > >
> > > On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> > >
> > > > Greetings Hadoop Development Community,
> > > >
> > > > As some of you are aware, we've been looking to bring the upstream 2.X
> > > > series into the Fedora channels. We believe that there are several
> > > > benefits that this can bring to the community:
> > > >
> > > > - Updated & streamlined rpm packaging
> > > > - Deeper level of system integration
> > > > - System managed dependencies (security + defect tracking)
> > > > - Proving ground for OpenJDK7
> > > > - Greater exposure for the community (both adoption and defect
> > tracking)
> > > > ...
> > > >
> > > > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop
> > ),
> > > > we've come across some build issues as they relate to Fedora packaging
> > > > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of
> > these
> > > > issues (listed below) are very minor dependency changes, and we were
> > > > wondering if there are folks who would be willing to review.
> > > >
> > > > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > > > fedora-patch-collections:
> > https://issues.apache.org/jira/browse/HADOOP-9610
> > > > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > > > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > > > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > > > (uber deprecated)
> > > >
> > > > There are other patches we have in the queue but they still need to be
> > > > flushed out, and in an effort to follow the KISS principles we figure
> > this
> > > > would be a good start.
> > > >
> > > > Cheers,
> > > > Tim
> > >
> > > --
> > > Arun C. Murthy
> > > Hortonworks Inc.
> > > http://hortonworks.com/
> > >
> > >
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Bringing Hadoop to Fedora
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
[moving bigtop to bcc]
Tim,
Except from HADOOP-9680 which has significant code changes and some false
changes (which I did not go thru), all other changes seem OK.
* Have you had a change to run ALL Hadoop testcases with them applied to
make sure there are not regression?
* Have you look at the output of running 'mvn dependency:tree' without/with
the patches to verify unwanted dependencies are not sneaking in?
On HADOOP-9650 (not included in the above list):
Moving from Jetty6 to Jetty9 should be OK if there are not regressions. I
was chatting with ToddL about the issues we had before in and according to
Jetty guys they've been fixed in newer Jetty versions. Also, we are not
using Jetty for shuffle anymore (we use Netty), so Jetty is not 'stress'
that much anymore as it is used for the web UIs and for NN-SNN
checkpointing.
>From the patch some code changes are required, once that is take care we
should repeat the bullet item above for this patch.
In addition, we have to see how this will play with other projects like
HBase that are using HttpServer from hadoop-common. I'll forward this part
of the email to their dev@ so they can watch/jump-in if necessary in the
JIRA.
Thanks.
On Mon, Jul 8, 2013 at 11:28 AM, Tim St Clair <ts...@redhat.com> wrote:
> Arun,
>
> I realize you're probably pretty busy (or on vacation), but I figured I
> would re-ping this thread to inquire about the status of the patch set
> listed below.?.?
>
> https://issues.apache.org/jira/browse/HADOOP-9594
> https://issues.apache.org/jira/browse/HADOOP-9610
> https://issues.apache.org/jira/browse/HADOOP-9611
> https://issues.apache.org/jira/browse/HADOOP-9613
> https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
>
> Cheers,
> Tim
>
> ----- Original Message -----
> > From: "Arun C Murthy" <ac...@hortonworks.com>
> > To: yarn-dev@hadoop.apache.org
> > Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org,
> hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> > Sent: Tuesday, June 11, 2013 12:54:11 PM
> > Subject: Re: Bringing Hadoop to Fedora
> >
> > Tim,
> >
> > This is great! I'll take a look at some of these patches, welcome!
> >
> > Arun
> >
> > On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> >
> > > Greetings Hadoop Development Community,
> > >
> > > As some of you are aware, we've been looking to bring the upstream 2.X
> > > series into the Fedora channels. We believe that there are several
> > > benefits that this can bring to the community:
> > >
> > > - Updated & streamlined rpm packaging
> > > - Deeper level of system integration
> > > - System managed dependencies (security + defect tracking)
> > > - Proving ground for OpenJDK7
> > > - Greater exposure for the community (both adoption and defect
> tracking)
> > > ...
> > >
> > > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop
> ),
> > > we've come across some build issues as they relate to Fedora packaging
> > > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of
> these
> > > issues (listed below) are very minor dependency changes, and we were
> > > wondering if there are folks who would be willing to review.
> > >
> > > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > > fedora-patch-collections:
> https://issues.apache.org/jira/browse/HADOOP-9610
> > > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > > (uber deprecated)
> > >
> > > There are other patches we have in the queue but they still need to be
> > > flushed out, and in an effort to follow the KISS principles we figure
> this
> > > would be a good start.
> > >
> > > Cheers,
> > > Tim
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
> >
>
--
Alejandro
Re: Bringing Hadoop to Fedora
Posted by Steve Loughran <st...@hortonworks.com>.
On 8 July 2013 19:28, Tim St Clair <ts...@redhat.com> wrote:
> Arun,
>
>
> https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
>
>
fixing S3 is something that is niggling me as something I want to sit down
and do once the Swift stuff is in -and once we've clarified some quirks w/
the FS API, and got some more FS contract tests.
There's been about 3 move to JetS3t 9 JIRAs, they'll need to be merged in,
along with a test that tries to do a many GB file upload; the kind of test
you'd only do in-EC2.
I'm certainly not going to go near this until august; if other people can
play with it too that'd be great -but it does need some good stressing.
Re: Bringing Hadoop to Fedora
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
[moving bigtop to bcc]
Tim,
Except from HADOOP-9680 which has significant code changes and some false
changes (which I did not go thru), all other changes seem OK.
* Have you had a change to run ALL Hadoop testcases with them applied to
make sure there are not regression?
* Have you look at the output of running 'mvn dependency:tree' without/with
the patches to verify unwanted dependencies are not sneaking in?
On HADOOP-9650 (not included in the above list):
Moving from Jetty6 to Jetty9 should be OK if there are not regressions. I
was chatting with ToddL about the issues we had before in and according to
Jetty guys they've been fixed in newer Jetty versions. Also, we are not
using Jetty for shuffle anymore (we use Netty), so Jetty is not 'stress'
that much anymore as it is used for the web UIs and for NN-SNN
checkpointing.
>From the patch some code changes are required, once that is take care we
should repeat the bullet item above for this patch.
In addition, we have to see how this will play with other projects like
HBase that are using HttpServer from hadoop-common. I'll forward this part
of the email to their dev@ so they can watch/jump-in if necessary in the
JIRA.
Thanks.
On Mon, Jul 8, 2013 at 11:28 AM, Tim St Clair <ts...@redhat.com> wrote:
> Arun,
>
> I realize you're probably pretty busy (or on vacation), but I figured I
> would re-ping this thread to inquire about the status of the patch set
> listed below.?.?
>
> https://issues.apache.org/jira/browse/HADOOP-9594
> https://issues.apache.org/jira/browse/HADOOP-9610
> https://issues.apache.org/jira/browse/HADOOP-9611
> https://issues.apache.org/jira/browse/HADOOP-9613
> https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
>
> Cheers,
> Tim
>
> ----- Original Message -----
> > From: "Arun C Murthy" <ac...@hortonworks.com>
> > To: yarn-dev@hadoop.apache.org
> > Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org,
> hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> > Sent: Tuesday, June 11, 2013 12:54:11 PM
> > Subject: Re: Bringing Hadoop to Fedora
> >
> > Tim,
> >
> > This is great! I'll take a look at some of these patches, welcome!
> >
> > Arun
> >
> > On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
> >
> > > Greetings Hadoop Development Community,
> > >
> > > As some of you are aware, we've been looking to bring the upstream 2.X
> > > series into the Fedora channels. We believe that there are several
> > > benefits that this can bring to the community:
> > >
> > > - Updated & streamlined rpm packaging
> > > - Deeper level of system integration
> > > - System managed dependencies (security + defect tracking)
> > > - Proving ground for OpenJDK7
> > > - Greater exposure for the community (both adoption and defect
> tracking)
> > > ...
> > >
> > > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop
> ),
> > > we've come across some build issues as they relate to Fedora packaging
> > > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of
> these
> > > issues (listed below) are very minor dependency changes, and we were
> > > wondering if there are folks who would be willing to review.
> > >
> > > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > > fedora-patch-collections:
> https://issues.apache.org/jira/browse/HADOOP-9610
> > > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > > (uber deprecated)
> > >
> > > There are other patches we have in the queue but they still need to be
> > > flushed out, and in an effort to follow the KISS principles we figure
> this
> > > would be a good start.
> > >
> > > Cheers,
> > > Tim
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
> >
>
--
Alejandro
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Arun,
I realize you're probably pretty busy (or on vacation), but I figured I would re-ping this thread to inquire about the status of the patch set listed below.?.?
https://issues.apache.org/jira/browse/HADOOP-9594
https://issues.apache.org/jira/browse/HADOOP-9610
https://issues.apache.org/jira/browse/HADOOP-9611
https://issues.apache.org/jira/browse/HADOOP-9613
https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
Cheers,
Tim
----- Original Message -----
> From: "Arun C Murthy" <ac...@hortonworks.com>
> To: yarn-dev@hadoop.apache.org
> Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> Sent: Tuesday, June 11, 2013 12:54:11 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> Tim,
>
> This is great! I'll take a look at some of these patches, welcome!
>
> Arun
>
> On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
>
> > Greetings Hadoop Development Community,
> >
> > As some of you are aware, we've been looking to bring the upstream 2.X
> > series into the Fedora channels. We believe that there are several
> > benefits that this can bring to the community:
> >
> > - Updated & streamlined rpm packaging
> > - Deeper level of system integration
> > - System managed dependencies (security + defect tracking)
> > - Proving ground for OpenJDK7
> > - Greater exposure for the community (both adoption and defect tracking)
> > ...
> >
> > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop),
> > we've come across some build issues as they relate to Fedora packaging
> > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these
> > issues (listed below) are very minor dependency changes, and we were
> > wondering if there are folks who would be willing to review.
> >
> > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > (uber deprecated)
> >
> > There are other patches we have in the queue but they still need to be
> > flushed out, and in an effort to follow the KISS principles we figure this
> > would be a good start.
> >
> > Cheers,
> > Tim
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
Re: Bringing Hadoop to Fedora
Posted by Tim St Clair <ts...@redhat.com>.
Arun,
I realize you're probably pretty busy (or on vacation), but I figured I would re-ping this thread to inquire about the status of the patch set listed below.?.?
https://issues.apache.org/jira/browse/HADOOP-9594
https://issues.apache.org/jira/browse/HADOOP-9610
https://issues.apache.org/jira/browse/HADOOP-9611
https://issues.apache.org/jira/browse/HADOOP-9613
https://issues.apache.org/jira/browse/HADOOP-9680 / 9623
Cheers,
Tim
----- Original Message -----
> From: "Arun C Murthy" <ac...@hortonworks.com>
> To: yarn-dev@hadoop.apache.org
> Cc: common-dev@hadoop.apache.org, mapreduce-dev@hadoop.apache.org, hdfs-dev@hadoop.apache.org, dev@bigtop.apache.org
> Sent: Tuesday, June 11, 2013 12:54:11 PM
> Subject: Re: Bringing Hadoop to Fedora
>
> Tim,
>
> This is great! I'll take a look at some of these patches, welcome!
>
> Arun
>
> On Jun 11, 2013, at 8:54 AM, Tim St Clair wrote:
>
> > Greetings Hadoop Development Community,
> >
> > As some of you are aware, we've been looking to bring the upstream 2.X
> > series into the Fedora channels. We believe that there are several
> > benefits that this can bring to the community:
> >
> > - Updated & streamlined rpm packaging
> > - Deeper level of system integration
> > - System managed dependencies (security + defect tracking)
> > - Proving ground for OpenJDK7
> > - Greater exposure for the community (both adoption and defect tracking)
> > ...
> >
> > During our evaluation (https://fedoraproject.org/wiki/Features/Hadoop),
> > we've come across some build issues as they relate to Fedora packaging
> > guidelines (http://fedoraproject.org/wiki/Packaging:Java). Most of these
> > issues (listed below) are very minor dependency changes, and we were
> > wondering if there are folks who would be willing to review.
> >
> > fedora-patch-math: https://issues.apache.org/jira/browse/HADOOP-9594
> > fedora-patch-collections: https://issues.apache.org/jira/browse/HADOOP-9610
> > fedora-patch-cglib: https://issues.apache.org/jira/browse/HADOOP-9611
> > fedora-patch-jersey: https://issues.apache.org/jira/browse/HADOOP-9613
> > fedora-patch-jets3t: https://issues.apache.org/jira/browse/HADOOP-9623
> > (uber deprecated)
> >
> > There are other patches we have in the queue but they still need to be
> > flushed out, and in an effort to follow the KISS principles we figure this
> > would be a good start.
> >
> > Cheers,
> > Tim
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>