Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Folks,

 The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!

 I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.

 For now the remaining work is to wrap up loose ends i.e. last minute api work (e.g. YARN-759 showed up last night for consideration), bug-fixes etc.; then run this through a battery of unit/system/integration tests and do a final review before we ship. There is more work remaining on documentation (e.g. HADOOP-9517) and I plan to personally focus on it this week - obviously help reviewing docs is very welcome.

 Committers, from now, please please exercise your judgement on where you commit. Typically, features should go into branch-2 with 2.3.0 as the version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0 will be limited to content in branch-2.1-beta and we stick to stabilizing it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).

thanks,
Arun

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Mon, Jun 10, 2013 at 9:35 AM, Ramya Sunil <ra...@hortonworks.com> wrote:
> We had some minor hiccups with some of our e2e tests breaking due to
> additional setsid info being logged by the bin scripts and errors while
> building with snappy turned on.

Any chance some (all?) of these tests can be donated to Bigtop so we
can run them regularly? I would love to help you guys push the code
into Bigtop test artifacts.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Mon, Jun 10, 2013 at 9:35 AM, Ramya Sunil <ra...@hortonworks.com> wrote:
> We had some minor hiccups with some of our e2e tests breaking due to
> additional setsid info being logged by the bin scripts and errors while
> building with snappy turned on.

Any chance some (all?) of these tests can be donated to Bigtop so we
can run them regularly? I would love to help you guys push the code
into Bigtop test artifacts.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Ramya Sunil <ra...@hortonworks.com>]

We have started testing branch-2.1-beta and for most parts the code looks
very stable. We have deployed both secure and non-secure multinode clusters.

We had some minor hiccups with some of our e2e tests breaking due to
additional setsid info being logged by the bin scripts and errors while
building with snappy turned on. Both are now fixed. Some of the e2e tests
had to be tweaked due to configuration changes in branch-2.1-beta such as
yarn.nodemanager.delete.debug-delay-sec has to be set to 0 to invoke the
cleanup of container tmp files, fs.trash.interval is now a server side
config etc.

We are now seeing failures while configuring short circuit read and job
summary logs(most likely configuration issues) and tests which use hftp://
and webhdfs:// are failing on secure clusters. Will file jiras after
initial investigation.

Thanks,
Ramya


On Tue, Jun 4, 2013 at 8:32 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
>
>  The vast majority of of the planned features and API work is complete,
> thanks to everyone who contributed!
>
>  I've created a branch-2.1-beta branch from which I anticipate I can make
> the first of our beta releases very shortly.
>
>  For now the remaining work is to wrap up loose ends i.e. last minute api
> work (e.g. YARN-759 showed up last night for consideration), bug-fixes
> etc.; then run this through a battery of unit/system/integration tests and
> do a final review before we ship. There is more work remaining on
> documentation (e.g. HADOOP-9517) and I plan to personally focus on it this
> week - obviously help reviewing docs is very welcome.
>
>  Committers, from now, please please exercise your judgement on where you
> commit. Typically, features should go into branch-2 with 2.3.0 as the
> version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0
> will be limited to content in branch-2.1-beta and we stick to stabilizing
> it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
>
> thanks,
> Arun
>

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>
>> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>> based on Hadoop 2.0.x codeline we plan to start running the same battery
>> of integration tests on the branch-2.1-beta.
>>
>> We plan to simply file JIRAs if anything gets detected and I will also
>> publish the URL of the Jenkins job once it gets created.
>
> Thanks Roman. Is there an ETA for this? Also, please file jiras with Blocker priority to catch attention.

Now that we've got the branch building on ASF it should be by the
end of this week.

I'll send a heads up once everything is hooked up and working.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Karthik,

 My apologies - I'll definitely spend time on this *today*.

Arun

On Jun 15, 2013, at 4:18 PM, Karthik Kambatla wrote:

> Re-posting here to the wider audience:
> 
> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
> all suggestions so far. It would be great if people could take another look
> at it. I ll iterate fast on any comments so we get this in by the time rest
> of the code pieces are committed.
> 
> Thanks
> Karthik
> 
> 
> 
> 
> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
> 
>> Just curious of your procedures. Given that there is at least one blocker
>> JIRA out there that has yet to be fully resolved, do you intend to release
>> anyway?
>> 
>> 
>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>> 
>>> If the intention is to get the release out in time for the Hadoop Summit
>> we
>>> have a very tight schedule.
>>> 
>>> Because the release vote runs for 7 days, we should have an RC latest
>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>> consider is that if the changes on an RC are corrections that do not
>> affect
>>> code, we could agree on not reseting the voting period clock if we need
>> to
>>> cut a new RC (ie doc, build, notes changes).
>>> 
>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>> are
>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>> 
>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>> 
>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>> things for the folks doing QA.
>>> 
>>> Arun, as you are coordinating the work for this release, please do commit
>>> them or give me the go ahead and I'll commit.
>>> 
>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>> the changes on the milli-slot calculations do not affect the current
>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>> 
>>> I'll be checking my email over the weekend and I can take care of some
>>> stuff if needed (while the monkeys sleep).
>>> 
>>> Thx
>>> 
>>> 
>>> 
>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>> 
>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>> 2.1 release:
>>>> 
>>>> From the 1st group I think all 3 should make.
>>>> 
>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>> MAPREDUCE-5130.
>>>> 
>>>> From the 3rd group, I don't think this JIRA will make it.
>>>> 
>>>> From the 4th group, we don't need to worry about this or 2.1
>>>> 
>>>> Thanks
>>>> 
>>>> Alejandro
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that are in shape to make it to 2.1
>>>> 
>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>> for requested nodes
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: patch avail, +1ed.
>>>> 
>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>> 
>>>> impact: Addition to MRAM HTTP API
>>>> 
>>>> status: patch avail, +1ed, needs to be committed
>>>> 
>>>> * YARN-787: Remove resource min from Yarn client API
>>>> 
>>>> impact: Yarn client API change
>>>> 
>>>> status: patch avail, needs to be reviewed. (the calculation of
>> slot-millis
>>>> is not affected, the MIN is taken from conf for now)
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that require minor work to make it to 2.1
>>>> 
>>>> * YARN-521: Augment AM - RM client module to be able to request
>> containers
>>>> only at specific locations
>>>> 
>>>> impact: AMRM client API change
>>>> 
>>>> status: patch not avail yet (requires YARN-752)
>>>> 
>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>> with
>>>> /nodes REST API
>>>> 
>>>> impact: Yarn client API & proto change
>>>> 
>>>> status: patch avail, review in progress
>>>> 
>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: patch avail but some tests are failing
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that require significant work to make it to 2.1 and may not make
>> it
>>>> 
>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>> 
>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>> does not change API)
>>>> 
>>>> status: patch avail, review in progress
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that don't need to make it to 2.1
>>>> 
>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>> counter constants
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: per discussion we should first add memory-millis and
>> vcores-millis
>>>> 
>>>> ------------------------------------------------------
>>>> 
>>>> 
>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>> wrote:
>>>> 
>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>> wrote:
>>>>>> 
>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>> 
>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>> platform
>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>> battery
>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>> 
>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>> also
>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>> 
>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>> Blocker priority to catch attention.
>>>>> 
>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>> platforms!):
>>>>>   http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>> 
>>>>> The immediate benefit here is that we get to see that the
>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>> 
>>>>> Starting from next week, I'll start running regular tests
>>>>> on these bits and will keep you guys posted!
>>>>> 
>>>>> Thanks,
>>>>> Roman.
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Alejandro
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Alejandro
>> 
>> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Karthik,

 My apologies - I'll definitely spend time on this *today*.

Arun

On Jun 15, 2013, at 4:18 PM, Karthik Kambatla wrote:

> Re-posting here to the wider audience:
> 
> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
> all suggestions so far. It would be great if people could take another look
> at it. I ll iterate fast on any comments so we get this in by the time rest
> of the code pieces are committed.
> 
> Thanks
> Karthik
> 
> 
> 
> 
> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
> 
>> Just curious of your procedures. Given that there is at least one blocker
>> JIRA out there that has yet to be fully resolved, do you intend to release
>> anyway?
>> 
>> 
>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>> 
>>> If the intention is to get the release out in time for the Hadoop Summit
>> we
>>> have a very tight schedule.
>>> 
>>> Because the release vote runs for 7 days, we should have an RC latest
>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>> consider is that if the changes on an RC are corrections that do not
>> affect
>>> code, we could agree on not reseting the voting period clock if we need
>> to
>>> cut a new RC (ie doc, build, notes changes).
>>> 
>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>> are
>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>> 
>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>> 
>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>> things for the folks doing QA.
>>> 
>>> Arun, as you are coordinating the work for this release, please do commit
>>> them or give me the go ahead and I'll commit.
>>> 
>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>> the changes on the milli-slot calculations do not affect the current
>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>> 
>>> I'll be checking my email over the weekend and I can take care of some
>>> stuff if needed (while the monkeys sleep).
>>> 
>>> Thx
>>> 
>>> 
>>> 
>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>> 
>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>> 2.1 release:
>>>> 
>>>> From the 1st group I think all 3 should make.
>>>> 
>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>> MAPREDUCE-5130.
>>>> 
>>>> From the 3rd group, I don't think this JIRA will make it.
>>>> 
>>>> From the 4th group, we don't need to worry about this or 2.1
>>>> 
>>>> Thanks
>>>> 
>>>> Alejandro
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that are in shape to make it to 2.1
>>>> 
>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>> for requested nodes
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: patch avail, +1ed.
>>>> 
>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>> 
>>>> impact: Addition to MRAM HTTP API
>>>> 
>>>> status: patch avail, +1ed, needs to be committed
>>>> 
>>>> * YARN-787: Remove resource min from Yarn client API
>>>> 
>>>> impact: Yarn client API change
>>>> 
>>>> status: patch avail, needs to be reviewed. (the calculation of
>> slot-millis
>>>> is not affected, the MIN is taken from conf for now)
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that require minor work to make it to 2.1
>>>> 
>>>> * YARN-521: Augment AM - RM client module to be able to request
>> containers
>>>> only at specific locations
>>>> 
>>>> impact: AMRM client API change
>>>> 
>>>> status: patch not avail yet (requires YARN-752)
>>>> 
>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>> with
>>>> /nodes REST API
>>>> 
>>>> impact: Yarn client API & proto change
>>>> 
>>>> status: patch avail, review in progress
>>>> 
>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: patch avail but some tests are failing
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that require significant work to make it to 2.1 and may not make
>> it
>>>> 
>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>> 
>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>> does not change API)
>>>> 
>>>> status: patch avail, review in progress
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that don't need to make it to 2.1
>>>> 
>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>> counter constants
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: per discussion we should first add memory-millis and
>> vcores-millis
>>>> 
>>>> ------------------------------------------------------
>>>> 
>>>> 
>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>> wrote:
>>>> 
>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>> wrote:
>>>>>> 
>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>> 
>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>> platform
>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>> battery
>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>> 
>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>> also
>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>> 
>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>> Blocker priority to catch attention.
>>>>> 
>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>> platforms!):
>>>>>   http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>> 
>>>>> The immediate benefit here is that we get to see that the
>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>> 
>>>>> Starting from next week, I'll start running regular tests
>>>>> on these bits and will keep you guys posted!
>>>>> 
>>>>> Thanks,
>>>>> Roman.
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Alejandro
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Alejandro
>> 
>> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Karthik,

 My apologies - I'll definitely spend time on this *today*.

Arun

On Jun 15, 2013, at 4:18 PM, Karthik Kambatla wrote:

> Re-posting here to the wider audience:
> 
> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
> all suggestions so far. It would be great if people could take another look
> at it. I ll iterate fast on any comments so we get this in by the time rest
> of the code pieces are committed.
> 
> Thanks
> Karthik
> 
> 
> 
> 
> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
> 
>> Just curious of your procedures. Given that there is at least one blocker
>> JIRA out there that has yet to be fully resolved, do you intend to release
>> anyway?
>> 
>> 
>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>> 
>>> If the intention is to get the release out in time for the Hadoop Summit
>> we
>>> have a very tight schedule.
>>> 
>>> Because the release vote runs for 7 days, we should have an RC latest
>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>> consider is that if the changes on an RC are corrections that do not
>> affect
>>> code, we could agree on not reseting the voting period clock if we need
>> to
>>> cut a new RC (ie doc, build, notes changes).
>>> 
>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>> are
>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>> 
>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>> 
>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>> things for the folks doing QA.
>>> 
>>> Arun, as you are coordinating the work for this release, please do commit
>>> them or give me the go ahead and I'll commit.
>>> 
>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>> the changes on the milli-slot calculations do not affect the current
>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>> 
>>> I'll be checking my email over the weekend and I can take care of some
>>> stuff if needed (while the monkeys sleep).
>>> 
>>> Thx
>>> 
>>> 
>>> 
>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>> 
>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>> 2.1 release:
>>>> 
>>>> From the 1st group I think all 3 should make.
>>>> 
>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>> MAPREDUCE-5130.
>>>> 
>>>> From the 3rd group, I don't think this JIRA will make it.
>>>> 
>>>> From the 4th group, we don't need to worry about this or 2.1
>>>> 
>>>> Thanks
>>>> 
>>>> Alejandro
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that are in shape to make it to 2.1
>>>> 
>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>> for requested nodes
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: patch avail, +1ed.
>>>> 
>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>> 
>>>> impact: Addition to MRAM HTTP API
>>>> 
>>>> status: patch avail, +1ed, needs to be committed
>>>> 
>>>> * YARN-787: Remove resource min from Yarn client API
>>>> 
>>>> impact: Yarn client API change
>>>> 
>>>> status: patch avail, needs to be reviewed. (the calculation of
>> slot-millis
>>>> is not affected, the MIN is taken from conf for now)
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that require minor work to make it to 2.1
>>>> 
>>>> * YARN-521: Augment AM - RM client module to be able to request
>> containers
>>>> only at specific locations
>>>> 
>>>> impact: AMRM client API change
>>>> 
>>>> status: patch not avail yet (requires YARN-752)
>>>> 
>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>> with
>>>> /nodes REST API
>>>> 
>>>> impact: Yarn client API & proto change
>>>> 
>>>> status: patch avail, review in progress
>>>> 
>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: patch avail but some tests are failing
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that require significant work to make it to 2.1 and may not make
>> it
>>>> 
>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>> 
>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>> does not change API)
>>>> 
>>>> status: patch avail, review in progress
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that don't need to make it to 2.1
>>>> 
>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>> counter constants
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: per discussion we should first add memory-millis and
>> vcores-millis
>>>> 
>>>> ------------------------------------------------------
>>>> 
>>>> 
>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>> wrote:
>>>> 
>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>> wrote:
>>>>>> 
>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>> 
>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>> platform
>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>> battery
>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>> 
>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>> also
>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>> 
>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>> Blocker priority to catch attention.
>>>>> 
>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>> platforms!):
>>>>>   http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>> 
>>>>> The immediate benefit here is that we get to see that the
>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>> 
>>>>> Starting from next week, I'll start running regular tests
>>>>> on these bits and will keep you guys posted!
>>>>> 
>>>>> Thanks,
>>>>> Roman.
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Alejandro
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Alejandro
>> 
>> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Karthik,

 My apologies - I'll definitely spend time on this *today*.

Arun

On Jun 15, 2013, at 4:18 PM, Karthik Kambatla wrote:

> Re-posting here to the wider audience:
> 
> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
> all suggestions so far. It would be great if people could take another look
> at it. I ll iterate fast on any comments so we get this in by the time rest
> of the code pieces are committed.
> 
> Thanks
> Karthik
> 
> 
> 
> 
> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
> 
>> Just curious of your procedures. Given that there is at least one blocker
>> JIRA out there that has yet to be fully resolved, do you intend to release
>> anyway?
>> 
>> 
>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>> 
>>> If the intention is to get the release out in time for the Hadoop Summit
>> we
>>> have a very tight schedule.
>>> 
>>> Because the release vote runs for 7 days, we should have an RC latest
>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>> consider is that if the changes on an RC are corrections that do not
>> affect
>>> code, we could agree on not reseting the voting period clock if we need
>> to
>>> cut a new RC (ie doc, build, notes changes).
>>> 
>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>> are
>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>> 
>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>> 
>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>> things for the folks doing QA.
>>> 
>>> Arun, as you are coordinating the work for this release, please do commit
>>> them or give me the go ahead and I'll commit.
>>> 
>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>> the changes on the milli-slot calculations do not affect the current
>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>> 
>>> I'll be checking my email over the weekend and I can take care of some
>>> stuff if needed (while the monkeys sleep).
>>> 
>>> Thx
>>> 
>>> 
>>> 
>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>> 
>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>> 2.1 release:
>>>> 
>>>> From the 1st group I think all 3 should make.
>>>> 
>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>> MAPREDUCE-5130.
>>>> 
>>>> From the 3rd group, I don't think this JIRA will make it.
>>>> 
>>>> From the 4th group, we don't need to worry about this or 2.1
>>>> 
>>>> Thanks
>>>> 
>>>> Alejandro
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that are in shape to make it to 2.1
>>>> 
>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>> for requested nodes
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: patch avail, +1ed.
>>>> 
>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>> 
>>>> impact: Addition to MRAM HTTP API
>>>> 
>>>> status: patch avail, +1ed, needs to be committed
>>>> 
>>>> * YARN-787: Remove resource min from Yarn client API
>>>> 
>>>> impact: Yarn client API change
>>>> 
>>>> status: patch avail, needs to be reviewed. (the calculation of
>> slot-millis
>>>> is not affected, the MIN is taken from conf for now)
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that require minor work to make it to 2.1
>>>> 
>>>> * YARN-521: Augment AM - RM client module to be able to request
>> containers
>>>> only at specific locations
>>>> 
>>>> impact: AMRM client API change
>>>> 
>>>> status: patch not avail yet (requires YARN-752)
>>>> 
>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>> with
>>>> /nodes REST API
>>>> 
>>>> impact: Yarn client API & proto change
>>>> 
>>>> status: patch avail, review in progress
>>>> 
>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: patch avail but some tests are failing
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that require significant work to make it to 2.1 and may not make
>> it
>>>> 
>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>> 
>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>> does not change API)
>>>> 
>>>> status: patch avail, review in progress
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that don't need to make it to 2.1
>>>> 
>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>> counter constants
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: per discussion we should first add memory-millis and
>> vcores-millis
>>>> 
>>>> ------------------------------------------------------
>>>> 
>>>> 
>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>> wrote:
>>>> 
>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>> wrote:
>>>>>> 
>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>> 
>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>> platform
>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>> battery
>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>> 
>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>> also
>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>> 
>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>> Blocker priority to catch attention.
>>>>> 
>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>> platforms!):
>>>>>   http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>> 
>>>>> The immediate benefit here is that we get to see that the
>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>> 
>>>>> Starting from next week, I'll start running regular tests
>>>>> on these bits and will keep you guys posted!
>>>>> 
>>>>> Thanks,
>>>>> Roman.
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Alejandro
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Alejandro
>> 
>> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Ralph Castain <ra...@castain.org>]

Will do

FWIW: at least two people reported errors on that ticket once the patch had been applied, so I'm not convinced it is correct.


On Jun 16, 2013, at 12:04 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> HDFS-4866 was committed a couple of days ago - hence isn't showing up in my filters.
> 
> Can you please file a new jira for the linker issues?
> 
> thanks,
> Arun
> 
> On Jun 16, 2013, at 10:16 AM, Ralph Castain wrote:
> 
>> HDFS-4866 is marked as a blocker. A patch was submitted and applied, but it only fixes compilation. As I marked on the ticket, the results remain unusable due to linker issues.
>> 
>> 
>> On Jun 16, 2013, at 5:33 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> 
>>> Which one are you talking about Ralph? This doesn't show up on any blocker list.
>>> 
>>> http://s.apache.org/hadoop-blocker-bugs
>>> 
>>> Arun
>>> 
>>> 
>>> On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:
>>> 
>>>> Not trying to be a pain, but I am trying to get clarification. The protocol buffer support is still broken. Do you intend to release 2.1 with that unfixed?
>>>> 
>>>> 
>>>> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
>>>> 
>>>>> Re-posting here to the wider audience:
>>>>> 
>>>>> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
>>>>> all suggestions so far. It would be great if people could take another look
>>>>> at it. I ll iterate fast on any comments so we get this in by the time rest
>>>>> of the code pieces are committed.
>>>>> 
>>>>> Thanks
>>>>> Karthik
>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
>>>>> 
>>>>>> Just curious of your procedures. Given that there is at least one blocker
>>>>>> JIRA out there that has yet to be fully resolved, do you intend to release
>>>>>> anyway?
>>>>>> 
>>>>>> 
>>>>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>>>>> 
>>>>>>> If the intention is to get the release out in time for the Hadoop Summit
>>>>>> we
>>>>>>> have a very tight schedule.
>>>>>>> 
>>>>>>> Because the release vote runs for 7 days, we should have an RC latest
>>>>>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>>>>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>>>>>> consider is that if the changes on an RC are corrections that do not
>>>>>> affect
>>>>>>> code, we could agree on not reseting the voting period clock if we need
>>>>>> to
>>>>>>> cut a new RC (ie doc, build, notes changes).
>>>>>>> 
>>>>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>>>>>> are
>>>>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>>>>>> 
>>>>>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>>>>>> 
>>>>>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>>>>>> things for the folks doing QA.
>>>>>>> 
>>>>>>> Arun, as you are coordinating the work for this release, please do commit
>>>>>>> them or give me the go ahead and I'll commit.
>>>>>>> 
>>>>>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>>>>>> the changes on the milli-slot calculations do not affect the current
>>>>>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>>>>>> 
>>>>>>> I'll be checking my email over the weekend and I can take care of some
>>>>>>> stuff if needed (while the monkeys sleep).
>>>>>>> 
>>>>>>> Thx
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>>>>>> 2.1 release:
>>>>>>>> 
>>>>>>>> From the 1st group I think all 3 should make.
>>>>>>>> 
>>>>>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>>>>>> MAPREDUCE-5130.
>>>>>>>> 
>>>>>>>> From the 3rd group, I don't think this JIRA will make it.
>>>>>>>> 
>>>>>>>> From the 4th group, we don't need to worry about this or 2.1
>>>>>>>> 
>>>>>>>> Thanks
>>>>>>>> 
>>>>>>>> Alejandro
>>>>>>>> 
>>>>>>>> ------------------------------------------------------
>>>>>>>> JIRAs that are in shape to make it to 2.1
>>>>>>>> 
>>>>>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>>>>>> for requested nodes
>>>>>>>> 
>>>>>>>> impact: behavior change
>>>>>>>> 
>>>>>>>> status: patch avail, +1ed.
>>>>>>>> 
>>>>>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>>>>>> 
>>>>>>>> impact: Addition to MRAM HTTP API
>>>>>>>> 
>>>>>>>> status: patch avail, +1ed, needs to be committed
>>>>>>>> 
>>>>>>>> * YARN-787: Remove resource min from Yarn client API
>>>>>>>> 
>>>>>>>> impact: Yarn client API change
>>>>>>>> 
>>>>>>>> status: patch avail, needs to be reviewed. (the calculation of
>>>>>> slot-millis
>>>>>>>> is not affected, the MIN is taken from conf for now)
>>>>>>>> 
>>>>>>>> ------------------------------------------------------
>>>>>>>> JIRAs that require minor work to make it to 2.1
>>>>>>>> 
>>>>>>>> * YARN-521: Augment AM - RM client module to be able to request
>>>>>> containers
>>>>>>>> only at specific locations
>>>>>>>> 
>>>>>>>> impact: AMRM client API change
>>>>>>>> 
>>>>>>>> status: patch not avail yet (requires YARN-752)
>>>>>>>> 
>>>>>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>>>>>> with
>>>>>>>> /nodes REST API
>>>>>>>> 
>>>>>>>> impact: Yarn client API & proto change
>>>>>>>> 
>>>>>>>> status: patch avail, review in progress
>>>>>>>> 
>>>>>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>>>>>> 
>>>>>>>> impact: behavior change
>>>>>>>> 
>>>>>>>> status: patch avail but some tests are failing
>>>>>>>> 
>>>>>>>> ------------------------------------------------------
>>>>>>>> JIRAs that require significant work to make it to 2.1 and may not make
>>>>>> it
>>>>>>>> 
>>>>>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>>>>>> 
>>>>>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>>>>>> does not change API)
>>>>>>>> 
>>>>>>>> status: patch avail, review in progress
>>>>>>>> 
>>>>>>>> ------------------------------------------------------
>>>>>>>> JIRAs that don't need to make it to 2.1
>>>>>>>> 
>>>>>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>>>>>> counter constants
>>>>>>>> 
>>>>>>>> impact: behavior change
>>>>>>>> 
>>>>>>>> status: per discussion we should first add memory-millis and
>>>>>> vcores-millis
>>>>>>>> 
>>>>>>>> ------------------------------------------------------
>>>>>>>> 
>>>>>>>> 
>>>>>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>>>>>> wrote:
>>>>>>>> 
>>>>>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>>>>>> 
>>>>>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>>>>>> platform
>>>>>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>>>>>> battery
>>>>>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>>>>>> 
>>>>>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>>>>>> also
>>>>>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>>>>>> 
>>>>>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>>>>>> Blocker priority to catch attention.
>>>>>>>>> 
>>>>>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>>>>>> platforms!):
>>>>>>>>> http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>>>>>> 
>>>>>>>>> The immediate benefit here is that we get to see that the
>>>>>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>>>>>> 
>>>>>>>>> Starting from next week, I'll start running regular tests
>>>>>>>>> on these bits and will keep you guys posted!
>>>>>>>>> 
>>>>>>>>> Thanks,
>>>>>>>>> Roman.
>>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>> --
>>>>>>>> Alejandro
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Alejandro
>>>>>> 
>>>>>> 
>>>> 
>>> 
>>> --
>>> Arun C. Murthy
>>> Hortonworks Inc.
>>> http://hortonworks.com/
>>> 
>>> 
>> 
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

HDFS-4866 was committed a couple of days ago - hence isn't showing up in my filters.

Can you please file a new jira for the linker issues?

thanks,
Arun

On Jun 16, 2013, at 10:16 AM, Ralph Castain wrote:

> HDFS-4866 is marked as a blocker. A patch was submitted and applied, but it only fixes compilation. As I marked on the ticket, the results remain unusable due to linker issues.
> 
> 
> On Jun 16, 2013, at 5:33 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> 
>> Which one are you talking about Ralph? This doesn't show up on any blocker list.
>> 
>> http://s.apache.org/hadoop-blocker-bugs
>> 
>> Arun
>> 
>> 
>> On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:
>> 
>>> Not trying to be a pain, but I am trying to get clarification. The protocol buffer support is still broken. Do you intend to release 2.1 with that unfixed?
>>> 
>>> 
>>> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
>>> 
>>>> Re-posting here to the wider audience:
>>>> 
>>>> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
>>>> all suggestions so far. It would be great if people could take another look
>>>> at it. I ll iterate fast on any comments so we get this in by the time rest
>>>> of the code pieces are committed.
>>>> 
>>>> Thanks
>>>> Karthik
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
>>>> 
>>>>> Just curious of your procedures. Given that there is at least one blocker
>>>>> JIRA out there that has yet to be fully resolved, do you intend to release
>>>>> anyway?
>>>>> 
>>>>> 
>>>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>>>> 
>>>>>> If the intention is to get the release out in time for the Hadoop Summit
>>>>> we
>>>>>> have a very tight schedule.
>>>>>> 
>>>>>> Because the release vote runs for 7 days, we should have an RC latest
>>>>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>>>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>>>>> consider is that if the changes on an RC are corrections that do not
>>>>> affect
>>>>>> code, we could agree on not reseting the voting period clock if we need
>>>>> to
>>>>>> cut a new RC (ie doc, build, notes changes).
>>>>>> 
>>>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>>>>> are
>>>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>>>>> 
>>>>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>>>>> 
>>>>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>>>>> things for the folks doing QA.
>>>>>> 
>>>>>> Arun, as you are coordinating the work for this release, please do commit
>>>>>> them or give me the go ahead and I'll commit.
>>>>>> 
>>>>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>>>>> the changes on the milli-slot calculations do not affect the current
>>>>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>>>>> 
>>>>>> I'll be checking my email over the weekend and I can take care of some
>>>>>> stuff if needed (while the monkeys sleep).
>>>>>> 
>>>>>> Thx
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>>>> wrote:
>>>>>> 
>>>>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>>>>> 2.1 release:
>>>>>>> 
>>>>>>> From the 1st group I think all 3 should make.
>>>>>>> 
>>>>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>>>>> MAPREDUCE-5130.
>>>>>>> 
>>>>>>> From the 3rd group, I don't think this JIRA will make it.
>>>>>>> 
>>>>>>> From the 4th group, we don't need to worry about this or 2.1
>>>>>>> 
>>>>>>> Thanks
>>>>>>> 
>>>>>>> Alejandro
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that are in shape to make it to 2.1
>>>>>>> 
>>>>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>>>>> for requested nodes
>>>>>>> 
>>>>>>> impact: behavior change
>>>>>>> 
>>>>>>> status: patch avail, +1ed.
>>>>>>> 
>>>>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>>>>> 
>>>>>>> impact: Addition to MRAM HTTP API
>>>>>>> 
>>>>>>> status: patch avail, +1ed, needs to be committed
>>>>>>> 
>>>>>>> * YARN-787: Remove resource min from Yarn client API
>>>>>>> 
>>>>>>> impact: Yarn client API change
>>>>>>> 
>>>>>>> status: patch avail, needs to be reviewed. (the calculation of
>>>>> slot-millis
>>>>>>> is not affected, the MIN is taken from conf for now)
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that require minor work to make it to 2.1
>>>>>>> 
>>>>>>> * YARN-521: Augment AM - RM client module to be able to request
>>>>> containers
>>>>>>> only at specific locations
>>>>>>> 
>>>>>>> impact: AMRM client API change
>>>>>>> 
>>>>>>> status: patch not avail yet (requires YARN-752)
>>>>>>> 
>>>>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>>>>> with
>>>>>>> /nodes REST API
>>>>>>> 
>>>>>>> impact: Yarn client API & proto change
>>>>>>> 
>>>>>>> status: patch avail, review in progress
>>>>>>> 
>>>>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>>>>> 
>>>>>>> impact: behavior change
>>>>>>> 
>>>>>>> status: patch avail but some tests are failing
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that require significant work to make it to 2.1 and may not make
>>>>> it
>>>>>>> 
>>>>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>>>>> 
>>>>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>>>>> does not change API)
>>>>>>> 
>>>>>>> status: patch avail, review in progress
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that don't need to make it to 2.1
>>>>>>> 
>>>>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>>>>> counter constants
>>>>>>> 
>>>>>>> impact: behavior change
>>>>>>> 
>>>>>>> status: per discussion we should first add memory-millis and
>>>>> vcores-millis
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>>>>> wrote:
>>>>>>> 
>>>>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>>>>> 
>>>>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>>>>> platform
>>>>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>>>>> battery
>>>>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>>>>> 
>>>>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>>>>> also
>>>>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>>>>> 
>>>>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>>>>> Blocker priority to catch attention.
>>>>>>>> 
>>>>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>>>>> platforms!):
>>>>>>>> http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>>>>> 
>>>>>>>> The immediate benefit here is that we get to see that the
>>>>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>>>>> 
>>>>>>>> Starting from next week, I'll start running regular tests
>>>>>>>> on these bits and will keep you guys posted!
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Roman.
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Alejandro
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Alejandro
>>>>> 
>>>>> 
>>> 
>> 
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>> 
>> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

HDFS-4866 was committed a couple of days ago - hence isn't showing up in my filters.

Can you please file a new jira for the linker issues?

thanks,
Arun

On Jun 16, 2013, at 10:16 AM, Ralph Castain wrote:

> HDFS-4866 is marked as a blocker. A patch was submitted and applied, but it only fixes compilation. As I marked on the ticket, the results remain unusable due to linker issues.
> 
> 
> On Jun 16, 2013, at 5:33 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> 
>> Which one are you talking about Ralph? This doesn't show up on any blocker list.
>> 
>> http://s.apache.org/hadoop-blocker-bugs
>> 
>> Arun
>> 
>> 
>> On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:
>> 
>>> Not trying to be a pain, but I am trying to get clarification. The protocol buffer support is still broken. Do you intend to release 2.1 with that unfixed?
>>> 
>>> 
>>> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
>>> 
>>>> Re-posting here to the wider audience:
>>>> 
>>>> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
>>>> all suggestions so far. It would be great if people could take another look
>>>> at it. I ll iterate fast on any comments so we get this in by the time rest
>>>> of the code pieces are committed.
>>>> 
>>>> Thanks
>>>> Karthik
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
>>>> 
>>>>> Just curious of your procedures. Given that there is at least one blocker
>>>>> JIRA out there that has yet to be fully resolved, do you intend to release
>>>>> anyway?
>>>>> 
>>>>> 
>>>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>>>> 
>>>>>> If the intention is to get the release out in time for the Hadoop Summit
>>>>> we
>>>>>> have a very tight schedule.
>>>>>> 
>>>>>> Because the release vote runs for 7 days, we should have an RC latest
>>>>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>>>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>>>>> consider is that if the changes on an RC are corrections that do not
>>>>> affect
>>>>>> code, we could agree on not reseting the voting period clock if we need
>>>>> to
>>>>>> cut a new RC (ie doc, build, notes changes).
>>>>>> 
>>>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>>>>> are
>>>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>>>>> 
>>>>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>>>>> 
>>>>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>>>>> things for the folks doing QA.
>>>>>> 
>>>>>> Arun, as you are coordinating the work for this release, please do commit
>>>>>> them or give me the go ahead and I'll commit.
>>>>>> 
>>>>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>>>>> the changes on the milli-slot calculations do not affect the current
>>>>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>>>>> 
>>>>>> I'll be checking my email over the weekend and I can take care of some
>>>>>> stuff if needed (while the monkeys sleep).
>>>>>> 
>>>>>> Thx
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>>>> wrote:
>>>>>> 
>>>>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>>>>> 2.1 release:
>>>>>>> 
>>>>>>> From the 1st group I think all 3 should make.
>>>>>>> 
>>>>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>>>>> MAPREDUCE-5130.
>>>>>>> 
>>>>>>> From the 3rd group, I don't think this JIRA will make it.
>>>>>>> 
>>>>>>> From the 4th group, we don't need to worry about this or 2.1
>>>>>>> 
>>>>>>> Thanks
>>>>>>> 
>>>>>>> Alejandro
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that are in shape to make it to 2.1
>>>>>>> 
>>>>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>>>>> for requested nodes
>>>>>>> 
>>>>>>> impact: behavior change
>>>>>>> 
>>>>>>> status: patch avail, +1ed.
>>>>>>> 
>>>>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>>>>> 
>>>>>>> impact: Addition to MRAM HTTP API
>>>>>>> 
>>>>>>> status: patch avail, +1ed, needs to be committed
>>>>>>> 
>>>>>>> * YARN-787: Remove resource min from Yarn client API
>>>>>>> 
>>>>>>> impact: Yarn client API change
>>>>>>> 
>>>>>>> status: patch avail, needs to be reviewed. (the calculation of
>>>>> slot-millis
>>>>>>> is not affected, the MIN is taken from conf for now)
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that require minor work to make it to 2.1
>>>>>>> 
>>>>>>> * YARN-521: Augment AM - RM client module to be able to request
>>>>> containers
>>>>>>> only at specific locations
>>>>>>> 
>>>>>>> impact: AMRM client API change
>>>>>>> 
>>>>>>> status: patch not avail yet (requires YARN-752)
>>>>>>> 
>>>>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>>>>> with
>>>>>>> /nodes REST API
>>>>>>> 
>>>>>>> impact: Yarn client API & proto change
>>>>>>> 
>>>>>>> status: patch avail, review in progress
>>>>>>> 
>>>>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>>>>> 
>>>>>>> impact: behavior change
>>>>>>> 
>>>>>>> status: patch avail but some tests are failing
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that require significant work to make it to 2.1 and may not make
>>>>> it
>>>>>>> 
>>>>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>>>>> 
>>>>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>>>>> does not change API)
>>>>>>> 
>>>>>>> status: patch avail, review in progress
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that don't need to make it to 2.1
>>>>>>> 
>>>>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>>>>> counter constants
>>>>>>> 
>>>>>>> impact: behavior change
>>>>>>> 
>>>>>>> status: per discussion we should first add memory-millis and
>>>>> vcores-millis
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>>>>> wrote:
>>>>>>> 
>>>>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>>>>> 
>>>>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>>>>> platform
>>>>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>>>>> battery
>>>>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>>>>> 
>>>>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>>>>> also
>>>>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>>>>> 
>>>>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>>>>> Blocker priority to catch attention.
>>>>>>>> 
>>>>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>>>>> platforms!):
>>>>>>>> http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>>>>> 
>>>>>>>> The immediate benefit here is that we get to see that the
>>>>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>>>>> 
>>>>>>>> Starting from next week, I'll start running regular tests
>>>>>>>> on these bits and will keep you guys posted!
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Roman.
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Alejandro
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Alejandro
>>>>> 
>>>>> 
>>> 
>> 
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>> 
>> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

HDFS-4866 was committed a couple of days ago - hence isn't showing up in my filters.

Can you please file a new jira for the linker issues?

thanks,
Arun

On Jun 16, 2013, at 10:16 AM, Ralph Castain wrote:

> HDFS-4866 is marked as a blocker. A patch was submitted and applied, but it only fixes compilation. As I marked on the ticket, the results remain unusable due to linker issues.
> 
> 
> On Jun 16, 2013, at 5:33 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> 
>> Which one are you talking about Ralph? This doesn't show up on any blocker list.
>> 
>> http://s.apache.org/hadoop-blocker-bugs
>> 
>> Arun
>> 
>> 
>> On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:
>> 
>>> Not trying to be a pain, but I am trying to get clarification. The protocol buffer support is still broken. Do you intend to release 2.1 with that unfixed?
>>> 
>>> 
>>> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
>>> 
>>>> Re-posting here to the wider audience:
>>>> 
>>>> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
>>>> all suggestions so far. It would be great if people could take another look
>>>> at it. I ll iterate fast on any comments so we get this in by the time rest
>>>> of the code pieces are committed.
>>>> 
>>>> Thanks
>>>> Karthik
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
>>>> 
>>>>> Just curious of your procedures. Given that there is at least one blocker
>>>>> JIRA out there that has yet to be fully resolved, do you intend to release
>>>>> anyway?
>>>>> 
>>>>> 
>>>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>>>> 
>>>>>> If the intention is to get the release out in time for the Hadoop Summit
>>>>> we
>>>>>> have a very tight schedule.
>>>>>> 
>>>>>> Because the release vote runs for 7 days, we should have an RC latest
>>>>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>>>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>>>>> consider is that if the changes on an RC are corrections that do not
>>>>> affect
>>>>>> code, we could agree on not reseting the voting period clock if we need
>>>>> to
>>>>>> cut a new RC (ie doc, build, notes changes).
>>>>>> 
>>>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>>>>> are
>>>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>>>>> 
>>>>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>>>>> 
>>>>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>>>>> things for the folks doing QA.
>>>>>> 
>>>>>> Arun, as you are coordinating the work for this release, please do commit
>>>>>> them or give me the go ahead and I'll commit.
>>>>>> 
>>>>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>>>>> the changes on the milli-slot calculations do not affect the current
>>>>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>>>>> 
>>>>>> I'll be checking my email over the weekend and I can take care of some
>>>>>> stuff if needed (while the monkeys sleep).
>>>>>> 
>>>>>> Thx
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>>>> wrote:
>>>>>> 
>>>>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>>>>> 2.1 release:
>>>>>>> 
>>>>>>> From the 1st group I think all 3 should make.
>>>>>>> 
>>>>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>>>>> MAPREDUCE-5130.
>>>>>>> 
>>>>>>> From the 3rd group, I don't think this JIRA will make it.
>>>>>>> 
>>>>>>> From the 4th group, we don't need to worry about this or 2.1
>>>>>>> 
>>>>>>> Thanks
>>>>>>> 
>>>>>>> Alejandro
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that are in shape to make it to 2.1
>>>>>>> 
>>>>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>>>>> for requested nodes
>>>>>>> 
>>>>>>> impact: behavior change
>>>>>>> 
>>>>>>> status: patch avail, +1ed.
>>>>>>> 
>>>>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>>>>> 
>>>>>>> impact: Addition to MRAM HTTP API
>>>>>>> 
>>>>>>> status: patch avail, +1ed, needs to be committed
>>>>>>> 
>>>>>>> * YARN-787: Remove resource min from Yarn client API
>>>>>>> 
>>>>>>> impact: Yarn client API change
>>>>>>> 
>>>>>>> status: patch avail, needs to be reviewed. (the calculation of
>>>>> slot-millis
>>>>>>> is not affected, the MIN is taken from conf for now)
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that require minor work to make it to 2.1
>>>>>>> 
>>>>>>> * YARN-521: Augment AM - RM client module to be able to request
>>>>> containers
>>>>>>> only at specific locations
>>>>>>> 
>>>>>>> impact: AMRM client API change
>>>>>>> 
>>>>>>> status: patch not avail yet (requires YARN-752)
>>>>>>> 
>>>>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>>>>> with
>>>>>>> /nodes REST API
>>>>>>> 
>>>>>>> impact: Yarn client API & proto change
>>>>>>> 
>>>>>>> status: patch avail, review in progress
>>>>>>> 
>>>>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>>>>> 
>>>>>>> impact: behavior change
>>>>>>> 
>>>>>>> status: patch avail but some tests are failing
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that require significant work to make it to 2.1 and may not make
>>>>> it
>>>>>>> 
>>>>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>>>>> 
>>>>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>>>>> does not change API)
>>>>>>> 
>>>>>>> status: patch avail, review in progress
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that don't need to make it to 2.1
>>>>>>> 
>>>>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>>>>> counter constants
>>>>>>> 
>>>>>>> impact: behavior change
>>>>>>> 
>>>>>>> status: per discussion we should first add memory-millis and
>>>>> vcores-millis
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>>>>> wrote:
>>>>>>> 
>>>>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>>>>> 
>>>>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>>>>> platform
>>>>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>>>>> battery
>>>>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>>>>> 
>>>>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>>>>> also
>>>>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>>>>> 
>>>>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>>>>> Blocker priority to catch attention.
>>>>>>>> 
>>>>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>>>>> platforms!):
>>>>>>>> http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>>>>> 
>>>>>>>> The immediate benefit here is that we get to see that the
>>>>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>>>>> 
>>>>>>>> Starting from next week, I'll start running regular tests
>>>>>>>> on these bits and will keep you guys posted!
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Roman.
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Alejandro
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Alejandro
>>>>> 
>>>>> 
>>> 
>> 
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>> 
>> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

HDFS-4866 was committed a couple of days ago - hence isn't showing up in my filters.

Can you please file a new jira for the linker issues?

thanks,
Arun

On Jun 16, 2013, at 10:16 AM, Ralph Castain wrote:

> HDFS-4866 is marked as a blocker. A patch was submitted and applied, but it only fixes compilation. As I marked on the ticket, the results remain unusable due to linker issues.
> 
> 
> On Jun 16, 2013, at 5:33 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> 
>> Which one are you talking about Ralph? This doesn't show up on any blocker list.
>> 
>> http://s.apache.org/hadoop-blocker-bugs
>> 
>> Arun
>> 
>> 
>> On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:
>> 
>>> Not trying to be a pain, but I am trying to get clarification. The protocol buffer support is still broken. Do you intend to release 2.1 with that unfixed?
>>> 
>>> 
>>> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
>>> 
>>>> Re-posting here to the wider audience:
>>>> 
>>>> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
>>>> all suggestions so far. It would be great if people could take another look
>>>> at it. I ll iterate fast on any comments so we get this in by the time rest
>>>> of the code pieces are committed.
>>>> 
>>>> Thanks
>>>> Karthik
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
>>>> 
>>>>> Just curious of your procedures. Given that there is at least one blocker
>>>>> JIRA out there that has yet to be fully resolved, do you intend to release
>>>>> anyway?
>>>>> 
>>>>> 
>>>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>>>> 
>>>>>> If the intention is to get the release out in time for the Hadoop Summit
>>>>> we
>>>>>> have a very tight schedule.
>>>>>> 
>>>>>> Because the release vote runs for 7 days, we should have an RC latest
>>>>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>>>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>>>>> consider is that if the changes on an RC are corrections that do not
>>>>> affect
>>>>>> code, we could agree on not reseting the voting period clock if we need
>>>>> to
>>>>>> cut a new RC (ie doc, build, notes changes).
>>>>>> 
>>>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>>>>> are
>>>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>>>>> 
>>>>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>>>>> 
>>>>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>>>>> things for the folks doing QA.
>>>>>> 
>>>>>> Arun, as you are coordinating the work for this release, please do commit
>>>>>> them or give me the go ahead and I'll commit.
>>>>>> 
>>>>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>>>>> the changes on the milli-slot calculations do not affect the current
>>>>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>>>>> 
>>>>>> I'll be checking my email over the weekend and I can take care of some
>>>>>> stuff if needed (while the monkeys sleep).
>>>>>> 
>>>>>> Thx
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>>>> wrote:
>>>>>> 
>>>>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>>>>> 2.1 release:
>>>>>>> 
>>>>>>> From the 1st group I think all 3 should make.
>>>>>>> 
>>>>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>>>>> MAPREDUCE-5130.
>>>>>>> 
>>>>>>> From the 3rd group, I don't think this JIRA will make it.
>>>>>>> 
>>>>>>> From the 4th group, we don't need to worry about this or 2.1
>>>>>>> 
>>>>>>> Thanks
>>>>>>> 
>>>>>>> Alejandro
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that are in shape to make it to 2.1
>>>>>>> 
>>>>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>>>>> for requested nodes
>>>>>>> 
>>>>>>> impact: behavior change
>>>>>>> 
>>>>>>> status: patch avail, +1ed.
>>>>>>> 
>>>>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>>>>> 
>>>>>>> impact: Addition to MRAM HTTP API
>>>>>>> 
>>>>>>> status: patch avail, +1ed, needs to be committed
>>>>>>> 
>>>>>>> * YARN-787: Remove resource min from Yarn client API
>>>>>>> 
>>>>>>> impact: Yarn client API change
>>>>>>> 
>>>>>>> status: patch avail, needs to be reviewed. (the calculation of
>>>>> slot-millis
>>>>>>> is not affected, the MIN is taken from conf for now)
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that require minor work to make it to 2.1
>>>>>>> 
>>>>>>> * YARN-521: Augment AM - RM client module to be able to request
>>>>> containers
>>>>>>> only at specific locations
>>>>>>> 
>>>>>>> impact: AMRM client API change
>>>>>>> 
>>>>>>> status: patch not avail yet (requires YARN-752)
>>>>>>> 
>>>>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>>>>> with
>>>>>>> /nodes REST API
>>>>>>> 
>>>>>>> impact: Yarn client API & proto change
>>>>>>> 
>>>>>>> status: patch avail, review in progress
>>>>>>> 
>>>>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>>>>> 
>>>>>>> impact: behavior change
>>>>>>> 
>>>>>>> status: patch avail but some tests are failing
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that require significant work to make it to 2.1 and may not make
>>>>> it
>>>>>>> 
>>>>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>>>>> 
>>>>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>>>>> does not change API)
>>>>>>> 
>>>>>>> status: patch avail, review in progress
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> JIRAs that don't need to make it to 2.1
>>>>>>> 
>>>>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>>>>> counter constants
>>>>>>> 
>>>>>>> impact: behavior change
>>>>>>> 
>>>>>>> status: per discussion we should first add memory-millis and
>>>>> vcores-millis
>>>>>>> 
>>>>>>> ------------------------------------------------------
>>>>>>> 
>>>>>>> 
>>>>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>>>>> wrote:
>>>>>>> 
>>>>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>>>>> 
>>>>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>>>>> platform
>>>>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>>>>> battery
>>>>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>>>>> 
>>>>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>>>>> also
>>>>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>>>>> 
>>>>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>>>>> Blocker priority to catch attention.
>>>>>>>> 
>>>>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>>>>> platforms!):
>>>>>>>> http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>>>>> 
>>>>>>>> The immediate benefit here is that we get to see that the
>>>>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>>>>> 
>>>>>>>> Starting from next week, I'll start running regular tests
>>>>>>>> on these bits and will keep you guys posted!
>>>>>>>> 
>>>>>>>> Thanks,
>>>>>>>> Roman.
>>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> --
>>>>>>> Alejandro
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Alejandro
>>>>> 
>>>>> 
>>> 
>> 
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>> 
>> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Ralph Castain <rh...@open-mpi.org>]

HDFS-4866 is marked as a blocker. A patch was submitted and applied, but it only fixes compilation. As I marked on the ticket, the results remain unusable due to linker issues.


On Jun 16, 2013, at 5:33 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Which one are you talking about Ralph? This doesn't show up on any blocker list.
> 
> http://s.apache.org/hadoop-blocker-bugs
> 
> Arun
> 
> 
> On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:
> 
>> Not trying to be a pain, but I am trying to get clarification. The protocol buffer support is still broken. Do you intend to release 2.1 with that unfixed?
>> 
>> 
>> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
>> 
>>> Re-posting here to the wider audience:
>>> 
>>> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
>>> all suggestions so far. It would be great if people could take another look
>>> at it. I ll iterate fast on any comments so we get this in by the time rest
>>> of the code pieces are committed.
>>> 
>>> Thanks
>>> Karthik
>>> 
>>> 
>>> 
>>> 
>>> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
>>> 
>>>> Just curious of your procedures. Given that there is at least one blocker
>>>> JIRA out there that has yet to be fully resolved, do you intend to release
>>>> anyway?
>>>> 
>>>> 
>>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>>> 
>>>>> If the intention is to get the release out in time for the Hadoop Summit
>>>> we
>>>>> have a very tight schedule.
>>>>> 
>>>>> Because the release vote runs for 7 days, we should have an RC latest
>>>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>>>> consider is that if the changes on an RC are corrections that do not
>>>> affect
>>>>> code, we could agree on not reseting the voting period clock if we need
>>>> to
>>>>> cut a new RC (ie doc, build, notes changes).
>>>>> 
>>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>>>> are
>>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>>>> 
>>>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>>>> 
>>>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>>>> things for the folks doing QA.
>>>>> 
>>>>> Arun, as you are coordinating the work for this release, please do commit
>>>>> them or give me the go ahead and I'll commit.
>>>>> 
>>>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>>>> the changes on the milli-slot calculations do not affect the current
>>>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>>>> 
>>>>> I'll be checking my email over the weekend and I can take care of some
>>>>> stuff if needed (while the monkeys sleep).
>>>>> 
>>>>> Thx
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>>> wrote:
>>>>> 
>>>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>>>> 2.1 release:
>>>>>> 
>>>>>> From the 1st group I think all 3 should make.
>>>>>> 
>>>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>>>> MAPREDUCE-5130.
>>>>>> 
>>>>>> From the 3rd group, I don't think this JIRA will make it.
>>>>>> 
>>>>>> From the 4th group, we don't need to worry about this or 2.1
>>>>>> 
>>>>>> Thanks
>>>>>> 
>>>>>> Alejandro
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> JIRAs that are in shape to make it to 2.1
>>>>>> 
>>>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>>>> for requested nodes
>>>>>> 
>>>>>> impact: behavior change
>>>>>> 
>>>>>> status: patch avail, +1ed.
>>>>>> 
>>>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>>>> 
>>>>>> impact: Addition to MRAM HTTP API
>>>>>> 
>>>>>> status: patch avail, +1ed, needs to be committed
>>>>>> 
>>>>>> * YARN-787: Remove resource min from Yarn client API
>>>>>> 
>>>>>> impact: Yarn client API change
>>>>>> 
>>>>>> status: patch avail, needs to be reviewed. (the calculation of
>>>> slot-millis
>>>>>> is not affected, the MIN is taken from conf for now)
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> JIRAs that require minor work to make it to 2.1
>>>>>> 
>>>>>> * YARN-521: Augment AM - RM client module to be able to request
>>>> containers
>>>>>> only at specific locations
>>>>>> 
>>>>>> impact: AMRM client API change
>>>>>> 
>>>>>> status: patch not avail yet (requires YARN-752)
>>>>>> 
>>>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>>>> with
>>>>>> /nodes REST API
>>>>>> 
>>>>>> impact: Yarn client API & proto change
>>>>>> 
>>>>>> status: patch avail, review in progress
>>>>>> 
>>>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>>>> 
>>>>>> impact: behavior change
>>>>>> 
>>>>>> status: patch avail but some tests are failing
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> JIRAs that require significant work to make it to 2.1 and may not make
>>>> it
>>>>>> 
>>>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>>>> 
>>>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>>>> does not change API)
>>>>>> 
>>>>>> status: patch avail, review in progress
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> JIRAs that don't need to make it to 2.1
>>>>>> 
>>>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>>>> counter constants
>>>>>> 
>>>>>> impact: behavior change
>>>>>> 
>>>>>> status: per discussion we should first add memory-millis and
>>>> vcores-millis
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> 
>>>>>> 
>>>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>>>> wrote:
>>>>>> 
>>>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>>>> 
>>>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>>>> platform
>>>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>>>> battery
>>>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>>>> 
>>>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>>>> also
>>>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>>>> 
>>>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>>>> Blocker priority to catch attention.
>>>>>>> 
>>>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>>>> platforms!):
>>>>>>> http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>>>> 
>>>>>>> The immediate benefit here is that we get to see that the
>>>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>>>> 
>>>>>>> Starting from next week, I'll start running regular tests
>>>>>>> on these bits and will keep you guys posted!
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Roman.
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Alejandro
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Alejandro
>>>> 
>>>> 
>> 
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

Re: Heads up: branch-2.1-beta

[Ralph Castain <rh...@open-mpi.org>]

HDFS-4866 is marked as a blocker. A patch was submitted and applied, but it only fixes compilation. As I marked on the ticket, the results remain unusable due to linker issues.


On Jun 16, 2013, at 5:33 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Which one are you talking about Ralph? This doesn't show up on any blocker list.
> 
> http://s.apache.org/hadoop-blocker-bugs
> 
> Arun
> 
> 
> On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:
> 
>> Not trying to be a pain, but I am trying to get clarification. The protocol buffer support is still broken. Do you intend to release 2.1 with that unfixed?
>> 
>> 
>> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
>> 
>>> Re-posting here to the wider audience:
>>> 
>>> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
>>> all suggestions so far. It would be great if people could take another look
>>> at it. I ll iterate fast on any comments so we get this in by the time rest
>>> of the code pieces are committed.
>>> 
>>> Thanks
>>> Karthik
>>> 
>>> 
>>> 
>>> 
>>> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
>>> 
>>>> Just curious of your procedures. Given that there is at least one blocker
>>>> JIRA out there that has yet to be fully resolved, do you intend to release
>>>> anyway?
>>>> 
>>>> 
>>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>>> 
>>>>> If the intention is to get the release out in time for the Hadoop Summit
>>>> we
>>>>> have a very tight schedule.
>>>>> 
>>>>> Because the release vote runs for 7 days, we should have an RC latest
>>>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>>>> consider is that if the changes on an RC are corrections that do not
>>>> affect
>>>>> code, we could agree on not reseting the voting period clock if we need
>>>> to
>>>>> cut a new RC (ie doc, build, notes changes).
>>>>> 
>>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>>>> are
>>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>>>> 
>>>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>>>> 
>>>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>>>> things for the folks doing QA.
>>>>> 
>>>>> Arun, as you are coordinating the work for this release, please do commit
>>>>> them or give me the go ahead and I'll commit.
>>>>> 
>>>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>>>> the changes on the milli-slot calculations do not affect the current
>>>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>>>> 
>>>>> I'll be checking my email over the weekend and I can take care of some
>>>>> stuff if needed (while the monkeys sleep).
>>>>> 
>>>>> Thx
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>>> wrote:
>>>>> 
>>>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>>>> 2.1 release:
>>>>>> 
>>>>>> From the 1st group I think all 3 should make.
>>>>>> 
>>>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>>>> MAPREDUCE-5130.
>>>>>> 
>>>>>> From the 3rd group, I don't think this JIRA will make it.
>>>>>> 
>>>>>> From the 4th group, we don't need to worry about this or 2.1
>>>>>> 
>>>>>> Thanks
>>>>>> 
>>>>>> Alejandro
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> JIRAs that are in shape to make it to 2.1
>>>>>> 
>>>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>>>> for requested nodes
>>>>>> 
>>>>>> impact: behavior change
>>>>>> 
>>>>>> status: patch avail, +1ed.
>>>>>> 
>>>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>>>> 
>>>>>> impact: Addition to MRAM HTTP API
>>>>>> 
>>>>>> status: patch avail, +1ed, needs to be committed
>>>>>> 
>>>>>> * YARN-787: Remove resource min from Yarn client API
>>>>>> 
>>>>>> impact: Yarn client API change
>>>>>> 
>>>>>> status: patch avail, needs to be reviewed. (the calculation of
>>>> slot-millis
>>>>>> is not affected, the MIN is taken from conf for now)
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> JIRAs that require minor work to make it to 2.1
>>>>>> 
>>>>>> * YARN-521: Augment AM - RM client module to be able to request
>>>> containers
>>>>>> only at specific locations
>>>>>> 
>>>>>> impact: AMRM client API change
>>>>>> 
>>>>>> status: patch not avail yet (requires YARN-752)
>>>>>> 
>>>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>>>> with
>>>>>> /nodes REST API
>>>>>> 
>>>>>> impact: Yarn client API & proto change
>>>>>> 
>>>>>> status: patch avail, review in progress
>>>>>> 
>>>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>>>> 
>>>>>> impact: behavior change
>>>>>> 
>>>>>> status: patch avail but some tests are failing
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> JIRAs that require significant work to make it to 2.1 and may not make
>>>> it
>>>>>> 
>>>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>>>> 
>>>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>>>> does not change API)
>>>>>> 
>>>>>> status: patch avail, review in progress
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> JIRAs that don't need to make it to 2.1
>>>>>> 
>>>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>>>> counter constants
>>>>>> 
>>>>>> impact: behavior change
>>>>>> 
>>>>>> status: per discussion we should first add memory-millis and
>>>> vcores-millis
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> 
>>>>>> 
>>>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>>>> wrote:
>>>>>> 
>>>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>>>> 
>>>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>>>> platform
>>>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>>>> battery
>>>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>>>> 
>>>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>>>> also
>>>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>>>> 
>>>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>>>> Blocker priority to catch attention.
>>>>>>> 
>>>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>>>> platforms!):
>>>>>>> http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>>>> 
>>>>>>> The immediate benefit here is that we get to see that the
>>>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>>>> 
>>>>>>> Starting from next week, I'll start running regular tests
>>>>>>> on these bits and will keep you guys posted!
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Roman.
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Alejandro
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Alejandro
>>>> 
>>>> 
>> 
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

Re: Heads up: branch-2.1-beta

[Ralph Castain <ra...@castain.org>]

HDFS-4866 is marked as a blocker. A patch was submitted that lets it compile, but it still results in an unusable state because you can't link to the resulting code.

On Jun 16, 2013, at 5:33 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Which one are you talking about Ralph? This doesn't show up on any blocker list.
> 
> http://s.apache.org/hadoop-blocker-bugs
> 
> Arun
> 
> 
> On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:
> 
>> Not trying to be a pain, but I am trying to get clarification. The protocol buffer support is still broken. Do you intend to release 2.1 with that unfixed?
>> 
>> 
>> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
>> 
>>> Re-posting here to the wider audience:
>>> 
>>> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
>>> all suggestions so far. It would be great if people could take another look
>>> at it. I ll iterate fast on any comments so we get this in by the time rest
>>> of the code pieces are committed.
>>> 
>>> Thanks
>>> Karthik
>>> 
>>> 
>>> 
>>> 
>>> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
>>> 
>>>> Just curious of your procedures. Given that there is at least one blocker
>>>> JIRA out there that has yet to be fully resolved, do you intend to release
>>>> anyway?
>>>> 
>>>> 
>>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>>> 
>>>>> If the intention is to get the release out in time for the Hadoop Summit
>>>> we
>>>>> have a very tight schedule.
>>>>> 
>>>>> Because the release vote runs for 7 days, we should have an RC latest
>>>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>>>> consider is that if the changes on an RC are corrections that do not
>>>> affect
>>>>> code, we could agree on not reseting the voting period clock if we need
>>>> to
>>>>> cut a new RC (ie doc, build, notes changes).
>>>>> 
>>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>>>> are
>>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>>>> 
>>>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>>>> 
>>>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>>>> things for the folks doing QA.
>>>>> 
>>>>> Arun, as you are coordinating the work for this release, please do commit
>>>>> them or give me the go ahead and I'll commit.
>>>>> 
>>>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>>>> the changes on the milli-slot calculations do not affect the current
>>>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>>>> 
>>>>> I'll be checking my email over the weekend and I can take care of some
>>>>> stuff if needed (while the monkeys sleep).
>>>>> 
>>>>> Thx
>>>>> 
>>>>> 
>>>>> 
>>>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>>> wrote:
>>>>> 
>>>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>>>> 2.1 release:
>>>>>> 
>>>>>> From the 1st group I think all 3 should make.
>>>>>> 
>>>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>>>> MAPREDUCE-5130.
>>>>>> 
>>>>>> From the 3rd group, I don't think this JIRA will make it.
>>>>>> 
>>>>>> From the 4th group, we don't need to worry about this or 2.1
>>>>>> 
>>>>>> Thanks
>>>>>> 
>>>>>> Alejandro
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> JIRAs that are in shape to make it to 2.1
>>>>>> 
>>>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>>>> for requested nodes
>>>>>> 
>>>>>> impact: behavior change
>>>>>> 
>>>>>> status: patch avail, +1ed.
>>>>>> 
>>>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>>>> 
>>>>>> impact: Addition to MRAM HTTP API
>>>>>> 
>>>>>> status: patch avail, +1ed, needs to be committed
>>>>>> 
>>>>>> * YARN-787: Remove resource min from Yarn client API
>>>>>> 
>>>>>> impact: Yarn client API change
>>>>>> 
>>>>>> status: patch avail, needs to be reviewed. (the calculation of
>>>> slot-millis
>>>>>> is not affected, the MIN is taken from conf for now)
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> JIRAs that require minor work to make it to 2.1
>>>>>> 
>>>>>> * YARN-521: Augment AM - RM client module to be able to request
>>>> containers
>>>>>> only at specific locations
>>>>>> 
>>>>>> impact: AMRM client API change
>>>>>> 
>>>>>> status: patch not avail yet (requires YARN-752)
>>>>>> 
>>>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>>>> with
>>>>>> /nodes REST API
>>>>>> 
>>>>>> impact: Yarn client API & proto change
>>>>>> 
>>>>>> status: patch avail, review in progress
>>>>>> 
>>>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>>>> 
>>>>>> impact: behavior change
>>>>>> 
>>>>>> status: patch avail but some tests are failing
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> JIRAs that require significant work to make it to 2.1 and may not make
>>>> it
>>>>>> 
>>>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>>>> 
>>>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>>>> does not change API)
>>>>>> 
>>>>>> status: patch avail, review in progress
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> JIRAs that don't need to make it to 2.1
>>>>>> 
>>>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>>>> counter constants
>>>>>> 
>>>>>> impact: behavior change
>>>>>> 
>>>>>> status: per discussion we should first add memory-millis and
>>>> vcores-millis
>>>>>> 
>>>>>> ------------------------------------------------------
>>>>>> 
>>>>>> 
>>>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>>>> wrote:
>>>>>> 
>>>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>>>> 
>>>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>>>> platform
>>>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>>>> battery
>>>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>>>> 
>>>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>>>> also
>>>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>>>> 
>>>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>>>> Blocker priority to catch attention.
>>>>>>> 
>>>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>>>> platforms!):
>>>>>>> http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>>>> 
>>>>>>> The immediate benefit here is that we get to see that the
>>>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>>>> 
>>>>>>> Starting from next week, I'll start running regular tests
>>>>>>> on these bits and will keep you guys posted!
>>>>>>> 
>>>>>>> Thanks,
>>>>>>> Roman.
>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> --
>>>>>> Alejandro
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Alejandro
>>>> 
>>>> 
>> 
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Which one are you talking about Ralph? This doesn't show up on any blocker list.

http://s.apache.org/hadoop-blocker-bugs

Arun


On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:

> Not trying to be a pain, but I am trying to get clarification. The protocol buffer support is still broken. Do you intend to release 2.1 with that unfixed?
> 
> 
> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
> 
>> Re-posting here to the wider audience:
>> 
>> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
>> all suggestions so far. It would be great if people could take another look
>> at it. I ll iterate fast on any comments so we get this in by the time rest
>> of the code pieces are committed.
>> 
>> Thanks
>> Karthik
>> 
>> 
>> 
>> 
>> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
>> 
>>> Just curious of your procedures. Given that there is at least one blocker
>>> JIRA out there that has yet to be fully resolved, do you intend to release
>>> anyway?
>>> 
>>> 
>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>> 
>>>> If the intention is to get the release out in time for the Hadoop Summit
>>> we
>>>> have a very tight schedule.
>>>> 
>>>> Because the release vote runs for 7 days, we should have an RC latest
>>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>>> consider is that if the changes on an RC are corrections that do not
>>> affect
>>>> code, we could agree on not reseting the voting period clock if we need
>>> to
>>>> cut a new RC (ie doc, build, notes changes).
>>>> 
>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>>> are
>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>>> 
>>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>>> 
>>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>>> things for the folks doing QA.
>>>> 
>>>> Arun, as you are coordinating the work for this release, please do commit
>>>> them or give me the go ahead and I'll commit.
>>>> 
>>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>>> the changes on the milli-slot calculations do not affect the current
>>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>>> 
>>>> I'll be checking my email over the weekend and I can take care of some
>>>> stuff if needed (while the monkeys sleep).
>>>> 
>>>> Thx
>>>> 
>>>> 
>>>> 
>>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>> wrote:
>>>> 
>>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>>> 2.1 release:
>>>>> 
>>>>> From the 1st group I think all 3 should make.
>>>>> 
>>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>>> MAPREDUCE-5130.
>>>>> 
>>>>> From the 3rd group, I don't think this JIRA will make it.
>>>>> 
>>>>> From the 4th group, we don't need to worry about this or 2.1
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> Alejandro
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that are in shape to make it to 2.1
>>>>> 
>>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>>> for requested nodes
>>>>> 
>>>>> impact: behavior change
>>>>> 
>>>>> status: patch avail, +1ed.
>>>>> 
>>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>>> 
>>>>> impact: Addition to MRAM HTTP API
>>>>> 
>>>>> status: patch avail, +1ed, needs to be committed
>>>>> 
>>>>> * YARN-787: Remove resource min from Yarn client API
>>>>> 
>>>>> impact: Yarn client API change
>>>>> 
>>>>> status: patch avail, needs to be reviewed. (the calculation of
>>> slot-millis
>>>>> is not affected, the MIN is taken from conf for now)
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that require minor work to make it to 2.1
>>>>> 
>>>>> * YARN-521: Augment AM - RM client module to be able to request
>>> containers
>>>>> only at specific locations
>>>>> 
>>>>> impact: AMRM client API change
>>>>> 
>>>>> status: patch not avail yet (requires YARN-752)
>>>>> 
>>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>>> with
>>>>> /nodes REST API
>>>>> 
>>>>> impact: Yarn client API & proto change
>>>>> 
>>>>> status: patch avail, review in progress
>>>>> 
>>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>>> 
>>>>> impact: behavior change
>>>>> 
>>>>> status: patch avail but some tests are failing
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that require significant work to make it to 2.1 and may not make
>>> it
>>>>> 
>>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>>> 
>>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>>> does not change API)
>>>>> 
>>>>> status: patch avail, review in progress
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that don't need to make it to 2.1
>>>>> 
>>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>>> counter constants
>>>>> 
>>>>> impact: behavior change
>>>>> 
>>>>> status: per discussion we should first add memory-millis and
>>> vcores-millis
>>>>> 
>>>>> ------------------------------------------------------
>>>>> 
>>>>> 
>>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>>> wrote:
>>>>> 
>>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>>> 
>>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>>> platform
>>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>>> battery
>>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>>> 
>>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>>> also
>>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>>> 
>>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>>> Blocker priority to catch attention.
>>>>>> 
>>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>>> platforms!):
>>>>>>  http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>>> 
>>>>>> The immediate benefit here is that we get to see that the
>>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>>> 
>>>>>> Starting from next week, I'll start running regular tests
>>>>>> on these bits and will keep you guys posted!
>>>>>> 
>>>>>> Thanks,
>>>>>> Roman.
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Alejandro
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Alejandro
>>> 
>>> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Which one are you talking about Ralph? This doesn't show up on any blocker list.

http://s.apache.org/hadoop-blocker-bugs

Arun


On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:

> Not trying to be a pain, but I am trying to get clarification. The protocol buffer support is still broken. Do you intend to release 2.1 with that unfixed?
> 
> 
> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
> 
>> Re-posting here to the wider audience:
>> 
>> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
>> all suggestions so far. It would be great if people could take another look
>> at it. I ll iterate fast on any comments so we get this in by the time rest
>> of the code pieces are committed.
>> 
>> Thanks
>> Karthik
>> 
>> 
>> 
>> 
>> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
>> 
>>> Just curious of your procedures. Given that there is at least one blocker
>>> JIRA out there that has yet to be fully resolved, do you intend to release
>>> anyway?
>>> 
>>> 
>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>> 
>>>> If the intention is to get the release out in time for the Hadoop Summit
>>> we
>>>> have a very tight schedule.
>>>> 
>>>> Because the release vote runs for 7 days, we should have an RC latest
>>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>>> consider is that if the changes on an RC are corrections that do not
>>> affect
>>>> code, we could agree on not reseting the voting period clock if we need
>>> to
>>>> cut a new RC (ie doc, build, notes changes).
>>>> 
>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>>> are
>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>>> 
>>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>>> 
>>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>>> things for the folks doing QA.
>>>> 
>>>> Arun, as you are coordinating the work for this release, please do commit
>>>> them or give me the go ahead and I'll commit.
>>>> 
>>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>>> the changes on the milli-slot calculations do not affect the current
>>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>>> 
>>>> I'll be checking my email over the weekend and I can take care of some
>>>> stuff if needed (while the monkeys sleep).
>>>> 
>>>> Thx
>>>> 
>>>> 
>>>> 
>>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>> wrote:
>>>> 
>>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>>> 2.1 release:
>>>>> 
>>>>> From the 1st group I think all 3 should make.
>>>>> 
>>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>>> MAPREDUCE-5130.
>>>>> 
>>>>> From the 3rd group, I don't think this JIRA will make it.
>>>>> 
>>>>> From the 4th group, we don't need to worry about this or 2.1
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> Alejandro
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that are in shape to make it to 2.1
>>>>> 
>>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>>> for requested nodes
>>>>> 
>>>>> impact: behavior change
>>>>> 
>>>>> status: patch avail, +1ed.
>>>>> 
>>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>>> 
>>>>> impact: Addition to MRAM HTTP API
>>>>> 
>>>>> status: patch avail, +1ed, needs to be committed
>>>>> 
>>>>> * YARN-787: Remove resource min from Yarn client API
>>>>> 
>>>>> impact: Yarn client API change
>>>>> 
>>>>> status: patch avail, needs to be reviewed. (the calculation of
>>> slot-millis
>>>>> is not affected, the MIN is taken from conf for now)
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that require minor work to make it to 2.1
>>>>> 
>>>>> * YARN-521: Augment AM - RM client module to be able to request
>>> containers
>>>>> only at specific locations
>>>>> 
>>>>> impact: AMRM client API change
>>>>> 
>>>>> status: patch not avail yet (requires YARN-752)
>>>>> 
>>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>>> with
>>>>> /nodes REST API
>>>>> 
>>>>> impact: Yarn client API & proto change
>>>>> 
>>>>> status: patch avail, review in progress
>>>>> 
>>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>>> 
>>>>> impact: behavior change
>>>>> 
>>>>> status: patch avail but some tests are failing
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that require significant work to make it to 2.1 and may not make
>>> it
>>>>> 
>>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>>> 
>>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>>> does not change API)
>>>>> 
>>>>> status: patch avail, review in progress
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that don't need to make it to 2.1
>>>>> 
>>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>>> counter constants
>>>>> 
>>>>> impact: behavior change
>>>>> 
>>>>> status: per discussion we should first add memory-millis and
>>> vcores-millis
>>>>> 
>>>>> ------------------------------------------------------
>>>>> 
>>>>> 
>>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>>> wrote:
>>>>> 
>>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>>> 
>>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>>> platform
>>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>>> battery
>>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>>> 
>>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>>> also
>>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>>> 
>>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>>> Blocker priority to catch attention.
>>>>>> 
>>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>>> platforms!):
>>>>>>  http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>>> 
>>>>>> The immediate benefit here is that we get to see that the
>>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>>> 
>>>>>> Starting from next week, I'll start running regular tests
>>>>>> on these bits and will keep you guys posted!
>>>>>> 
>>>>>> Thanks,
>>>>>> Roman.
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Alejandro
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Alejandro
>>> 
>>> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Which one are you talking about Ralph? This doesn't show up on any blocker list.

http://s.apache.org/hadoop-blocker-bugs

Arun


On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:

> Not trying to be a pain, but I am trying to get clarification. The protocol buffer support is still broken. Do you intend to release 2.1 with that unfixed?
> 
> 
> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
> 
>> Re-posting here to the wider audience:
>> 
>> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
>> all suggestions so far. It would be great if people could take another look
>> at it. I ll iterate fast on any comments so we get this in by the time rest
>> of the code pieces are committed.
>> 
>> Thanks
>> Karthik
>> 
>> 
>> 
>> 
>> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
>> 
>>> Just curious of your procedures. Given that there is at least one blocker
>>> JIRA out there that has yet to be fully resolved, do you intend to release
>>> anyway?
>>> 
>>> 
>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>> 
>>>> If the intention is to get the release out in time for the Hadoop Summit
>>> we
>>>> have a very tight schedule.
>>>> 
>>>> Because the release vote runs for 7 days, we should have an RC latest
>>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>>> consider is that if the changes on an RC are corrections that do not
>>> affect
>>>> code, we could agree on not reseting the voting period clock if we need
>>> to
>>>> cut a new RC (ie doc, build, notes changes).
>>>> 
>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>>> are
>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>>> 
>>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>>> 
>>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>>> things for the folks doing QA.
>>>> 
>>>> Arun, as you are coordinating the work for this release, please do commit
>>>> them or give me the go ahead and I'll commit.
>>>> 
>>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>>> the changes on the milli-slot calculations do not affect the current
>>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>>> 
>>>> I'll be checking my email over the weekend and I can take care of some
>>>> stuff if needed (while the monkeys sleep).
>>>> 
>>>> Thx
>>>> 
>>>> 
>>>> 
>>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>> wrote:
>>>> 
>>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>>> 2.1 release:
>>>>> 
>>>>> From the 1st group I think all 3 should make.
>>>>> 
>>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>>> MAPREDUCE-5130.
>>>>> 
>>>>> From the 3rd group, I don't think this JIRA will make it.
>>>>> 
>>>>> From the 4th group, we don't need to worry about this or 2.1
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> Alejandro
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that are in shape to make it to 2.1
>>>>> 
>>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>>> for requested nodes
>>>>> 
>>>>> impact: behavior change
>>>>> 
>>>>> status: patch avail, +1ed.
>>>>> 
>>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>>> 
>>>>> impact: Addition to MRAM HTTP API
>>>>> 
>>>>> status: patch avail, +1ed, needs to be committed
>>>>> 
>>>>> * YARN-787: Remove resource min from Yarn client API
>>>>> 
>>>>> impact: Yarn client API change
>>>>> 
>>>>> status: patch avail, needs to be reviewed. (the calculation of
>>> slot-millis
>>>>> is not affected, the MIN is taken from conf for now)
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that require minor work to make it to 2.1
>>>>> 
>>>>> * YARN-521: Augment AM - RM client module to be able to request
>>> containers
>>>>> only at specific locations
>>>>> 
>>>>> impact: AMRM client API change
>>>>> 
>>>>> status: patch not avail yet (requires YARN-752)
>>>>> 
>>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>>> with
>>>>> /nodes REST API
>>>>> 
>>>>> impact: Yarn client API & proto change
>>>>> 
>>>>> status: patch avail, review in progress
>>>>> 
>>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>>> 
>>>>> impact: behavior change
>>>>> 
>>>>> status: patch avail but some tests are failing
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that require significant work to make it to 2.1 and may not make
>>> it
>>>>> 
>>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>>> 
>>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>>> does not change API)
>>>>> 
>>>>> status: patch avail, review in progress
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that don't need to make it to 2.1
>>>>> 
>>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>>> counter constants
>>>>> 
>>>>> impact: behavior change
>>>>> 
>>>>> status: per discussion we should first add memory-millis and
>>> vcores-millis
>>>>> 
>>>>> ------------------------------------------------------
>>>>> 
>>>>> 
>>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>>> wrote:
>>>>> 
>>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>>> 
>>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>>> platform
>>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>>> battery
>>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>>> 
>>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>>> also
>>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>>> 
>>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>>> Blocker priority to catch attention.
>>>>>> 
>>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>>> platforms!):
>>>>>>  http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>>> 
>>>>>> The immediate benefit here is that we get to see that the
>>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>>> 
>>>>>> Starting from next week, I'll start running regular tests
>>>>>> on these bits and will keep you guys posted!
>>>>>> 
>>>>>> Thanks,
>>>>>> Roman.
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Alejandro
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Alejandro
>>> 
>>> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Which one are you talking about Ralph? This doesn't show up on any blocker list.

http://s.apache.org/hadoop-blocker-bugs

Arun


On Jun 15, 2013, at 4:44 PM, Ralph Castain wrote:

> Not trying to be a pain, but I am trying to get clarification. The protocol buffer support is still broken. Do you intend to release 2.1 with that unfixed?
> 
> 
> On Jun 15, 2013, at 4:18 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
> 
>> Re-posting here to the wider audience:
>> 
>> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
>> all suggestions so far. It would be great if people could take another look
>> at it. I ll iterate fast on any comments so we get this in by the time rest
>> of the code pieces are committed.
>> 
>> Thanks
>> Karthik
>> 
>> 
>> 
>> 
>> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
>> 
>>> Just curious of your procedures. Given that there is at least one blocker
>>> JIRA out there that has yet to be fully resolved, do you intend to release
>>> anyway?
>>> 
>>> 
>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>> 
>>>> If the intention is to get the release out in time for the Hadoop Summit
>>> we
>>>> have a very tight schedule.
>>>> 
>>>> Because the release vote runs for 7 days, we should have an RC latest
>>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>>> consider is that if the changes on an RC are corrections that do not
>>> affect
>>>> code, we could agree on not reseting the voting period clock if we need
>>> to
>>>> cut a new RC (ie doc, build, notes changes).
>>>> 
>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>>> are
>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>>> 
>>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>>> 
>>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>>> things for the folks doing QA.
>>>> 
>>>> Arun, as you are coordinating the work for this release, please do commit
>>>> them or give me the go ahead and I'll commit.
>>>> 
>>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>>> the changes on the milli-slot calculations do not affect the current
>>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>>> 
>>>> I'll be checking my email over the weekend and I can take care of some
>>>> stuff if needed (while the monkeys sleep).
>>>> 
>>>> Thx
>>>> 
>>>> 
>>>> 
>>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>>> wrote:
>>>> 
>>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>>> 2.1 release:
>>>>> 
>>>>> From the 1st group I think all 3 should make.
>>>>> 
>>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>>> MAPREDUCE-5130.
>>>>> 
>>>>> From the 3rd group, I don't think this JIRA will make it.
>>>>> 
>>>>> From the 4th group, we don't need to worry about this or 2.1
>>>>> 
>>>>> Thanks
>>>>> 
>>>>> Alejandro
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that are in shape to make it to 2.1
>>>>> 
>>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>>> for requested nodes
>>>>> 
>>>>> impact: behavior change
>>>>> 
>>>>> status: patch avail, +1ed.
>>>>> 
>>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>>> 
>>>>> impact: Addition to MRAM HTTP API
>>>>> 
>>>>> status: patch avail, +1ed, needs to be committed
>>>>> 
>>>>> * YARN-787: Remove resource min from Yarn client API
>>>>> 
>>>>> impact: Yarn client API change
>>>>> 
>>>>> status: patch avail, needs to be reviewed. (the calculation of
>>> slot-millis
>>>>> is not affected, the MIN is taken from conf for now)
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that require minor work to make it to 2.1
>>>>> 
>>>>> * YARN-521: Augment AM - RM client module to be able to request
>>> containers
>>>>> only at specific locations
>>>>> 
>>>>> impact: AMRM client API change
>>>>> 
>>>>> status: patch not avail yet (requires YARN-752)
>>>>> 
>>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>>> with
>>>>> /nodes REST API
>>>>> 
>>>>> impact: Yarn client API & proto change
>>>>> 
>>>>> status: patch avail, review in progress
>>>>> 
>>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>>> 
>>>>> impact: behavior change
>>>>> 
>>>>> status: patch avail but some tests are failing
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that require significant work to make it to 2.1 and may not make
>>> it
>>>>> 
>>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>>> 
>>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>>> does not change API)
>>>>> 
>>>>> status: patch avail, review in progress
>>>>> 
>>>>> ------------------------------------------------------
>>>>> JIRAs that don't need to make it to 2.1
>>>>> 
>>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>>> counter constants
>>>>> 
>>>>> impact: behavior change
>>>>> 
>>>>> status: per discussion we should first add memory-millis and
>>> vcores-millis
>>>>> 
>>>>> ------------------------------------------------------
>>>>> 
>>>>> 
>>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>>> wrote:
>>>>> 
>>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>> 
>>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>>> 
>>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>>> platform
>>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>>> battery
>>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>>> 
>>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>>> also
>>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>>> 
>>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>>> Blocker priority to catch attention.
>>>>>> 
>>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>>> platforms!):
>>>>>>  http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>>> 
>>>>>> The immediate benefit here is that we get to see that the
>>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>>> 
>>>>>> Starting from next week, I'll start running regular tests
>>>>>> on these bits and will keep you guys posted!
>>>>>> 
>>>>>> Thanks,
>>>>>> Roman.
>>>>>> 
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> Alejandro
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Alejandro
>>> 
>>> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Ralph Castain <ra...@castain.org>]

Not trying to be a pain, but I am trying to get clarification. The protocol buffer support is still broken. Do you intend to release 2.1 with that unfixed?


On Jun 15, 2013, at 4:18 PM, Karthik Kambatla <ka...@cloudera.com> wrote:

> Re-posting here to the wider audience:
> 
> HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
> all suggestions so far. It would be great if people could take another look
> at it. I ll iterate fast on any comments so we get this in by the time rest
> of the code pieces are committed.
> 
> Thanks
> Karthik
> 
> 
> 
> 
> On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:
> 
>> Just curious of your procedures. Given that there is at least one blocker
>> JIRA out there that has yet to be fully resolved, do you intend to release
>> anyway?
>> 
>> 
>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>> 
>>> If the intention is to get the release out in time for the Hadoop Summit
>> we
>>> have a very tight schedule.
>>> 
>>> Because the release vote runs for 7 days, we should have an RC latest
>>> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
>>> if we need to cut a new RC we can do it on Tuesday. Another thing to
>>> consider is that if the changes on an RC are corrections that do not
>> affect
>>> code, we could agree on not reseting the voting period clock if we need
>> to
>>> cut a new RC (ie doc, build, notes changes).
>>> 
>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>> are
>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>> 
>>> The first 2 are already +1ed, the last one needs to be reviewed.
>>> 
>>> I have not committed the first 2 ones yet because I don't want to disrupt
>>> things for the folks doing QA.
>>> 
>>> Arun, as you are coordinating the work for this release, please do commit
>>> them or give me the go ahead and I'll commit.
>>> 
>>> Also, it  would be great if you can review YARN-787 (as per discussions,
>>> the changes on the milli-slot calculations do not affect the current
>>> calculations, that would be left for MAPREDUCE-5311 to do).
>>> 
>>> I'll be checking my email over the weekend and I can take care of some
>>> stuff if needed (while the monkeys sleep).
>>> 
>>> Thx
>>> 
>>> 
>>> 
>>> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>> 
>>>> Following is a revisited assessment of JIRAs I would like to get in the
>>>> 2.1 release:
>>>> 
>>>> From the 1st group I think all 3 should make.
>>>> 
>>>> From the 2nd group I think YARN-791 should make it for sure and ideally
>>>> MAPREDUCE-5130.
>>>> 
>>>> From the 3rd group, I don't think this JIRA will make it.
>>>> 
>>>> From the 4th group, we don't need to worry about this or 2.1
>>>> 
>>>> Thanks
>>>> 
>>>> Alejandro
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that are in shape to make it to 2.1
>>>> 
>>>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>>>> for requested nodes
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: patch avail, +1ed.
>>>> 
>>>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>>>> 
>>>> impact: Addition to MRAM HTTP API
>>>> 
>>>> status: patch avail, +1ed, needs to be committed
>>>> 
>>>> * YARN-787: Remove resource min from Yarn client API
>>>> 
>>>> impact: Yarn client API change
>>>> 
>>>> status: patch avail, needs to be reviewed. (the calculation of
>> slot-millis
>>>> is not affected, the MIN is taken from conf for now)
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that require minor work to make it to 2.1
>>>> 
>>>> * YARN-521: Augment AM - RM client module to be able to request
>> containers
>>>> only at specific locations
>>>> 
>>>> impact: AMRM client API change
>>>> 
>>>> status: patch not avail yet (requires YARN-752)
>>>> 
>>>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
>> with
>>>> /nodes REST API
>>>> 
>>>> impact: Yarn client API & proto change
>>>> 
>>>> status: patch avail, review in progress
>>>> 
>>>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: patch avail but some tests are failing
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that require significant work to make it to 2.1 and may not make
>> it
>>>> 
>>>> * YARN-649: Make container logs available over HTTP in plain text
>>>> 
>>>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>>>> does not change API)
>>>> 
>>>> status: patch avail, review in progress
>>>> 
>>>> ------------------------------------------------------
>>>> JIRAs that don't need to make it to 2.1
>>>> 
>>>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>>>> counter constants
>>>> 
>>>> impact: behavior change
>>>> 
>>>> status: per discussion we should first add memory-millis and
>> vcores-millis
>>>> 
>>>> ------------------------------------------------------
>>>> 
>>>> 
>>>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
>> wrote:
>>>> 
>>>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>>>> wrote:
>>>>>> 
>>>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>>>> 
>>>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
>> platform
>>>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>>>> battery
>>>>>>> of integration tests on the branch-2.1-beta.
>>>>>>> 
>>>>>>> We plan to simply file JIRAs if anything gets detected and I will
>> also
>>>>>>> publish the URL of the Jenkins job once it gets created.
>>>>>> 
>>>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>>>> Blocker priority to catch attention.
>>>>> 
>>>>> The build is up and running (and all green on all of the 9 Linux
>>>>> platforms!):
>>>>>   http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>>>> 
>>>>> The immediate benefit here is that we get to see that the
>>>>> build is ok on all these Linuxes and all anybody can easily
>>>>> install packaged Hadoop 2.1.0 nightly builds.
>>>>> 
>>>>> Starting from next week, I'll start running regular tests
>>>>> on these bits and will keep you guys posted!
>>>>> 
>>>>> Thanks,
>>>>> Roman.
>>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Alejandro
>>>> 
>>> 
>>> 
>>> 
>>> --
>>> Alejandro
>> 
>> 

Re: Heads up: branch-2.1-beta

[Karthik Kambatla <ka...@cloudera.com>]

Re-posting here to the wider audience:

HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
all suggestions so far. It would be great if people could take another look
at it. I ll iterate fast on any comments so we get this in by the time rest
of the code pieces are committed.

Thanks
Karthik




On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:

> Just curious of your procedures. Given that there is at least one blocker
> JIRA out there that has yet to be fully resolved, do you intend to release
> anyway?
>
>
> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > If the intention is to get the release out in time for the Hadoop Summit
> we
> > have a very tight schedule.
> >
> > Because the release vote runs for 7 days, we should have an RC latest
> > Monday afternoon, and we should encourage folks to verify & vote ASAP, so
> > if we need to cut a new RC we can do it on Tuesday. Another thing to
> > consider is that if the changes on an RC are corrections that do not
> affect
> > code, we could agree on not reseting the voting period clock if we need
> to
> > cut a new RC (ie doc, build, notes changes).
> >
> > Of the JIRAs in my laundry list for 2.1 the ones I would really want in
> are
> > YARN-752, MAPREDUCE-5171 & YARN-787.
> >
> > The first 2 are already +1ed, the last one needs to be reviewed.
> >
> > I have not committed the first 2 ones yet because I don't want to disrupt
> > things for the folks doing QA.
> >
> > Arun, as you are coordinating the work for this release, please do commit
> > them or give me the go ahead and I'll commit.
> >
> > Also, it  would be great if you can review YARN-787 (as per discussions,
> > the changes on the milli-slot calculations do not affect the current
> > calculations, that would be left for MAPREDUCE-5311 to do).
> >
> > I'll be checking my email over the weekend and I can take care of some
> > stuff if needed (while the monkeys sleep).
> >
> > Thx
> >
> >
> >
> > On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >
> >> Following is a revisited assessment of JIRAs I would like to get in the
> >> 2.1 release:
> >>
> >> From the 1st group I think all 3 should make.
> >>
> >> From the 2nd group I think YARN-791 should make it for sure and ideally
> >> MAPREDUCE-5130.
> >>
> >> From the 3rd group, I don't think this JIRA will make it.
> >>
> >> From the 4th group, we don't need to worry about this or 2.1
> >>
> >> Thanks
> >>
> >> Alejandro
> >>
> >> ------------------------------------------------------
> >> JIRAs that are in shape to make it to 2.1
> >>
> >> * YARN-752: In AMRMClient, automatically add corresponding rack requests
> >> for requested nodes
> >>
> >> impact: behavior change
> >>
> >> status: patch avail, +1ed.
> >>
> >> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
> >>
> >> impact: Addition to MRAM HTTP API
> >>
> >> status: patch avail, +1ed, needs to be committed
> >>
> >> * YARN-787: Remove resource min from Yarn client API
> >>
> >> impact: Yarn client API change
> >>
> >> status: patch avail, needs to be reviewed. (the calculation of
> slot-millis
> >> is not affected, the MIN is taken from conf for now)
> >>
> >> ------------------------------------------------------
> >> JIRAs that require minor work to make it to 2.1
> >>
> >> * YARN-521: Augment AM - RM client module to be able to request
> containers
> >> only at specific locations
> >>
> >> impact: AMRM client API change
> >>
> >> status: patch not avail yet (requires YARN-752)
> >>
> >> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
> with
> >> /nodes REST API
> >>
> >> impact: Yarn client API & proto change
> >>
> >> status: patch avail, review in progress
> >>
> >> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
> >>
> >> impact: behavior change
> >>
> >> status: patch avail but some tests are failing
> >>
> >> ------------------------------------------------------
> >> JIRAs that require significant work to make it to 2.1 and may not make
> it
> >>
> >> * YARN-649: Make container logs available over HTTP in plain text
> >>
> >> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
> >> does not change API)
> >>
> >> status: patch avail, review in progress
> >>
> >> ------------------------------------------------------
> >> JIRAs that don't need to make it to 2.1
> >>
> >> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
> >> counter constants
> >>
> >> impact: behavior change
> >>
> >> status: per discussion we should first add memory-millis and
> vcores-millis
> >>
> >> ------------------------------------------------------
> >>
> >>
> >> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
> wrote:
> >>
> >>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
> >>> wrote:
> >>>>
> >>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
> >>>>>
> >>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
> platform
> >>>>> based on Hadoop 2.0.x codeline we plan to start running the same
> >>> battery
> >>>>> of integration tests on the branch-2.1-beta.
> >>>>>
> >>>>> We plan to simply file JIRAs if anything gets detected and I will
> also
> >>>>> publish the URL of the Jenkins job once it gets created.
> >>>>
> >>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
> >>> Blocker priority to catch attention.
> >>>
> >>> The build is up and running (and all green on all of the 9 Linux
> >>> platforms!):
> >>>    http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
> >>>
> >>> The immediate benefit here is that we get to see that the
> >>> build is ok on all these Linuxes and all anybody can easily
> >>> install packaged Hadoop 2.1.0 nightly builds.
> >>>
> >>> Starting from next week, I'll start running regular tests
> >>> on these bits and will keep you guys posted!
> >>>
> >>> Thanks,
> >>> Roman.
> >>>
> >>
> >>
> >>
> >> --
> >> Alejandro
> >>
> >
> >
> >
> > --
> > Alejandro
>
>

Re: Heads up: branch-2.1-beta

[Karthik Kambatla <ka...@cloudera.com>]

Re-posting here to the wider audience:

HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
all suggestions so far. It would be great if people could take another look
at it. I ll iterate fast on any comments so we get this in by the time rest
of the code pieces are committed.

Thanks
Karthik




On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:

> Just curious of your procedures. Given that there is at least one blocker
> JIRA out there that has yet to be fully resolved, do you intend to release
> anyway?
>
>
> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > If the intention is to get the release out in time for the Hadoop Summit
> we
> > have a very tight schedule.
> >
> > Because the release vote runs for 7 days, we should have an RC latest
> > Monday afternoon, and we should encourage folks to verify & vote ASAP, so
> > if we need to cut a new RC we can do it on Tuesday. Another thing to
> > consider is that if the changes on an RC are corrections that do not
> affect
> > code, we could agree on not reseting the voting period clock if we need
> to
> > cut a new RC (ie doc, build, notes changes).
> >
> > Of the JIRAs in my laundry list for 2.1 the ones I would really want in
> are
> > YARN-752, MAPREDUCE-5171 & YARN-787.
> >
> > The first 2 are already +1ed, the last one needs to be reviewed.
> >
> > I have not committed the first 2 ones yet because I don't want to disrupt
> > things for the folks doing QA.
> >
> > Arun, as you are coordinating the work for this release, please do commit
> > them or give me the go ahead and I'll commit.
> >
> > Also, it  would be great if you can review YARN-787 (as per discussions,
> > the changes on the milli-slot calculations do not affect the current
> > calculations, that would be left for MAPREDUCE-5311 to do).
> >
> > I'll be checking my email over the weekend and I can take care of some
> > stuff if needed (while the monkeys sleep).
> >
> > Thx
> >
> >
> >
> > On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >
> >> Following is a revisited assessment of JIRAs I would like to get in the
> >> 2.1 release:
> >>
> >> From the 1st group I think all 3 should make.
> >>
> >> From the 2nd group I think YARN-791 should make it for sure and ideally
> >> MAPREDUCE-5130.
> >>
> >> From the 3rd group, I don't think this JIRA will make it.
> >>
> >> From the 4th group, we don't need to worry about this or 2.1
> >>
> >> Thanks
> >>
> >> Alejandro
> >>
> >> ------------------------------------------------------
> >> JIRAs that are in shape to make it to 2.1
> >>
> >> * YARN-752: In AMRMClient, automatically add corresponding rack requests
> >> for requested nodes
> >>
> >> impact: behavior change
> >>
> >> status: patch avail, +1ed.
> >>
> >> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
> >>
> >> impact: Addition to MRAM HTTP API
> >>
> >> status: patch avail, +1ed, needs to be committed
> >>
> >> * YARN-787: Remove resource min from Yarn client API
> >>
> >> impact: Yarn client API change
> >>
> >> status: patch avail, needs to be reviewed. (the calculation of
> slot-millis
> >> is not affected, the MIN is taken from conf for now)
> >>
> >> ------------------------------------------------------
> >> JIRAs that require minor work to make it to 2.1
> >>
> >> * YARN-521: Augment AM - RM client module to be able to request
> containers
> >> only at specific locations
> >>
> >> impact: AMRM client API change
> >>
> >> status: patch not avail yet (requires YARN-752)
> >>
> >> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
> with
> >> /nodes REST API
> >>
> >> impact: Yarn client API & proto change
> >>
> >> status: patch avail, review in progress
> >>
> >> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
> >>
> >> impact: behavior change
> >>
> >> status: patch avail but some tests are failing
> >>
> >> ------------------------------------------------------
> >> JIRAs that require significant work to make it to 2.1 and may not make
> it
> >>
> >> * YARN-649: Make container logs available over HTTP in plain text
> >>
> >> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
> >> does not change API)
> >>
> >> status: patch avail, review in progress
> >>
> >> ------------------------------------------------------
> >> JIRAs that don't need to make it to 2.1
> >>
> >> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
> >> counter constants
> >>
> >> impact: behavior change
> >>
> >> status: per discussion we should first add memory-millis and
> vcores-millis
> >>
> >> ------------------------------------------------------
> >>
> >>
> >> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
> wrote:
> >>
> >>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
> >>> wrote:
> >>>>
> >>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
> >>>>>
> >>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
> platform
> >>>>> based on Hadoop 2.0.x codeline we plan to start running the same
> >>> battery
> >>>>> of integration tests on the branch-2.1-beta.
> >>>>>
> >>>>> We plan to simply file JIRAs if anything gets detected and I will
> also
> >>>>> publish the URL of the Jenkins job once it gets created.
> >>>>
> >>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
> >>> Blocker priority to catch attention.
> >>>
> >>> The build is up and running (and all green on all of the 9 Linux
> >>> platforms!):
> >>>    http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
> >>>
> >>> The immediate benefit here is that we get to see that the
> >>> build is ok on all these Linuxes and all anybody can easily
> >>> install packaged Hadoop 2.1.0 nightly builds.
> >>>
> >>> Starting from next week, I'll start running regular tests
> >>> on these bits and will keep you guys posted!
> >>>
> >>> Thanks,
> >>> Roman.
> >>>
> >>
> >>
> >>
> >> --
> >> Alejandro
> >>
> >
> >
> >
> > --
> > Alejandro
>
>

Re: Heads up: branch-2.1-beta

[Karthik Kambatla <ka...@cloudera.com>]

Re-posting here to the wider audience:

HADOOP-9517 (Document Hadoop Compatibility): I believe I have incorporated
all suggestions so far. It would be great if people could take another look
at it. I ll iterate fast on any comments so we get this in by the time rest
of the code pieces are committed.

Thanks
Karthik




On Sat, Jun 15, 2013 at 11:46 AM, Ralph Castain <rh...@open-mpi.org> wrote:

> Just curious of your procedures. Given that there is at least one blocker
> JIRA out there that has yet to be fully resolved, do you intend to release
> anyway?
>
>
> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > If the intention is to get the release out in time for the Hadoop Summit
> we
> > have a very tight schedule.
> >
> > Because the release vote runs for 7 days, we should have an RC latest
> > Monday afternoon, and we should encourage folks to verify & vote ASAP, so
> > if we need to cut a new RC we can do it on Tuesday. Another thing to
> > consider is that if the changes on an RC are corrections that do not
> affect
> > code, we could agree on not reseting the voting period clock if we need
> to
> > cut a new RC (ie doc, build, notes changes).
> >
> > Of the JIRAs in my laundry list for 2.1 the ones I would really want in
> are
> > YARN-752, MAPREDUCE-5171 & YARN-787.
> >
> > The first 2 are already +1ed, the last one needs to be reviewed.
> >
> > I have not committed the first 2 ones yet because I don't want to disrupt
> > things for the folks doing QA.
> >
> > Arun, as you are coordinating the work for this release, please do commit
> > them or give me the go ahead and I'll commit.
> >
> > Also, it  would be great if you can review YARN-787 (as per discussions,
> > the changes on the milli-slot calculations do not affect the current
> > calculations, that would be left for MAPREDUCE-5311 to do).
> >
> > I'll be checking my email over the weekend and I can take care of some
> > stuff if needed (while the monkeys sleep).
> >
> > Thx
> >
> >
> >
> > On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >
> >> Following is a revisited assessment of JIRAs I would like to get in the
> >> 2.1 release:
> >>
> >> From the 1st group I think all 3 should make.
> >>
> >> From the 2nd group I think YARN-791 should make it for sure and ideally
> >> MAPREDUCE-5130.
> >>
> >> From the 3rd group, I don't think this JIRA will make it.
> >>
> >> From the 4th group, we don't need to worry about this or 2.1
> >>
> >> Thanks
> >>
> >> Alejandro
> >>
> >> ------------------------------------------------------
> >> JIRAs that are in shape to make it to 2.1
> >>
> >> * YARN-752: In AMRMClient, automatically add corresponding rack requests
> >> for requested nodes
> >>
> >> impact: behavior change
> >>
> >> status: patch avail, +1ed.
> >>
> >> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
> >>
> >> impact: Addition to MRAM HTTP API
> >>
> >> status: patch avail, +1ed, needs to be committed
> >>
> >> * YARN-787: Remove resource min from Yarn client API
> >>
> >> impact: Yarn client API change
> >>
> >> status: patch avail, needs to be reviewed. (the calculation of
> slot-millis
> >> is not affected, the MIN is taken from conf for now)
> >>
> >> ------------------------------------------------------
> >> JIRAs that require minor work to make it to 2.1
> >>
> >> * YARN-521: Augment AM - RM client module to be able to request
> containers
> >> only at specific locations
> >>
> >> impact: AMRM client API change
> >>
> >> status: patch not avail yet (requires YARN-752)
> >>
> >> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent
> with
> >> /nodes REST API
> >>
> >> impact: Yarn client API & proto change
> >>
> >> status: patch avail, review in progress
> >>
> >> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
> >>
> >> impact: behavior change
> >>
> >> status: patch avail but some tests are failing
> >>
> >> ------------------------------------------------------
> >> JIRAs that require significant work to make it to 2.1 and may not make
> it
> >>
> >> * YARN-649: Make container logs available over HTTP in plain text
> >>
> >> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
> >> does not change API)
> >>
> >> status: patch avail, review in progress
> >>
> >> ------------------------------------------------------
> >> JIRAs that don't need to make it to 2.1
> >>
> >> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
> >> counter constants
> >>
> >> impact: behavior change
> >>
> >> status: per discussion we should first add memory-millis and
> vcores-millis
> >>
> >> ------------------------------------------------------
> >>
> >>
> >> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org>
> wrote:
> >>
> >>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
> >>> wrote:
> >>>>
> >>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
> >>>>>
> >>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0
> platform
> >>>>> based on Hadoop 2.0.x codeline we plan to start running the same
> >>> battery
> >>>>> of integration tests on the branch-2.1-beta.
> >>>>>
> >>>>> We plan to simply file JIRAs if anything gets detected and I will
> also
> >>>>> publish the URL of the Jenkins job once it gets created.
> >>>>
> >>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
> >>> Blocker priority to catch attention.
> >>>
> >>> The build is up and running (and all green on all of the 9 Linux
> >>> platforms!):
> >>>    http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
> >>>
> >>> The immediate benefit here is that we get to see that the
> >>> build is ok on all these Linuxes and all anybody can easily
> >>> install packaged Hadoop 2.1.0 nightly builds.
> >>>
> >>> Starting from next week, I'll start running regular tests
> >>> on these bits and will keep you guys posted!
> >>>
> >>> Thanks,
> >>> Roman.
> >>>
> >>
> >>
> >>
> >> --
> >> Alejandro
> >>
> >
> >
> >
> > --
> > Alejandro
>
>

Re: Heads up: branch-2.1-beta

[Ralph Castain <rh...@open-mpi.org>]

Just curious of your procedures. Given that there is at least one blocker JIRA out there that has yet to be fully resolved, do you intend to release anyway?


On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:

> If the intention is to get the release out in time for the Hadoop Summit we
> have a very tight schedule.
> 
> Because the release vote runs for 7 days, we should have an RC latest
> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
> if we need to cut a new RC we can do it on Tuesday. Another thing to
> consider is that if the changes on an RC are corrections that do not affect
> code, we could agree on not reseting the voting period clock if we need to
> cut a new RC (ie doc, build, notes changes).
> 
> Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
> YARN-752, MAPREDUCE-5171 & YARN-787.
> 
> The first 2 are already +1ed, the last one needs to be reviewed.
> 
> I have not committed the first 2 ones yet because I don't want to disrupt
> things for the folks doing QA.
> 
> Arun, as you are coordinating the work for this release, please do commit
> them or give me the go ahead and I'll commit.
> 
> Also, it  would be great if you can review YARN-787 (as per discussions,
> the changes on the milli-slot calculations do not affect the current
> calculations, that would be left for MAPREDUCE-5311 to do).
> 
> I'll be checking my email over the weekend and I can take care of some
> stuff if needed (while the monkeys sleep).
> 
> Thx
> 
> 
> 
> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> 
>> Following is a revisited assessment of JIRAs I would like to get in the
>> 2.1 release:
>> 
>> From the 1st group I think all 3 should make.
>> 
>> From the 2nd group I think YARN-791 should make it for sure and ideally
>> MAPREDUCE-5130.
>> 
>> From the 3rd group, I don't think this JIRA will make it.
>> 
>> From the 4th group, we don't need to worry about this or 2.1
>> 
>> Thanks
>> 
>> Alejandro
>> 
>> ------------------------------------------------------
>> JIRAs that are in shape to make it to 2.1
>> 
>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>> for requested nodes
>> 
>> impact: behavior change
>> 
>> status: patch avail, +1ed.
>> 
>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>> 
>> impact: Addition to MRAM HTTP API
>> 
>> status: patch avail, +1ed, needs to be committed
>> 
>> * YARN-787: Remove resource min from Yarn client API
>> 
>> impact: Yarn client API change
>> 
>> status: patch avail, needs to be reviewed. (the calculation of slot-millis
>> is not affected, the MIN is taken from conf for now)
>> 
>> ------------------------------------------------------
>> JIRAs that require minor work to make it to 2.1
>> 
>> * YARN-521: Augment AM - RM client module to be able to request containers
>> only at specific locations
>> 
>> impact: AMRM client API change
>> 
>> status: patch not avail yet (requires YARN-752)
>> 
>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
>> /nodes REST API
>> 
>> impact: Yarn client API & proto change
>> 
>> status: patch avail, review in progress
>> 
>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>> 
>> impact: behavior change
>> 
>> status: patch avail but some tests are failing
>> 
>> ------------------------------------------------------
>> JIRAs that require significant work to make it to 2.1 and may not make it
>> 
>> * YARN-649: Make container logs available over HTTP in plain text
>> 
>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>> does not change API)
>> 
>> status: patch avail, review in progress
>> 
>> ------------------------------------------------------
>> JIRAs that don't need to make it to 2.1
>> 
>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>> counter constants
>> 
>> impact: behavior change
>> 
>> status: per discussion we should first add memory-millis and vcores-millis
>> 
>> ------------------------------------------------------
>> 
>> 
>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>> 
>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>> wrote:
>>>> 
>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>> 
>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>> battery
>>>>> of integration tests on the branch-2.1-beta.
>>>>> 
>>>>> We plan to simply file JIRAs if anything gets detected and I will also
>>>>> publish the URL of the Jenkins job once it gets created.
>>>> 
>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>> Blocker priority to catch attention.
>>> 
>>> The build is up and running (and all green on all of the 9 Linux
>>> platforms!):
>>>    http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>> 
>>> The immediate benefit here is that we get to see that the
>>> build is ok on all these Linuxes and all anybody can easily
>>> install packaged Hadoop 2.1.0 nightly builds.
>>> 
>>> Starting from next week, I'll start running regular tests
>>> on these bits and will keep you guys posted!
>>> 
>>> Thanks,
>>> Roman.
>>> 
>> 
>> 
>> 
>> --
>> Alejandro
>> 
> 
> 
> 
> -- 
> Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun, actually YARN-787 fixed the problem. Run those tests from trunk and
branch-2.1-beta HEADs without issues. Without YARN-787 they fail.

Thx


On Sun, Jun 16, 2013 at 12:03 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Alejandro,
>
>  Can you please take a look at MAPREDUCE-5327? This is related to YARN-787.
>
> thanks,
> Arun
>
> On Jun 16, 2013, at 8:56 AM, Alejandro Abdelnur wrote:
>
> > Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
> > around noon today (SUN noon PST).
> >
> > What is your take on YARN-791 & MAPREDUCE-5130?
> >
> >
> >
> > On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
> >
> >>
> >> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
> >>
> >>>
> >>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> >>>>
> >>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want
> in
> >> are
> >>>> YARN-752, MAPREDUCE-5171 & YARN-787.
> >>>
> >>> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
> >> right now.
> >>
> >> I committed YARN-787. Thanks.
> >>
> >> Arun
> >>
> >>
> >
> >
> > --
> > Alejandro
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun, actually YARN-787 fixed the problem. Run those tests from trunk and
branch-2.1-beta HEADs without issues. Without YARN-787 they fail.

Thx


On Sun, Jun 16, 2013 at 12:03 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Alejandro,
>
>  Can you please take a look at MAPREDUCE-5327? This is related to YARN-787.
>
> thanks,
> Arun
>
> On Jun 16, 2013, at 8:56 AM, Alejandro Abdelnur wrote:
>
> > Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
> > around noon today (SUN noon PST).
> >
> > What is your take on YARN-791 & MAPREDUCE-5130?
> >
> >
> >
> > On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
> >
> >>
> >> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
> >>
> >>>
> >>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> >>>>
> >>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want
> in
> >> are
> >>>> YARN-752, MAPREDUCE-5171 & YARN-787.
> >>>
> >>> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
> >> right now.
> >>
> >> I committed YARN-787. Thanks.
> >>
> >> Arun
> >>
> >>
> >
> >
> > --
> > Alejandro
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun, actually YARN-787 fixed the problem. Run those tests from trunk and
branch-2.1-beta HEADs without issues. Without YARN-787 they fail.

Thx


On Sun, Jun 16, 2013 at 12:03 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Alejandro,
>
>  Can you please take a look at MAPREDUCE-5327? This is related to YARN-787.
>
> thanks,
> Arun
>
> On Jun 16, 2013, at 8:56 AM, Alejandro Abdelnur wrote:
>
> > Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
> > around noon today (SUN noon PST).
> >
> > What is your take on YARN-791 & MAPREDUCE-5130?
> >
> >
> >
> > On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
> >
> >>
> >> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
> >>
> >>>
> >>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> >>>>
> >>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want
> in
> >> are
> >>>> YARN-752, MAPREDUCE-5171 & YARN-787.
> >>>
> >>> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
> >> right now.
> >>
> >> I committed YARN-787. Thanks.
> >>
> >> Arun
> >>
> >>
> >
> >
> > --
> > Alejandro
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun, actually YARN-787 fixed the problem. Run those tests from trunk and
branch-2.1-beta HEADs without issues. Without YARN-787 they fail.

Thx


On Sun, Jun 16, 2013 at 12:03 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Alejandro,
>
>  Can you please take a look at MAPREDUCE-5327? This is related to YARN-787.
>
> thanks,
> Arun
>
> On Jun 16, 2013, at 8:56 AM, Alejandro Abdelnur wrote:
>
> > Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
> > around noon today (SUN noon PST).
> >
> > What is your take on YARN-791 & MAPREDUCE-5130?
> >
> >
> >
> > On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
> >
> >>
> >> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
> >>
> >>>
> >>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> >>>>
> >>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want
> in
> >> are
> >>>> YARN-752, MAPREDUCE-5171 & YARN-787.
> >>>
> >>> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
> >> right now.
> >>
> >> I committed YARN-787. Thanks.
> >>
> >> Arun
> >>
> >>
> >
> >
> > --
> > Alejandro
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Alejandro,

 Can you please take a look at MAPREDUCE-5327? This is related to YARN-787.

thanks,
Arun

On Jun 16, 2013, at 8:56 AM, Alejandro Abdelnur wrote:

> Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
> around noon today (SUN noon PST).
> 
> What is your take on YARN-791 & MAPREDUCE-5130?
> 
> 
> 
> On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> 
>> 
>> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
>> 
>>> 
>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
>>>> 
>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>> are
>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>> 
>>> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
>> right now.
>> 
>> I committed YARN-787. Thanks.
>> 
>> Arun
>> 
>> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Alejandro,

 Can you please take a look at MAPREDUCE-5327? This is related to YARN-787.

thanks,
Arun

On Jun 16, 2013, at 8:56 AM, Alejandro Abdelnur wrote:

> Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
> around noon today (SUN noon PST).
> 
> What is your take on YARN-791 & MAPREDUCE-5130?
> 
> 
> 
> On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> 
>> 
>> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
>> 
>>> 
>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
>>>> 
>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>> are
>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>> 
>>> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
>> right now.
>> 
>> I committed YARN-787. Thanks.
>> 
>> Arun
>> 
>> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Alejandro,

 Can you please take a look at MAPREDUCE-5327? This is related to YARN-787.

thanks,
Arun

On Jun 16, 2013, at 8:56 AM, Alejandro Abdelnur wrote:

> Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
> around noon today (SUN noon PST).
> 
> What is your take on YARN-791 & MAPREDUCE-5130?
> 
> 
> 
> On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> 
>> 
>> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
>> 
>>> 
>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
>>>> 
>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>> are
>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>> 
>>> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
>> right now.
>> 
>> I committed YARN-787. Thanks.
>> 
>> Arun
>> 
>> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Alejandro,

 Can you please take a look at MAPREDUCE-5327? This is related to YARN-787.

thanks,
Arun

On Jun 16, 2013, at 8:56 AM, Alejandro Abdelnur wrote:

> Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
> around noon today (SUN noon PST).
> 
> What is your take on YARN-791 & MAPREDUCE-5130?
> 
> 
> 
> On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> 
>> 
>> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
>> 
>>> 
>>> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
>>>> 
>>>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
>> are
>>>> YARN-752, MAPREDUCE-5171 & YARN-787.
>>> 
>>> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
>> right now.
>> 
>> I committed YARN-787. Thanks.
>> 
>> Arun
>> 
>> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
around noon today (SUN noon PST).

What is your take on YARN-791 & MAPREDUCE-5130?



On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

>
> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
>
> >
> > On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> >>
> >> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
> are
> >> YARN-752, MAPREDUCE-5171 & YARN-787.
> >
> > I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
> right now.
>
> I committed YARN-787. Thanks.
>
> Arun
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
around noon today (SUN noon PST).

What is your take on YARN-791 & MAPREDUCE-5130?



On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

>
> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
>
> >
> > On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> >>
> >> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
> are
> >> YARN-752, MAPREDUCE-5171 & YARN-787.
> >
> > I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
> right now.
>
> I committed YARN-787. Thanks.
>
> Arun
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
around noon today (SUN noon PST).

What is your take on YARN-791 & MAPREDUCE-5130?



On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

>
> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
>
> >
> > On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> >>
> >> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
> are
> >> YARN-752, MAPREDUCE-5171 & YARN-787.
> >
> > I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
> right now.
>
> I committed YARN-787. Thanks.
>
> Arun
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Thanks Arun, I'll take care of committing YARN-752 and MAPREDUCE-5171
around noon today (SUN noon PST).

What is your take on YARN-791 & MAPREDUCE-5130?



On Sun, Jun 16, 2013 at 7:02 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

>
> On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:
>
> >
> > On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> >>
> >> Of the JIRAs in my laundry list for 2.1 the ones I would really want in
> are
> >> YARN-752, MAPREDUCE-5171 & YARN-787.
> >
> > I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it
> right now.
>
> I committed YARN-787. Thanks.
>
> Arun
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:

> 
> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
>> 
>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
>> YARN-752, MAPREDUCE-5171 & YARN-787.
> 
> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it right now.

I committed YARN-787. Thanks.

Arun

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:

> 
> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
>> 
>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
>> YARN-752, MAPREDUCE-5171 & YARN-787.
> 
> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it right now.

I committed YARN-787. Thanks.

Arun

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:

> 
> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
>> 
>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
>> YARN-752, MAPREDUCE-5171 & YARN-787.
> 
> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it right now.

I committed YARN-787. Thanks.

Arun

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 16, 2013, at 5:39 AM, Arun C Murthy wrote:

> 
> On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
>> 
>> Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
>> YARN-752, MAPREDUCE-5171 & YARN-787.
> 
> I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it right now.

I committed YARN-787. Thanks.

Arun

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> 
> Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
> YARN-752, MAPREDUCE-5171 & YARN-787.

I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it right now.

I'm ok with YARN-752 & MAPREDUCE-5171 going in, but don't consider either a blocker.

Suresh - will HDFS-4777 go in soon too?

Vinod - How about YARN-386?

thanks,
Arun

> 
> Thx
> 
> 
> 
> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> 
>> Following is a revisited assessment of JIRAs I would like to get in the
>> 2.1 release:
>> 
>> From the 1st group I think all 3 should make.
>> 
>> From the 2nd group I think YARN-791 should make it for sure and ideally
>> MAPREDUCE-5130.
>> 
>> From the 3rd group, I don't think this JIRA will make it.
>> 
>> From the 4th group, we don't need to worry about this or 2.1
>> 
>> Thanks
>> 
>> Alejandro
>> 
>> ------------------------------------------------------
>> JIRAs that are in shape to make it to 2.1
>> 
>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>> for requested nodes
>> 
>> impact: behavior change
>> 
>> status: patch avail, +1ed.
>> 
>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>> 
>> impact: Addition to MRAM HTTP API
>> 
>> status: patch avail, +1ed, needs to be committed
>> 
>> * YARN-787: Remove resource min from Yarn client API
>> 
>> impact: Yarn client API change
>> 
>> status: patch avail, needs to be reviewed. (the calculation of slot-millis
>> is not affected, the MIN is taken from conf for now)
>> 
>> ------------------------------------------------------
>> JIRAs that require minor work to make it to 2.1
>> 
>> * YARN-521: Augment AM - RM client module to be able to request containers
>> only at specific locations
>> 
>> impact: AMRM client API change
>> 
>> status: patch not avail yet (requires YARN-752)
>> 
>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
>> /nodes REST API
>> 
>> impact: Yarn client API & proto change
>> 
>> status: patch avail, review in progress
>> 
>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>> 
>> impact: behavior change
>> 
>> status: patch avail but some tests are failing
>> 
>> ------------------------------------------------------
>> JIRAs that require significant work to make it to 2.1 and may not make it
>> 
>> * YARN-649: Make container logs available over HTTP in plain text
>> 
>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>> does not change API)
>> 
>> status: patch avail, review in progress
>> 
>> ------------------------------------------------------
>> JIRAs that don't need to make it to 2.1
>> 
>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>> counter constants
>> 
>> impact: behavior change
>> 
>> status: per discussion we should first add memory-millis and vcores-millis
>> 
>> ------------------------------------------------------
>> 
>> 
>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>> 
>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>> wrote:
>>>> 
>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>> 
>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>> battery
>>>>> of integration tests on the branch-2.1-beta.
>>>>> 
>>>>> We plan to simply file JIRAs if anything gets detected and I will also
>>>>> publish the URL of the Jenkins job once it gets created.
>>>> 
>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>> Blocker priority to catch attention.
>>> 
>>> The build is up and running (and all green on all of the 9 Linux
>>> platforms!):
>>>    http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>> 
>>> The immediate benefit here is that we get to see that the
>>> build is ok on all these Linuxes and all anybody can easily
>>> install packaged Hadoop 2.1.0 nightly builds.
>>> 
>>> Starting from next week, I'll start running regular tests
>>> on these bits and will keep you guys posted!
>>> 
>>> Thanks,
>>> Roman.
>>> 
>> 
>> 
>> 
>> --
>> Alejandro
>> 
> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

RE: Heads up: branch-2.1-beta

[Devaraj k <de...@huawei.com>]

Hi Arun,

	Is there any possibility of including YARN-41 in this release.
 
Thanks
Devaraj K

-----Original Message-----
From: Arun C Murthy [mailto:acm@hortonworks.com] 
Sent: 19 June 2013 12:29
To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Heads up: branch-2.1-beta

Ping. Any luck?

On Jun 17, 2013, at 4:06 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Sun, Jun 16, 2013 at 5:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> Roman,
>> 
>> Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?
> 
> I will try to kick off the full build today. And deploy/test tomorrow.
> It is all pretty automated, but takes a long time. Hope the results 
> will still be useful for you guys wrt. 2.1 release.
> 
> Thanks,
> Roman.

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

RE: Heads up: branch-2.1-beta

[Devaraj k <de...@huawei.com>]

Hi Arun,

	Is there any possibility of including YARN-41 in this release.
 
Thanks
Devaraj K

-----Original Message-----
From: Arun C Murthy [mailto:acm@hortonworks.com] 
Sent: 19 June 2013 12:29
To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Heads up: branch-2.1-beta

Ping. Any luck?

On Jun 17, 2013, at 4:06 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Sun, Jun 16, 2013 at 5:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> Roman,
>> 
>> Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?
> 
> I will try to kick off the full build today. And deploy/test tomorrow.
> It is all pretty automated, but takes a long time. Hope the results 
> will still be useful for you guys wrt. 2.1 release.
> 
> Thanks,
> Roman.

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

RE: Heads up: branch-2.1-beta

[Devaraj k <de...@huawei.com>]

Hi Arun,

	Is there any possibility of including YARN-41 in this release.
 
Thanks
Devaraj K

-----Original Message-----
From: Arun C Murthy [mailto:acm@hortonworks.com] 
Sent: 19 June 2013 12:29
To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Heads up: branch-2.1-beta

Ping. Any luck?

On Jun 17, 2013, at 4:06 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Sun, Jun 16, 2013 at 5:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> Roman,
>> 
>> Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?
> 
> I will try to kick off the full build today. And deploy/test tomorrow.
> It is all pretty automated, but takes a long time. Hope the results 
> will still be useful for you guys wrt. 2.1 release.
> 
> Thanks,
> Roman.

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Ping. Any luck?

The full bigtop build failed because of Giraph:
     http://bigtop01.cloudera.org:8080/view/Upstream-tests/job/Hadoop-2.1.0/label=centos5/4/console

Now, I'm pretty sure that's an issue with Giraph
itself (or more precisely Giraph-on-YARN profile)
rather than Hadoop. I'll follow up with the Giraph
community.

Unfortunately, since the full build failed the automated
pipe-line didn't pick up the build for testing. I'll manually
kick it off right away (with Giraph taken from Bigtop 0.6.0)
and will report the results by COB 06/19.

Stay tuned!

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Have anybody from the HDFS side of things had a chance to look at the OOM
that Roman was reporting?

Thanks.


On Wed, Jun 26, 2013 at 12:26 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Ok, the last of the blockers is done.
>
> I'll roll an RC tonight.
>
> thanks,
> Arun
>
> On Jun 20, 2013, at 2:40 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> > I think I've shared this before, but here you go again…
> >
> > http://s.apache.org/hadoop-2.1.0-beta-blockers
> >
> > At this point, HADOOP-9421 seems like the most important.
> >
> > thanks,
> > Arun
> >
> > On Jun 20, 2013, at 8:31 AM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
> >
> >> Arun,
> >>
> >> It seems there are still a few things to iron out before getting 2.1
> out of
> >> the door.
> >>
> >> As RM for the release, would you mind sharing the current state of
> things
> >> and your estimate on when it could happen?
> >>
> >> Thanks.
> >>
> >>
> >> On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org>
> wrote:
> >>
> >>> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
> >>> wrote:
> >>>> Ping. Any luck?
> >>>
> >>> Unfortunately I've ran into:
> >>>   https://issues.apache.org/jira/browse/HADOOP-9654
> >>>
> >>> which was caused by the unrelated memory pressure on the NN,
> >>> but it had an unfortunate side effect of making the rest of the
> >>> testing stuck. I'll correct the problem now and re-run.
> >>>
> >>> Thanks,
> >>> Roman.
> >>>
> >>
> >>
> >>
> >> --
> >> Alejandro
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Have anybody from the HDFS side of things had a chance to look at the OOM
that Roman was reporting?

Thanks.


On Wed, Jun 26, 2013 at 12:26 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Ok, the last of the blockers is done.
>
> I'll roll an RC tonight.
>
> thanks,
> Arun
>
> On Jun 20, 2013, at 2:40 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> > I think I've shared this before, but here you go again…
> >
> > http://s.apache.org/hadoop-2.1.0-beta-blockers
> >
> > At this point, HADOOP-9421 seems like the most important.
> >
> > thanks,
> > Arun
> >
> > On Jun 20, 2013, at 8:31 AM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
> >
> >> Arun,
> >>
> >> It seems there are still a few things to iron out before getting 2.1
> out of
> >> the door.
> >>
> >> As RM for the release, would you mind sharing the current state of
> things
> >> and your estimate on when it could happen?
> >>
> >> Thanks.
> >>
> >>
> >> On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org>
> wrote:
> >>
> >>> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
> >>> wrote:
> >>>> Ping. Any luck?
> >>>
> >>> Unfortunately I've ran into:
> >>>   https://issues.apache.org/jira/browse/HADOOP-9654
> >>>
> >>> which was caused by the unrelated memory pressure on the NN,
> >>> but it had an unfortunate side effect of making the rest of the
> >>> testing stuck. I'll correct the problem now and re-run.
> >>>
> >>> Thanks,
> >>> Roman.
> >>>
> >>
> >>
> >>
> >> --
> >> Alejandro
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Have anybody from the HDFS side of things had a chance to look at the OOM
that Roman was reporting?

Thanks.


On Wed, Jun 26, 2013 at 12:26 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Ok, the last of the blockers is done.
>
> I'll roll an RC tonight.
>
> thanks,
> Arun
>
> On Jun 20, 2013, at 2:40 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> > I think I've shared this before, but here you go again…
> >
> > http://s.apache.org/hadoop-2.1.0-beta-blockers
> >
> > At this point, HADOOP-9421 seems like the most important.
> >
> > thanks,
> > Arun
> >
> > On Jun 20, 2013, at 8:31 AM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
> >
> >> Arun,
> >>
> >> It seems there are still a few things to iron out before getting 2.1
> out of
> >> the door.
> >>
> >> As RM for the release, would you mind sharing the current state of
> things
> >> and your estimate on when it could happen?
> >>
> >> Thanks.
> >>
> >>
> >> On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org>
> wrote:
> >>
> >>> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
> >>> wrote:
> >>>> Ping. Any luck?
> >>>
> >>> Unfortunately I've ran into:
> >>>   https://issues.apache.org/jira/browse/HADOOP-9654
> >>>
> >>> which was caused by the unrelated memory pressure on the NN,
> >>> but it had an unfortunate side effect of making the rest of the
> >>> testing stuck. I'll correct the problem now and re-run.
> >>>
> >>> Thanks,
> >>> Roman.
> >>>
> >>
> >>
> >>
> >> --
> >> Alejandro
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Have anybody from the HDFS side of things had a chance to look at the OOM
that Roman was reporting?

Thanks.


On Wed, Jun 26, 2013 at 12:26 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Ok, the last of the blockers is done.
>
> I'll roll an RC tonight.
>
> thanks,
> Arun
>
> On Jun 20, 2013, at 2:40 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> > I think I've shared this before, but here you go again…
> >
> > http://s.apache.org/hadoop-2.1.0-beta-blockers
> >
> > At this point, HADOOP-9421 seems like the most important.
> >
> > thanks,
> > Arun
> >
> > On Jun 20, 2013, at 8:31 AM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
> >
> >> Arun,
> >>
> >> It seems there are still a few things to iron out before getting 2.1
> out of
> >> the door.
> >>
> >> As RM for the release, would you mind sharing the current state of
> things
> >> and your estimate on when it could happen?
> >>
> >> Thanks.
> >>
> >>
> >> On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org>
> wrote:
> >>
> >>> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
> >>> wrote:
> >>>> Ping. Any luck?
> >>>
> >>> Unfortunately I've ran into:
> >>>   https://issues.apache.org/jira/browse/HADOOP-9654
> >>>
> >>> which was caused by the unrelated memory pressure on the NN,
> >>> but it had an unfortunate side effect of making the rest of the
> >>> testing stuck. I'll correct the problem now and re-run.
> >>>
> >>> Thanks,
> >>> Roman.
> >>>
> >>
> >>
> >>
> >> --
> >> Alejandro
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Ok, the last of the blockers is done.

I'll roll an RC tonight.

thanks,
Arun

On Jun 20, 2013, at 2:40 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> I think I've shared this before, but here you go again…
> 
> http://s.apache.org/hadoop-2.1.0-beta-blockers
> 
> At this point, HADOOP-9421 seems like the most important.
> 
> thanks,
> Arun
> 
> On Jun 20, 2013, at 8:31 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> 
>> Arun,
>> 
>> It seems there are still a few things to iron out before getting 2.1 out of
>> the door.
>> 
>> As RM for the release, would you mind sharing the current state of things
>> and your estimate on when it could happen?
>> 
>> Thanks.
>> 
>> 
>> On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>> 
>>> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
>>> wrote:
>>>> Ping. Any luck?
>>> 
>>> Unfortunately I've ran into:
>>>   https://issues.apache.org/jira/browse/HADOOP-9654
>>> 
>>> which was caused by the unrelated memory pressure on the NN,
>>> but it had an unfortunate side effect of making the rest of the
>>> testing stuck. I'll correct the problem now and re-run.
>>> 
>>> Thanks,
>>> Roman.
>>> 
>> 
>> 
>> 
>> -- 
>> Alejandro
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Ok, the last of the blockers is done.

I'll roll an RC tonight.

thanks,
Arun

On Jun 20, 2013, at 2:40 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> I think I've shared this before, but here you go again…
> 
> http://s.apache.org/hadoop-2.1.0-beta-blockers
> 
> At this point, HADOOP-9421 seems like the most important.
> 
> thanks,
> Arun
> 
> On Jun 20, 2013, at 8:31 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> 
>> Arun,
>> 
>> It seems there are still a few things to iron out before getting 2.1 out of
>> the door.
>> 
>> As RM for the release, would you mind sharing the current state of things
>> and your estimate on when it could happen?
>> 
>> Thanks.
>> 
>> 
>> On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>> 
>>> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
>>> wrote:
>>>> Ping. Any luck?
>>> 
>>> Unfortunately I've ran into:
>>>   https://issues.apache.org/jira/browse/HADOOP-9654
>>> 
>>> which was caused by the unrelated memory pressure on the NN,
>>> but it had an unfortunate side effect of making the rest of the
>>> testing stuck. I'll correct the problem now and re-run.
>>> 
>>> Thanks,
>>> Roman.
>>> 
>> 
>> 
>> 
>> -- 
>> Alejandro
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Ok, the last of the blockers is done.

I'll roll an RC tonight.

thanks,
Arun

On Jun 20, 2013, at 2:40 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> I think I've shared this before, but here you go again…
> 
> http://s.apache.org/hadoop-2.1.0-beta-blockers
> 
> At this point, HADOOP-9421 seems like the most important.
> 
> thanks,
> Arun
> 
> On Jun 20, 2013, at 8:31 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> 
>> Arun,
>> 
>> It seems there are still a few things to iron out before getting 2.1 out of
>> the door.
>> 
>> As RM for the release, would you mind sharing the current state of things
>> and your estimate on when it could happen?
>> 
>> Thanks.
>> 
>> 
>> On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>> 
>>> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
>>> wrote:
>>>> Ping. Any luck?
>>> 
>>> Unfortunately I've ran into:
>>>   https://issues.apache.org/jira/browse/HADOOP-9654
>>> 
>>> which was caused by the unrelated memory pressure on the NN,
>>> but it had an unfortunate side effect of making the rest of the
>>> testing stuck. I'll correct the problem now and re-run.
>>> 
>>> Thanks,
>>> Roman.
>>> 
>> 
>> 
>> 
>> -- 
>> Alejandro
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Ok, the last of the blockers is done.

I'll roll an RC tonight.

thanks,
Arun

On Jun 20, 2013, at 2:40 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> I think I've shared this before, but here you go again…
> 
> http://s.apache.org/hadoop-2.1.0-beta-blockers
> 
> At this point, HADOOP-9421 seems like the most important.
> 
> thanks,
> Arun
> 
> On Jun 20, 2013, at 8:31 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> 
>> Arun,
>> 
>> It seems there are still a few things to iron out before getting 2.1 out of
>> the door.
>> 
>> As RM for the release, would you mind sharing the current state of things
>> and your estimate on when it could happen?
>> 
>> Thanks.
>> 
>> 
>> On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>> 
>>> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
>>> wrote:
>>>> Ping. Any luck?
>>> 
>>> Unfortunately I've ran into:
>>>   https://issues.apache.org/jira/browse/HADOOP-9654
>>> 
>>> which was caused by the unrelated memory pressure on the NN,
>>> but it had an unfortunate side effect of making the rest of the
>>> testing stuck. I'll correct the problem now and re-run.
>>> 
>>> Thanks,
>>> Roman.
>>> 
>> 
>> 
>> 
>> -- 
>> Alejandro
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

I think I've shared this before, but here you go again…

http://s.apache.org/hadoop-2.1.0-beta-blockers

At this point, HADOOP-9421 seems like the most important.

thanks,
Arun

On Jun 20, 2013, at 8:31 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:

> Arun,
> 
> It seems there are still a few things to iron out before getting 2.1 out of
> the door.
> 
> As RM for the release, would you mind sharing the current state of things
> and your estimate on when it could happen?
> 
> Thanks.
> 
> 
> On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> 
>> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
>> wrote:
>>> Ping. Any luck?
>> 
>> Unfortunately I've ran into:
>>    https://issues.apache.org/jira/browse/HADOOP-9654
>> 
>> which was caused by the unrelated memory pressure on the NN,
>> but it had an unfortunate side effect of making the rest of the
>> testing stuck. I'll correct the problem now and re-run.
>> 
>> Thanks,
>> Roman.
>> 
> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

I think I've shared this before, but here you go again…

http://s.apache.org/hadoop-2.1.0-beta-blockers

At this point, HADOOP-9421 seems like the most important.

thanks,
Arun

On Jun 20, 2013, at 8:31 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:

> Arun,
> 
> It seems there are still a few things to iron out before getting 2.1 out of
> the door.
> 
> As RM for the release, would you mind sharing the current state of things
> and your estimate on when it could happen?
> 
> Thanks.
> 
> 
> On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> 
>> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
>> wrote:
>>> Ping. Any luck?
>> 
>> Unfortunately I've ran into:
>>    https://issues.apache.org/jira/browse/HADOOP-9654
>> 
>> which was caused by the unrelated memory pressure on the NN,
>> but it had an unfortunate side effect of making the rest of the
>> testing stuck. I'll correct the problem now and re-run.
>> 
>> Thanks,
>> Roman.
>> 
> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

I think I've shared this before, but here you go again…

http://s.apache.org/hadoop-2.1.0-beta-blockers

At this point, HADOOP-9421 seems like the most important.

thanks,
Arun

On Jun 20, 2013, at 8:31 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:

> Arun,
> 
> It seems there are still a few things to iron out before getting 2.1 out of
> the door.
> 
> As RM for the release, would you mind sharing the current state of things
> and your estimate on when it could happen?
> 
> Thanks.
> 
> 
> On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> 
>> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
>> wrote:
>>> Ping. Any luck?
>> 
>> Unfortunately I've ran into:
>>    https://issues.apache.org/jira/browse/HADOOP-9654
>> 
>> which was caused by the unrelated memory pressure on the NN,
>> but it had an unfortunate side effect of making the rest of the
>> testing stuck. I'll correct the problem now and re-run.
>> 
>> Thanks,
>> Roman.
>> 
> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

I think I've shared this before, but here you go again…

http://s.apache.org/hadoop-2.1.0-beta-blockers

At this point, HADOOP-9421 seems like the most important.

thanks,
Arun

On Jun 20, 2013, at 8:31 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:

> Arun,
> 
> It seems there are still a few things to iron out before getting 2.1 out of
> the door.
> 
> As RM for the release, would you mind sharing the current state of things
> and your estimate on when it could happen?
> 
> Thanks.
> 
> 
> On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> 
>> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
>> wrote:
>>> Ping. Any luck?
>> 
>> Unfortunately I've ran into:
>>    https://issues.apache.org/jira/browse/HADOOP-9654
>> 
>> which was caused by the unrelated memory pressure on the NN,
>> but it had an unfortunate side effect of making the rest of the
>> testing stuck. I'll correct the problem now and re-run.
>> 
>> Thanks,
>> Roman.
>> 
> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun,

It seems there are still a few things to iron out before getting 2.1 out of
the door.

As RM for the release, would you mind sharing the current state of things
and your estimate on when it could happen?

Thanks.


On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
> > Ping. Any luck?
>
> Unfortunately I've ran into:
>     https://issues.apache.org/jira/browse/HADOOP-9654
>
> which was caused by the unrelated memory pressure on the NN,
> but it had an unfortunate side effect of making the rest of the
> testing stuck. I'll correct the problem now and re-run.
>
> Thanks,
> Roman.
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun,

It seems there are still a few things to iron out before getting 2.1 out of
the door.

As RM for the release, would you mind sharing the current state of things
and your estimate on when it could happen?

Thanks.


On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
> > Ping. Any luck?
>
> Unfortunately I've ran into:
>     https://issues.apache.org/jira/browse/HADOOP-9654
>
> which was caused by the unrelated memory pressure on the NN,
> but it had an unfortunate side effect of making the rest of the
> testing stuck. I'll correct the problem now and re-run.
>
> Thanks,
> Roman.
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun,

It seems there are still a few things to iron out before getting 2.1 out of
the door.

As RM for the release, would you mind sharing the current state of things
and your estimate on when it could happen?

Thanks.


On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
> > Ping. Any luck?
>
> Unfortunately I've ran into:
>     https://issues.apache.org/jira/browse/HADOOP-9654
>
> which was caused by the unrelated memory pressure on the NN,
> but it had an unfortunate side effect of making the rest of the
> testing stuck. I'll correct the problem now and re-run.
>
> Thanks,
> Roman.
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun,

It seems there are still a few things to iron out before getting 2.1 out of
the door.

As RM for the release, would you mind sharing the current state of things
and your estimate on when it could happen?

Thanks.


On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
> > Ping. Any luck?
>
> Unfortunately I've ran into:
>     https://issues.apache.org/jira/browse/HADOOP-9654
>
> which was caused by the unrelated memory pressure on the NN,
> but it had an unfortunate side effect of making the rest of the
> testing stuck. I'll correct the problem now and re-run.
>
> Thanks,
> Roman.
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Suresh Srinivas <su...@hortonworks.com>]

Did you change the java version? Can you please provide the JVM
configuration from hadoop-env.sh?



On Wed, Jun 19, 2013 at 10:02 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Wed, Jun 19, 2013 at 5:21 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> > On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org>
> wrote:
> >> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
> >>> Ping. Any luck?
> >
> > One final question -- has the memory requirements for NN change? I've
> > been always running my Bigtop tests with 1G of heap and I've never
> > seen any issues. In this test run NN kept OOMing till I doubled its
> > heap size. Is this expected?
>
> To follow: NN keeps OOMing. It seems that a reliable way to make it happen
> is to run Bigtop's version of TestCLI against a fully distributed
> cluster. At this
> point I'm actually reasonably concerned and will follow up tomorrow.
>
> Thanks,
> Roman.
>



-- 
http://hortonworks.com/download/

Re: Heads up: branch-2.1-beta

[Todd Lipcon <to...@cloudera.com>]

Have you tried running with -XX:+HeapDumpOnOutOfMemoryError? Got a heap
dump folks can look at?

The new snapshot stuff may be at fault... other than that, no particular
ideas.

On Wed, Jun 19, 2013 at 10:02 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Wed, Jun 19, 2013 at 5:21 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> > On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org>
> wrote:
> >> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
> >>> Ping. Any luck?
> >
> > One final question -- has the memory requirements for NN change? I've
> > been always running my Bigtop tests with 1G of heap and I've never
> > seen any issues. In this test run NN kept OOMing till I doubled its
> > heap size. Is this expected?
>
> To follow: NN keeps OOMing. It seems that a reliable way to make it happen
> is to run Bigtop's version of TestCLI against a fully distributed
> cluster. At this
> point I'm actually reasonably concerned and will follow up tomorrow.
>
> Thanks,
> Roman.
>



-- 
Todd Lipcon
Software Engineer, Cloudera

Re: Heads up: branch-2.1-beta

[Suresh Srinivas <su...@hortonworks.com>]

I still need to look at hadoop-env.sh. 

I suspect this is due to not passing Xms (starting java heap size) set to same value as Xmx (max java heap size). The recent inode id changes create hash apps based on max heap size and I have seen this problem without Xms. 

Sent from a mobile device

On Jun 20, 2013, at 5:33 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> ||| Did you change the java version? Can you please provide the JVM
> ||| configuration from hadoop-env.sh?
> 
> Nope I didn't. In fact since Bigtop uses clean VMs every single time
> it runs tests there's also no chance of env. contamination. As far
> as the version of JVM it is still the same as I used for Bigtop 0.6.0
> (Hadoop 2.0.5 based) which is Orcale's JDK jdk1.6.0_21
> 
> Thanks,
> Roman.

Re: Heads up: branch-2.1-beta

[Suresh Srinivas <su...@hortonworks.com>]

Can you please run the tests with -Xms2G added?
Sent from a mobile device

On Jun 20, 2013, at 6:02 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Thu, Jun 20, 2013 at 5:59 PM, Suresh Srinivas <su...@hortonworks.com> wrote:
>> I still need look at hadoop-env.sh
> 
> Bigtopt deployments don't utilize hadoop-env.sh we use /etc/default/hadoop*
> for the more precise configuration.
> 
>> I suspect this is due to not passing Xms (starting java heap size) set to same
>> value as Xmx (max java heap size). The recent inode id changes create hash
>> apps based on max heap size and I have seen this problem without Xms.
> 
> Here's the output of ps:
> 
> $ ps -aelf | grep NameNode
> 0 S hdfs     28482     1  1  80   0 - 661539 futex_ 00:42 ?
> 00:00:13 /usr/java/jdk1.6.0_21/bin/java -Dproc_namenode -Xmx2G
> -Dhadoop.log.dir=/var/log/hadoop-hdfs -Dhadoop.log.file=hadoop.log
> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
> -Dhadoop.root.logger=INFO,console
> -Djava.library.path=/usr/lib/hadoop/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -Dhadoop.log.dir=/var/log/hadoop-hdfs
> -Dhadoop.log.file=hadoop-hdfs-namenode-ip-10-144-82-213.log
> -Dhadoop.home.dir=/usr/lib/hadoop -Dhadoop.id.str=hdfs
> -Dhadoop.root.logger=INFO,RFA
> -Djava.library.path=/usr/lib/hadoop/lib/native
> -Dhadoop.policy.file=hadoop-policy.xml -Djava.net.preferIPv4Stack=true
> -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote
> -Dcom.sun.management.jmxremote -Dhadoop.security.logger=INFO,RFAS
> org.apache.hadoop.hdfs.server.namenode.NameNode
> 
> 
> Thanks,
> Roman.

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Wed, Jun 19, 2013 at 5:21 PM, Roman Shaposhnik <rv...@apache.org> wrote:
> On Wed, Jun 19, 2013 at 4:29 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>> On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>>> Ping. Any luck?
>
> One final question -- has the memory requirements for NN change? I've
> been always running my Bigtop tests with 1G of heap and I've never
> seen any issues. In this test run NN kept OOMing till I doubled its
> heap size. Is this expected?

To follow: NN keeps OOMing. It seems that a reliable way to make it happen
is to run Bigtop's version of TestCLI against a fully distributed
cluster. At this
point I'm actually reasonably concerned and will follow up tomorrow.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Ping. Any luck?

Unfortunately I've ran into:
    https://issues.apache.org/jira/browse/HADOOP-9654

which was caused by the unrelated memory pressure on the NN,
but it had an unfortunate side effect of making the rest of the
testing stuck. I'll correct the problem now and re-run.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Ping. Any luck?

Unfortunately I've ran into:
    https://issues.apache.org/jira/browse/HADOOP-9654

which was caused by the unrelated memory pressure on the NN,
but it had an unfortunate side effect of making the rest of the
testing stuck. I'll correct the problem now and re-run.

Thanks,
Roman.

RE: Heads up: branch-2.1-beta

[Devaraj k <de...@huawei.com>]

Hi Arun,

	Is there any possibility of including YARN-41 in this release.
 
Thanks
Devaraj K

-----Original Message-----
From: Arun C Murthy [mailto:acm@hortonworks.com] 
Sent: 19 June 2013 12:29
To: mapreduce-dev@hadoop.apache.org; common-dev@hadoop.apache.org; hdfs-dev@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: Heads up: branch-2.1-beta

Ping. Any luck?

On Jun 17, 2013, at 4:06 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Sun, Jun 16, 2013 at 5:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> Roman,
>> 
>> Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?
> 
> I will try to kick off the full build today. And deploy/test tomorrow.
> It is all pretty automated, but takes a long time. Hope the results 
> will still be useful for you guys wrt. 2.1 release.
> 
> Thanks,
> Roman.

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Ping. Any luck?

Unfortunately I've ran into:
    https://issues.apache.org/jira/browse/HADOOP-9654

which was caused by the unrelated memory pressure on the NN,
but it had an unfortunate side effect of making the rest of the
testing stuck. I'll correct the problem now and re-run.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Tue, Jun 18, 2013 at 11:58 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Ping. Any luck?

The full bigtop build failed because of Giraph:
     http://bigtop01.cloudera.org:8080/view/Upstream-tests/job/Hadoop-2.1.0/label=centos5/4/console

Now, I'm pretty sure that's an issue with Giraph
itself (or more precisely Giraph-on-YARN profile)
rather than Hadoop. I'll follow up with the Giraph
community.

Unfortunately, since the full build failed the automated
pipe-line didn't pick up the build for testing. I'll manually
kick it off right away (with Giraph taken from Bigtop 0.6.0)
and will report the results by COB 06/19.

Stay tuned!

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Ping. Any luck?

On Jun 17, 2013, at 4:06 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Sun, Jun 16, 2013 at 5:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> Roman,
>> 
>> Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?
> 
> I will try to kick off the full build today. And deploy/test tomorrow.
> It is all pretty automated, but takes a long time. Hope the results
> will still be useful for you guys wrt. 2.1 release.
> 
> Thanks,
> Roman.

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Ping. Any luck?

On Jun 17, 2013, at 4:06 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Sun, Jun 16, 2013 at 5:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> Roman,
>> 
>> Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?
> 
> I will try to kick off the full build today. And deploy/test tomorrow.
> It is all pretty automated, but takes a long time. Hope the results
> will still be useful for you guys wrt. 2.1 release.
> 
> Thanks,
> Roman.

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Ping. Any luck?

On Jun 17, 2013, at 4:06 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Sun, Jun 16, 2013 at 5:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> Roman,
>> 
>> Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?
> 
> I will try to kick off the full build today. And deploy/test tomorrow.
> It is all pretty automated, but takes a long time. Hope the results
> will still be useful for you guys wrt. 2.1 release.
> 
> Thanks,
> Roman.

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Ping. Any luck?

On Jun 17, 2013, at 4:06 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Sun, Jun 16, 2013 at 5:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> Roman,
>> 
>> Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?
> 
> I will try to kick off the full build today. And deploy/test tomorrow.
> It is all pretty automated, but takes a long time. Hope the results
> will still be useful for you guys wrt. 2.1 release.
> 
> Thanks,
> Roman.

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Sun, Jun 16, 2013 at 5:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Roman,
>
>  Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?

I will try to kick off the full build today. And deploy/test tomorrow.
It is all pretty automated, but takes a long time. Hope the results
will still be useful for you guys wrt. 2.1 release.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Sun, Jun 16, 2013 at 5:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Roman,
>
>  Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?

I will try to kick off the full build today. And deploy/test tomorrow.
It is all pretty automated, but takes a long time. Hope the results
will still be useful for you guys wrt. 2.1 release.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Roman,

 Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?

thanks,
Arun

On Jun 16, 2013, at 4:50 PM, Arun C Murthy wrote:

> Responses inline:
> 
> On Jun 16, 2013, at 1:04 PM, Roman Shaposhnik wrote:
> 
>> But there's a bit of bad news too (or at least the news that need to
>> be triaged). At
>> this point I don't know whether the Hadoop code is to blame or the
>> tests/components
>> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>>   1. HDFS append integration tests failed:
>>            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>>       It seems that there's some incompatibility between the client code
>>       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
>>       and the current Hadoop 2.1.0.
> 
> This is well known, we need to recompile against hadoop-2.1.0-beta.
> 
>> 
>>    2. Quite a few Sqoop tests ended up failing because of what seems
>> like AM not
>>    realizing that one of the tasks exited and waiting for it to
>> timeout. In the end
>>    the task is getting killed like this:
>>         AttemptID:attempt_1371348647940_0030_m_000000_2 Timed out
>> after 600 secsContainer killed by the ApplicationMaster.
>>     but it takes a VERY long time (on the task side the log is
>> attached bellow).
> 
> Essentially, this is a Sqoop error - we could investigate why it took 600s, but doesn't look like a blocker to me.
> 
> We'll probably need another 2.1.1-beta anyway...
> 
>> 
>>     3. There's a couple of Hive tests (out of more than a dozen) that
>> failed in a pretty odd way
>>          (scroll to the very bottom of every page to see the excpetion):
>>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
>>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
>>         What's peculiar here is that nothing has failed *before* or
>> *after* these particular
>>         tests. Hence I don't think that the state of the cluster
>> deployment is to blame.
> 
> Both errors seemed to be related to Hive unit tests failing since MiniHDFSCluster didn't come up:
> 
> Job Submission failed with exception 'org.apache.hadoop.ipc.RemoteException(File /user/jenkins/.staging/job_1371348647940_0308/job.split could only be replicated to 0 nodes instead of minReplication (=1).  There are 4 datanode(s) running and no node(s) are excluded in this operation.
> 
> 
>> 
>>      4. All of the Mahout tests failed with the following:
>>              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>>          This odd, and as I said -- all I know at this point is that
>> the very same
>>           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.
> 
> Again, this is related to fact that we need to recompile Mahout against 2.1.0-beta - in particular this was due to the compatibility work done via MAPREDUCE-5156 (part of MAPREDUCE-5108).
> 
> 
> Arun
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

Hi Arun,

thanks for chiming in!

On Sun, Jun 16, 2013 at 4:50 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> But there's a bit of bad news too (or at least the news that need to
>> be triaged). At
>> this point I don't know whether the Hadoop code is to blame or the
>> tests/components
>> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>>   1. HDFS append integration tests failed:
>>            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>>       It seems that there's some incompatibility between the client code
>>       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
>>       and the current Hadoop 2.1.0.
>
> This is well known, we need to recompile against hadoop-2.1.0-beta.

That's what I suspected. Thanks for confirming. Like I said -- given
the tight schedule I didn't have the time to recompile the entire
Bigtop stack against Hadoop 2.1.0-beta (including tests).

> Essentially, this is a Sqoop error - we could investigate why it took 600s, but doesn't look like a blocker to me.
>
> We'll probably need another 2.1.1-beta anyway...

Agreed. There seems to be some weird interaction between
Sqoop and RM, though. But it definitely doesn't qualify as
a blocker from what I see in the logs and my debugger
and whether it ends up being a Sqoop issue or Hadoop
issue we can tackle it a tad later.

> Both errors seemed to be related to Hive unit tests failing since MiniHDFSCluster didn't come up:

Bigtop Hive tests don't use MiniHDFSCluster -- they are executed against
an honest fully distributed cluster. Like I said -- the weird part here is
that the failing two are in the middle of a whole bunch that worked
just fine. And I haven't really seen them fail before like that ever.
I wish I had more time to investigate.

>>      4. All of the Mahout tests failed with the following:
>>              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>>          This odd, and as I said -- all I know at this point is that
>> the very same
>>           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.
>
> Again, this is related to fact that we need to recompile Mahout against 2.1.0-beta - in
> particular this was due to the compatibility work done via MAPREDUCE-5156 (part of
> MAPREDUCE-5108).

Ah! Didn't know that.

Well, it sounds like we've got enough mileage out of of the quick
application of Bigtop stack/tests from Bigtop 0.6.0. At this point
I feel like the first level of smoke testing passed and if we want
to drill down we'd have to spend time and rebuild everything
against Hadoop 2.1.0. That'll take some time and should happen
sometimes this week.

Hope the amount of validation I was capable of was useful so far.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

Hi Arun,

thanks for chiming in!

On Sun, Jun 16, 2013 at 4:50 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> But there's a bit of bad news too (or at least the news that need to
>> be triaged). At
>> this point I don't know whether the Hadoop code is to blame or the
>> tests/components
>> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>>   1. HDFS append integration tests failed:
>>            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>>       It seems that there's some incompatibility between the client code
>>       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
>>       and the current Hadoop 2.1.0.
>
> This is well known, we need to recompile against hadoop-2.1.0-beta.

That's what I suspected. Thanks for confirming. Like I said -- given
the tight schedule I didn't have the time to recompile the entire
Bigtop stack against Hadoop 2.1.0-beta (including tests).

> Essentially, this is a Sqoop error - we could investigate why it took 600s, but doesn't look like a blocker to me.
>
> We'll probably need another 2.1.1-beta anyway...

Agreed. There seems to be some weird interaction between
Sqoop and RM, though. But it definitely doesn't qualify as
a blocker from what I see in the logs and my debugger
and whether it ends up being a Sqoop issue or Hadoop
issue we can tackle it a tad later.

> Both errors seemed to be related to Hive unit tests failing since MiniHDFSCluster didn't come up:

Bigtop Hive tests don't use MiniHDFSCluster -- they are executed against
an honest fully distributed cluster. Like I said -- the weird part here is
that the failing two are in the middle of a whole bunch that worked
just fine. And I haven't really seen them fail before like that ever.
I wish I had more time to investigate.

>>      4. All of the Mahout tests failed with the following:
>>              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>>          This odd, and as I said -- all I know at this point is that
>> the very same
>>           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.
>
> Again, this is related to fact that we need to recompile Mahout against 2.1.0-beta - in
> particular this was due to the compatibility work done via MAPREDUCE-5156 (part of
> MAPREDUCE-5108).

Ah! Didn't know that.

Well, it sounds like we've got enough mileage out of of the quick
application of Bigtop stack/tests from Bigtop 0.6.0. At this point
I feel like the first level of smoke testing passed and if we want
to drill down we'd have to spend time and rebuild everything
against Hadoop 2.1.0. That'll take some time and should happen
sometimes this week.

Hope the amount of validation I was capable of was useful so far.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Roman,

 Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?

thanks,
Arun

On Jun 16, 2013, at 4:50 PM, Arun C Murthy wrote:

> Responses inline:
> 
> On Jun 16, 2013, at 1:04 PM, Roman Shaposhnik wrote:
> 
>> But there's a bit of bad news too (or at least the news that need to
>> be triaged). At
>> this point I don't know whether the Hadoop code is to blame or the
>> tests/components
>> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>>   1. HDFS append integration tests failed:
>>            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>>       It seems that there's some incompatibility between the client code
>>       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
>>       and the current Hadoop 2.1.0.
> 
> This is well known, we need to recompile against hadoop-2.1.0-beta.
> 
>> 
>>    2. Quite a few Sqoop tests ended up failing because of what seems
>> like AM not
>>    realizing that one of the tasks exited and waiting for it to
>> timeout. In the end
>>    the task is getting killed like this:
>>         AttemptID:attempt_1371348647940_0030_m_000000_2 Timed out
>> after 600 secsContainer killed by the ApplicationMaster.
>>     but it takes a VERY long time (on the task side the log is
>> attached bellow).
> 
> Essentially, this is a Sqoop error - we could investigate why it took 600s, but doesn't look like a blocker to me.
> 
> We'll probably need another 2.1.1-beta anyway...
> 
>> 
>>     3. There's a couple of Hive tests (out of more than a dozen) that
>> failed in a pretty odd way
>>          (scroll to the very bottom of every page to see the excpetion):
>>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
>>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
>>         What's peculiar here is that nothing has failed *before* or
>> *after* these particular
>>         tests. Hence I don't think that the state of the cluster
>> deployment is to blame.
> 
> Both errors seemed to be related to Hive unit tests failing since MiniHDFSCluster didn't come up:
> 
> Job Submission failed with exception 'org.apache.hadoop.ipc.RemoteException(File /user/jenkins/.staging/job_1371348647940_0308/job.split could only be replicated to 0 nodes instead of minReplication (=1).  There are 4 datanode(s) running and no node(s) are excluded in this operation.
> 
> 
>> 
>>      4. All of the Mahout tests failed with the following:
>>              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>>          This odd, and as I said -- all I know at this point is that
>> the very same
>>           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.
> 
> Again, this is related to fact that we need to recompile Mahout against 2.1.0-beta - in particular this was due to the compatibility work done via MAPREDUCE-5156 (part of MAPREDUCE-5108).
> 
> 
> Arun
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Roman,

 Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?

thanks,
Arun

On Jun 16, 2013, at 4:50 PM, Arun C Murthy wrote:

> Responses inline:
> 
> On Jun 16, 2013, at 1:04 PM, Roman Shaposhnik wrote:
> 
>> But there's a bit of bad news too (or at least the news that need to
>> be triaged). At
>> this point I don't know whether the Hadoop code is to blame or the
>> tests/components
>> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>>   1. HDFS append integration tests failed:
>>            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>>       It seems that there's some incompatibility between the client code
>>       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
>>       and the current Hadoop 2.1.0.
> 
> This is well known, we need to recompile against hadoop-2.1.0-beta.
> 
>> 
>>    2. Quite a few Sqoop tests ended up failing because of what seems
>> like AM not
>>    realizing that one of the tasks exited and waiting for it to
>> timeout. In the end
>>    the task is getting killed like this:
>>         AttemptID:attempt_1371348647940_0030_m_000000_2 Timed out
>> after 600 secsContainer killed by the ApplicationMaster.
>>     but it takes a VERY long time (on the task side the log is
>> attached bellow).
> 
> Essentially, this is a Sqoop error - we could investigate why it took 600s, but doesn't look like a blocker to me.
> 
> We'll probably need another 2.1.1-beta anyway...
> 
>> 
>>     3. There's a couple of Hive tests (out of more than a dozen) that
>> failed in a pretty odd way
>>          (scroll to the very bottom of every page to see the excpetion):
>>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
>>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
>>         What's peculiar here is that nothing has failed *before* or
>> *after* these particular
>>         tests. Hence I don't think that the state of the cluster
>> deployment is to blame.
> 
> Both errors seemed to be related to Hive unit tests failing since MiniHDFSCluster didn't come up:
> 
> Job Submission failed with exception 'org.apache.hadoop.ipc.RemoteException(File /user/jenkins/.staging/job_1371348647940_0308/job.split could only be replicated to 0 nodes instead of minReplication (=1).  There are 4 datanode(s) running and no node(s) are excluded in this operation.
> 
> 
>> 
>>      4. All of the Mahout tests failed with the following:
>>              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>>          This odd, and as I said -- all I know at this point is that
>> the very same
>>           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.
> 
> Again, this is related to fact that we need to recompile Mahout against 2.1.0-beta - in particular this was due to the compatibility work done via MAPREDUCE-5156 (part of MAPREDUCE-5108).
> 
> 
> Arun
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Roman,

 Is there a chance you can run the tests with the full stack built against branch-2.1-beta and help us know where we are?

thanks,
Arun

On Jun 16, 2013, at 4:50 PM, Arun C Murthy wrote:

> Responses inline:
> 
> On Jun 16, 2013, at 1:04 PM, Roman Shaposhnik wrote:
> 
>> But there's a bit of bad news too (or at least the news that need to
>> be triaged). At
>> this point I don't know whether the Hadoop code is to blame or the
>> tests/components
>> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>>   1. HDFS append integration tests failed:
>>            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>>       It seems that there's some incompatibility between the client code
>>       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
>>       and the current Hadoop 2.1.0.
> 
> This is well known, we need to recompile against hadoop-2.1.0-beta.
> 
>> 
>>    2. Quite a few Sqoop tests ended up failing because of what seems
>> like AM not
>>    realizing that one of the tasks exited and waiting for it to
>> timeout. In the end
>>    the task is getting killed like this:
>>         AttemptID:attempt_1371348647940_0030_m_000000_2 Timed out
>> after 600 secsContainer killed by the ApplicationMaster.
>>     but it takes a VERY long time (on the task side the log is
>> attached bellow).
> 
> Essentially, this is a Sqoop error - we could investigate why it took 600s, but doesn't look like a blocker to me.
> 
> We'll probably need another 2.1.1-beta anyway...
> 
>> 
>>     3. There's a couple of Hive tests (out of more than a dozen) that
>> failed in a pretty odd way
>>          (scroll to the very bottom of every page to see the excpetion):
>>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
>>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
>>         What's peculiar here is that nothing has failed *before* or
>> *after* these particular
>>         tests. Hence I don't think that the state of the cluster
>> deployment is to blame.
> 
> Both errors seemed to be related to Hive unit tests failing since MiniHDFSCluster didn't come up:
> 
> Job Submission failed with exception 'org.apache.hadoop.ipc.RemoteException(File /user/jenkins/.staging/job_1371348647940_0308/job.split could only be replicated to 0 nodes instead of minReplication (=1).  There are 4 datanode(s) running and no node(s) are excluded in this operation.
> 
> 
>> 
>>      4. All of the Mahout tests failed with the following:
>>              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>>          This odd, and as I said -- all I know at this point is that
>> the very same
>>           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.
> 
> Again, this is related to fact that we need to recompile Mahout against 2.1.0-beta - in particular this was due to the compatibility work done via MAPREDUCE-5156 (part of MAPREDUCE-5108).
> 
> 
> Arun
> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Responses inline:

On Jun 16, 2013, at 1:04 PM, Roman Shaposhnik wrote:

> But there's a bit of bad news too (or at least the news that need to
> be triaged). At
> this point I don't know whether the Hadoop code is to blame or the
> tests/components
> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>   1. HDFS append integration tests failed:
>            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>       It seems that there's some incompatibility between the client code
>       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
>       and the current Hadoop 2.1.0.

This is well known, we need to recompile against hadoop-2.1.0-beta.

> 
>    2. Quite a few Sqoop tests ended up failing because of what seems
> like AM not
>    realizing that one of the tasks exited and waiting for it to
> timeout. In the end
>    the task is getting killed like this:
>         AttemptID:attempt_1371348647940_0030_m_000000_2 Timed out
> after 600 secsContainer killed by the ApplicationMaster.
>     but it takes a VERY long time (on the task side the log is
> attached bellow).

Essentially, this is a Sqoop error - we could investigate why it took 600s, but doesn't look like a blocker to me.

We'll probably need another 2.1.1-beta anyway...

> 
>     3. There's a couple of Hive tests (out of more than a dozen) that
> failed in a pretty odd way
>          (scroll to the very bottom of every page to see the excpetion):
>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
>         What's peculiar here is that nothing has failed *before* or
> *after* these particular
>         tests. Hence I don't think that the state of the cluster
> deployment is to blame.

Both errors seemed to be related to Hive unit tests failing since MiniHDFSCluster didn't come up:

Job Submission failed with exception 'org.apache.hadoop.ipc.RemoteException(File /user/jenkins/.staging/job_1371348647940_0308/job.split could only be replicated to 0 nodes instead of minReplication (=1).  There are 4 datanode(s) running and no node(s) are excluded in this operation.


> 
>      4. All of the Mahout tests failed with the following:
>              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>          This odd, and as I said -- all I know at this point is that
> the very same
>           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.

Again, this is related to fact that we need to recompile Mahout against 2.1.0-beta - in particular this was due to the compatibility work done via MAPREDUCE-5156 (part of MAPREDUCE-5108).


Arun

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Responses inline:

On Jun 16, 2013, at 1:04 PM, Roman Shaposhnik wrote:

> But there's a bit of bad news too (or at least the news that need to
> be triaged). At
> this point I don't know whether the Hadoop code is to blame or the
> tests/components
> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>   1. HDFS append integration tests failed:
>            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>       It seems that there's some incompatibility between the client code
>       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
>       and the current Hadoop 2.1.0.

This is well known, we need to recompile against hadoop-2.1.0-beta.

> 
>    2. Quite a few Sqoop tests ended up failing because of what seems
> like AM not
>    realizing that one of the tasks exited and waiting for it to
> timeout. In the end
>    the task is getting killed like this:
>         AttemptID:attempt_1371348647940_0030_m_000000_2 Timed out
> after 600 secsContainer killed by the ApplicationMaster.
>     but it takes a VERY long time (on the task side the log is
> attached bellow).

Essentially, this is a Sqoop error - we could investigate why it took 600s, but doesn't look like a blocker to me.

We'll probably need another 2.1.1-beta anyway...

> 
>     3. There's a couple of Hive tests (out of more than a dozen) that
> failed in a pretty odd way
>          (scroll to the very bottom of every page to see the excpetion):
>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
>         What's peculiar here is that nothing has failed *before* or
> *after* these particular
>         tests. Hence I don't think that the state of the cluster
> deployment is to blame.

Both errors seemed to be related to Hive unit tests failing since MiniHDFSCluster didn't come up:

Job Submission failed with exception 'org.apache.hadoop.ipc.RemoteException(File /user/jenkins/.staging/job_1371348647940_0308/job.split could only be replicated to 0 nodes instead of minReplication (=1).  There are 4 datanode(s) running and no node(s) are excluded in this operation.


> 
>      4. All of the Mahout tests failed with the following:
>              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>          This odd, and as I said -- all I know at this point is that
> the very same
>           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.

Again, this is related to fact that we need to recompile Mahout against 2.1.0-beta - in particular this was due to the compatibility work done via MAPREDUCE-5156 (part of MAPREDUCE-5108).


Arun

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Responses inline:

On Jun 16, 2013, at 1:04 PM, Roman Shaposhnik wrote:

> But there's a bit of bad news too (or at least the news that need to
> be triaged). At
> this point I don't know whether the Hadoop code is to blame or the
> tests/components
> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>   1. HDFS append integration tests failed:
>            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>       It seems that there's some incompatibility between the client code
>       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
>       and the current Hadoop 2.1.0.

This is well known, we need to recompile against hadoop-2.1.0-beta.

> 
>    2. Quite a few Sqoop tests ended up failing because of what seems
> like AM not
>    realizing that one of the tasks exited and waiting for it to
> timeout. In the end
>    the task is getting killed like this:
>         AttemptID:attempt_1371348647940_0030_m_000000_2 Timed out
> after 600 secsContainer killed by the ApplicationMaster.
>     but it takes a VERY long time (on the task side the log is
> attached bellow).

Essentially, this is a Sqoop error - we could investigate why it took 600s, but doesn't look like a blocker to me.

We'll probably need another 2.1.1-beta anyway...

> 
>     3. There's a couple of Hive tests (out of more than a dozen) that
> failed in a pretty odd way
>          (scroll to the very bottom of every page to see the excpetion):
>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
>         What's peculiar here is that nothing has failed *before* or
> *after* these particular
>         tests. Hence I don't think that the state of the cluster
> deployment is to blame.

Both errors seemed to be related to Hive unit tests failing since MiniHDFSCluster didn't come up:

Job Submission failed with exception 'org.apache.hadoop.ipc.RemoteException(File /user/jenkins/.staging/job_1371348647940_0308/job.split could only be replicated to 0 nodes instead of minReplication (=1).  There are 4 datanode(s) running and no node(s) are excluded in this operation.


> 
>      4. All of the Mahout tests failed with the following:
>              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>          This odd, and as I said -- all I know at this point is that
> the very same
>           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.

Again, this is related to fact that we need to recompile Mahout against 2.1.0-beta - in particular this was due to the compatibility work done via MAPREDUCE-5156 (part of MAPREDUCE-5108).


Arun

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

Responses inline:

On Jun 16, 2013, at 1:04 PM, Roman Shaposhnik wrote:

> But there's a bit of bad news too (or at least the news that need to
> be triaged). At
> this point I don't know whether the Hadoop code is to blame or the
> tests/components
> themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
>   1. HDFS append integration tests failed:
>            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
>       It seems that there's some incompatibility between the client code
>       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
>       and the current Hadoop 2.1.0.

This is well known, we need to recompile against hadoop-2.1.0-beta.

> 
>    2. Quite a few Sqoop tests ended up failing because of what seems
> like AM not
>    realizing that one of the tasks exited and waiting for it to
> timeout. In the end
>    the task is getting killed like this:
>         AttemptID:attempt_1371348647940_0030_m_000000_2 Timed out
> after 600 secsContainer killed by the ApplicationMaster.
>     but it takes a VERY long time (on the task side the log is
> attached bellow).

Essentially, this is a Sqoop error - we could investigate why it took 600s, but doesn't look like a blocker to me.

We'll probably need another 2.1.1-beta anyway...

> 
>     3. There's a couple of Hive tests (out of more than a dozen) that
> failed in a pretty odd way
>          (scroll to the very bottom of every page to see the excpetion):
>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
>             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
>         What's peculiar here is that nothing has failed *before* or
> *after* these particular
>         tests. Hence I don't think that the state of the cluster
> deployment is to blame.

Both errors seemed to be related to Hive unit tests failing since MiniHDFSCluster didn't come up:

Job Submission failed with exception 'org.apache.hadoop.ipc.RemoteException(File /user/jenkins/.staging/job_1371348647940_0308/job.split could only be replicated to 0 nodes instead of minReplication (=1).  There are 4 datanode(s) running and no node(s) are excluded in this operation.


> 
>      4. All of the Mahout tests failed with the following:
>              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
>          This odd, and as I said -- all I know at this point is that
> the very same
>           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.

Again, this is related to fact that we need to recompile Mahout against 2.1.0-beta - in particular this was due to the compatibility work done via MAPREDUCE-5156 (part of MAPREDUCE-5108).


Arun

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Sat, Jun 15, 2013 at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> If the intention is to get the release out in time for the Hadoop Summit we
> have a very tight schedule.
>
> Because the release vote runs for 7 days, we should have an RC latest
> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
> if we need to cut a new RC we can do it on Tuesday. Another thing to
> consider is that if the changes on an RC are corrections that do not affect
> code, we could agree on not reseting the voting period clock if we need to
> cut a new RC (ie doc, build, notes changes).

I didn't realize the schedule was tight. Hence I tried my best to run the
integration tests on Hadoop 2.1.0 as quickly as possible. Here are the
results:
     http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/

The good news is that quite a few tests passed.

But there's a bit of bad news too (or at least the news that need to
be triaged). At
this point I don't know whether the Hadoop code is to blame or the
tests/components
themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
   1. HDFS append integration tests failed:
            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
       It seems that there's some incompatibility between the client code
       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
       and the current Hadoop 2.1.0.

    2. Quite a few Sqoop tests ended up failing because of what seems
like AM not
    realizing that one of the tasks exited and waiting for it to
timeout. In the end
    the task is getting killed like this:
         AttemptID:attempt_1371348647940_0030_m_000000_2 Timed out
after 600 secsContainer killed by the ApplicationMaster.
     but it takes a VERY long time (on the task side the log is
attached bellow).

     3. There's a couple of Hive tests (out of more than a dozen) that
failed in a pretty odd way
          (scroll to the very bottom of every page to see the excpetion):
             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
         What's peculiar here is that nothing has failed *before* or
*after* these particular
         tests. Hence I don't think that the state of the cluster
deployment is to blame.

      4. All of the Mahout tests failed with the following:
              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
          This odd, and as I said -- all I know at this point is that
the very same
           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.

Given the tight schedule any help in triaging these results will be appreciated.

In the meantime I'll run the same set of tests against a fully secure cluster.

Stay tuned!

Thanks,
Roman.

2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.mr.SqoopMapper: Running extractor class
org.apache.sqoop.job.etl.HdfsExportExtractor
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Working on partition:
{hdfs://ip-10-34-97-106.ec2.internal:17020/tmp/input-dir/t_date/part-m-00000
(offset=0, end=8, length=8)}
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Extracting file
hdfs://ip-10-34-97-106.ec2.internal:17020/tmp/input-dir/t_date/part-m-00000
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: 	 from offset 0
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: 	 to offset 8
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: 	 of length 8
2013-06-16 03:16:14,373 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Extracting text file
2013-06-16 03:16:14,542 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Start position: 0
2013-06-16 03:16:14,550 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Extracting ended on
position: 77
2013-06-16 03:16:14,550 INFO [main]
org.apache.sqoop.job.mr.SqoopMapper: Extractor has finished
2013-06-16 03:16:14,552 INFO [main]
org.apache.sqoop.job.mr.SqoopMapper: Stopping progress service
2013-06-16 03:16:14,571 INFO [main]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor:
SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2013-06-16 03:16:15,300 ERROR [OutputFormatLoader-consumer]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: Error while
loading data out of MR job.
java.lang.NumberFormatException: For input string: ""9999-12-31""
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
	at java.lang.Long.parseLong(Long.java:410)
	at java.lang.Long.parseLong(Long.java:468)
	at org.apache.sqoop.job.io.Data.parseField(Data.java:449)
	at org.apache.sqoop.job.io.Data.parse(Data.java:374)
	at org.apache.sqoop.job.io.Data.getContent(Data.java:88)
	at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$OutputFormatDataReader.readContent(SqoopOutputFormatLoadExecutor.java:169)
	at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$OutputFormatDataReader.readArrayRecord(SqoopOutputFormatLoadExecutor.java:145)
	at org.apache.sqoop.connector.jdbc.GenericJdbcExportLoader.load(GenericJdbcExportLoader.java:48)
	at org.apache.sqoop.connector.jdbc.GenericJdbcExportLoader.load(GenericJdbcExportLoader.java:25)
	at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:216)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
2013-06-16 03:16:15,303 INFO [main]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor:
SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> 
> Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
> YARN-752, MAPREDUCE-5171 & YARN-787.

I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it right now.

I'm ok with YARN-752 & MAPREDUCE-5171 going in, but don't consider either a blocker.

Suresh - will HDFS-4777 go in soon too?

Vinod - How about YARN-386?

thanks,
Arun

> 
> Thx
> 
> 
> 
> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> 
>> Following is a revisited assessment of JIRAs I would like to get in the
>> 2.1 release:
>> 
>> From the 1st group I think all 3 should make.
>> 
>> From the 2nd group I think YARN-791 should make it for sure and ideally
>> MAPREDUCE-5130.
>> 
>> From the 3rd group, I don't think this JIRA will make it.
>> 
>> From the 4th group, we don't need to worry about this or 2.1
>> 
>> Thanks
>> 
>> Alejandro
>> 
>> ------------------------------------------------------
>> JIRAs that are in shape to make it to 2.1
>> 
>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>> for requested nodes
>> 
>> impact: behavior change
>> 
>> status: patch avail, +1ed.
>> 
>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>> 
>> impact: Addition to MRAM HTTP API
>> 
>> status: patch avail, +1ed, needs to be committed
>> 
>> * YARN-787: Remove resource min from Yarn client API
>> 
>> impact: Yarn client API change
>> 
>> status: patch avail, needs to be reviewed. (the calculation of slot-millis
>> is not affected, the MIN is taken from conf for now)
>> 
>> ------------------------------------------------------
>> JIRAs that require minor work to make it to 2.1
>> 
>> * YARN-521: Augment AM - RM client module to be able to request containers
>> only at specific locations
>> 
>> impact: AMRM client API change
>> 
>> status: patch not avail yet (requires YARN-752)
>> 
>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
>> /nodes REST API
>> 
>> impact: Yarn client API & proto change
>> 
>> status: patch avail, review in progress
>> 
>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>> 
>> impact: behavior change
>> 
>> status: patch avail but some tests are failing
>> 
>> ------------------------------------------------------
>> JIRAs that require significant work to make it to 2.1 and may not make it
>> 
>> * YARN-649: Make container logs available over HTTP in plain text
>> 
>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>> does not change API)
>> 
>> status: patch avail, review in progress
>> 
>> ------------------------------------------------------
>> JIRAs that don't need to make it to 2.1
>> 
>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>> counter constants
>> 
>> impact: behavior change
>> 
>> status: per discussion we should first add memory-millis and vcores-millis
>> 
>> ------------------------------------------------------
>> 
>> 
>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>> 
>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>> wrote:
>>>> 
>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>> 
>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>> battery
>>>>> of integration tests on the branch-2.1-beta.
>>>>> 
>>>>> We plan to simply file JIRAs if anything gets detected and I will also
>>>>> publish the URL of the Jenkins job once it gets created.
>>>> 
>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>> Blocker priority to catch attention.
>>> 
>>> The build is up and running (and all green on all of the 9 Linux
>>> platforms!):
>>>    http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>> 
>>> The immediate benefit here is that we get to see that the
>>> build is ok on all these Linuxes and all anybody can easily
>>> install packaged Hadoop 2.1.0 nightly builds.
>>> 
>>> Starting from next week, I'll start running regular tests
>>> on these bits and will keep you guys posted!
>>> 
>>> Thanks,
>>> Roman.
>>> 
>> 
>> 
>> 
>> --
>> Alejandro
>> 
> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Sat, Jun 15, 2013 at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> If the intention is to get the release out in time for the Hadoop Summit we
> have a very tight schedule.
>
> Because the release vote runs for 7 days, we should have an RC latest
> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
> if we need to cut a new RC we can do it on Tuesday. Another thing to
> consider is that if the changes on an RC are corrections that do not affect
> code, we could agree on not reseting the voting period clock if we need to
> cut a new RC (ie doc, build, notes changes).

I didn't realize the schedule was tight. Hence I tried my best to run the
integration tests on Hadoop 2.1.0 as quickly as possible. Here are the
results:
     http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/

The good news is that quite a few tests passed.

But there's a bit of bad news too (or at least the news that need to
be triaged). At
this point I don't know whether the Hadoop code is to blame or the
tests/components
themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
   1. HDFS append integration tests failed:
            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
       It seems that there's some incompatibility between the client code
       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
       and the current Hadoop 2.1.0.

    2. Quite a few Sqoop tests ended up failing because of what seems
like AM not
    realizing that one of the tasks exited and waiting for it to
timeout. In the end
    the task is getting killed like this:
         AttemptID:attempt_1371348647940_0030_m_000000_2 Timed out
after 600 secsContainer killed by the ApplicationMaster.
     but it takes a VERY long time (on the task side the log is
attached bellow).

     3. There's a couple of Hive tests (out of more than a dozen) that
failed in a pretty odd way
          (scroll to the very bottom of every page to see the excpetion):
             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
         What's peculiar here is that nothing has failed *before* or
*after* these particular
         tests. Hence I don't think that the state of the cluster
deployment is to blame.

      4. All of the Mahout tests failed with the following:
              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
          This odd, and as I said -- all I know at this point is that
the very same
           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.

Given the tight schedule any help in triaging these results will be appreciated.

In the meantime I'll run the same set of tests against a fully secure cluster.

Stay tuned!

Thanks,
Roman.

2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.mr.SqoopMapper: Running extractor class
org.apache.sqoop.job.etl.HdfsExportExtractor
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Working on partition:
{hdfs://ip-10-34-97-106.ec2.internal:17020/tmp/input-dir/t_date/part-m-00000
(offset=0, end=8, length=8)}
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Extracting file
hdfs://ip-10-34-97-106.ec2.internal:17020/tmp/input-dir/t_date/part-m-00000
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: 	 from offset 0
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: 	 to offset 8
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: 	 of length 8
2013-06-16 03:16:14,373 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Extracting text file
2013-06-16 03:16:14,542 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Start position: 0
2013-06-16 03:16:14,550 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Extracting ended on
position: 77
2013-06-16 03:16:14,550 INFO [main]
org.apache.sqoop.job.mr.SqoopMapper: Extractor has finished
2013-06-16 03:16:14,552 INFO [main]
org.apache.sqoop.job.mr.SqoopMapper: Stopping progress service
2013-06-16 03:16:14,571 INFO [main]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor:
SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2013-06-16 03:16:15,300 ERROR [OutputFormatLoader-consumer]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: Error while
loading data out of MR job.
java.lang.NumberFormatException: For input string: ""9999-12-31""
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
	at java.lang.Long.parseLong(Long.java:410)
	at java.lang.Long.parseLong(Long.java:468)
	at org.apache.sqoop.job.io.Data.parseField(Data.java:449)
	at org.apache.sqoop.job.io.Data.parse(Data.java:374)
	at org.apache.sqoop.job.io.Data.getContent(Data.java:88)
	at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$OutputFormatDataReader.readContent(SqoopOutputFormatLoadExecutor.java:169)
	at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$OutputFormatDataReader.readArrayRecord(SqoopOutputFormatLoadExecutor.java:145)
	at org.apache.sqoop.connector.jdbc.GenericJdbcExportLoader.load(GenericJdbcExportLoader.java:48)
	at org.apache.sqoop.connector.jdbc.GenericJdbcExportLoader.load(GenericJdbcExportLoader.java:25)
	at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:216)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
2013-06-16 03:16:15,303 INFO [main]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor:
SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> 
> Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
> YARN-752, MAPREDUCE-5171 & YARN-787.

I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it right now.

I'm ok with YARN-752 & MAPREDUCE-5171 going in, but don't consider either a blocker.

Suresh - will HDFS-4777 go in soon too?

Vinod - How about YARN-386?

thanks,
Arun

> 
> Thx
> 
> 
> 
> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> 
>> Following is a revisited assessment of JIRAs I would like to get in the
>> 2.1 release:
>> 
>> From the 1st group I think all 3 should make.
>> 
>> From the 2nd group I think YARN-791 should make it for sure and ideally
>> MAPREDUCE-5130.
>> 
>> From the 3rd group, I don't think this JIRA will make it.
>> 
>> From the 4th group, we don't need to worry about this or 2.1
>> 
>> Thanks
>> 
>> Alejandro
>> 
>> ------------------------------------------------------
>> JIRAs that are in shape to make it to 2.1
>> 
>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>> for requested nodes
>> 
>> impact: behavior change
>> 
>> status: patch avail, +1ed.
>> 
>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>> 
>> impact: Addition to MRAM HTTP API
>> 
>> status: patch avail, +1ed, needs to be committed
>> 
>> * YARN-787: Remove resource min from Yarn client API
>> 
>> impact: Yarn client API change
>> 
>> status: patch avail, needs to be reviewed. (the calculation of slot-millis
>> is not affected, the MIN is taken from conf for now)
>> 
>> ------------------------------------------------------
>> JIRAs that require minor work to make it to 2.1
>> 
>> * YARN-521: Augment AM - RM client module to be able to request containers
>> only at specific locations
>> 
>> impact: AMRM client API change
>> 
>> status: patch not avail yet (requires YARN-752)
>> 
>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
>> /nodes REST API
>> 
>> impact: Yarn client API & proto change
>> 
>> status: patch avail, review in progress
>> 
>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>> 
>> impact: behavior change
>> 
>> status: patch avail but some tests are failing
>> 
>> ------------------------------------------------------
>> JIRAs that require significant work to make it to 2.1 and may not make it
>> 
>> * YARN-649: Make container logs available over HTTP in plain text
>> 
>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>> does not change API)
>> 
>> status: patch avail, review in progress
>> 
>> ------------------------------------------------------
>> JIRAs that don't need to make it to 2.1
>> 
>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>> counter constants
>> 
>> impact: behavior change
>> 
>> status: per discussion we should first add memory-millis and vcores-millis
>> 
>> ------------------------------------------------------
>> 
>> 
>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>> 
>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>> wrote:
>>>> 
>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>> 
>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>> battery
>>>>> of integration tests on the branch-2.1-beta.
>>>>> 
>>>>> We plan to simply file JIRAs if anything gets detected and I will also
>>>>> publish the URL of the Jenkins job once it gets created.
>>>> 
>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>> Blocker priority to catch attention.
>>> 
>>> The build is up and running (and all green on all of the 9 Linux
>>> platforms!):
>>>    http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>> 
>>> The immediate benefit here is that we get to see that the
>>> build is ok on all these Linuxes and all anybody can easily
>>> install packaged Hadoop 2.1.0 nightly builds.
>>> 
>>> Starting from next week, I'll start running regular tests
>>> on these bits and will keep you guys posted!
>>> 
>>> Thanks,
>>> Roman.
>>> 
>> 
>> 
>> 
>> --
>> Alejandro
>> 
> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Sat, Jun 15, 2013 at 8:19 AM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> If the intention is to get the release out in time for the Hadoop Summit we
> have a very tight schedule.
>
> Because the release vote runs for 7 days, we should have an RC latest
> Monday afternoon, and we should encourage folks to verify & vote ASAP, so
> if we need to cut a new RC we can do it on Tuesday. Another thing to
> consider is that if the changes on an RC are corrections that do not affect
> code, we could agree on not reseting the voting period clock if we need to
> cut a new RC (ie doc, build, notes changes).

I didn't realize the schedule was tight. Hence I tried my best to run the
integration tests on Hadoop 2.1.0 as quickly as possible. Here are the
results:
     http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/

The good news is that quite a few tests passed.

But there's a bit of bad news too (or at least the news that need to
be triaged). At
this point I don't know whether the Hadoop code is to blame or the
tests/components
themselves -- all I know is that these tests passed with Hadoop 2.0.5-alpha:
   1. HDFS append integration tests failed:
            http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/lastCompletedBuild/testReport/org.apache.bigtop.itest.hadoop.hdfs/TestFileAppend/testMultipleOutputStreamFailure/
       It seems that there's some incompatibility between the client code
       that was compiled against Hadoop 2.0.5 (as part of Bigtop 0.6.0 release)
       and the current Hadoop 2.1.0.

    2. Quite a few Sqoop tests ended up failing because of what seems
like AM not
    realizing that one of the tasks exited and waiting for it to
timeout. In the end
    the task is getting killed like this:
         AttemptID:attempt_1371348647940_0030_m_000000_2 Timed out
after 600 secsContainer killed by the ApplicationMaster.
     but it takes a VERY long time (on the task side the log is
attached bellow).

     3. There's a couple of Hive tests (out of more than a dozen) that
failed in a pretty odd way
          (scroll to the very bottom of every page to see the excpetion):
             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_auto_join20_/
             http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.hivesmoke/TestHiveSmokeBulk/testHiveBulk_union3_/
         What's peculiar here is that nothing has failed *before* or
*after* these particular
         tests. Hence I don't think that the state of the cluster
deployment is to blame.

      4. All of the Mahout tests failed with the following:
              http://bigtop01.cloudera.org:8080/view/Bigtop-trunk/job/Bigtop-trunk-runsmokes/59/testReport/org.apache.bigtop.itest.mahout.smoke/TestMahoutExamples/clusterControlDataWithCanopy/
          This odd, and as I said -- all I know at this point is that
the very same
           tests running the very same Mahout pass with Hadoop 2.0.5-alpha.

Given the tight schedule any help in triaging these results will be appreciated.

In the meantime I'll run the same set of tests against a fully secure cluster.

Stay tuned!

Thanks,
Roman.

2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.mr.SqoopMapper: Running extractor class
org.apache.sqoop.job.etl.HdfsExportExtractor
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Working on partition:
{hdfs://ip-10-34-97-106.ec2.internal:17020/tmp/input-dir/t_date/part-m-00000
(offset=0, end=8, length=8)}
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Extracting file
hdfs://ip-10-34-97-106.ec2.internal:17020/tmp/input-dir/t_date/part-m-00000
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: 	 from offset 0
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: 	 to offset 8
2013-06-16 03:16:14,195 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: 	 of length 8
2013-06-16 03:16:14,373 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Extracting text file
2013-06-16 03:16:14,542 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Start position: 0
2013-06-16 03:16:14,550 INFO [main]
org.apache.sqoop.job.etl.HdfsExportExtractor: Extracting ended on
position: 77
2013-06-16 03:16:14,550 INFO [main]
org.apache.sqoop.job.mr.SqoopMapper: Extractor has finished
2013-06-16 03:16:14,552 INFO [main]
org.apache.sqoop.job.mr.SqoopMapper: Stopping progress service
2013-06-16 03:16:14,571 INFO [main]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor:
SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed
2013-06-16 03:16:15,300 ERROR [OutputFormatLoader-consumer]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor: Error while
loading data out of MR job.
java.lang.NumberFormatException: For input string: ""9999-12-31""
	at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
	at java.lang.Long.parseLong(Long.java:410)
	at java.lang.Long.parseLong(Long.java:468)
	at org.apache.sqoop.job.io.Data.parseField(Data.java:449)
	at org.apache.sqoop.job.io.Data.parse(Data.java:374)
	at org.apache.sqoop.job.io.Data.getContent(Data.java:88)
	at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$OutputFormatDataReader.readContent(SqoopOutputFormatLoadExecutor.java:169)
	at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$OutputFormatDataReader.readArrayRecord(SqoopOutputFormatLoadExecutor.java:145)
	at org.apache.sqoop.connector.jdbc.GenericJdbcExportLoader.load(GenericJdbcExportLoader.java:48)
	at org.apache.sqoop.connector.jdbc.GenericJdbcExportLoader.load(GenericJdbcExportLoader.java:25)
	at org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor$ConsumerThread.run(SqoopOutputFormatLoadExecutor.java:216)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
2013-06-16 03:16:15,303 INFO [main]
org.apache.sqoop.job.mr.SqoopOutputFormatLoadExecutor:
SqoopOutputFormatLoadExecutor::SqoopRecordWriter is about to be closed

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 15, 2013, at 8:19 AM, Alejandro Abdelnur wrote:
> 
> Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
> YARN-752, MAPREDUCE-5171 & YARN-787.

I agree YARN-787 needs to go in ASAP & is a blocker - I'm looking at it right now.

I'm ok with YARN-752 & MAPREDUCE-5171 going in, but don't consider either a blocker.

Suresh - will HDFS-4777 go in soon too?

Vinod - How about YARN-386?

thanks,
Arun

> 
> Thx
> 
> 
> 
> On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> 
>> Following is a revisited assessment of JIRAs I would like to get in the
>> 2.1 release:
>> 
>> From the 1st group I think all 3 should make.
>> 
>> From the 2nd group I think YARN-791 should make it for sure and ideally
>> MAPREDUCE-5130.
>> 
>> From the 3rd group, I don't think this JIRA will make it.
>> 
>> From the 4th group, we don't need to worry about this or 2.1
>> 
>> Thanks
>> 
>> Alejandro
>> 
>> ------------------------------------------------------
>> JIRAs that are in shape to make it to 2.1
>> 
>> * YARN-752: In AMRMClient, automatically add corresponding rack requests
>> for requested nodes
>> 
>> impact: behavior change
>> 
>> status: patch avail, +1ed.
>> 
>> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>> 
>> impact: Addition to MRAM HTTP API
>> 
>> status: patch avail, +1ed, needs to be committed
>> 
>> * YARN-787: Remove resource min from Yarn client API
>> 
>> impact: Yarn client API change
>> 
>> status: patch avail, needs to be reviewed. (the calculation of slot-millis
>> is not affected, the MIN is taken from conf for now)
>> 
>> ------------------------------------------------------
>> JIRAs that require minor work to make it to 2.1
>> 
>> * YARN-521: Augment AM - RM client module to be able to request containers
>> only at specific locations
>> 
>> impact: AMRM client API change
>> 
>> status: patch not avail yet (requires YARN-752)
>> 
>> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
>> /nodes REST API
>> 
>> impact: Yarn client API & proto change
>> 
>> status: patch avail, review in progress
>> 
>> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>> 
>> impact: behavior change
>> 
>> status: patch avail but some tests are failing
>> 
>> ------------------------------------------------------
>> JIRAs that require significant work to make it to 2.1 and may not make it
>> 
>> * YARN-649: Make container logs available over HTTP in plain text
>> 
>> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
>> does not change API)
>> 
>> status: patch avail, review in progress
>> 
>> ------------------------------------------------------
>> JIRAs that don't need to make it to 2.1
>> 
>> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
>> counter constants
>> 
>> impact: behavior change
>> 
>> status: per discussion we should first add memory-millis and vcores-millis
>> 
>> ------------------------------------------------------
>> 
>> 
>> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>> 
>>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>>> wrote:
>>>> 
>>>> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>>>> 
>>>>> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>>>>> based on Hadoop 2.0.x codeline we plan to start running the same
>>> battery
>>>>> of integration tests on the branch-2.1-beta.
>>>>> 
>>>>> We plan to simply file JIRAs if anything gets detected and I will also
>>>>> publish the URL of the Jenkins job once it gets created.
>>>> 
>>>> Thanks Roman. Is there an ETA for this? Also, please file jiras with
>>> Blocker priority to catch attention.
>>> 
>>> The build is up and running (and all green on all of the 9 Linux
>>> platforms!):
>>>    http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>> 
>>> The immediate benefit here is that we get to see that the
>>> build is ok on all these Linuxes and all anybody can easily
>>> install packaged Hadoop 2.1.0 nightly builds.
>>> 
>>> Starting from next week, I'll start running regular tests
>>> on these bits and will keep you guys posted!
>>> 
>>> Thanks,
>>> Roman.
>>> 
>> 
>> 
>> 
>> --
>> Alejandro
>> 
> 
> 
> 
> -- 
> Alejandro

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

If the intention is to get the release out in time for the Hadoop Summit we
have a very tight schedule.

Because the release vote runs for 7 days, we should have an RC latest
Monday afternoon, and we should encourage folks to verify & vote ASAP, so
if we need to cut a new RC we can do it on Tuesday. Another thing to
consider is that if the changes on an RC are corrections that do not affect
code, we could agree on not reseting the voting period clock if we need to
cut a new RC (ie doc, build, notes changes).

Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
YARN-752, MAPREDUCE-5171 & YARN-787.

The first 2 are already +1ed, the last one needs to be reviewed.

I have not committed the first 2 ones yet because I don't want to disrupt
things for the folks doing QA.

Arun, as you are coordinating the work for this release, please do commit
them or give me the go ahead and I'll commit.

Also, it  would be great if you can review YARN-787 (as per discussions,
the changes on the milli-slot calculations do not affect the current
calculations, that would be left for MAPREDUCE-5311 to do).

I'll be checking my email over the weekend and I can take care of some
stuff if needed (while the monkeys sleep).

Thx



On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Following is a revisited assessment of JIRAs I would like to get in the
> 2.1 release:
>
> From the 1st group I think all 3 should make.
>
> From the 2nd group I think YARN-791 should make it for sure and ideally
> MAPREDUCE-5130.
>
> From the 3rd group, I don't think this JIRA will make it.
>
> From the 4th group, we don't need to worry about this or 2.1
>
> Thanks
>
> Alejandro
>
> ------------------------------------------------------
> JIRAs that are in shape to make it to 2.1
>
> * YARN-752: In AMRMClient, automatically add corresponding rack requests
> for requested nodes
>
> impact: behavior change
>
> status: patch avail, +1ed.
>
> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>
> impact: Addition to MRAM HTTP API
>
> status: patch avail, +1ed, needs to be committed
>
> * YARN-787: Remove resource min from Yarn client API
>
> impact: Yarn client API change
>
> status: patch avail, needs to be reviewed. (the calculation of slot-millis
> is not affected, the MIN is taken from conf for now)
>
> ------------------------------------------------------
>  JIRAs that require minor work to make it to 2.1
>
> * YARN-521: Augment AM - RM client module to be able to request containers
> only at specific locations
>
> impact: AMRM client API change
>
> status: patch not avail yet (requires YARN-752)
>
> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
> /nodes REST API
>
> impact: Yarn client API & proto change
>
> status: patch avail, review in progress
>
> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>
> impact: behavior change
>
> status: patch avail but some tests are failing
>
> ------------------------------------------------------
> JIRAs that require significant work to make it to 2.1 and may not make it
>
> * YARN-649: Make container logs available over HTTP in plain text
>
> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
> does not change API)
>
> status: patch avail, review in progress
>
> ------------------------------------------------------
> JIRAs that don't need to make it to 2.1
>
> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
> counter constants
>
> impact: behavior change
>
> status: per discussion we should first add memory-millis and vcores-millis
>
> ------------------------------------------------------
>
>
> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>
>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>> wrote:
>> >
>> > On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>> >>
>> >> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>> >> based on Hadoop 2.0.x codeline we plan to start running the same
>> battery
>> >> of integration tests on the branch-2.1-beta.
>> >>
>> >> We plan to simply file JIRAs if anything gets detected and I will also
>> >> publish the URL of the Jenkins job once it gets created.
>> >
>> > Thanks Roman. Is there an ETA for this? Also, please file jiras with
>> Blocker priority to catch attention.
>>
>> The build is up and running (and all green on all of the 9 Linux
>> platforms!):
>>     http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>
>> The immediate benefit here is that we get to see that the
>> build is ok on all these Linuxes and all anybody can easily
>> install packaged Hadoop 2.1.0 nightly builds.
>>
>> Starting from next week, I'll start running regular tests
>> on these bits and will keep you guys posted!
>>
>> Thanks,
>> Roman.
>>
>
>
>
> --
> Alejandro
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

If the intention is to get the release out in time for the Hadoop Summit we
have a very tight schedule.

Because the release vote runs for 7 days, we should have an RC latest
Monday afternoon, and we should encourage folks to verify & vote ASAP, so
if we need to cut a new RC we can do it on Tuesday. Another thing to
consider is that if the changes on an RC are corrections that do not affect
code, we could agree on not reseting the voting period clock if we need to
cut a new RC (ie doc, build, notes changes).

Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
YARN-752, MAPREDUCE-5171 & YARN-787.

The first 2 are already +1ed, the last one needs to be reviewed.

I have not committed the first 2 ones yet because I don't want to disrupt
things for the folks doing QA.

Arun, as you are coordinating the work for this release, please do commit
them or give me the go ahead and I'll commit.

Also, it  would be great if you can review YARN-787 (as per discussions,
the changes on the milli-slot calculations do not affect the current
calculations, that would be left for MAPREDUCE-5311 to do).

I'll be checking my email over the weekend and I can take care of some
stuff if needed (while the monkeys sleep).

Thx



On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Following is a revisited assessment of JIRAs I would like to get in the
> 2.1 release:
>
> From the 1st group I think all 3 should make.
>
> From the 2nd group I think YARN-791 should make it for sure and ideally
> MAPREDUCE-5130.
>
> From the 3rd group, I don't think this JIRA will make it.
>
> From the 4th group, we don't need to worry about this or 2.1
>
> Thanks
>
> Alejandro
>
> ------------------------------------------------------
> JIRAs that are in shape to make it to 2.1
>
> * YARN-752: In AMRMClient, automatically add corresponding rack requests
> for requested nodes
>
> impact: behavior change
>
> status: patch avail, +1ed.
>
> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>
> impact: Addition to MRAM HTTP API
>
> status: patch avail, +1ed, needs to be committed
>
> * YARN-787: Remove resource min from Yarn client API
>
> impact: Yarn client API change
>
> status: patch avail, needs to be reviewed. (the calculation of slot-millis
> is not affected, the MIN is taken from conf for now)
>
> ------------------------------------------------------
>  JIRAs that require minor work to make it to 2.1
>
> * YARN-521: Augment AM - RM client module to be able to request containers
> only at specific locations
>
> impact: AMRM client API change
>
> status: patch not avail yet (requires YARN-752)
>
> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
> /nodes REST API
>
> impact: Yarn client API & proto change
>
> status: patch avail, review in progress
>
> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>
> impact: behavior change
>
> status: patch avail but some tests are failing
>
> ------------------------------------------------------
> JIRAs that require significant work to make it to 2.1 and may not make it
>
> * YARN-649: Make container logs available over HTTP in plain text
>
> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
> does not change API)
>
> status: patch avail, review in progress
>
> ------------------------------------------------------
> JIRAs that don't need to make it to 2.1
>
> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
> counter constants
>
> impact: behavior change
>
> status: per discussion we should first add memory-millis and vcores-millis
>
> ------------------------------------------------------
>
>
> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>
>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>> wrote:
>> >
>> > On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>> >>
>> >> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>> >> based on Hadoop 2.0.x codeline we plan to start running the same
>> battery
>> >> of integration tests on the branch-2.1-beta.
>> >>
>> >> We plan to simply file JIRAs if anything gets detected and I will also
>> >> publish the URL of the Jenkins job once it gets created.
>> >
>> > Thanks Roman. Is there an ETA for this? Also, please file jiras with
>> Blocker priority to catch attention.
>>
>> The build is up and running (and all green on all of the 9 Linux
>> platforms!):
>>     http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>
>> The immediate benefit here is that we get to see that the
>> build is ok on all these Linuxes and all anybody can easily
>> install packaged Hadoop 2.1.0 nightly builds.
>>
>> Starting from next week, I'll start running regular tests
>> on these bits and will keep you guys posted!
>>
>> Thanks,
>> Roman.
>>
>
>
>
> --
> Alejandro
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

If the intention is to get the release out in time for the Hadoop Summit we
have a very tight schedule.

Because the release vote runs for 7 days, we should have an RC latest
Monday afternoon, and we should encourage folks to verify & vote ASAP, so
if we need to cut a new RC we can do it on Tuesday. Another thing to
consider is that if the changes on an RC are corrections that do not affect
code, we could agree on not reseting the voting period clock if we need to
cut a new RC (ie doc, build, notes changes).

Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
YARN-752, MAPREDUCE-5171 & YARN-787.

The first 2 are already +1ed, the last one needs to be reviewed.

I have not committed the first 2 ones yet because I don't want to disrupt
things for the folks doing QA.

Arun, as you are coordinating the work for this release, please do commit
them or give me the go ahead and I'll commit.

Also, it  would be great if you can review YARN-787 (as per discussions,
the changes on the milli-slot calculations do not affect the current
calculations, that would be left for MAPREDUCE-5311 to do).

I'll be checking my email over the weekend and I can take care of some
stuff if needed (while the monkeys sleep).

Thx



On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Following is a revisited assessment of JIRAs I would like to get in the
> 2.1 release:
>
> From the 1st group I think all 3 should make.
>
> From the 2nd group I think YARN-791 should make it for sure and ideally
> MAPREDUCE-5130.
>
> From the 3rd group, I don't think this JIRA will make it.
>
> From the 4th group, we don't need to worry about this or 2.1
>
> Thanks
>
> Alejandro
>
> ------------------------------------------------------
> JIRAs that are in shape to make it to 2.1
>
> * YARN-752: In AMRMClient, automatically add corresponding rack requests
> for requested nodes
>
> impact: behavior change
>
> status: patch avail, +1ed.
>
> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>
> impact: Addition to MRAM HTTP API
>
> status: patch avail, +1ed, needs to be committed
>
> * YARN-787: Remove resource min from Yarn client API
>
> impact: Yarn client API change
>
> status: patch avail, needs to be reviewed. (the calculation of slot-millis
> is not affected, the MIN is taken from conf for now)
>
> ------------------------------------------------------
>  JIRAs that require minor work to make it to 2.1
>
> * YARN-521: Augment AM - RM client module to be able to request containers
> only at specific locations
>
> impact: AMRM client API change
>
> status: patch not avail yet (requires YARN-752)
>
> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
> /nodes REST API
>
> impact: Yarn client API & proto change
>
> status: patch avail, review in progress
>
> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>
> impact: behavior change
>
> status: patch avail but some tests are failing
>
> ------------------------------------------------------
> JIRAs that require significant work to make it to 2.1 and may not make it
>
> * YARN-649: Make container logs available over HTTP in plain text
>
> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
> does not change API)
>
> status: patch avail, review in progress
>
> ------------------------------------------------------
> JIRAs that don't need to make it to 2.1
>
> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
> counter constants
>
> impact: behavior change
>
> status: per discussion we should first add memory-millis and vcores-millis
>
> ------------------------------------------------------
>
>
> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>
>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>> wrote:
>> >
>> > On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>> >>
>> >> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>> >> based on Hadoop 2.0.x codeline we plan to start running the same
>> battery
>> >> of integration tests on the branch-2.1-beta.
>> >>
>> >> We plan to simply file JIRAs if anything gets detected and I will also
>> >> publish the URL of the Jenkins job once it gets created.
>> >
>> > Thanks Roman. Is there an ETA for this? Also, please file jiras with
>> Blocker priority to catch attention.
>>
>> The build is up and running (and all green on all of the 9 Linux
>> platforms!):
>>     http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>
>> The immediate benefit here is that we get to see that the
>> build is ok on all these Linuxes and all anybody can easily
>> install packaged Hadoop 2.1.0 nightly builds.
>>
>> Starting from next week, I'll start running regular tests
>> on these bits and will keep you guys posted!
>>
>> Thanks,
>> Roman.
>>
>
>
>
> --
> Alejandro
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

If the intention is to get the release out in time for the Hadoop Summit we
have a very tight schedule.

Because the release vote runs for 7 days, we should have an RC latest
Monday afternoon, and we should encourage folks to verify & vote ASAP, so
if we need to cut a new RC we can do it on Tuesday. Another thing to
consider is that if the changes on an RC are corrections that do not affect
code, we could agree on not reseting the voting period clock if we need to
cut a new RC (ie doc, build, notes changes).

Of the JIRAs in my laundry list for 2.1 the ones I would really want in are
YARN-752, MAPREDUCE-5171 & YARN-787.

The first 2 are already +1ed, the last one needs to be reviewed.

I have not committed the first 2 ones yet because I don't want to disrupt
things for the folks doing QA.

Arun, as you are coordinating the work for this release, please do commit
them or give me the go ahead and I'll commit.

Also, it  would be great if you can review YARN-787 (as per discussions,
the changes on the milli-slot calculations do not affect the current
calculations, that would be left for MAPREDUCE-5311 to do).

I'll be checking my email over the weekend and I can take care of some
stuff if needed (while the monkeys sleep).

Thx



On Fri, Jun 14, 2013 at 9:56 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Following is a revisited assessment of JIRAs I would like to get in the
> 2.1 release:
>
> From the 1st group I think all 3 should make.
>
> From the 2nd group I think YARN-791 should make it for sure and ideally
> MAPREDUCE-5130.
>
> From the 3rd group, I don't think this JIRA will make it.
>
> From the 4th group, we don't need to worry about this or 2.1
>
> Thanks
>
> Alejandro
>
> ------------------------------------------------------
> JIRAs that are in shape to make it to 2.1
>
> * YARN-752: In AMRMClient, automatically add corresponding rack requests
> for requested nodes
>
> impact: behavior change
>
> status: patch avail, +1ed.
>
> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>
> impact: Addition to MRAM HTTP API
>
> status: patch avail, +1ed, needs to be committed
>
> * YARN-787: Remove resource min from Yarn client API
>
> impact: Yarn client API change
>
> status: patch avail, needs to be reviewed. (the calculation of slot-millis
> is not affected, the MIN is taken from conf for now)
>
> ------------------------------------------------------
>  JIRAs that require minor work to make it to 2.1
>
> * YARN-521: Augment AM - RM client module to be able to request containers
> only at specific locations
>
> impact: AMRM client API change
>
> status: patch not avail yet (requires YARN-752)
>
> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
> /nodes REST API
>
> impact: Yarn client API & proto change
>
> status: patch avail, review in progress
>
> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>
> impact: behavior change
>
> status: patch avail but some tests are failing
>
> ------------------------------------------------------
> JIRAs that require significant work to make it to 2.1 and may not make it
>
> * YARN-649: Make container logs available over HTTP in plain text
>
> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
> does not change API)
>
> status: patch avail, review in progress
>
> ------------------------------------------------------
> JIRAs that don't need to make it to 2.1
>
> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
> counter constants
>
> impact: behavior change
>
> status: per discussion we should first add memory-millis and vcores-millis
>
> ------------------------------------------------------
>
>
> On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:
>
>> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com>
>> wrote:
>> >
>> > On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>> >>
>> >> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>> >> based on Hadoop 2.0.x codeline we plan to start running the same
>> battery
>> >> of integration tests on the branch-2.1-beta.
>> >>
>> >> We plan to simply file JIRAs if anything gets detected and I will also
>> >> publish the URL of the Jenkins job once it gets created.
>> >
>> > Thanks Roman. Is there an ETA for this? Also, please file jiras with
>> Blocker priority to catch attention.
>>
>> The build is up and running (and all green on all of the 9 Linux
>> platforms!):
>>     http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>>
>> The immediate benefit here is that we get to see that the
>> build is ok on all these Linuxes and all anybody can easily
>> install packaged Hadoop 2.1.0 nightly builds.
>>
>> Starting from next week, I'll start running regular tests
>> on these bits and will keep you guys posted!
>>
>> Thanks,
>> Roman.
>>
>
>
>
> --
> Alejandro
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Following is a revisited assessment of JIRAs I would like to get in the 2.1
release:

>From the 1st group I think all 3 should make.

>From the 2nd group I think YARN-791 should make it for sure and ideally
MAPREDUCE-5130.

>From the 3rd group, I don't think this JIRA will make it.

>From the 4th group, we don't need to worry about this or 2.1

Thanks

Alejandro

------------------------------------------------------
JIRAs that are in shape to make it to 2.1

* YARN-752: In AMRMClient, automatically add corresponding rack requests
for requested nodes

impact: behavior change

status: patch avail, +1ed.

* MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API

impact: Addition to MRAM HTTP API

status: patch avail, +1ed, needs to be committed

* YARN-787: Remove resource min from Yarn client API

impact: Yarn client API change

status: patch avail, needs to be reviewed. (the calculation of slot-millis
is not affected, the MIN is taken from conf for now)

------------------------------------------------------
JIRAs that require minor work to make it to 2.1

* YARN-521: Augment AM - RM client module to be able to request containers
only at specific locations

impact: AMRM client API change

status: patch not avail yet (requires YARN-752)

* YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
/nodes REST API

impact: Yarn client API & proto change

status: patch avail, review in progress

* MAPREDUCE-5130: Add missing job config options to mapred-default.xml

impact: behavior change

status: patch avail but some tests are failing

------------------------------------------------------
JIRAs that require significant work to make it to 2.1 and may not make it

* YARN-649: Make container logs available over HTTP in plain text

impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which does
not change API)

status: patch avail, review in progress

------------------------------------------------------
JIRAs that don't need to make it to 2.1

* MAPREDUCE-5311: Remove slot millis computation logic and deprecate
counter constants

impact: behavior change

status: per discussion we should first add memory-millis and vcores-millis

------------------------------------------------------


On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> >
> > On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
> >>
> >> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
> >> based on Hadoop 2.0.x codeline we plan to start running the same battery
> >> of integration tests on the branch-2.1-beta.
> >>
> >> We plan to simply file JIRAs if anything gets detected and I will also
> >> publish the URL of the Jenkins job once it gets created.
> >
> > Thanks Roman. Is there an ETA for this? Also, please file jiras with
> Blocker priority to catch attention.
>
> The build is up and running (and all green on all of the 9 Linux
> platforms!):
>     http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>
> The immediate benefit here is that we get to see that the
> build is ok on all these Linuxes and all anybody can easily
> install packaged Hadoop 2.1.0 nightly builds.
>
> Starting from next week, I'll start running regular tests
> on these bits and will keep you guys posted!
>
> Thanks,
> Roman.
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Following is a revisited assessment of JIRAs I would like to get in the 2.1
release:

>From the 1st group I think all 3 should make.

>From the 2nd group I think YARN-791 should make it for sure and ideally
MAPREDUCE-5130.

>From the 3rd group, I don't think this JIRA will make it.

>From the 4th group, we don't need to worry about this or 2.1

Thanks

Alejandro

------------------------------------------------------
JIRAs that are in shape to make it to 2.1

* YARN-752: In AMRMClient, automatically add corresponding rack requests
for requested nodes

impact: behavior change

status: patch avail, +1ed.

* MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API

impact: Addition to MRAM HTTP API

status: patch avail, +1ed, needs to be committed

* YARN-787: Remove resource min from Yarn client API

impact: Yarn client API change

status: patch avail, needs to be reviewed. (the calculation of slot-millis
is not affected, the MIN is taken from conf for now)

------------------------------------------------------
JIRAs that require minor work to make it to 2.1

* YARN-521: Augment AM - RM client module to be able to request containers
only at specific locations

impact: AMRM client API change

status: patch not avail yet (requires YARN-752)

* YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
/nodes REST API

impact: Yarn client API & proto change

status: patch avail, review in progress

* MAPREDUCE-5130: Add missing job config options to mapred-default.xml

impact: behavior change

status: patch avail but some tests are failing

------------------------------------------------------
JIRAs that require significant work to make it to 2.1 and may not make it

* YARN-649: Make container logs available over HTTP in plain text

impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which does
not change API)

status: patch avail, review in progress

------------------------------------------------------
JIRAs that don't need to make it to 2.1

* MAPREDUCE-5311: Remove slot millis computation logic and deprecate
counter constants

impact: behavior change

status: per discussion we should first add memory-millis and vcores-millis

------------------------------------------------------


On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> >
> > On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
> >>
> >> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
> >> based on Hadoop 2.0.x codeline we plan to start running the same battery
> >> of integration tests on the branch-2.1-beta.
> >>
> >> We plan to simply file JIRAs if anything gets detected and I will also
> >> publish the URL of the Jenkins job once it gets created.
> >
> > Thanks Roman. Is there an ETA for this? Also, please file jiras with
> Blocker priority to catch attention.
>
> The build is up and running (and all green on all of the 9 Linux
> platforms!):
>     http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>
> The immediate benefit here is that we get to see that the
> build is ok on all these Linuxes and all anybody can easily
> install packaged Hadoop 2.1.0 nightly builds.
>
> Starting from next week, I'll start running regular tests
> on these bits and will keep you guys posted!
>
> Thanks,
> Roman.
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Following is a revisited assessment of JIRAs I would like to get in the 2.1
release:

>From the 1st group I think all 3 should make.

>From the 2nd group I think YARN-791 should make it for sure and ideally
MAPREDUCE-5130.

>From the 3rd group, I don't think this JIRA will make it.

>From the 4th group, we don't need to worry about this or 2.1

Thanks

Alejandro

------------------------------------------------------
JIRAs that are in shape to make it to 2.1

* YARN-752: In AMRMClient, automatically add corresponding rack requests
for requested nodes

impact: behavior change

status: patch avail, +1ed.

* MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API

impact: Addition to MRAM HTTP API

status: patch avail, +1ed, needs to be committed

* YARN-787: Remove resource min from Yarn client API

impact: Yarn client API change

status: patch avail, needs to be reviewed. (the calculation of slot-millis
is not affected, the MIN is taken from conf for now)

------------------------------------------------------
JIRAs that require minor work to make it to 2.1

* YARN-521: Augment AM - RM client module to be able to request containers
only at specific locations

impact: AMRM client API change

status: patch not avail yet (requires YARN-752)

* YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
/nodes REST API

impact: Yarn client API & proto change

status: patch avail, review in progress

* MAPREDUCE-5130: Add missing job config options to mapred-default.xml

impact: behavior change

status: patch avail but some tests are failing

------------------------------------------------------
JIRAs that require significant work to make it to 2.1 and may not make it

* YARN-649: Make container logs available over HTTP in plain text

impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which does
not change API)

status: patch avail, review in progress

------------------------------------------------------
JIRAs that don't need to make it to 2.1

* MAPREDUCE-5311: Remove slot millis computation logic and deprecate
counter constants

impact: behavior change

status: per discussion we should first add memory-millis and vcores-millis

------------------------------------------------------


On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> >
> > On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
> >>
> >> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
> >> based on Hadoop 2.0.x codeline we plan to start running the same battery
> >> of integration tests on the branch-2.1-beta.
> >>
> >> We plan to simply file JIRAs if anything gets detected and I will also
> >> publish the URL of the Jenkins job once it gets created.
> >
> > Thanks Roman. Is there an ETA for this? Also, please file jiras with
> Blocker priority to catch attention.
>
> The build is up and running (and all green on all of the 9 Linux
> platforms!):
>     http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>
> The immediate benefit here is that we get to see that the
> build is ok on all these Linuxes and all anybody can easily
> install packaged Hadoop 2.1.0 nightly builds.
>
> Starting from next week, I'll start running regular tests
> on these bits and will keep you guys posted!
>
> Thanks,
> Roman.
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Following is a revisited assessment of JIRAs I would like to get in the 2.1
release:

>From the 1st group I think all 3 should make.

>From the 2nd group I think YARN-791 should make it for sure and ideally
MAPREDUCE-5130.

>From the 3rd group, I don't think this JIRA will make it.

>From the 4th group, we don't need to worry about this or 2.1

Thanks

Alejandro

------------------------------------------------------
JIRAs that are in shape to make it to 2.1

* YARN-752: In AMRMClient, automatically add corresponding rack requests
for requested nodes

impact: behavior change

status: patch avail, +1ed.

* MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API

impact: Addition to MRAM HTTP API

status: patch avail, +1ed, needs to be committed

* YARN-787: Remove resource min from Yarn client API

impact: Yarn client API change

status: patch avail, needs to be reviewed. (the calculation of slot-millis
is not affected, the MIN is taken from conf for now)

------------------------------------------------------
JIRAs that require minor work to make it to 2.1

* YARN-521: Augment AM - RM client module to be able to request containers
only at specific locations

impact: AMRM client API change

status: patch not avail yet (requires YARN-752)

* YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
/nodes REST API

impact: Yarn client API & proto change

status: patch avail, review in progress

* MAPREDUCE-5130: Add missing job config options to mapred-default.xml

impact: behavior change

status: patch avail but some tests are failing

------------------------------------------------------
JIRAs that require significant work to make it to 2.1 and may not make it

* YARN-649: Make container logs available over HTTP in plain text

impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which does
not change API)

status: patch avail, review in progress

------------------------------------------------------
JIRAs that don't need to make it to 2.1

* MAPREDUCE-5311: Remove slot millis computation logic and deprecate
counter constants

impact: behavior change

status: per discussion we should first add memory-millis and vcores-millis

------------------------------------------------------


On Fri, Jun 14, 2013 at 7:17 PM, Roman Shaposhnik <rv...@apache.org> wrote:

> On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> >
> > On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
> >>
> >> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
> >> based on Hadoop 2.0.x codeline we plan to start running the same battery
> >> of integration tests on the branch-2.1-beta.
> >>
> >> We plan to simply file JIRAs if anything gets detected and I will also
> >> publish the URL of the Jenkins job once it gets created.
> >
> > Thanks Roman. Is there an ETA for this? Also, please file jiras with
> Blocker priority to catch attention.
>
> The build is up and running (and all green on all of the 9 Linux
> platforms!):
>     http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/
>
> The immediate benefit here is that we get to see that the
> build is ok on all these Linuxes and all anybody can easily
> install packaged Hadoop 2.1.0 nightly builds.
>
> Starting from next week, I'll start running regular tests
> on these bits and will keep you guys posted!
>
> Thanks,
> Roman.
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>
>> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>> based on Hadoop 2.0.x codeline we plan to start running the same battery
>> of integration tests on the branch-2.1-beta.
>>
>> We plan to simply file JIRAs if anything gets detected and I will also
>> publish the URL of the Jenkins job once it gets created.
>
> Thanks Roman. Is there an ETA for this? Also, please file jiras with Blocker priority to catch attention.

The build is up and running (and all green on all of the 9 Linux platforms!):
    http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/

The immediate benefit here is that we get to see that the
build is ok on all these Linuxes and all anybody can easily
install packaged Hadoop 2.1.0 nightly builds.

Starting from next week, I'll start running regular tests
on these bits and will keep you guys posted!

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>
>> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>> based on Hadoop 2.0.x codeline we plan to start running the same battery
>> of integration tests on the branch-2.1-beta.
>>
>> We plan to simply file JIRAs if anything gets detected and I will also
>> publish the URL of the Jenkins job once it gets created.
>
> Thanks Roman. Is there an ETA for this? Also, please file jiras with Blocker priority to catch attention.

The build is up and running (and all green on all of the 9 Linux platforms!):
    http://bigtop01.cloudera.org:8080/job/Hadoop-2.1.0/

The immediate benefit here is that we get to see that the
build is ok on all these Linuxes and all anybody can easily
install packaged Hadoop 2.1.0 nightly builds.

Starting from next week, I'll start running regular tests
on these bits and will keep you guys posted!

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>
>> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>> based on Hadoop 2.0.x codeline we plan to start running the same battery
>> of integration tests on the branch-2.1-beta.
>>
>> We plan to simply file JIRAs if anything gets detected and I will also
>> publish the URL of the Jenkins job once it gets created.
>
> Thanks Roman. Is there an ETA for this? Also, please file jiras with Blocker priority to catch attention.

Now that we've got the branch building on ASF it should be by the
end of this week.

I'll send a heads up once everything is hooked up and working.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Thu, Jun 6, 2013 at 4:48 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
>>
>> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
>> based on Hadoop 2.0.x codeline we plan to start running the same battery
>> of integration tests on the branch-2.1-beta.
>>
>> We plan to simply file JIRAs if anything gets detected and I will also
>> publish the URL of the Jenkins job once it gets created.
>
> Thanks Roman. Is there an ETA for this? Also, please file jiras with Blocker priority to catch attention.

Now that we've got the branch building on ASF it should be by the
end of this week.

I'll send a heads up once everything is hooked up and working.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
> 
> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
> based on Hadoop 2.0.x codeline we plan to start running the same battery
> of integration tests on the branch-2.1-beta.
> 
> We plan to simply file JIRAs if anything gets detected and I will also
> publish the URL of the Jenkins job once it gets created.

Thanks Roman. Is there an ETA for this? Also, please file jiras with Blocker priority to catch attention.

thanks,
Arun

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
> 
> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
> based on Hadoop 2.0.x codeline we plan to start running the same battery
> of integration tests on the branch-2.1-beta.
> 
> We plan to simply file JIRAs if anything gets detected and I will also
> publish the URL of the Jenkins job once it gets created.

Thanks Roman. Is there an ETA for this? Also, please file jiras with Blocker priority to catch attention.

thanks,
Arun

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
> 
> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
> based on Hadoop 2.0.x codeline we plan to start running the same battery
> of integration tests on the branch-2.1-beta.
> 
> We plan to simply file JIRAs if anything gets detected and I will also
> publish the URL of the Jenkins job once it gets created.

Thanks Roman. Is there an ETA for this? Also, please file jiras with Blocker priority to catch attention.

thanks,
Arun

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 5, 2013, at 11:04 AM, Roman Shaposhnik wrote
> 
> On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
> based on Hadoop 2.0.x codeline we plan to start running the same battery
> of integration tests on the branch-2.1-beta.
> 
> We plan to simply file JIRAs if anything gets detected and I will also
> publish the URL of the Jenkins job once it gets created.

Thanks Roman. Is there an ETA for this? Also, please file jiras with Blocker priority to catch attention.

thanks,
Arun

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Tue, Jun 4, 2013 at 8:32 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Folks,
>
>  The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!
>
>  I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.
>
>  For now the remaining work is to wrap up loose ends i.e. last minute api work (e.g. YARN-759 showed up last night for consideration),
>  bug-fixes etc.; then run this through a battery of unit/system/integration tests and do a final review before we ship. There is more
> work remaining on documentation (e.g. HADOOP-9517) and I plan to personally focus on it this week - obviously help reviewing docs
> is very welcome.

On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
based on Hadoop 2.0.x codeline we plan to start running the same battery
of integration tests on the branch-2.1-beta.

We plan to simply file JIRAs if anything gets detected and I will also
publish the URL of the Jenkins job once it gets created.

Let me know if there's anything else Bigtop can help with to make 2.1-beta
a success.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Roman Shaposhnik <rv...@apache.org>]

On Tue, Jun 4, 2013 at 8:32 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Folks,
>
>  The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!
>
>  I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.
>
>  For now the remaining work is to wrap up loose ends i.e. last minute api work (e.g. YARN-759 showed up last night for consideration),
>  bug-fixes etc.; then run this through a battery of unit/system/integration tests and do a final review before we ship. There is more
> work remaining on documentation (e.g. HADOOP-9517) and I plan to personally focus on it this week - obviously help reviewing docs
> is very welcome.

On the Bigtop side of things, once we have stable Bigtop 0.6.0 platform
based on Hadoop 2.0.x codeline we plan to start running the same battery
of integration tests on the branch-2.1-beta.

We plan to simply file JIRAs if anything gets detected and I will also
publish the URL of the Jenkins job once it gets created.

Let me know if there's anything else Bigtop can help with to make 2.1-beta
a success.

Thanks,
Roman.

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun

Forgot to make it explicit in previous email, I'll be happy to help so this
is done ASAP.

Please, let me know how you want to proceed

Thx


On Fri, Jun 14, 2013 at 2:02 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Arun,
>
> This sounds great. Following is the list of JIRAs I'd like to get in. Note
> that the are ready or almost ready, my estimate is that they can be taken
> care of in a couple of day.
>
> Thanks.
>
> * YARN-752: In AMRMClient, automatically add corresponding rack requests
> for requested nodes
>
> impact: behavior change
>
> status: patch avail, reviewed by Bikas. As Bikas did some changes it needs
> another committer to look at it.
>
> * YARN-521: Augment AM - RM client module to be able to request containers
> only at specific locations
>
> impact: AMRM client API change
>
> status: patch avail, needs to be reviewed, needs YARN-752
>
> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
> /nodes REST API
>
> impact: Yarn client API & proto change
>
> status: patch avail, review in progress
>
> * YARN-649: Make container logs available over HTTP in plain text
>
> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
> does not change API)
>
> status: patch avail, review in progress
>
> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>
> impact: Addition to MRAM HTTP API
>
> status: patch avail, +1ed, needs to be committed
>
> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>
> impact: behavior change
>
> status: patch avail, needs to be reviewed
>
> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
> counter constants
>
> impact: behavior change
>
> status: patch avail, needs to be reviewed
>
> * YARN-787: Remove resource min from Yarn client API
>
> impact: Yarn client API change
>
> status: patch needs rebase, depends on MAPREDUCE-5311
>
>
>
>
> On Fri, Jun 14, 2013 at 1:17 PM, Arun C Murthy <ac...@hortonworks.com>wrote:
>
>> As Ramya noted, things are looking good on branch-2.1-beta ATM.
>>
>> Henceforth, can I please ask committers to hold off non-blocker fixes for
>> the final set of tests?
>>
>> thanks,
>> Arun
>>
>> On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:
>>
>> > Folks,
>> >
>> > The vast majority of of the planned features and API work is complete,
>> thanks to everyone who contributed!
>> >
>> > I've created a branch-2.1-beta branch from which I anticipate I can
>> make the first of our beta releases very shortly.
>> >
>> > For now the remaining work is to wrap up loose ends i.e. last minute
>> api work (e.g. YARN-759 showed up last night for consideration), bug-fixes
>> etc.; then run this through a battery of unit/system/integration tests and
>> do a final review before we ship. There is more work remaining on
>> documentation (e.g. HADOOP-9517) and I plan to personally focus on it this
>> week - obviously help reviewing docs is very welcome.
>> >
>> > Committers, from now, please please exercise your judgement on where
>> you commit. Typically, features should go into branch-2 with 2.3.0 as the
>> version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0
>> will be limited to content in branch-2.1-beta and we stick to stabilizing
>> it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
>> >
>> > thanks,
>> > Arun
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>
>
>
> --
> Alejandro
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun

Forgot to make it explicit in previous email, I'll be happy to help so this
is done ASAP.

Please, let me know how you want to proceed

Thx


On Fri, Jun 14, 2013 at 2:02 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Arun,
>
> This sounds great. Following is the list of JIRAs I'd like to get in. Note
> that the are ready or almost ready, my estimate is that they can be taken
> care of in a couple of day.
>
> Thanks.
>
> * YARN-752: In AMRMClient, automatically add corresponding rack requests
> for requested nodes
>
> impact: behavior change
>
> status: patch avail, reviewed by Bikas. As Bikas did some changes it needs
> another committer to look at it.
>
> * YARN-521: Augment AM - RM client module to be able to request containers
> only at specific locations
>
> impact: AMRM client API change
>
> status: patch avail, needs to be reviewed, needs YARN-752
>
> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
> /nodes REST API
>
> impact: Yarn client API & proto change
>
> status: patch avail, review in progress
>
> * YARN-649: Make container logs available over HTTP in plain text
>
> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
> does not change API)
>
> status: patch avail, review in progress
>
> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>
> impact: Addition to MRAM HTTP API
>
> status: patch avail, +1ed, needs to be committed
>
> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>
> impact: behavior change
>
> status: patch avail, needs to be reviewed
>
> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
> counter constants
>
> impact: behavior change
>
> status: patch avail, needs to be reviewed
>
> * YARN-787: Remove resource min from Yarn client API
>
> impact: Yarn client API change
>
> status: patch needs rebase, depends on MAPREDUCE-5311
>
>
>
>
> On Fri, Jun 14, 2013 at 1:17 PM, Arun C Murthy <ac...@hortonworks.com>wrote:
>
>> As Ramya noted, things are looking good on branch-2.1-beta ATM.
>>
>> Henceforth, can I please ask committers to hold off non-blocker fixes for
>> the final set of tests?
>>
>> thanks,
>> Arun
>>
>> On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:
>>
>> > Folks,
>> >
>> > The vast majority of of the planned features and API work is complete,
>> thanks to everyone who contributed!
>> >
>> > I've created a branch-2.1-beta branch from which I anticipate I can
>> make the first of our beta releases very shortly.
>> >
>> > For now the remaining work is to wrap up loose ends i.e. last minute
>> api work (e.g. YARN-759 showed up last night for consideration), bug-fixes
>> etc.; then run this through a battery of unit/system/integration tests and
>> do a final review before we ship. There is more work remaining on
>> documentation (e.g. HADOOP-9517) and I plan to personally focus on it this
>> week - obviously help reviewing docs is very welcome.
>> >
>> > Committers, from now, please please exercise your judgement on where
>> you commit. Typically, features should go into branch-2 with 2.3.0 as the
>> version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0
>> will be limited to content in branch-2.1-beta and we stick to stabilizing
>> it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
>> >
>> > thanks,
>> > Arun
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>
>
>
> --
> Alejandro
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun

Forgot to make it explicit in previous email, I'll be happy to help so this
is done ASAP.

Please, let me know how you want to proceed

Thx


On Fri, Jun 14, 2013 at 2:02 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Arun,
>
> This sounds great. Following is the list of JIRAs I'd like to get in. Note
> that the are ready or almost ready, my estimate is that they can be taken
> care of in a couple of day.
>
> Thanks.
>
> * YARN-752: In AMRMClient, automatically add corresponding rack requests
> for requested nodes
>
> impact: behavior change
>
> status: patch avail, reviewed by Bikas. As Bikas did some changes it needs
> another committer to look at it.
>
> * YARN-521: Augment AM - RM client module to be able to request containers
> only at specific locations
>
> impact: AMRM client API change
>
> status: patch avail, needs to be reviewed, needs YARN-752
>
> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
> /nodes REST API
>
> impact: Yarn client API & proto change
>
> status: patch avail, review in progress
>
> * YARN-649: Make container logs available over HTTP in plain text
>
> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
> does not change API)
>
> status: patch avail, review in progress
>
> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>
> impact: Addition to MRAM HTTP API
>
> status: patch avail, +1ed, needs to be committed
>
> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>
> impact: behavior change
>
> status: patch avail, needs to be reviewed
>
> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
> counter constants
>
> impact: behavior change
>
> status: patch avail, needs to be reviewed
>
> * YARN-787: Remove resource min from Yarn client API
>
> impact: Yarn client API change
>
> status: patch needs rebase, depends on MAPREDUCE-5311
>
>
>
>
> On Fri, Jun 14, 2013 at 1:17 PM, Arun C Murthy <ac...@hortonworks.com>wrote:
>
>> As Ramya noted, things are looking good on branch-2.1-beta ATM.
>>
>> Henceforth, can I please ask committers to hold off non-blocker fixes for
>> the final set of tests?
>>
>> thanks,
>> Arun
>>
>> On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:
>>
>> > Folks,
>> >
>> > The vast majority of of the planned features and API work is complete,
>> thanks to everyone who contributed!
>> >
>> > I've created a branch-2.1-beta branch from which I anticipate I can
>> make the first of our beta releases very shortly.
>> >
>> > For now the remaining work is to wrap up loose ends i.e. last minute
>> api work (e.g. YARN-759 showed up last night for consideration), bug-fixes
>> etc.; then run this through a battery of unit/system/integration tests and
>> do a final review before we ship. There is more work remaining on
>> documentation (e.g. HADOOP-9517) and I plan to personally focus on it this
>> week - obviously help reviewing docs is very welcome.
>> >
>> > Committers, from now, please please exercise your judgement on where
>> you commit. Typically, features should go into branch-2 with 2.3.0 as the
>> version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0
>> will be limited to content in branch-2.1-beta and we stick to stabilizing
>> it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
>> >
>> > thanks,
>> > Arun
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>
>
>
> --
> Alejandro
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun

Forgot to make it explicit in previous email, I'll be happy to help so this
is done ASAP.

Please, let me know how you want to proceed

Thx


On Fri, Jun 14, 2013 at 2:02 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> Arun,
>
> This sounds great. Following is the list of JIRAs I'd like to get in. Note
> that the are ready or almost ready, my estimate is that they can be taken
> care of in a couple of day.
>
> Thanks.
>
> * YARN-752: In AMRMClient, automatically add corresponding rack requests
> for requested nodes
>
> impact: behavior change
>
> status: patch avail, reviewed by Bikas. As Bikas did some changes it needs
> another committer to look at it.
>
> * YARN-521: Augment AM - RM client module to be able to request containers
> only at specific locations
>
> impact: AMRM client API change
>
> status: patch avail, needs to be reviewed, needs YARN-752
>
> * YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
> /nodes REST API
>
> impact: Yarn client API & proto change
>
> status: patch avail, review in progress
>
> * YARN-649: Make container logs available over HTTP in plain text
>
> impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which
> does not change API)
>
> status: patch avail, review in progress
>
> * MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API
>
> impact: Addition to MRAM HTTP API
>
> status: patch avail, +1ed, needs to be committed
>
> * MAPREDUCE-5130: Add missing job config options to mapred-default.xml
>
> impact: behavior change
>
> status: patch avail, needs to be reviewed
>
> * MAPREDUCE-5311: Remove slot millis computation logic and deprecate
> counter constants
>
> impact: behavior change
>
> status: patch avail, needs to be reviewed
>
> * YARN-787: Remove resource min from Yarn client API
>
> impact: Yarn client API change
>
> status: patch needs rebase, depends on MAPREDUCE-5311
>
>
>
>
> On Fri, Jun 14, 2013 at 1:17 PM, Arun C Murthy <ac...@hortonworks.com>wrote:
>
>> As Ramya noted, things are looking good on branch-2.1-beta ATM.
>>
>> Henceforth, can I please ask committers to hold off non-blocker fixes for
>> the final set of tests?
>>
>> thanks,
>> Arun
>>
>> On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:
>>
>> > Folks,
>> >
>> > The vast majority of of the planned features and API work is complete,
>> thanks to everyone who contributed!
>> >
>> > I've created a branch-2.1-beta branch from which I anticipate I can
>> make the first of our beta releases very shortly.
>> >
>> > For now the remaining work is to wrap up loose ends i.e. last minute
>> api work (e.g. YARN-759 showed up last night for consideration), bug-fixes
>> etc.; then run this through a battery of unit/system/integration tests and
>> do a final review before we ship. There is more work remaining on
>> documentation (e.g. HADOOP-9517) and I plan to personally focus on it this
>> week - obviously help reviewing docs is very welcome.
>> >
>> > Committers, from now, please please exercise your judgement on where
>> you commit. Typically, features should go into branch-2 with 2.3.0 as the
>> version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0
>> will be limited to content in branch-2.1-beta and we stick to stabilizing
>> it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
>> >
>> > thanks,
>> > Arun
>>
>> --
>> Arun C. Murthy
>> Hortonworks Inc.
>> http://hortonworks.com/
>>
>>
>>
>
>
> --
> Alejandro
>



-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun,

This sounds great. Following is the list of JIRAs I'd like to get in. Note
that the are ready or almost ready, my estimate is that they can be taken
care of in a couple of day.

Thanks.

* YARN-752: In AMRMClient, automatically add corresponding rack requests
for requested nodes

impact: behavior change

status: patch avail, reviewed by Bikas. As Bikas did some changes it needs
another committer to look at it.

* YARN-521: Augment AM - RM client module to be able to request containers
only at specific locations

impact: AMRM client API change

status: patch avail, needs to be reviewed, needs YARN-752

* YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
/nodes REST API

impact: Yarn client API & proto change

status: patch avail, review in progress

* YARN-649: Make container logs available over HTTP in plain text

impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which does
not change API)

status: patch avail, review in progress

* MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API

impact: Addition to MRAM HTTP API

status: patch avail, +1ed, needs to be committed

* MAPREDUCE-5130: Add missing job config options to mapred-default.xml

impact: behavior change

status: patch avail, needs to be reviewed

* MAPREDUCE-5311: Remove slot millis computation logic and deprecate
counter constants

impact: behavior change

status: patch avail, needs to be reviewed

* YARN-787: Remove resource min from Yarn client API

impact: Yarn client API change

status: patch needs rebase, depends on MAPREDUCE-5311




On Fri, Jun 14, 2013 at 1:17 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> As Ramya noted, things are looking good on branch-2.1-beta ATM.
>
> Henceforth, can I please ask committers to hold off non-blocker fixes for
> the final set of tests?
>
> thanks,
> Arun
>
> On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:
>
> > Folks,
> >
> > The vast majority of of the planned features and API work is complete,
> thanks to everyone who contributed!
> >
> > I've created a branch-2.1-beta branch from which I anticipate I can make
> the first of our beta releases very shortly.
> >
> > For now the remaining work is to wrap up loose ends i.e. last minute api
> work (e.g. YARN-759 showed up last night for consideration), bug-fixes
> etc.; then run this through a battery of unit/system/integration tests and
> do a final review before we ship. There is more work remaining on
> documentation (e.g. HADOOP-9517) and I plan to personally focus on it this
> week - obviously help reviewing docs is very welcome.
> >
> > Committers, from now, please please exercise your judgement on where you
> commit. Typically, features should go into branch-2 with 2.3.0 as the
> version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0
> will be limited to content in branch-2.1-beta and we stick to stabilizing
> it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
> >
> > thanks,
> > Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun,

This sounds great. Following is the list of JIRAs I'd like to get in. Note
that the are ready or almost ready, my estimate is that they can be taken
care of in a couple of day.

Thanks.

* YARN-752: In AMRMClient, automatically add corresponding rack requests
for requested nodes

impact: behavior change

status: patch avail, reviewed by Bikas. As Bikas did some changes it needs
another committer to look at it.

* YARN-521: Augment AM - RM client module to be able to request containers
only at specific locations

impact: AMRM client API change

status: patch avail, needs to be reviewed, needs YARN-752

* YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
/nodes REST API

impact: Yarn client API & proto change

status: patch avail, review in progress

* YARN-649: Make container logs available over HTTP in plain text

impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which does
not change API)

status: patch avail, review in progress

* MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API

impact: Addition to MRAM HTTP API

status: patch avail, +1ed, needs to be committed

* MAPREDUCE-5130: Add missing job config options to mapred-default.xml

impact: behavior change

status: patch avail, needs to be reviewed

* MAPREDUCE-5311: Remove slot millis computation logic and deprecate
counter constants

impact: behavior change

status: patch avail, needs to be reviewed

* YARN-787: Remove resource min from Yarn client API

impact: Yarn client API change

status: patch needs rebase, depends on MAPREDUCE-5311




On Fri, Jun 14, 2013 at 1:17 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> As Ramya noted, things are looking good on branch-2.1-beta ATM.
>
> Henceforth, can I please ask committers to hold off non-blocker fixes for
> the final set of tests?
>
> thanks,
> Arun
>
> On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:
>
> > Folks,
> >
> > The vast majority of of the planned features and API work is complete,
> thanks to everyone who contributed!
> >
> > I've created a branch-2.1-beta branch from which I anticipate I can make
> the first of our beta releases very shortly.
> >
> > For now the remaining work is to wrap up loose ends i.e. last minute api
> work (e.g. YARN-759 showed up last night for consideration), bug-fixes
> etc.; then run this through a battery of unit/system/integration tests and
> do a final review before we ship. There is more work remaining on
> documentation (e.g. HADOOP-9517) and I plan to personally focus on it this
> week - obviously help reviewing docs is very welcome.
> >
> > Committers, from now, please please exercise your judgement on where you
> commit. Typically, features should go into branch-2 with 2.3.0 as the
> version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0
> will be limited to content in branch-2.1-beta and we stick to stabilizing
> it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
> >
> > thanks,
> > Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun,

This sounds great. Following is the list of JIRAs I'd like to get in. Note
that the are ready or almost ready, my estimate is that they can be taken
care of in a couple of day.

Thanks.

* YARN-752: In AMRMClient, automatically add corresponding rack requests
for requested nodes

impact: behavior change

status: patch avail, reviewed by Bikas. As Bikas did some changes it needs
another committer to look at it.

* YARN-521: Augment AM - RM client module to be able to request containers
only at specific locations

impact: AMRM client API change

status: patch avail, needs to be reviewed, needs YARN-752

* YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
/nodes REST API

impact: Yarn client API & proto change

status: patch avail, review in progress

* YARN-649: Make container logs available over HTTP in plain text

impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which does
not change API)

status: patch avail, review in progress

* MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API

impact: Addition to MRAM HTTP API

status: patch avail, +1ed, needs to be committed

* MAPREDUCE-5130: Add missing job config options to mapred-default.xml

impact: behavior change

status: patch avail, needs to be reviewed

* MAPREDUCE-5311: Remove slot millis computation logic and deprecate
counter constants

impact: behavior change

status: patch avail, needs to be reviewed

* YARN-787: Remove resource min from Yarn client API

impact: Yarn client API change

status: patch needs rebase, depends on MAPREDUCE-5311




On Fri, Jun 14, 2013 at 1:17 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> As Ramya noted, things are looking good on branch-2.1-beta ATM.
>
> Henceforth, can I please ask committers to hold off non-blocker fixes for
> the final set of tests?
>
> thanks,
> Arun
>
> On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:
>
> > Folks,
> >
> > The vast majority of of the planned features and API work is complete,
> thanks to everyone who contributed!
> >
> > I've created a branch-2.1-beta branch from which I anticipate I can make
> the first of our beta releases very shortly.
> >
> > For now the remaining work is to wrap up loose ends i.e. last minute api
> work (e.g. YARN-759 showed up last night for consideration), bug-fixes
> etc.; then run this through a battery of unit/system/integration tests and
> do a final review before we ship. There is more work remaining on
> documentation (e.g. HADOOP-9517) and I plan to personally focus on it this
> week - obviously help reviewing docs is very welcome.
> >
> > Committers, from now, please please exercise your judgement on where you
> commit. Typically, features should go into branch-2 with 2.3.0 as the
> version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0
> will be limited to content in branch-2.1-beta and we stick to stabilizing
> it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
> >
> > thanks,
> > Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Alejandro Abdelnur <tu...@cloudera.com>]

Arun,

This sounds great. Following is the list of JIRAs I'd like to get in. Note
that the are ready or almost ready, my estimate is that they can be taken
care of in a couple of day.

Thanks.

* YARN-752: In AMRMClient, automatically add corresponding rack requests
for requested nodes

impact: behavior change

status: patch avail, reviewed by Bikas. As Bikas did some changes it needs
another committer to look at it.

* YARN-521: Augment AM - RM client module to be able to request containers
only at specific locations

impact: AMRM client API change

status: patch avail, needs to be reviewed, needs YARN-752

* YARN-791: Ensure that RM RPC APIs that return nodes are consistent with
/nodes REST API

impact: Yarn client API & proto change

status: patch avail, review in progress

* YARN-649: Make container logs available over HTTP in plain text

impact: Addition to NM HTTP REST API. Needed for MAPREDUCE-4362 (which does
not change API)

status: patch avail, review in progress

* MAPREDUCE-5171: Expose blacklisted nodes from the MR AM REST API

impact: Addition to MRAM HTTP API

status: patch avail, +1ed, needs to be committed

* MAPREDUCE-5130: Add missing job config options to mapred-default.xml

impact: behavior change

status: patch avail, needs to be reviewed

* MAPREDUCE-5311: Remove slot millis computation logic and deprecate
counter constants

impact: behavior change

status: patch avail, needs to be reviewed

* YARN-787: Remove resource min from Yarn client API

impact: Yarn client API change

status: patch needs rebase, depends on MAPREDUCE-5311




On Fri, Jun 14, 2013 at 1:17 PM, Arun C Murthy <ac...@hortonworks.com> wrote:

> As Ramya noted, things are looking good on branch-2.1-beta ATM.
>
> Henceforth, can I please ask committers to hold off non-blocker fixes for
> the final set of tests?
>
> thanks,
> Arun
>
> On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:
>
> > Folks,
> >
> > The vast majority of of the planned features and API work is complete,
> thanks to everyone who contributed!
> >
> > I've created a branch-2.1-beta branch from which I anticipate I can make
> the first of our beta releases very shortly.
> >
> > For now the remaining work is to wrap up loose ends i.e. last minute api
> work (e.g. YARN-759 showed up last night for consideration), bug-fixes
> etc.; then run this through a battery of unit/system/integration tests and
> do a final review before we ship. There is more work remaining on
> documentation (e.g. HADOOP-9517) and I plan to personally focus on it this
> week - obviously help reviewing docs is very welcome.
> >
> > Committers, from now, please please exercise your judgement on where you
> commit. Typically, features should go into branch-2 with 2.3.0 as the
> version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0
> will be limited to content in branch-2.1-beta and we stick to stabilizing
> it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
> >
> > thanks,
> > Arun
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>


-- 
Alejandro

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

As Ramya noted, things are looking good on branch-2.1-beta ATM.

Henceforth, can I please ask committers to hold off non-blocker fixes for the final set of tests?

thanks,
Arun

On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:

> Folks,
> 
> The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!
> 
> I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.
> 
> For now the remaining work is to wrap up loose ends i.e. last minute api work (e.g. YARN-759 showed up last night for consideration), bug-fixes etc.; then run this through a battery of unit/system/integration tests and do a final review before we ship. There is more work remaining on documentation (e.g. HADOOP-9517) and I plan to personally focus on it this week - obviously help reviewing docs is very welcome.
> 
> Committers, from now, please please exercise your judgement on where you commit. Typically, features should go into branch-2 with 2.3.0 as the version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0 will be limited to content in branch-2.1-beta and we stick to stabilizing it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
> 
> thanks,
> Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

As Ramya noted, things are looking good on branch-2.1-beta ATM.

Henceforth, can I please ask committers to hold off non-blocker fixes for the final set of tests?

thanks,
Arun

On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:

> Folks,
> 
> The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!
> 
> I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.
> 
> For now the remaining work is to wrap up loose ends i.e. last minute api work (e.g. YARN-759 showed up last night for consideration), bug-fixes etc.; then run this through a battery of unit/system/integration tests and do a final review before we ship. There is more work remaining on documentation (e.g. HADOOP-9517) and I plan to personally focus on it this week - obviously help reviewing docs is very welcome.
> 
> Committers, from now, please please exercise your judgement on where you commit. Typically, features should go into branch-2 with 2.3.0 as the version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0 will be limited to content in branch-2.1-beta and we stick to stabilizing it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
> 
> thanks,
> Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

As Ramya noted, things are looking good on branch-2.1-beta ATM.

Henceforth, can I please ask committers to hold off non-blocker fixes for the final set of tests?

thanks,
Arun

On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:

> Folks,
> 
> The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!
> 
> I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.
> 
> For now the remaining work is to wrap up loose ends i.e. last minute api work (e.g. YARN-759 showed up last night for consideration), bug-fixes etc.; then run this through a battery of unit/system/integration tests and do a final review before we ship. There is more work remaining on documentation (e.g. HADOOP-9517) and I plan to personally focus on it this week - obviously help reviewing docs is very welcome.
> 
> Committers, from now, please please exercise your judgement on where you commit. Typically, features should go into branch-2 with 2.3.0 as the version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0 will be limited to content in branch-2.1-beta and we stick to stabilizing it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
> 
> thanks,
> Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Ramya Sunil <ra...@hortonworks.com>]

We have started testing branch-2.1-beta and for most parts the code looks
very stable. We have deployed both secure and non-secure multinode clusters.

We had some minor hiccups with some of our e2e tests breaking due to
additional setsid info being logged by the bin scripts and errors while
building with snappy turned on. Both are now fixed. Some of the e2e tests
had to be tweaked due to configuration changes in branch-2.1-beta such as
yarn.nodemanager.delete.debug-delay-sec has to be set to 0 to invoke the
cleanup of container tmp files, fs.trash.interval is now a server side
config etc.

We are now seeing failures while configuring short circuit read and job
summary logs(most likely configuration issues) and tests which use hftp://
and webhdfs:// are failing on secure clusters. Will file jiras after
initial investigation.

Thanks,
Ramya


On Tue, Jun 4, 2013 at 8:32 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
>
>  The vast majority of of the planned features and API work is complete,
> thanks to everyone who contributed!
>
>  I've created a branch-2.1-beta branch from which I anticipate I can make
> the first of our beta releases very shortly.
>
>  For now the remaining work is to wrap up loose ends i.e. last minute api
> work (e.g. YARN-759 showed up last night for consideration), bug-fixes
> etc.; then run this through a battery of unit/system/integration tests and
> do a final review before we ship. There is more work remaining on
> documentation (e.g. HADOOP-9517) and I plan to personally focus on it this
> week - obviously help reviewing docs is very welcome.
>
>  Committers, from now, please please exercise your judgement on where you
> commit. Typically, features should go into branch-2 with 2.3.0 as the
> version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0
> will be limited to content in branch-2.1-beta and we stick to stabilizing
> it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
>
> thanks,
> Arun
>

Re: Heads up: branch-2.1-beta

[Arpit Gupta <ar...@hortonworks.com>]

We are running into an issue where link to the AM leads to a 500

https://issues.apache.org/jira/browse/YARN-800

We should fix this before we release 2.1.0 beta.

--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/

On Jun 10, 2013, at 4:49 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:
> 
>> Folks,
>> 
>> The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!
>> 
>> I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.
> 
> 
> Thanks to Giri we now have a jenkins for daily branch-2.1-beta builds:
> 
> https://builds.apache.org/job/Hadoop-branch-2.1-beta/
> 
> Arun
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

Re: Heads up: branch-2.1-beta

[Arpit Gupta <ar...@hortonworks.com>]

We are running into an issue where link to the AM leads to a 500

https://issues.apache.org/jira/browse/YARN-800

We should fix this before we release 2.1.0 beta.

--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/

On Jun 10, 2013, at 4:49 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:
> 
>> Folks,
>> 
>> The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!
>> 
>> I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.
> 
> 
> Thanks to Giri we now have a jenkins for daily branch-2.1-beta builds:
> 
> https://builds.apache.org/job/Hadoop-branch-2.1-beta/
> 
> Arun
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

Re: Heads up: branch-2.1-beta

[Arpit Gupta <ar...@hortonworks.com>]

We are running into an issue where link to the AM leads to a 500

https://issues.apache.org/jira/browse/YARN-800

We should fix this before we release 2.1.0 beta.

--
Arpit Gupta
Hortonworks Inc.
http://hortonworks.com/

On Jun 10, 2013, at 4:49 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:
> 
>> Folks,
>> 
>> The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!
>> 
>> I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.
> 
> 
> Thanks to Giri we now have a jenkins for daily branch-2.1-beta builds:
> 
> https://builds.apache.org/job/Hadoop-branch-2.1-beta/
> 
> Arun
> 
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
> 
> 

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:

> Folks,
> 
> The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!
> 
> I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.


Thanks to Giri we now have a jenkins for daily branch-2.1-beta builds:

https://builds.apache.org/job/Hadoop-branch-2.1-beta/

Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Ramya Sunil <ra...@hortonworks.com>]

We have started testing branch-2.1-beta and for most parts the code looks
very stable. We have deployed both secure and non-secure multinode clusters.

We had some minor hiccups with some of our e2e tests breaking due to
additional setsid info being logged by the bin scripts and errors while
building with snappy turned on. Both are now fixed. Some of the e2e tests
had to be tweaked due to configuration changes in branch-2.1-beta such as
yarn.nodemanager.delete.debug-delay-sec has to be set to 0 to invoke the
cleanup of container tmp files, fs.trash.interval is now a server side
config etc.

We are now seeing failures while configuring short circuit read and job
summary logs(most likely configuration issues) and tests which use hftp://
and webhdfs:// are failing on secure clusters. Will file jiras after
initial investigation.

Thanks,
Ramya


On Tue, Jun 4, 2013 at 8:32 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
>
>  The vast majority of of the planned features and API work is complete,
> thanks to everyone who contributed!
>
>  I've created a branch-2.1-beta branch from which I anticipate I can make
> the first of our beta releases very shortly.
>
>  For now the remaining work is to wrap up loose ends i.e. last minute api
> work (e.g. YARN-759 showed up last night for consideration), bug-fixes
> etc.; then run this through a battery of unit/system/integration tests and
> do a final review before we ship. There is more work remaining on
> documentation (e.g. HADOOP-9517) and I plan to personally focus on it this
> week - obviously help reviewing docs is very welcome.
>
>  Committers, from now, please please exercise your judgement on where you
> commit. Typically, features should go into branch-2 with 2.3.0 as the
> version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0
> will be limited to content in branch-2.1-beta and we stick to stabilizing
> it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
>
> thanks,
> Arun
>

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:

> Folks,
> 
> The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!
> 
> I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.


Thanks to Giri we now have a jenkins for daily branch-2.1-beta builds:

https://builds.apache.org/job/Hadoop-branch-2.1-beta/

Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Steve Loughran <st...@hortonworks.com>]

Arun,

What do I do if I have I minor patch that I'd like to get out into 2.1, but
where I don't want to introduce instability into that 2.1 branch if you are
trying to make a beta release? For example,
HADOOP-9651<https://issues.apache.org/jira/browse/HADOOP-9651> changes
the exception that RawLocalFS throws when you try to create a file that
already exists, so that it is consistent with HDFS.

If I wanted to get this into a 2.1-beta-2, what should the strategy be? Get
it into trunk and then tag it as something to be considered for the
followup beta?

-steve

On 4 June 2013 16:32, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
>
>  The vast majority of of the planned features and API work is complete,
> thanks to everyone who contributed!
>
>  I've created a branch-2.1-beta branch from which I anticipate I can make
> the first of our beta releases very shortly.
>
>  For now the remaining work is to wrap up loose ends i.e. last minute api
> work (e.g. YARN-759 showed up last night for consideration), bug-fixes
> etc.; then run this through a battery of unit/system/integration tests and
> do a final review before we ship. There is more work remaining on
> documentation (e.g. HADOOP-9517) and I plan to personally focus on it this
> week - obviously help reviewing docs is very welcome.
>
>  Committers, from now, please please exercise your judgement on where you
> commit. Typically, features should go into branch-2 with 2.3.0 as the
> version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0
> will be limited to content in branch-2.1-beta and we stick to stabilizing
> it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
>
> thanks,
> Arun
>

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

As Ramya noted, things are looking good on branch-2.1-beta ATM.

Henceforth, can I please ask committers to hold off non-blocker fixes for the final set of tests?

thanks,
Arun

On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:

> Folks,
> 
> The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!
> 
> I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.
> 
> For now the remaining work is to wrap up loose ends i.e. last minute api work (e.g. YARN-759 showed up last night for consideration), bug-fixes etc.; then run this through a battery of unit/system/integration tests and do a final review before we ship. There is more work remaining on documentation (e.g. HADOOP-9517) and I plan to personally focus on it this week - obviously help reviewing docs is very welcome.
> 
> Committers, from now, please please exercise your judgement on where you commit. Typically, features should go into branch-2 with 2.3.0 as the version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0 will be limited to content in branch-2.1-beta and we stick to stabilizing it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
> 
> thanks,
> Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Ramya Sunil <ra...@hortonworks.com>]

We have started testing branch-2.1-beta and for most parts the code looks
very stable. We have deployed both secure and non-secure multinode clusters.

We had some minor hiccups with some of our e2e tests breaking due to
additional setsid info being logged by the bin scripts and errors while
building with snappy turned on. Both are now fixed. Some of the e2e tests
had to be tweaked due to configuration changes in branch-2.1-beta such as
yarn.nodemanager.delete.debug-delay-sec has to be set to 0 to invoke the
cleanup of container tmp files, fs.trash.interval is now a server side
config etc.

We are now seeing failures while configuring short circuit read and job
summary logs(most likely configuration issues) and tests which use hftp://
and webhdfs:// are failing on secure clusters. Will file jiras after
initial investigation.

Thanks,
Ramya


On Tue, Jun 4, 2013 at 8:32 AM, Arun C Murthy <ac...@hortonworks.com> wrote:

> Folks,
>
>  The vast majority of of the planned features and API work is complete,
> thanks to everyone who contributed!
>
>  I've created a branch-2.1-beta branch from which I anticipate I can make
> the first of our beta releases very shortly.
>
>  For now the remaining work is to wrap up loose ends i.e. last minute api
> work (e.g. YARN-759 showed up last night for consideration), bug-fixes
> etc.; then run this through a battery of unit/system/integration tests and
> do a final review before we ship. There is more work remaining on
> documentation (e.g. HADOOP-9517) and I plan to personally focus on it this
> week - obviously help reviewing docs is very welcome.
>
>  Committers, from now, please please exercise your judgement on where you
> commit. Typically, features should go into branch-2 with 2.3.0 as the
> version on jira (fix-version 2.3.0 is ready). The expectation is that 2.2.0
> will be limited to content in branch-2.1-beta and we stick to stabilizing
> it henceforth (I've deliberately not created 2.2.0 fix-version on jira yet).
>
> thanks,
> Arun
>

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:

> Folks,
> 
> The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!
> 
> I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.


Thanks to Giri we now have a jenkins for daily branch-2.1-beta builds:

https://builds.apache.org/job/Hadoop-branch-2.1-beta/

Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Heads up: branch-2.1-beta

[Arun C Murthy <ac...@hortonworks.com>]

On Jun 4, 2013, at 8:32 AM, Arun C Murthy wrote:

> Folks,
> 
> The vast majority of of the planned features and API work is complete, thanks to everyone who contributed!
> 
> I've created a branch-2.1-beta branch from which I anticipate I can make the first of our beta releases very shortly.


Thanks to Giri we now have a jenkins for daily branch-2.1-beta builds:

https://builds.apache.org/job/Hadoop-branch-2.1-beta/

Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/