You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Arun C Murthy <ac...@hortonworks.com> on 2014/02/01 01:29:42 UTC
Re: Re-swizzle 2.3
Thanks Vinod, appreciate it!
I think we are very close.
Here is a handy ref. to the list of blockers: http://s.apache.org/hadoop-2.3.0-blockers
I'd appreciate if folks can help expedite these fixes, and, equally importantly bring up others they feel should be blockers for 2.3.0.
thanks,
Arun
On Jan 30, 2014, at 12:42 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
> That was quite some exercise, but I'm done with it now. Updated YARN's and MAPREDUCE's CHANGES.txt on trunk, branch-2 and branch-2.3. Let me know if you find some inaccuracies.
>
> Thanks,
> +Vinod
>
> On Jan 29, 2014, at 10:49 PM, Vinod Kumar Vavilapalli <vi...@apache.org> wrote:
>
>>
>> Okay, I'll look at YARN and MR CHANGES.txt problems. Seems like they aren't addressed yet.
>>
>> +Vinod
>>
>>
>> On Jan 29, 2014, at 3:24 PM, Andrew Wang <an...@cloudera.com> wrote:
>>
>>> I just finished tuning up branch-2.3 and fixing up the HDFS and Common
>>> CHANGES.txt in trunk, branch-2, and branch-2.3. I had to merge back a few
>>> JIRAs committed between the swizzle and now where the fix version was 2.3
>>> but weren't in branch-2.3.
>>>
>>> I think the only two HDFS and Common JIRAs that are marked for 2.4 are
>>> these:
>>>
>>> HDFS-5842 Cannot create hftp filesystem when using a proxy user ugi and a
>>> doAs on a secure cluster
>>> HDFS-5781 Use an array to record the mapping between FSEditLogOpCode and
>>> the corresponding byte value
>>>
>>> Jing, these both look safe to me if you want to merge them back, or I can
>>> just do it.
>>>
>>> Thanks,
>>> Andrew
>>>
>>> On Wed, Jan 29, 2014 at 1:21 PM, Doug Cutting <cu...@apache.org> wrote:
>>>>
>>>> On Wed, Jan 29, 2014 at 12:30 PM, Jason Lowe <jl...@yahoo-inc.com> wrote:
>>>>> It is a bit concerning that the JIRA history showed that the target
>>> version
>>>>> was set at some point in the past but no record of it being cleared.
>>>>
>>>> Perhaps the version itself was renamed?
>>>>
>>>> Doug
>>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
> <signature.asc>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Konstantin Boudnik <co...@apache.org>.
Do you guys think that committing
https://issues.apache.org/jira/browse/HDFS-4858
to branch-2.3 is still Ok? It is a small change that bring fixes broken
timeout behavior of DN to NN RPC.
We have been testing this fix on top of 2.0.6 for a long time now and it seems
to be a real help.
Appreciate the feedback on 2.3 scope. If it is too late then I will commit it
to trunk, branch-2.4 and branch-2 only.
Regards,
Cos
On Thu, Feb 06, 2014 at 05:44PM, Alejandro Abdelnur wrote:
> yep, the idea is to pull all of them out from branch2.3. things go back to normal then.
>
> thanks
>
> Alejandro
> (phone typing)
>
> > On Feb 6, 2014, at 17:39, Zhijie Shen <zs...@hortonworks.com> wrote:
> >
> > Recently I brought 4 JIRAs to branch-2.3, which are MAPREDUCE-5743, YARN-1628,
> > YARN-1661 and YARN-1689. Recall that we mark test failure fixes as blockers
> > for pior releases as closing to release, thus I brought to branch-2.3
> > MAPREDUCE-5743
> > and YARN-1628 that are the fixes for the test failure on 2.3.0, but didn't
> > marked them as blockers. Please let me know if I should do that.
> >
> > YARN-1661 is a fix for exit log of DS AppMaster, otherwise the exit log of
> > it will always be failure, which sounds a critical issue to me. Feel free
> > to pull it out if any objects.
> >
> > YARN-1689 is brought to branch-2.3 as YARN-1493 is still in this branch. It
> > fixes one bug caused by YARN-1493. Those should be included or excluded
> > together upon the decision.
> >
> > Thanks,
> > Zhijie
> >
> >
> >> On Thu, Feb 6, 2014 at 5:15 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> >>
> >> +1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a
> >> problem caused by YARN-1493 I think we can revert it in branch-2.3 as well.
> >>
> >> I think we should leave them in branch-2 for now. We can revert if 2.4 is
> >> imminent and they're holding it up, but hopefully the issues they caused
> >> will be fixed by then.
> >>
> >> -Sandy
> >>
> >>
> >> On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur <tucu@cloudera.com
> >>> wrote:
> >>
> >>> Thanks Robert,
> >>>
> >>> All,
> >>>
> >>> So it seems that YARN-1493 and YARN-1490 are introducing serious
> >>> regressions.
> >>>
> >>> I would propose to revert them and the follow up JIRAs from the 2.3
> >> branch
> >>> and keep working on them on trunk/branch-2 until the are stable (I would
> >>> even prefer reverting them from branch-2 not to block a 2.4 if they are
> >> not
> >>> ready in time).
> >>>
> >>> As I've mentioned before, the list of JIRAs to revert were:
> >>>
> >>> YARN-1493
> >>> YARN-1490
> >>> YARN-1166
> >>> YARN-1041
> >>> YARN-1566
> >>>
> >>> Plus 2 additional JIRAs committed since my email on this issue 2 days
> >> ago:
> >>>
> >>> *YARN-1661
> >>> *YARN-1689 (not sure if this JIRA is related in functionality to the
> >>> previous ones but it is creating conflicts).
> >>>
> >>> I think we should hold on continuing work on top of something that is
> >>> broken until the broken stuff is fixed.
> >>>
> >>> Quoting Arun, "Committers - Henceforth, please use extreme caution while
> >>> committing to branch-2.3. Please commit *only* blockers to 2.3."
> >>>
> >>> YARN-1661 & YARN-1689 are not blockers.
> >>>
> >>> Unless there are objections, I'll revert all these JIRAs from branch-2.3
> >>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >>>
> >>> I'm inclined to revert them from branch-2 as well.
> >>>
> >>> Thoughts?
> >>>
> >>> Thanks.
> >>>
> >>>
> >>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> >>> wrote:
> >>>
> >>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> >>> was
> >>>> causing some strange behavior in the Oozie unit tests:
> >>>>
> >>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> >>> unit
> >>>> tests in a module. With YARN-1490 we saw that, regardless of test
> >> order,
> >>>> the last few tests would timeout waiting for an MR job to finish; on
> >>> slower
> >>>> machines, the entire test suite would timeout. Through some digging, I
> >>>> found that we were getting a ton of "Connection refused" Exceptions on
> >>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> >>>>
> >>>> After a bunch of investigation, I found that the problem went away once
> >>>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
> >>>> Even though this occurred in unit tests, it does make me concerned
> >> that
> >>> it
> >>>> could indicate some bigger issue in a long-running real cluster (where
> >>>> everything isn't running on the same machine) that we haven't seen yet.
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
> >> shortly.
> >>>> Will
> >>>>> pull it out of branch-2.3 if anyone objects.
> >>>>>
> >>>>>
> >>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> >>> aagarwal@hortonworks.com
> >>>>>> wrote:
> >>>>>
> >>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> >>>> aagarwal@hortonworks.com
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> >>>>>>>
> >>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> >>>> tucu@cloudera.com
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a
> >> very
> >>>> odd
> >>>>>>>> ways
> >>>>>>>> (to the point it seems un-deterministic).
> >>>>>>>>
> >>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
> >>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> >>>>> reverts)
> >>>>>>>> from Hadoop 2.3 branch before doing the release.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I've verified that after reverting those JIRAs things work fine
> >>> with
> >>>>>>>> unmanaged AMs.
> >>>>>>>>
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
> >>> acm@hortonworks.com
> >>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
> >>>>>>>>>
> >>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
> >>>>>> progress
> >>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
> >>>>>>>> objections?
> >>>>>>>>>
> >>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> >>>>>>>>>
> >>>>>>>>> Overall, I'll try get this out in next couple of days if we
> >> can
> >>>>> clear
> >>>>>>>> the
> >>>>>>>>> list.
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <
> >> acm@hortonworks.com
> >>>>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockerswe
> >>>> are
> >>>>>> now
> >>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >>>>>>>>>>
> >>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is
> >>> helping
> >>>>> out
> >>>>>>>> with
> >>>>>>>>> the YARN ones.
> >>>>>>>>>>
> >>>>>>>>>> thanks,
> >>>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Arun C. Murthy
> >>>>>>>>> Hortonworks Inc.
> >>>>>>>>> http://hortonworks.com/
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>> NOTICE: This message is intended for the use of the individual
> >>> or
> >>>>>>>> entity to
> >>>>>>>>> which it is addressed and may contain information that is
> >>>>>> confidential,
> >>>>>>>>> privileged and exempt from disclosure under applicable law. If
> >>> the
> >>>>>>>> reader
> >>>>>>>>> of this message is not the intended recipient, you are hereby
> >>>>> notified
> >>>>>>>> that
> >>>>>>>>> any printing, copying, dissemination, distribution, disclosure
> >>> or
> >>>>>>>>> forwarding of this communication is strictly prohibited. If
> >> you
> >>>> have
> >>>>>>>>> received this communication in error, please contact the
> >> sender
> >>>>>>>> immediately
> >>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Alejandro
> >>>>>>
> >>>>>> --
> >>>>>> CONFIDENTIALITY NOTICE
> >>>>>> NOTICE: This message is intended for the use of the individual or
> >>>> entity
> >>>>> to
> >>>>>> which it is addressed and may contain information that is
> >>> confidential,
> >>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>> reader
> >>>>>> of this message is not the intended recipient, you are hereby
> >>> notified
> >>>>> that
> >>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>> forwarding of this communication is strictly prohibited. If you
> >> have
> >>>>>> received this communication in error, please contact the sender
> >>>>> immediately
> >>>>>> and delete it from your system. Thank You.
> >>>
> >>>
> >>>
> >>> --
> >>> Alejandro
> >
> >
> >
> > --
> > Zhijie Shen
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender immediately
> > and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Konstantin Boudnik <co...@apache.org>.
Do you guys think that committing
https://issues.apache.org/jira/browse/HDFS-4858
to branch-2.3 is still Ok? It is a small change that bring fixes broken
timeout behavior of DN to NN RPC.
We have been testing this fix on top of 2.0.6 for a long time now and it seems
to be a real help.
Appreciate the feedback on 2.3 scope. If it is too late then I will commit it
to trunk, branch-2.4 and branch-2 only.
Regards,
Cos
On Thu, Feb 06, 2014 at 05:44PM, Alejandro Abdelnur wrote:
> yep, the idea is to pull all of them out from branch2.3. things go back to normal then.
>
> thanks
>
> Alejandro
> (phone typing)
>
> > On Feb 6, 2014, at 17:39, Zhijie Shen <zs...@hortonworks.com> wrote:
> >
> > Recently I brought 4 JIRAs to branch-2.3, which are MAPREDUCE-5743, YARN-1628,
> > YARN-1661 and YARN-1689. Recall that we mark test failure fixes as blockers
> > for pior releases as closing to release, thus I brought to branch-2.3
> > MAPREDUCE-5743
> > and YARN-1628 that are the fixes for the test failure on 2.3.0, but didn't
> > marked them as blockers. Please let me know if I should do that.
> >
> > YARN-1661 is a fix for exit log of DS AppMaster, otherwise the exit log of
> > it will always be failure, which sounds a critical issue to me. Feel free
> > to pull it out if any objects.
> >
> > YARN-1689 is brought to branch-2.3 as YARN-1493 is still in this branch. It
> > fixes one bug caused by YARN-1493. Those should be included or excluded
> > together upon the decision.
> >
> > Thanks,
> > Zhijie
> >
> >
> >> On Thu, Feb 6, 2014 at 5:15 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> >>
> >> +1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a
> >> problem caused by YARN-1493 I think we can revert it in branch-2.3 as well.
> >>
> >> I think we should leave them in branch-2 for now. We can revert if 2.4 is
> >> imminent and they're holding it up, but hopefully the issues they caused
> >> will be fixed by then.
> >>
> >> -Sandy
> >>
> >>
> >> On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur <tucu@cloudera.com
> >>> wrote:
> >>
> >>> Thanks Robert,
> >>>
> >>> All,
> >>>
> >>> So it seems that YARN-1493 and YARN-1490 are introducing serious
> >>> regressions.
> >>>
> >>> I would propose to revert them and the follow up JIRAs from the 2.3
> >> branch
> >>> and keep working on them on trunk/branch-2 until the are stable (I would
> >>> even prefer reverting them from branch-2 not to block a 2.4 if they are
> >> not
> >>> ready in time).
> >>>
> >>> As I've mentioned before, the list of JIRAs to revert were:
> >>>
> >>> YARN-1493
> >>> YARN-1490
> >>> YARN-1166
> >>> YARN-1041
> >>> YARN-1566
> >>>
> >>> Plus 2 additional JIRAs committed since my email on this issue 2 days
> >> ago:
> >>>
> >>> *YARN-1661
> >>> *YARN-1689 (not sure if this JIRA is related in functionality to the
> >>> previous ones but it is creating conflicts).
> >>>
> >>> I think we should hold on continuing work on top of something that is
> >>> broken until the broken stuff is fixed.
> >>>
> >>> Quoting Arun, "Committers - Henceforth, please use extreme caution while
> >>> committing to branch-2.3. Please commit *only* blockers to 2.3."
> >>>
> >>> YARN-1661 & YARN-1689 are not blockers.
> >>>
> >>> Unless there are objections, I'll revert all these JIRAs from branch-2.3
> >>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >>>
> >>> I'm inclined to revert them from branch-2 as well.
> >>>
> >>> Thoughts?
> >>>
> >>> Thanks.
> >>>
> >>>
> >>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> >>> wrote:
> >>>
> >>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> >>> was
> >>>> causing some strange behavior in the Oozie unit tests:
> >>>>
> >>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> >>> unit
> >>>> tests in a module. With YARN-1490 we saw that, regardless of test
> >> order,
> >>>> the last few tests would timeout waiting for an MR job to finish; on
> >>> slower
> >>>> machines, the entire test suite would timeout. Through some digging, I
> >>>> found that we were getting a ton of "Connection refused" Exceptions on
> >>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> >>>>
> >>>> After a bunch of investigation, I found that the problem went away once
> >>>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
> >>>> Even though this occurred in unit tests, it does make me concerned
> >> that
> >>> it
> >>>> could indicate some bigger issue in a long-running real cluster (where
> >>>> everything isn't running on the same machine) that we haven't seen yet.
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
> >> shortly.
> >>>> Will
> >>>>> pull it out of branch-2.3 if anyone objects.
> >>>>>
> >>>>>
> >>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> >>> aagarwal@hortonworks.com
> >>>>>> wrote:
> >>>>>
> >>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> >>>> aagarwal@hortonworks.com
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> >>>>>>>
> >>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> >>>> tucu@cloudera.com
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a
> >> very
> >>>> odd
> >>>>>>>> ways
> >>>>>>>> (to the point it seems un-deterministic).
> >>>>>>>>
> >>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
> >>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> >>>>> reverts)
> >>>>>>>> from Hadoop 2.3 branch before doing the release.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I've verified that after reverting those JIRAs things work fine
> >>> with
> >>>>>>>> unmanaged AMs.
> >>>>>>>>
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
> >>> acm@hortonworks.com
> >>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
> >>>>>>>>>
> >>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
> >>>>>> progress
> >>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
> >>>>>>>> objections?
> >>>>>>>>>
> >>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> >>>>>>>>>
> >>>>>>>>> Overall, I'll try get this out in next couple of days if we
> >> can
> >>>>> clear
> >>>>>>>> the
> >>>>>>>>> list.
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <
> >> acm@hortonworks.com
> >>>>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockerswe
> >>>> are
> >>>>>> now
> >>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >>>>>>>>>>
> >>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is
> >>> helping
> >>>>> out
> >>>>>>>> with
> >>>>>>>>> the YARN ones.
> >>>>>>>>>>
> >>>>>>>>>> thanks,
> >>>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Arun C. Murthy
> >>>>>>>>> Hortonworks Inc.
> >>>>>>>>> http://hortonworks.com/
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>> NOTICE: This message is intended for the use of the individual
> >>> or
> >>>>>>>> entity to
> >>>>>>>>> which it is addressed and may contain information that is
> >>>>>> confidential,
> >>>>>>>>> privileged and exempt from disclosure under applicable law. If
> >>> the
> >>>>>>>> reader
> >>>>>>>>> of this message is not the intended recipient, you are hereby
> >>>>> notified
> >>>>>>>> that
> >>>>>>>>> any printing, copying, dissemination, distribution, disclosure
> >>> or
> >>>>>>>>> forwarding of this communication is strictly prohibited. If
> >> you
> >>>> have
> >>>>>>>>> received this communication in error, please contact the
> >> sender
> >>>>>>>> immediately
> >>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Alejandro
> >>>>>>
> >>>>>> --
> >>>>>> CONFIDENTIALITY NOTICE
> >>>>>> NOTICE: This message is intended for the use of the individual or
> >>>> entity
> >>>>> to
> >>>>>> which it is addressed and may contain information that is
> >>> confidential,
> >>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>> reader
> >>>>>> of this message is not the intended recipient, you are hereby
> >>> notified
> >>>>> that
> >>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>> forwarding of this communication is strictly prohibited. If you
> >> have
> >>>>>> received this communication in error, please contact the sender
> >>>>> immediately
> >>>>>> and delete it from your system. Thank You.
> >>>
> >>>
> >>>
> >>> --
> >>> Alejandro
> >
> >
> >
> > --
> > Zhijie Shen
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender immediately
> > and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
sire, as sandy said, lets keep it in branch 2 for now and if not resolved by 2.4 timeframe we'll revert them there.
thx
Alejandro
(phone typing)
> On Feb 7, 2014, at 10:14, Steve Loughran <st...@hortonworks.com> wrote:
>
>> On 6 February 2014 17:07, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>
>> Thanks Robert,
>>
>> All,
>>
>>
>>
>> I'm inclined to revert them from branch-2 as well.
> -1 to that; if there are issues we should be able to find and fix them soon
> enough. Even if you aren't doing long-lived YARN services yet, even llama
> benefits from this zero-container-loss on AM restart.
>
> We do have Hoya using this (introspection code because the protobuf
> structures are hidden away), means that you can kill the AM and HBase &
> Accumulo clusters stay up in their YARN containers, the restarted AM gets
> that list of containers (and any pending events), rebuilds its data
> structures and carries on as before. Sweet!
>
> https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hoya/yarn/appmaster/HoyaAppMaster.java#L551
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
sire, as sandy said, lets keep it in branch 2 for now and if not resolved by 2.4 timeframe we'll revert them there.
thx
Alejandro
(phone typing)
> On Feb 7, 2014, at 10:14, Steve Loughran <st...@hortonworks.com> wrote:
>
>> On 6 February 2014 17:07, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>
>> Thanks Robert,
>>
>> All,
>>
>>
>>
>> I'm inclined to revert them from branch-2 as well.
> -1 to that; if there are issues we should be able to find and fix them soon
> enough. Even if you aren't doing long-lived YARN services yet, even llama
> benefits from this zero-container-loss on AM restart.
>
> We do have Hoya using this (introspection code because the protobuf
> structures are hidden away), means that you can kill the AM and HBase &
> Accumulo clusters stay up in their YARN containers, the restarted AM gets
> that list of containers (and any pending events), rebuilds its data
> structures and carries on as before. Sweet!
>
> https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hoya/yarn/appmaster/HoyaAppMaster.java#L551
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
sire, as sandy said, lets keep it in branch 2 for now and if not resolved by 2.4 timeframe we'll revert them there.
thx
Alejandro
(phone typing)
> On Feb 7, 2014, at 10:14, Steve Loughran <st...@hortonworks.com> wrote:
>
>> On 6 February 2014 17:07, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>
>> Thanks Robert,
>>
>> All,
>>
>>
>>
>> I'm inclined to revert them from branch-2 as well.
> -1 to that; if there are issues we should be able to find and fix them soon
> enough. Even if you aren't doing long-lived YARN services yet, even llama
> benefits from this zero-container-loss on AM restart.
>
> We do have Hoya using this (introspection code because the protobuf
> structures are hidden away), means that you can kill the AM and HBase &
> Accumulo clusters stay up in their YARN containers, the restarted AM gets
> that list of containers (and any pending events), rebuilds its data
> structures and carries on as before. Sweet!
>
> https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hoya/yarn/appmaster/HoyaAppMaster.java#L551
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
sire, as sandy said, lets keep it in branch 2 for now and if not resolved by 2.4 timeframe we'll revert them there.
thx
Alejandro
(phone typing)
> On Feb 7, 2014, at 10:14, Steve Loughran <st...@hortonworks.com> wrote:
>
>> On 6 February 2014 17:07, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>
>> Thanks Robert,
>>
>> All,
>>
>>
>>
>> I'm inclined to revert them from branch-2 as well.
> -1 to that; if there are issues we should be able to find and fix them soon
> enough. Even if you aren't doing long-lived YARN services yet, even llama
> benefits from this zero-container-loss on AM restart.
>
> We do have Hoya using this (introspection code because the protobuf
> structures are hidden away), means that you can kill the AM and HBase &
> Accumulo clusters stay up in their YARN containers, the restarted AM gets
> that list of containers (and any pending events), rebuilds its data
> structures and carries on as before. Sweet!
>
> https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hoya/yarn/appmaster/HoyaAppMaster.java#L551
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Steve Loughran <st...@hortonworks.com>.
On 6 February 2014 17:07, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Thanks Robert,
>
> All,
>
>
>
> I'm inclined to revert them from branch-2 as well.
>
>
-1 to that; if there are issues we should be able to find and fix them soon
enough. Even if you aren't doing long-lived YARN services yet, even llama
benefits from this zero-container-loss on AM restart.
We do have Hoya using this (introspection code because the protobuf
structures are hidden away), means that you can kill the AM and HBase &
Accumulo clusters stay up in their YARN containers, the restarted AM gets
that list of containers (and any pending events), rebuilds its data
structures and carries on as before. Sweet!
https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hoya/yarn/appmaster/HoyaAppMaster.java#L551
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Vinod, I have the patches to revert most of the JIRAs, the first batch,
I'll send them off line to you.
Thanks.
On Thu, Feb 6, 2014 at 8:56 PM, Vinod Kumar Vavilapalli
<vi...@apache.org>wrote:
>
> Thanks. please post your findings, Jian wrote this part of the code and
> between him/me, we can take care of those issues.
>
> +1 for going ahead with the revert on branch-2.3. I'll go do that tomorrow
> morning unless I hear otherwise from Jian.
>
> Thanks,
> +Vinod
>
>
> On Feb 6, 2014, at 8:28 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > Hi Vinod,
> >
> > Nothing confidential,
> >
> > * With umanaged AMs I'm seeing the trace I've posted a couple of days ago
> > in YARN-1577 (
> >
> https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853
> > ).
> >
> > * Also, Robert has been digging in Oozie testcases failing/getting suck
> > with several token renewer threads, this failures happened consistently
> at
> > different places around the same testcases (like some file descriptors
> > leaking out), reverting YARN-1490 fixes the problem. The potential issue
> > with this is that a long running client (oozie) my run into this
> situation
> > thus becoming unstable.
> >
> > *Robert,* mind posting to YARN-1490 the jvm thread dump at the time of
> test
> > hanging?
> >
> > After YARN-1493 & YARN-1490 we have a couple of JIRAs trying to fix
> issues
> > introduced by them, and we still didn't get them right.
> >
> > Because this, the improvements driven by YARN-1493 & YARN-1490 seem that
> > require more work before being stable.
> >
> > IMO, being conservative, we should do 2.3 without them and roll them with
> > 2.4. If we want to do regular releases we will have to make this kind of
> > calls, else we will start dragging the releases.
> >
> > Sounds like a plan?
> >
> > Thanks.
> >
> >
> >
> > On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli
> > <vi...@apache.org>wrote:
> >
> >> Hey
> >>
> >> I am not against removing them from 2.3 if that is helpful for progress.
> >> But I want to understand what the issues are before we make that
> decision.
> >>
> >> There is the issue with unmanaged AM that is clearly known and I was
> >> thinking of coming to the past two days, but couldn't. What is this new
> >> issue that we (confidently?) pinned down to YARN-1490?
> >>
> >> Thanks
> >> +Vinod
> >>
> >> On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
> >>
> >>> Thanks Robert,
> >>>
> >>> All,
> >>>
> >>> So it seems that YARN-1493 and YARN-1490 are introducing serious
> >>> regressions.
> >>>
> >>> I would propose to revert them and the follow up JIRAs from the 2.3
> >> branch
> >>> and keep working on them on trunk/branch-2 until the are stable (I
> would
> >>> even prefer reverting them from branch-2 not to block a 2.4 if they are
> >> not
> >>> ready in time).
> >>>
> >>> As I've mentioned before, the list of JIRAs to revert were:
> >>>
> >>> YARN-1493
> >>> YARN-1490
> >>> YARN-1166
> >>> YARN-1041
> >>> YARN-1566
> >>>
> >>> Plus 2 additional JIRAs committed since my email on this issue 2 days
> >> ago:
> >>>
> >>> *YARN-1661
> >>> *YARN-1689 (not sure if this JIRA is related in functionality to the
> >>> previous ones but it is creating conflicts).
> >>>
> >>> I think we should hold on continuing work on top of something that is
> >>> broken until the broken stuff is fixed.
> >>>
> >>> Quoting Arun, "Committers - Henceforth, please use extreme caution
> while
> >>> committing to branch-2.3. Please commit *only* blockers to 2.3."
> >>>
> >>> YARN-1661 & YARN-1689 are not blockers.
> >>>
> >>> Unless there are objections, I'll revert all these JIRAs from
> branch-2.3
> >>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >>>
> >>> I'm inclined to revert them from branch-2 as well.
> >>>
> >>> Thoughts?
> >>>
> >>> Thanks.
> >>>
> >>>
> >>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> >> wrote:
> >>>
> >>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> >> was
> >>>> causing some strange behavior in the Oozie unit tests:
> >>>>
> >>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> >> unit
> >>>> tests in a module. With YARN-1490 we saw that, regardless of test
> >> order,
> >>>> the last few tests would timeout waiting for an MR job to finish; on
> >> slower
> >>>> machines, the entire test suite would timeout. Through some digging,
> I
> >>>> found that we were getting a ton of "Connection refused" Exceptions on
> >>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> >>>>
> >>>> After a bunch of investigation, I found that the problem went away
> once
> >>>> YARN-1490 was removed. Though I couldn't figure out the exact
> problem.
> >>>> Even though this occurred in unit tests, it does make me concerned
> that
> >> it
> >>>> could indicate some bigger issue in a long-running real cluster (where
> >>>> everything isn't running on the same machine) that we haven't seen
> yet.
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
> shortly.
> >>>> Will
> >>>>> pull it out of branch-2.3 if anyone objects.
> >>>>>
> >>>>>
> >>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> >> aagarwal@hortonworks.com
> >>>>>> wrote:
> >>>>>
> >>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> >>>> aagarwal@hortonworks.com
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> >>>>>>>
> >>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> >>>> tucu@cloudera.com
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> >>>> odd
> >>>>>>>> ways
> >>>>>>>> (to the point it seems un-deterministic).
> >>>>>>>>
> >>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
> >>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> >>>>> reverts)
> >>>>>>>> from Hadoop 2.3 branch before doing the release.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I've verified that after reverting those JIRAs things work fine
> with
> >>>>>>>> unmanaged AMs.
> >>>>>>>>
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
> acm@hortonworks.com
> >>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
> >>>>>>>>>
> >>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
> >>>>>> progress
> >>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
> >>>>>>>> objections?
> >>>>>>>>>
> >>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> >>>>>>>>>
> >>>>>>>>> Overall, I'll try get this out in next couple of days if we can
> >>>>> clear
> >>>>>>>> the
> >>>>>>>>> list.
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> >>>> are
> >>>>>> now
> >>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >>>>>>>>>>
> >>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> >>>>> out
> >>>>>>>> with
> >>>>>>>>> the YARN ones.
> >>>>>>>>>>
> >>>>>>>>>> thanks,
> >>>>>>>>>> Arun
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Arun C. Murthy
> >>>>>>>>> Hortonworks Inc.
> >>>>>>>>> http://hortonworks.com/
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>>>> entity to
> >>>>>>>>> which it is addressed and may contain information that is
> >>>>>> confidential,
> >>>>>>>>> privileged and exempt from disclosure under applicable law. If
> the
> >>>>>>>> reader
> >>>>>>>>> of this message is not the intended recipient, you are hereby
> >>>>> notified
> >>>>>>>> that
> >>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>>>> forwarding of this communication is strictly prohibited. If you
> >>>> have
> >>>>>>>>> received this communication in error, please contact the sender
> >>>>>>>> immediately
> >>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Alejandro
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> CONFIDENTIALITY NOTICE
> >>>>>> NOTICE: This message is intended for the use of the individual or
> >>>> entity
> >>>>> to
> >>>>>> which it is addressed and may contain information that is
> >> confidential,
> >>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>> reader
> >>>>>> of this message is not the intended recipient, you are hereby
> notified
> >>>>> that
> >>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>> forwarding of this communication is strictly prohibited. If you have
> >>>>>> received this communication in error, please contact the sender
> >>>>> immediately
> >>>>>> and delete it from your system. Thank You.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Alejandro
> >>
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or
> entity to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> immediately
> >> and delete it from your system. Thank You.
> >>
> >
> >
> >
> > --
> > Alejandro
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Vinod, I have the patches to revert most of the JIRAs, the first batch,
I'll send them off line to you.
Thanks.
On Thu, Feb 6, 2014 at 8:56 PM, Vinod Kumar Vavilapalli
<vi...@apache.org>wrote:
>
> Thanks. please post your findings, Jian wrote this part of the code and
> between him/me, we can take care of those issues.
>
> +1 for going ahead with the revert on branch-2.3. I'll go do that tomorrow
> morning unless I hear otherwise from Jian.
>
> Thanks,
> +Vinod
>
>
> On Feb 6, 2014, at 8:28 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > Hi Vinod,
> >
> > Nothing confidential,
> >
> > * With umanaged AMs I'm seeing the trace I've posted a couple of days ago
> > in YARN-1577 (
> >
> https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853
> > ).
> >
> > * Also, Robert has been digging in Oozie testcases failing/getting suck
> > with several token renewer threads, this failures happened consistently
> at
> > different places around the same testcases (like some file descriptors
> > leaking out), reverting YARN-1490 fixes the problem. The potential issue
> > with this is that a long running client (oozie) my run into this
> situation
> > thus becoming unstable.
> >
> > *Robert,* mind posting to YARN-1490 the jvm thread dump at the time of
> test
> > hanging?
> >
> > After YARN-1493 & YARN-1490 we have a couple of JIRAs trying to fix
> issues
> > introduced by them, and we still didn't get them right.
> >
> > Because this, the improvements driven by YARN-1493 & YARN-1490 seem that
> > require more work before being stable.
> >
> > IMO, being conservative, we should do 2.3 without them and roll them with
> > 2.4. If we want to do regular releases we will have to make this kind of
> > calls, else we will start dragging the releases.
> >
> > Sounds like a plan?
> >
> > Thanks.
> >
> >
> >
> > On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli
> > <vi...@apache.org>wrote:
> >
> >> Hey
> >>
> >> I am not against removing them from 2.3 if that is helpful for progress.
> >> But I want to understand what the issues are before we make that
> decision.
> >>
> >> There is the issue with unmanaged AM that is clearly known and I was
> >> thinking of coming to the past two days, but couldn't. What is this new
> >> issue that we (confidently?) pinned down to YARN-1490?
> >>
> >> Thanks
> >> +Vinod
> >>
> >> On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
> >>
> >>> Thanks Robert,
> >>>
> >>> All,
> >>>
> >>> So it seems that YARN-1493 and YARN-1490 are introducing serious
> >>> regressions.
> >>>
> >>> I would propose to revert them and the follow up JIRAs from the 2.3
> >> branch
> >>> and keep working on them on trunk/branch-2 until the are stable (I
> would
> >>> even prefer reverting them from branch-2 not to block a 2.4 if they are
> >> not
> >>> ready in time).
> >>>
> >>> As I've mentioned before, the list of JIRAs to revert were:
> >>>
> >>> YARN-1493
> >>> YARN-1490
> >>> YARN-1166
> >>> YARN-1041
> >>> YARN-1566
> >>>
> >>> Plus 2 additional JIRAs committed since my email on this issue 2 days
> >> ago:
> >>>
> >>> *YARN-1661
> >>> *YARN-1689 (not sure if this JIRA is related in functionality to the
> >>> previous ones but it is creating conflicts).
> >>>
> >>> I think we should hold on continuing work on top of something that is
> >>> broken until the broken stuff is fixed.
> >>>
> >>> Quoting Arun, "Committers - Henceforth, please use extreme caution
> while
> >>> committing to branch-2.3. Please commit *only* blockers to 2.3."
> >>>
> >>> YARN-1661 & YARN-1689 are not blockers.
> >>>
> >>> Unless there are objections, I'll revert all these JIRAs from
> branch-2.3
> >>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >>>
> >>> I'm inclined to revert them from branch-2 as well.
> >>>
> >>> Thoughts?
> >>>
> >>> Thanks.
> >>>
> >>>
> >>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> >> wrote:
> >>>
> >>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> >> was
> >>>> causing some strange behavior in the Oozie unit tests:
> >>>>
> >>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> >> unit
> >>>> tests in a module. With YARN-1490 we saw that, regardless of test
> >> order,
> >>>> the last few tests would timeout waiting for an MR job to finish; on
> >> slower
> >>>> machines, the entire test suite would timeout. Through some digging,
> I
> >>>> found that we were getting a ton of "Connection refused" Exceptions on
> >>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> >>>>
> >>>> After a bunch of investigation, I found that the problem went away
> once
> >>>> YARN-1490 was removed. Though I couldn't figure out the exact
> problem.
> >>>> Even though this occurred in unit tests, it does make me concerned
> that
> >> it
> >>>> could indicate some bigger issue in a long-running real cluster (where
> >>>> everything isn't running on the same machine) that we haven't seen
> yet.
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
> shortly.
> >>>> Will
> >>>>> pull it out of branch-2.3 if anyone objects.
> >>>>>
> >>>>>
> >>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> >> aagarwal@hortonworks.com
> >>>>>> wrote:
> >>>>>
> >>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> >>>> aagarwal@hortonworks.com
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> >>>>>>>
> >>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> >>>> tucu@cloudera.com
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> >>>> odd
> >>>>>>>> ways
> >>>>>>>> (to the point it seems un-deterministic).
> >>>>>>>>
> >>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
> >>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> >>>>> reverts)
> >>>>>>>> from Hadoop 2.3 branch before doing the release.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I've verified that after reverting those JIRAs things work fine
> with
> >>>>>>>> unmanaged AMs.
> >>>>>>>>
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
> acm@hortonworks.com
> >>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
> >>>>>>>>>
> >>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
> >>>>>> progress
> >>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
> >>>>>>>> objections?
> >>>>>>>>>
> >>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> >>>>>>>>>
> >>>>>>>>> Overall, I'll try get this out in next couple of days if we can
> >>>>> clear
> >>>>>>>> the
> >>>>>>>>> list.
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> >>>> are
> >>>>>> now
> >>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >>>>>>>>>>
> >>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> >>>>> out
> >>>>>>>> with
> >>>>>>>>> the YARN ones.
> >>>>>>>>>>
> >>>>>>>>>> thanks,
> >>>>>>>>>> Arun
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Arun C. Murthy
> >>>>>>>>> Hortonworks Inc.
> >>>>>>>>> http://hortonworks.com/
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>>>> entity to
> >>>>>>>>> which it is addressed and may contain information that is
> >>>>>> confidential,
> >>>>>>>>> privileged and exempt from disclosure under applicable law. If
> the
> >>>>>>>> reader
> >>>>>>>>> of this message is not the intended recipient, you are hereby
> >>>>> notified
> >>>>>>>> that
> >>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>>>> forwarding of this communication is strictly prohibited. If you
> >>>> have
> >>>>>>>>> received this communication in error, please contact the sender
> >>>>>>>> immediately
> >>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Alejandro
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> CONFIDENTIALITY NOTICE
> >>>>>> NOTICE: This message is intended for the use of the individual or
> >>>> entity
> >>>>> to
> >>>>>> which it is addressed and may contain information that is
> >> confidential,
> >>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>> reader
> >>>>>> of this message is not the intended recipient, you are hereby
> notified
> >>>>> that
> >>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>> forwarding of this communication is strictly prohibited. If you have
> >>>>>> received this communication in error, please contact the sender
> >>>>> immediately
> >>>>>> and delete it from your system. Thank You.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Alejandro
> >>
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or
> entity to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> immediately
> >> and delete it from your system. Thank You.
> >>
> >
> >
> >
> > --
> > Alejandro
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Vinod, I have the patches to revert most of the JIRAs, the first batch,
I'll send them off line to you.
Thanks.
On Thu, Feb 6, 2014 at 8:56 PM, Vinod Kumar Vavilapalli
<vi...@apache.org>wrote:
>
> Thanks. please post your findings, Jian wrote this part of the code and
> between him/me, we can take care of those issues.
>
> +1 for going ahead with the revert on branch-2.3. I'll go do that tomorrow
> morning unless I hear otherwise from Jian.
>
> Thanks,
> +Vinod
>
>
> On Feb 6, 2014, at 8:28 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > Hi Vinod,
> >
> > Nothing confidential,
> >
> > * With umanaged AMs I'm seeing the trace I've posted a couple of days ago
> > in YARN-1577 (
> >
> https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853
> > ).
> >
> > * Also, Robert has been digging in Oozie testcases failing/getting suck
> > with several token renewer threads, this failures happened consistently
> at
> > different places around the same testcases (like some file descriptors
> > leaking out), reverting YARN-1490 fixes the problem. The potential issue
> > with this is that a long running client (oozie) my run into this
> situation
> > thus becoming unstable.
> >
> > *Robert,* mind posting to YARN-1490 the jvm thread dump at the time of
> test
> > hanging?
> >
> > After YARN-1493 & YARN-1490 we have a couple of JIRAs trying to fix
> issues
> > introduced by them, and we still didn't get them right.
> >
> > Because this, the improvements driven by YARN-1493 & YARN-1490 seem that
> > require more work before being stable.
> >
> > IMO, being conservative, we should do 2.3 without them and roll them with
> > 2.4. If we want to do regular releases we will have to make this kind of
> > calls, else we will start dragging the releases.
> >
> > Sounds like a plan?
> >
> > Thanks.
> >
> >
> >
> > On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli
> > <vi...@apache.org>wrote:
> >
> >> Hey
> >>
> >> I am not against removing them from 2.3 if that is helpful for progress.
> >> But I want to understand what the issues are before we make that
> decision.
> >>
> >> There is the issue with unmanaged AM that is clearly known and I was
> >> thinking of coming to the past two days, but couldn't. What is this new
> >> issue that we (confidently?) pinned down to YARN-1490?
> >>
> >> Thanks
> >> +Vinod
> >>
> >> On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
> >>
> >>> Thanks Robert,
> >>>
> >>> All,
> >>>
> >>> So it seems that YARN-1493 and YARN-1490 are introducing serious
> >>> regressions.
> >>>
> >>> I would propose to revert them and the follow up JIRAs from the 2.3
> >> branch
> >>> and keep working on them on trunk/branch-2 until the are stable (I
> would
> >>> even prefer reverting them from branch-2 not to block a 2.4 if they are
> >> not
> >>> ready in time).
> >>>
> >>> As I've mentioned before, the list of JIRAs to revert were:
> >>>
> >>> YARN-1493
> >>> YARN-1490
> >>> YARN-1166
> >>> YARN-1041
> >>> YARN-1566
> >>>
> >>> Plus 2 additional JIRAs committed since my email on this issue 2 days
> >> ago:
> >>>
> >>> *YARN-1661
> >>> *YARN-1689 (not sure if this JIRA is related in functionality to the
> >>> previous ones but it is creating conflicts).
> >>>
> >>> I think we should hold on continuing work on top of something that is
> >>> broken until the broken stuff is fixed.
> >>>
> >>> Quoting Arun, "Committers - Henceforth, please use extreme caution
> while
> >>> committing to branch-2.3. Please commit *only* blockers to 2.3."
> >>>
> >>> YARN-1661 & YARN-1689 are not blockers.
> >>>
> >>> Unless there are objections, I'll revert all these JIRAs from
> branch-2.3
> >>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >>>
> >>> I'm inclined to revert them from branch-2 as well.
> >>>
> >>> Thoughts?
> >>>
> >>> Thanks.
> >>>
> >>>
> >>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> >> wrote:
> >>>
> >>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> >> was
> >>>> causing some strange behavior in the Oozie unit tests:
> >>>>
> >>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> >> unit
> >>>> tests in a module. With YARN-1490 we saw that, regardless of test
> >> order,
> >>>> the last few tests would timeout waiting for an MR job to finish; on
> >> slower
> >>>> machines, the entire test suite would timeout. Through some digging,
> I
> >>>> found that we were getting a ton of "Connection refused" Exceptions on
> >>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> >>>>
> >>>> After a bunch of investigation, I found that the problem went away
> once
> >>>> YARN-1490 was removed. Though I couldn't figure out the exact
> problem.
> >>>> Even though this occurred in unit tests, it does make me concerned
> that
> >> it
> >>>> could indicate some bigger issue in a long-running real cluster (where
> >>>> everything isn't running on the same machine) that we haven't seen
> yet.
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
> shortly.
> >>>> Will
> >>>>> pull it out of branch-2.3 if anyone objects.
> >>>>>
> >>>>>
> >>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> >> aagarwal@hortonworks.com
> >>>>>> wrote:
> >>>>>
> >>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> >>>> aagarwal@hortonworks.com
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> >>>>>>>
> >>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> >>>> tucu@cloudera.com
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> >>>> odd
> >>>>>>>> ways
> >>>>>>>> (to the point it seems un-deterministic).
> >>>>>>>>
> >>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
> >>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> >>>>> reverts)
> >>>>>>>> from Hadoop 2.3 branch before doing the release.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I've verified that after reverting those JIRAs things work fine
> with
> >>>>>>>> unmanaged AMs.
> >>>>>>>>
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
> acm@hortonworks.com
> >>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
> >>>>>>>>>
> >>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
> >>>>>> progress
> >>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
> >>>>>>>> objections?
> >>>>>>>>>
> >>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> >>>>>>>>>
> >>>>>>>>> Overall, I'll try get this out in next couple of days if we can
> >>>>> clear
> >>>>>>>> the
> >>>>>>>>> list.
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> >>>> are
> >>>>>> now
> >>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >>>>>>>>>>
> >>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> >>>>> out
> >>>>>>>> with
> >>>>>>>>> the YARN ones.
> >>>>>>>>>>
> >>>>>>>>>> thanks,
> >>>>>>>>>> Arun
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Arun C. Murthy
> >>>>>>>>> Hortonworks Inc.
> >>>>>>>>> http://hortonworks.com/
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>>>> entity to
> >>>>>>>>> which it is addressed and may contain information that is
> >>>>>> confidential,
> >>>>>>>>> privileged and exempt from disclosure under applicable law. If
> the
> >>>>>>>> reader
> >>>>>>>>> of this message is not the intended recipient, you are hereby
> >>>>> notified
> >>>>>>>> that
> >>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>>>> forwarding of this communication is strictly prohibited. If you
> >>>> have
> >>>>>>>>> received this communication in error, please contact the sender
> >>>>>>>> immediately
> >>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Alejandro
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> CONFIDENTIALITY NOTICE
> >>>>>> NOTICE: This message is intended for the use of the individual or
> >>>> entity
> >>>>> to
> >>>>>> which it is addressed and may contain information that is
> >> confidential,
> >>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>> reader
> >>>>>> of this message is not the intended recipient, you are hereby
> notified
> >>>>> that
> >>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>> forwarding of this communication is strictly prohibited. If you have
> >>>>>> received this communication in error, please contact the sender
> >>>>> immediately
> >>>>>> and delete it from your system. Thank You.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Alejandro
> >>
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or
> entity to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> immediately
> >> and delete it from your system. Thank You.
> >>
> >
> >
> >
> > --
> > Alejandro
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Vinod, I have the patches to revert most of the JIRAs, the first batch,
I'll send them off line to you.
Thanks.
On Thu, Feb 6, 2014 at 8:56 PM, Vinod Kumar Vavilapalli
<vi...@apache.org>wrote:
>
> Thanks. please post your findings, Jian wrote this part of the code and
> between him/me, we can take care of those issues.
>
> +1 for going ahead with the revert on branch-2.3. I'll go do that tomorrow
> morning unless I hear otherwise from Jian.
>
> Thanks,
> +Vinod
>
>
> On Feb 6, 2014, at 8:28 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > Hi Vinod,
> >
> > Nothing confidential,
> >
> > * With umanaged AMs I'm seeing the trace I've posted a couple of days ago
> > in YARN-1577 (
> >
> https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853
> > ).
> >
> > * Also, Robert has been digging in Oozie testcases failing/getting suck
> > with several token renewer threads, this failures happened consistently
> at
> > different places around the same testcases (like some file descriptors
> > leaking out), reverting YARN-1490 fixes the problem. The potential issue
> > with this is that a long running client (oozie) my run into this
> situation
> > thus becoming unstable.
> >
> > *Robert,* mind posting to YARN-1490 the jvm thread dump at the time of
> test
> > hanging?
> >
> > After YARN-1493 & YARN-1490 we have a couple of JIRAs trying to fix
> issues
> > introduced by them, and we still didn't get them right.
> >
> > Because this, the improvements driven by YARN-1493 & YARN-1490 seem that
> > require more work before being stable.
> >
> > IMO, being conservative, we should do 2.3 without them and roll them with
> > 2.4. If we want to do regular releases we will have to make this kind of
> > calls, else we will start dragging the releases.
> >
> > Sounds like a plan?
> >
> > Thanks.
> >
> >
> >
> > On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli
> > <vi...@apache.org>wrote:
> >
> >> Hey
> >>
> >> I am not against removing them from 2.3 if that is helpful for progress.
> >> But I want to understand what the issues are before we make that
> decision.
> >>
> >> There is the issue with unmanaged AM that is clearly known and I was
> >> thinking of coming to the past two days, but couldn't. What is this new
> >> issue that we (confidently?) pinned down to YARN-1490?
> >>
> >> Thanks
> >> +Vinod
> >>
> >> On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com>
> wrote:
> >>
> >>> Thanks Robert,
> >>>
> >>> All,
> >>>
> >>> So it seems that YARN-1493 and YARN-1490 are introducing serious
> >>> regressions.
> >>>
> >>> I would propose to revert them and the follow up JIRAs from the 2.3
> >> branch
> >>> and keep working on them on trunk/branch-2 until the are stable (I
> would
> >>> even prefer reverting them from branch-2 not to block a 2.4 if they are
> >> not
> >>> ready in time).
> >>>
> >>> As I've mentioned before, the list of JIRAs to revert were:
> >>>
> >>> YARN-1493
> >>> YARN-1490
> >>> YARN-1166
> >>> YARN-1041
> >>> YARN-1566
> >>>
> >>> Plus 2 additional JIRAs committed since my email on this issue 2 days
> >> ago:
> >>>
> >>> *YARN-1661
> >>> *YARN-1689 (not sure if this JIRA is related in functionality to the
> >>> previous ones but it is creating conflicts).
> >>>
> >>> I think we should hold on continuing work on top of something that is
> >>> broken until the broken stuff is fixed.
> >>>
> >>> Quoting Arun, "Committers - Henceforth, please use extreme caution
> while
> >>> committing to branch-2.3. Please commit *only* blockers to 2.3."
> >>>
> >>> YARN-1661 & YARN-1689 are not blockers.
> >>>
> >>> Unless there are objections, I'll revert all these JIRAs from
> branch-2.3
> >>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >>>
> >>> I'm inclined to revert them from branch-2 as well.
> >>>
> >>> Thoughts?
> >>>
> >>> Thanks.
> >>>
> >>>
> >>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> >> wrote:
> >>>
> >>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> >> was
> >>>> causing some strange behavior in the Oozie unit tests:
> >>>>
> >>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> >> unit
> >>>> tests in a module. With YARN-1490 we saw that, regardless of test
> >> order,
> >>>> the last few tests would timeout waiting for an MR job to finish; on
> >> slower
> >>>> machines, the entire test suite would timeout. Through some digging,
> I
> >>>> found that we were getting a ton of "Connection refused" Exceptions on
> >>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> >>>>
> >>>> After a bunch of investigation, I found that the problem went away
> once
> >>>> YARN-1490 was removed. Though I couldn't figure out the exact
> problem.
> >>>> Even though this occurred in unit tests, it does make me concerned
> that
> >> it
> >>>> could indicate some bigger issue in a long-running real cluster (where
> >>>> everything isn't running on the same machine) that we haven't seen
> yet.
> >>>>
> >>>>
> >>>>
> >>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> >>>> wrote:
> >>>>
> >>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
> shortly.
> >>>> Will
> >>>>> pull it out of branch-2.3 if anyone objects.
> >>>>>
> >>>>>
> >>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> >> aagarwal@hortonworks.com
> >>>>>> wrote:
> >>>>>
> >>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
> >>>>>>
> >>>>>>
> >>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> >>>> aagarwal@hortonworks.com
> >>>>>>> wrote:
> >>>>>>
> >>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> >>>>>>>
> >>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> >>>>>>>
> >>>>>>>
> >>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> >>>> tucu@cloudera.com
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> >>>> odd
> >>>>>>>> ways
> >>>>>>>> (to the point it seems un-deterministic).
> >>>>>>>>
> >>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
> >>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> >>>>> reverts)
> >>>>>>>> from Hadoop 2.3 branch before doing the release.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> I've verified that after reverting those JIRAs things work fine
> with
> >>>>>>>> unmanaged AMs.
> >>>>>>>>
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
> acm@hortonworks.com
> >>>>>
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
> >>>>>>>>>
> >>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
> >>>>>> progress
> >>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
> >>>>>>>> objections?
> >>>>>>>>>
> >>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> >>>>>>>>>
> >>>>>>>>> Overall, I'll try get this out in next couple of days if we can
> >>>>> clear
> >>>>>>>> the
> >>>>>>>>> list.
> >>>>>>>>>
> >>>>>>>>> thanks,
> >>>>>>>>> Arun
> >>>>>>>>>
> >>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> >>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> >>>> are
> >>>>>> now
> >>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >>>>>>>>>>
> >>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> >>>>> out
> >>>>>>>> with
> >>>>>>>>> the YARN ones.
> >>>>>>>>>>
> >>>>>>>>>> thanks,
> >>>>>>>>>> Arun
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> Arun C. Murthy
> >>>>>>>>> Hortonworks Inc.
> >>>>>>>>> http://hortonworks.com/
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>>>> entity to
> >>>>>>>>> which it is addressed and may contain information that is
> >>>>>> confidential,
> >>>>>>>>> privileged and exempt from disclosure under applicable law. If
> the
> >>>>>>>> reader
> >>>>>>>>> of this message is not the intended recipient, you are hereby
> >>>>> notified
> >>>>>>>> that
> >>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>>>> forwarding of this communication is strictly prohibited. If you
> >>>> have
> >>>>>>>>> received this communication in error, please contact the sender
> >>>>>>>> immediately
> >>>>>>>>> and delete it from your system. Thank You.
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> --
> >>>>>>>> Alejandro
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> CONFIDENTIALITY NOTICE
> >>>>>> NOTICE: This message is intended for the use of the individual or
> >>>> entity
> >>>>> to
> >>>>>> which it is addressed and may contain information that is
> >> confidential,
> >>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>> reader
> >>>>>> of this message is not the intended recipient, you are hereby
> notified
> >>>>> that
> >>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>> forwarding of this communication is strictly prohibited. If you have
> >>>>>> received this communication in error, please contact the sender
> >>>>> immediately
> >>>>>> and delete it from your system. Thank You.
> >>>>>>
> >>>>>
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Alejandro
> >>
> >>
> >> --
> >> CONFIDENTIALITY NOTICE
> >> NOTICE: This message is intended for the use of the individual or
> entity to
> >> which it is addressed and may contain information that is confidential,
> >> privileged and exempt from disclosure under applicable law. If the
> reader
> >> of this message is not the intended recipient, you are hereby notified
> that
> >> any printing, copying, dissemination, distribution, disclosure or
> >> forwarding of this communication is strictly prohibited. If you have
> >> received this communication in error, please contact the sender
> immediately
> >> and delete it from your system. Thank You.
> >>
> >
> >
> >
> > --
> > Alejandro
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks. please post your findings, Jian wrote this part of the code and between him/me, we can take care of those issues.
+1 for going ahead with the revert on branch-2.3. I'll go do that tomorrow morning unless I hear otherwise from Jian.
Thanks,
+Vinod
On Feb 6, 2014, at 8:28 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Hi Vinod,
>
> Nothing confidential,
>
> * With umanaged AMs I'm seeing the trace I've posted a couple of days ago
> in YARN-1577 (
> https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853
> ).
>
> * Also, Robert has been digging in Oozie testcases failing/getting suck
> with several token renewer threads, this failures happened consistently at
> different places around the same testcases (like some file descriptors
> leaking out), reverting YARN-1490 fixes the problem. The potential issue
> with this is that a long running client (oozie) my run into this situation
> thus becoming unstable.
>
> *Robert,* mind posting to YARN-1490 the jvm thread dump at the time of test
> hanging?
>
> After YARN-1493 & YARN-1490 we have a couple of JIRAs trying to fix issues
> introduced by them, and we still didn't get them right.
>
> Because this, the improvements driven by YARN-1493 & YARN-1490 seem that
> require more work before being stable.
>
> IMO, being conservative, we should do 2.3 without them and roll them with
> 2.4. If we want to do regular releases we will have to make this kind of
> calls, else we will start dragging the releases.
>
> Sounds like a plan?
>
> Thanks.
>
>
>
> On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli
> <vi...@apache.org>wrote:
>
>> Hey
>>
>> I am not against removing them from 2.3 if that is helpful for progress.
>> But I want to understand what the issues are before we make that decision.
>>
>> There is the issue with unmanaged AM that is clearly known and I was
>> thinking of coming to the past two days, but couldn't. What is this new
>> issue that we (confidently?) pinned down to YARN-1490?
>>
>> Thanks
>> +Vinod
>>
>> On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>
>>> Thanks Robert,
>>>
>>> All,
>>>
>>> So it seems that YARN-1493 and YARN-1490 are introducing serious
>>> regressions.
>>>
>>> I would propose to revert them and the follow up JIRAs from the 2.3
>> branch
>>> and keep working on them on trunk/branch-2 until the are stable (I would
>>> even prefer reverting them from branch-2 not to block a 2.4 if they are
>> not
>>> ready in time).
>>>
>>> As I've mentioned before, the list of JIRAs to revert were:
>>>
>>> YARN-1493
>>> YARN-1490
>>> YARN-1166
>>> YARN-1041
>>> YARN-1566
>>>
>>> Plus 2 additional JIRAs committed since my email on this issue 2 days
>> ago:
>>>
>>> *YARN-1661
>>> *YARN-1689 (not sure if this JIRA is related in functionality to the
>>> previous ones but it is creating conflicts).
>>>
>>> I think we should hold on continuing work on top of something that is
>>> broken until the broken stuff is fixed.
>>>
>>> Quoting Arun, "Committers - Henceforth, please use extreme caution while
>>> committing to branch-2.3. Please commit *only* blockers to 2.3."
>>>
>>> YARN-1661 & YARN-1689 are not blockers.
>>>
>>> Unless there are objections, I'll revert all these JIRAs from branch-2.3
>>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>>>
>>> I'm inclined to revert them from branch-2 as well.
>>>
>>> Thoughts?
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
>> wrote:
>>>
>>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
>> was
>>>> causing some strange behavior in the Oozie unit tests:
>>>>
>>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
>> unit
>>>> tests in a module. With YARN-1490 we saw that, regardless of test
>> order,
>>>> the last few tests would timeout waiting for an MR job to finish; on
>> slower
>>>> machines, the entire test suite would timeout. Through some digging, I
>>>> found that we were getting a ton of "Connection refused" Exceptions on
>>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>>>>
>>>> After a bunch of investigation, I found that the problem went away once
>>>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
>>>> Even though this occurred in unit tests, it does make me concerned that
>> it
>>>> could indicate some bigger issue in a long-running real cluster (where
>>>> everything isn't running on the same machine) that we haven't seen yet.
>>>>
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
>>>> wrote:
>>>>
>>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
>>>> Will
>>>>> pull it out of branch-2.3 if anyone objects.
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
>> aagarwal@hortonworks.com
>>>>>> wrote:
>>>>>
>>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
>>>> aagarwal@hortonworks.com
>>>>>>> wrote:
>>>>>>
>>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>>>>>>>
>>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
>>>> odd
>>>>>>>> ways
>>>>>>>> (to the point it seems un-deterministic).
>>>>>>>>
>>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
>>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
>>>>> reverts)
>>>>>>>> from Hadoop 2.3 branch before doing the release.
>>>>>>>>
>>>>>>>>
>>>>>>>> I've verified that after reverting those JIRAs things work fine with
>>>>>>>> unmanaged AMs.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>>>>>>>>>
>>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
>>>>>> progress
>>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
>>>>>>>> objections?
>>>>>>>>>
>>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>>>>>>>>>
>>>>>>>>> Overall, I'll try get this out in next couple of days if we can
>>>>> clear
>>>>>>>> the
>>>>>>>>> list.
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
>>>> are
>>>>>> now
>>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>>>>>>>>>>
>>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
>>>>> out
>>>>>>>> with
>>>>>>>>> the YARN ones.
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>>> entity to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Alejandro
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>> to
>>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>> that
>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>>> and delete it from your system. Thank You.
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Alejandro
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> Alejandro
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks. please post your findings, Jian wrote this part of the code and between him/me, we can take care of those issues.
+1 for going ahead with the revert on branch-2.3. I'll go do that tomorrow morning unless I hear otherwise from Jian.
Thanks,
+Vinod
On Feb 6, 2014, at 8:28 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Hi Vinod,
>
> Nothing confidential,
>
> * With umanaged AMs I'm seeing the trace I've posted a couple of days ago
> in YARN-1577 (
> https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853
> ).
>
> * Also, Robert has been digging in Oozie testcases failing/getting suck
> with several token renewer threads, this failures happened consistently at
> different places around the same testcases (like some file descriptors
> leaking out), reverting YARN-1490 fixes the problem. The potential issue
> with this is that a long running client (oozie) my run into this situation
> thus becoming unstable.
>
> *Robert,* mind posting to YARN-1490 the jvm thread dump at the time of test
> hanging?
>
> After YARN-1493 & YARN-1490 we have a couple of JIRAs trying to fix issues
> introduced by them, and we still didn't get them right.
>
> Because this, the improvements driven by YARN-1493 & YARN-1490 seem that
> require more work before being stable.
>
> IMO, being conservative, we should do 2.3 without them and roll them with
> 2.4. If we want to do regular releases we will have to make this kind of
> calls, else we will start dragging the releases.
>
> Sounds like a plan?
>
> Thanks.
>
>
>
> On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli
> <vi...@apache.org>wrote:
>
>> Hey
>>
>> I am not against removing them from 2.3 if that is helpful for progress.
>> But I want to understand what the issues are before we make that decision.
>>
>> There is the issue with unmanaged AM that is clearly known and I was
>> thinking of coming to the past two days, but couldn't. What is this new
>> issue that we (confidently?) pinned down to YARN-1490?
>>
>> Thanks
>> +Vinod
>>
>> On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>
>>> Thanks Robert,
>>>
>>> All,
>>>
>>> So it seems that YARN-1493 and YARN-1490 are introducing serious
>>> regressions.
>>>
>>> I would propose to revert them and the follow up JIRAs from the 2.3
>> branch
>>> and keep working on them on trunk/branch-2 until the are stable (I would
>>> even prefer reverting them from branch-2 not to block a 2.4 if they are
>> not
>>> ready in time).
>>>
>>> As I've mentioned before, the list of JIRAs to revert were:
>>>
>>> YARN-1493
>>> YARN-1490
>>> YARN-1166
>>> YARN-1041
>>> YARN-1566
>>>
>>> Plus 2 additional JIRAs committed since my email on this issue 2 days
>> ago:
>>>
>>> *YARN-1661
>>> *YARN-1689 (not sure if this JIRA is related in functionality to the
>>> previous ones but it is creating conflicts).
>>>
>>> I think we should hold on continuing work on top of something that is
>>> broken until the broken stuff is fixed.
>>>
>>> Quoting Arun, "Committers - Henceforth, please use extreme caution while
>>> committing to branch-2.3. Please commit *only* blockers to 2.3."
>>>
>>> YARN-1661 & YARN-1689 are not blockers.
>>>
>>> Unless there are objections, I'll revert all these JIRAs from branch-2.3
>>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>>>
>>> I'm inclined to revert them from branch-2 as well.
>>>
>>> Thoughts?
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
>> wrote:
>>>
>>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
>> was
>>>> causing some strange behavior in the Oozie unit tests:
>>>>
>>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
>> unit
>>>> tests in a module. With YARN-1490 we saw that, regardless of test
>> order,
>>>> the last few tests would timeout waiting for an MR job to finish; on
>> slower
>>>> machines, the entire test suite would timeout. Through some digging, I
>>>> found that we were getting a ton of "Connection refused" Exceptions on
>>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>>>>
>>>> After a bunch of investigation, I found that the problem went away once
>>>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
>>>> Even though this occurred in unit tests, it does make me concerned that
>> it
>>>> could indicate some bigger issue in a long-running real cluster (where
>>>> everything isn't running on the same machine) that we haven't seen yet.
>>>>
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
>>>> wrote:
>>>>
>>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
>>>> Will
>>>>> pull it out of branch-2.3 if anyone objects.
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
>> aagarwal@hortonworks.com
>>>>>> wrote:
>>>>>
>>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
>>>> aagarwal@hortonworks.com
>>>>>>> wrote:
>>>>>>
>>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>>>>>>>
>>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
>>>> odd
>>>>>>>> ways
>>>>>>>> (to the point it seems un-deterministic).
>>>>>>>>
>>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
>>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
>>>>> reverts)
>>>>>>>> from Hadoop 2.3 branch before doing the release.
>>>>>>>>
>>>>>>>>
>>>>>>>> I've verified that after reverting those JIRAs things work fine with
>>>>>>>> unmanaged AMs.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>>>>>>>>>
>>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
>>>>>> progress
>>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
>>>>>>>> objections?
>>>>>>>>>
>>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>>>>>>>>>
>>>>>>>>> Overall, I'll try get this out in next couple of days if we can
>>>>> clear
>>>>>>>> the
>>>>>>>>> list.
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
>>>> are
>>>>>> now
>>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>>>>>>>>>>
>>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
>>>>> out
>>>>>>>> with
>>>>>>>>> the YARN ones.
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>>> entity to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Alejandro
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>> to
>>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>> that
>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>>> and delete it from your system. Thank You.
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Alejandro
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> Alejandro
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks. please post your findings, Jian wrote this part of the code and between him/me, we can take care of those issues.
+1 for going ahead with the revert on branch-2.3. I'll go do that tomorrow morning unless I hear otherwise from Jian.
Thanks,
+Vinod
On Feb 6, 2014, at 8:28 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Hi Vinod,
>
> Nothing confidential,
>
> * With umanaged AMs I'm seeing the trace I've posted a couple of days ago
> in YARN-1577 (
> https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853
> ).
>
> * Also, Robert has been digging in Oozie testcases failing/getting suck
> with several token renewer threads, this failures happened consistently at
> different places around the same testcases (like some file descriptors
> leaking out), reverting YARN-1490 fixes the problem. The potential issue
> with this is that a long running client (oozie) my run into this situation
> thus becoming unstable.
>
> *Robert,* mind posting to YARN-1490 the jvm thread dump at the time of test
> hanging?
>
> After YARN-1493 & YARN-1490 we have a couple of JIRAs trying to fix issues
> introduced by them, and we still didn't get them right.
>
> Because this, the improvements driven by YARN-1493 & YARN-1490 seem that
> require more work before being stable.
>
> IMO, being conservative, we should do 2.3 without them and roll them with
> 2.4. If we want to do regular releases we will have to make this kind of
> calls, else we will start dragging the releases.
>
> Sounds like a plan?
>
> Thanks.
>
>
>
> On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli
> <vi...@apache.org>wrote:
>
>> Hey
>>
>> I am not against removing them from 2.3 if that is helpful for progress.
>> But I want to understand what the issues are before we make that decision.
>>
>> There is the issue with unmanaged AM that is clearly known and I was
>> thinking of coming to the past two days, but couldn't. What is this new
>> issue that we (confidently?) pinned down to YARN-1490?
>>
>> Thanks
>> +Vinod
>>
>> On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>
>>> Thanks Robert,
>>>
>>> All,
>>>
>>> So it seems that YARN-1493 and YARN-1490 are introducing serious
>>> regressions.
>>>
>>> I would propose to revert them and the follow up JIRAs from the 2.3
>> branch
>>> and keep working on them on trunk/branch-2 until the are stable (I would
>>> even prefer reverting them from branch-2 not to block a 2.4 if they are
>> not
>>> ready in time).
>>>
>>> As I've mentioned before, the list of JIRAs to revert were:
>>>
>>> YARN-1493
>>> YARN-1490
>>> YARN-1166
>>> YARN-1041
>>> YARN-1566
>>>
>>> Plus 2 additional JIRAs committed since my email on this issue 2 days
>> ago:
>>>
>>> *YARN-1661
>>> *YARN-1689 (not sure if this JIRA is related in functionality to the
>>> previous ones but it is creating conflicts).
>>>
>>> I think we should hold on continuing work on top of something that is
>>> broken until the broken stuff is fixed.
>>>
>>> Quoting Arun, "Committers - Henceforth, please use extreme caution while
>>> committing to branch-2.3. Please commit *only* blockers to 2.3."
>>>
>>> YARN-1661 & YARN-1689 are not blockers.
>>>
>>> Unless there are objections, I'll revert all these JIRAs from branch-2.3
>>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>>>
>>> I'm inclined to revert them from branch-2 as well.
>>>
>>> Thoughts?
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
>> wrote:
>>>
>>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
>> was
>>>> causing some strange behavior in the Oozie unit tests:
>>>>
>>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
>> unit
>>>> tests in a module. With YARN-1490 we saw that, regardless of test
>> order,
>>>> the last few tests would timeout waiting for an MR job to finish; on
>> slower
>>>> machines, the entire test suite would timeout. Through some digging, I
>>>> found that we were getting a ton of "Connection refused" Exceptions on
>>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>>>>
>>>> After a bunch of investigation, I found that the problem went away once
>>>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
>>>> Even though this occurred in unit tests, it does make me concerned that
>> it
>>>> could indicate some bigger issue in a long-running real cluster (where
>>>> everything isn't running on the same machine) that we haven't seen yet.
>>>>
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
>>>> wrote:
>>>>
>>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
>>>> Will
>>>>> pull it out of branch-2.3 if anyone objects.
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
>> aagarwal@hortonworks.com
>>>>>> wrote:
>>>>>
>>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
>>>> aagarwal@hortonworks.com
>>>>>>> wrote:
>>>>>>
>>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>>>>>>>
>>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
>>>> odd
>>>>>>>> ways
>>>>>>>> (to the point it seems un-deterministic).
>>>>>>>>
>>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
>>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
>>>>> reverts)
>>>>>>>> from Hadoop 2.3 branch before doing the release.
>>>>>>>>
>>>>>>>>
>>>>>>>> I've verified that after reverting those JIRAs things work fine with
>>>>>>>> unmanaged AMs.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>>>>>>>>>
>>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
>>>>>> progress
>>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
>>>>>>>> objections?
>>>>>>>>>
>>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>>>>>>>>>
>>>>>>>>> Overall, I'll try get this out in next couple of days if we can
>>>>> clear
>>>>>>>> the
>>>>>>>>> list.
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
>>>> are
>>>>>> now
>>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>>>>>>>>>>
>>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
>>>>> out
>>>>>>>> with
>>>>>>>>> the YARN ones.
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>>> entity to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Alejandro
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>> to
>>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>> that
>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>>> and delete it from your system. Thank You.
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Alejandro
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> Alejandro
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Thanks. please post your findings, Jian wrote this part of the code and between him/me, we can take care of those issues.
+1 for going ahead with the revert on branch-2.3. I'll go do that tomorrow morning unless I hear otherwise from Jian.
Thanks,
+Vinod
On Feb 6, 2014, at 8:28 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Hi Vinod,
>
> Nothing confidential,
>
> * With umanaged AMs I'm seeing the trace I've posted a couple of days ago
> in YARN-1577 (
> https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853
> ).
>
> * Also, Robert has been digging in Oozie testcases failing/getting suck
> with several token renewer threads, this failures happened consistently at
> different places around the same testcases (like some file descriptors
> leaking out), reverting YARN-1490 fixes the problem. The potential issue
> with this is that a long running client (oozie) my run into this situation
> thus becoming unstable.
>
> *Robert,* mind posting to YARN-1490 the jvm thread dump at the time of test
> hanging?
>
> After YARN-1493 & YARN-1490 we have a couple of JIRAs trying to fix issues
> introduced by them, and we still didn't get them right.
>
> Because this, the improvements driven by YARN-1493 & YARN-1490 seem that
> require more work before being stable.
>
> IMO, being conservative, we should do 2.3 without them and roll them with
> 2.4. If we want to do regular releases we will have to make this kind of
> calls, else we will start dragging the releases.
>
> Sounds like a plan?
>
> Thanks.
>
>
>
> On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli
> <vi...@apache.org>wrote:
>
>> Hey
>>
>> I am not against removing them from 2.3 if that is helpful for progress.
>> But I want to understand what the issues are before we make that decision.
>>
>> There is the issue with unmanaged AM that is clearly known and I was
>> thinking of coming to the past two days, but couldn't. What is this new
>> issue that we (confidently?) pinned down to YARN-1490?
>>
>> Thanks
>> +Vinod
>>
>> On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>>
>>> Thanks Robert,
>>>
>>> All,
>>>
>>> So it seems that YARN-1493 and YARN-1490 are introducing serious
>>> regressions.
>>>
>>> I would propose to revert them and the follow up JIRAs from the 2.3
>> branch
>>> and keep working on them on trunk/branch-2 until the are stable (I would
>>> even prefer reverting them from branch-2 not to block a 2.4 if they are
>> not
>>> ready in time).
>>>
>>> As I've mentioned before, the list of JIRAs to revert were:
>>>
>>> YARN-1493
>>> YARN-1490
>>> YARN-1166
>>> YARN-1041
>>> YARN-1566
>>>
>>> Plus 2 additional JIRAs committed since my email on this issue 2 days
>> ago:
>>>
>>> *YARN-1661
>>> *YARN-1689 (not sure if this JIRA is related in functionality to the
>>> previous ones but it is creating conflicts).
>>>
>>> I think we should hold on continuing work on top of something that is
>>> broken until the broken stuff is fixed.
>>>
>>> Quoting Arun, "Committers - Henceforth, please use extreme caution while
>>> committing to branch-2.3. Please commit *only* blockers to 2.3."
>>>
>>> YARN-1661 & YARN-1689 are not blockers.
>>>
>>> Unless there are objections, I'll revert all these JIRAs from branch-2.3
>>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>>>
>>> I'm inclined to revert them from branch-2 as well.
>>>
>>> Thoughts?
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
>> wrote:
>>>
>>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
>> was
>>>> causing some strange behavior in the Oozie unit tests:
>>>>
>>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
>> unit
>>>> tests in a module. With YARN-1490 we saw that, regardless of test
>> order,
>>>> the last few tests would timeout waiting for an MR job to finish; on
>> slower
>>>> machines, the entire test suite would timeout. Through some digging, I
>>>> found that we were getting a ton of "Connection refused" Exceptions on
>>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>>>>
>>>> After a bunch of investigation, I found that the problem went away once
>>>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
>>>> Even though this occurred in unit tests, it does make me concerned that
>> it
>>>> could indicate some bigger issue in a long-running real cluster (where
>>>> everything isn't running on the same machine) that we haven't seen yet.
>>>>
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
>>>> wrote:
>>>>
>>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
>>>> Will
>>>>> pull it out of branch-2.3 if anyone objects.
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
>> aagarwal@hortonworks.com
>>>>>> wrote:
>>>>>
>>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
>>>> aagarwal@hortonworks.com
>>>>>>> wrote:
>>>>>>
>>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>>>>>>>
>>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
>>>> odd
>>>>>>>> ways
>>>>>>>> (to the point it seems un-deterministic).
>>>>>>>>
>>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
>>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
>>>>> reverts)
>>>>>>>> from Hadoop 2.3 branch before doing the release.
>>>>>>>>
>>>>>>>>
>>>>>>>> I've verified that after reverting those JIRAs things work fine with
>>>>>>>> unmanaged AMs.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>>>>>>>>>
>>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
>>>>>> progress
>>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
>>>>>>>> objections?
>>>>>>>>>
>>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>>>>>>>>>
>>>>>>>>> Overall, I'll try get this out in next couple of days if we can
>>>>> clear
>>>>>>>> the
>>>>>>>>> list.
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
>>>> are
>>>>>> now
>>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>>>>>>>>>>
>>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
>>>>> out
>>>>>>>> with
>>>>>>>>> the YARN ones.
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>>>> entity to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>>>> forwarding of this communication is strictly prohibited. If you
>>>> have
>>>>>>>>> received this communication in error, please contact the sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Alejandro
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>> to
>>>>>> which it is addressed and may contain information that is
>> confidential,
>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>>>> of this message is not the intended recipient, you are hereby notified
>>>>> that
>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>> forwarding of this communication is strictly prohibited. If you have
>>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>>> and delete it from your system. Thank You.
>>>>>>
>>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Alejandro
>>
>>
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>>
>
>
>
> --
> Alejandro
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Hi Vinod,
Nothing confidential,
* With umanaged AMs I'm seeing the trace I've posted a couple of days ago
in YARN-1577 (
https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853
).
* Also, Robert has been digging in Oozie testcases failing/getting suck
with several token renewer threads, this failures happened consistently at
different places around the same testcases (like some file descriptors
leaking out), reverting YARN-1490 fixes the problem. The potential issue
with this is that a long running client (oozie) my run into this situation
thus becoming unstable.
*Robert,* mind posting to YARN-1490 the jvm thread dump at the time of test
hanging?
After YARN-1493 & YARN-1490 we have a couple of JIRAs trying to fix issues
introduced by them, and we still didn't get them right.
Because this, the improvements driven by YARN-1493 & YARN-1490 seem that
require more work before being stable.
IMO, being conservative, we should do 2.3 without them and roll them with
2.4. If we want to do regular releases we will have to make this kind of
calls, else we will start dragging the releases.
Sounds like a plan?
Thanks.
On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli
<vi...@apache.org>wrote:
> Hey
>
> I am not against removing them from 2.3 if that is helpful for progress.
> But I want to understand what the issues are before we make that decision.
>
> There is the issue with unmanaged AM that is clearly known and I was
> thinking of coming to the past two days, but couldn't. What is this new
> issue that we (confidently?) pinned down to YARN-1490?
>
> Thanks
> +Vinod
>
> On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > Thanks Robert,
> >
> > All,
> >
> > So it seems that YARN-1493 and YARN-1490 are introducing serious
> > regressions.
> >
> > I would propose to revert them and the follow up JIRAs from the 2.3
> branch
> > and keep working on them on trunk/branch-2 until the are stable (I would
> > even prefer reverting them from branch-2 not to block a 2.4 if they are
> not
> > ready in time).
> >
> > As I've mentioned before, the list of JIRAs to revert were:
> >
> > YARN-1493
> > YARN-1490
> > YARN-1166
> > YARN-1041
> > YARN-1566
> >
> > Plus 2 additional JIRAs committed since my email on this issue 2 days
> ago:
> >
> > *YARN-1661
> > *YARN-1689 (not sure if this JIRA is related in functionality to the
> > previous ones but it is creating conflicts).
> >
> > I think we should hold on continuing work on top of something that is
> > broken until the broken stuff is fixed.
> >
> > Quoting Arun, "Committers - Henceforth, please use extreme caution while
> > committing to branch-2.3. Please commit *only* blockers to 2.3."
> >
> > YARN-1661 & YARN-1689 are not blockers.
> >
> > Unless there are objections, I'll revert all these JIRAs from branch-2.3
> > tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >
> > I'm inclined to revert them from branch-2 as well.
> >
> > Thoughts?
> >
> > Thanks.
> >
> >
> > On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
> >
> >> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> was
> >> causing some strange behavior in the Oozie unit tests:
> >>
> >> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> unit
> >> tests in a module. With YARN-1490 we saw that, regardless of test
> order,
> >> the last few tests would timeout waiting for an MR job to finish; on
> slower
> >> machines, the entire test suite would timeout. Through some digging, I
> >> found that we were getting a ton of "Connection refused" Exceptions on
> >> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> >>
> >> After a bunch of investigation, I found that the problem went away once
> >> YARN-1490 was removed. Though I couldn't figure out the exact problem.
> >> Even though this occurred in unit tests, it does make me concerned that
> it
> >> could indicate some bigger issue in a long-running real cluster (where
> >> everything isn't running on the same machine) that we haven't seen yet.
> >>
> >>
> >>
> >> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> >> wrote:
> >>
> >>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
> >> Will
> >>> pull it out of branch-2.3 if anyone objects.
> >>>
> >>>
> >>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> aagarwal@hortonworks.com
> >>>> wrote:
> >>>
> >>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
> >>>>
> >>>>
> >>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> >> aagarwal@hortonworks.com
> >>>>> wrote:
> >>>>
> >>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> >>>>>
> >>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> >>>>>
> >>>>>
> >>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> >> tucu@cloudera.com
> >>>>> wrote:
> >>>>>
> >>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> >> odd
> >>>>>> ways
> >>>>>> (to the point it seems un-deterministic).
> >>>>>>
> >>>>>> I'd say eiher YARN-1577 is fixed or we revert
> >>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> >>> reverts)
> >>>>>> from Hadoop 2.3 branch before doing the release.
> >>>>>>
> >>>>>>
> >>>>>> I've verified that after reverting those JIRAs things work fine with
> >>>>>> unmanaged AMs.
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
> >>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
> >>>>>>>
> >>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
> >>>> progress
> >>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
> >>>>>> objections?
> >>>>>>>
> >>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> >>>>>>>
> >>>>>>> Overall, I'll try get this out in next couple of days if we can
> >>> clear
> >>>>>> the
> >>>>>>> list.
> >>>>>>>
> >>>>>>> thanks,
> >>>>>>> Arun
> >>>>>>>
> >>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> >> are
> >>>> now
> >>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >>>>>>>>
> >>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> >>> out
> >>>>>> with
> >>>>>>> the YARN ones.
> >>>>>>>>
> >>>>>>>> thanks,
> >>>>>>>> Arun
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Arun C. Murthy
> >>>>>>> Hortonworks Inc.
> >>>>>>> http://hortonworks.com/
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>> entity to
> >>>>>>> which it is addressed and may contain information that is
> >>>> confidential,
> >>>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>>>> reader
> >>>>>>> of this message is not the intended recipient, you are hereby
> >>> notified
> >>>>>> that
> >>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>> forwarding of this communication is strictly prohibited. If you
> >> have
> >>>>>>> received this communication in error, please contact the sender
> >>>>>> immediately
> >>>>>>> and delete it from your system. Thank You.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Alejandro
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> >> entity
> >>> to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> >> reader
> >>>> of this message is not the intended recipient, you are hereby notified
> >>> that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> >>> immediately
> >>>> and delete it from your system. Thank You.
> >>>>
> >>>
> >>
> >
> >
> >
> > --
> > Alejandro
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Hi Vinod,
Nothing confidential,
* With umanaged AMs I'm seeing the trace I've posted a couple of days ago
in YARN-1577 (
https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853
).
* Also, Robert has been digging in Oozie testcases failing/getting suck
with several token renewer threads, this failures happened consistently at
different places around the same testcases (like some file descriptors
leaking out), reverting YARN-1490 fixes the problem. The potential issue
with this is that a long running client (oozie) my run into this situation
thus becoming unstable.
*Robert,* mind posting to YARN-1490 the jvm thread dump at the time of test
hanging?
After YARN-1493 & YARN-1490 we have a couple of JIRAs trying to fix issues
introduced by them, and we still didn't get them right.
Because this, the improvements driven by YARN-1493 & YARN-1490 seem that
require more work before being stable.
IMO, being conservative, we should do 2.3 without them and roll them with
2.4. If we want to do regular releases we will have to make this kind of
calls, else we will start dragging the releases.
Sounds like a plan?
Thanks.
On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli
<vi...@apache.org>wrote:
> Hey
>
> I am not against removing them from 2.3 if that is helpful for progress.
> But I want to understand what the issues are before we make that decision.
>
> There is the issue with unmanaged AM that is clearly known and I was
> thinking of coming to the past two days, but couldn't. What is this new
> issue that we (confidently?) pinned down to YARN-1490?
>
> Thanks
> +Vinod
>
> On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > Thanks Robert,
> >
> > All,
> >
> > So it seems that YARN-1493 and YARN-1490 are introducing serious
> > regressions.
> >
> > I would propose to revert them and the follow up JIRAs from the 2.3
> branch
> > and keep working on them on trunk/branch-2 until the are stable (I would
> > even prefer reverting them from branch-2 not to block a 2.4 if they are
> not
> > ready in time).
> >
> > As I've mentioned before, the list of JIRAs to revert were:
> >
> > YARN-1493
> > YARN-1490
> > YARN-1166
> > YARN-1041
> > YARN-1566
> >
> > Plus 2 additional JIRAs committed since my email on this issue 2 days
> ago:
> >
> > *YARN-1661
> > *YARN-1689 (not sure if this JIRA is related in functionality to the
> > previous ones but it is creating conflicts).
> >
> > I think we should hold on continuing work on top of something that is
> > broken until the broken stuff is fixed.
> >
> > Quoting Arun, "Committers - Henceforth, please use extreme caution while
> > committing to branch-2.3. Please commit *only* blockers to 2.3."
> >
> > YARN-1661 & YARN-1689 are not blockers.
> >
> > Unless there are objections, I'll revert all these JIRAs from branch-2.3
> > tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >
> > I'm inclined to revert them from branch-2 as well.
> >
> > Thoughts?
> >
> > Thanks.
> >
> >
> > On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
> >
> >> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> was
> >> causing some strange behavior in the Oozie unit tests:
> >>
> >> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> unit
> >> tests in a module. With YARN-1490 we saw that, regardless of test
> order,
> >> the last few tests would timeout waiting for an MR job to finish; on
> slower
> >> machines, the entire test suite would timeout. Through some digging, I
> >> found that we were getting a ton of "Connection refused" Exceptions on
> >> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> >>
> >> After a bunch of investigation, I found that the problem went away once
> >> YARN-1490 was removed. Though I couldn't figure out the exact problem.
> >> Even though this occurred in unit tests, it does make me concerned that
> it
> >> could indicate some bigger issue in a long-running real cluster (where
> >> everything isn't running on the same machine) that we haven't seen yet.
> >>
> >>
> >>
> >> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> >> wrote:
> >>
> >>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
> >> Will
> >>> pull it out of branch-2.3 if anyone objects.
> >>>
> >>>
> >>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> aagarwal@hortonworks.com
> >>>> wrote:
> >>>
> >>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
> >>>>
> >>>>
> >>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> >> aagarwal@hortonworks.com
> >>>>> wrote:
> >>>>
> >>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> >>>>>
> >>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> >>>>>
> >>>>>
> >>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> >> tucu@cloudera.com
> >>>>> wrote:
> >>>>>
> >>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> >> odd
> >>>>>> ways
> >>>>>> (to the point it seems un-deterministic).
> >>>>>>
> >>>>>> I'd say eiher YARN-1577 is fixed or we revert
> >>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> >>> reverts)
> >>>>>> from Hadoop 2.3 branch before doing the release.
> >>>>>>
> >>>>>>
> >>>>>> I've verified that after reverting those JIRAs things work fine with
> >>>>>> unmanaged AMs.
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
> >>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
> >>>>>>>
> >>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
> >>>> progress
> >>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
> >>>>>> objections?
> >>>>>>>
> >>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> >>>>>>>
> >>>>>>> Overall, I'll try get this out in next couple of days if we can
> >>> clear
> >>>>>> the
> >>>>>>> list.
> >>>>>>>
> >>>>>>> thanks,
> >>>>>>> Arun
> >>>>>>>
> >>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> >> are
> >>>> now
> >>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >>>>>>>>
> >>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> >>> out
> >>>>>> with
> >>>>>>> the YARN ones.
> >>>>>>>>
> >>>>>>>> thanks,
> >>>>>>>> Arun
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Arun C. Murthy
> >>>>>>> Hortonworks Inc.
> >>>>>>> http://hortonworks.com/
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>> entity to
> >>>>>>> which it is addressed and may contain information that is
> >>>> confidential,
> >>>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>>>> reader
> >>>>>>> of this message is not the intended recipient, you are hereby
> >>> notified
> >>>>>> that
> >>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>> forwarding of this communication is strictly prohibited. If you
> >> have
> >>>>>>> received this communication in error, please contact the sender
> >>>>>> immediately
> >>>>>>> and delete it from your system. Thank You.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Alejandro
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> >> entity
> >>> to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> >> reader
> >>>> of this message is not the intended recipient, you are hereby notified
> >>> that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> >>> immediately
> >>>> and delete it from your system. Thank You.
> >>>>
> >>>
> >>
> >
> >
> >
> > --
> > Alejandro
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Hi Vinod,
Nothing confidential,
* With umanaged AMs I'm seeing the trace I've posted a couple of days ago
in YARN-1577 (
https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853
).
* Also, Robert has been digging in Oozie testcases failing/getting suck
with several token renewer threads, this failures happened consistently at
different places around the same testcases (like some file descriptors
leaking out), reverting YARN-1490 fixes the problem. The potential issue
with this is that a long running client (oozie) my run into this situation
thus becoming unstable.
*Robert,* mind posting to YARN-1490 the jvm thread dump at the time of test
hanging?
After YARN-1493 & YARN-1490 we have a couple of JIRAs trying to fix issues
introduced by them, and we still didn't get them right.
Because this, the improvements driven by YARN-1493 & YARN-1490 seem that
require more work before being stable.
IMO, being conservative, we should do 2.3 without them and roll them with
2.4. If we want to do regular releases we will have to make this kind of
calls, else we will start dragging the releases.
Sounds like a plan?
Thanks.
On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli
<vi...@apache.org>wrote:
> Hey
>
> I am not against removing them from 2.3 if that is helpful for progress.
> But I want to understand what the issues are before we make that decision.
>
> There is the issue with unmanaged AM that is clearly known and I was
> thinking of coming to the past two days, but couldn't. What is this new
> issue that we (confidently?) pinned down to YARN-1490?
>
> Thanks
> +Vinod
>
> On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > Thanks Robert,
> >
> > All,
> >
> > So it seems that YARN-1493 and YARN-1490 are introducing serious
> > regressions.
> >
> > I would propose to revert them and the follow up JIRAs from the 2.3
> branch
> > and keep working on them on trunk/branch-2 until the are stable (I would
> > even prefer reverting them from branch-2 not to block a 2.4 if they are
> not
> > ready in time).
> >
> > As I've mentioned before, the list of JIRAs to revert were:
> >
> > YARN-1493
> > YARN-1490
> > YARN-1166
> > YARN-1041
> > YARN-1566
> >
> > Plus 2 additional JIRAs committed since my email on this issue 2 days
> ago:
> >
> > *YARN-1661
> > *YARN-1689 (not sure if this JIRA is related in functionality to the
> > previous ones but it is creating conflicts).
> >
> > I think we should hold on continuing work on top of something that is
> > broken until the broken stuff is fixed.
> >
> > Quoting Arun, "Committers - Henceforth, please use extreme caution while
> > committing to branch-2.3. Please commit *only* blockers to 2.3."
> >
> > YARN-1661 & YARN-1689 are not blockers.
> >
> > Unless there are objections, I'll revert all these JIRAs from branch-2.3
> > tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >
> > I'm inclined to revert them from branch-2 as well.
> >
> > Thoughts?
> >
> > Thanks.
> >
> >
> > On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
> >
> >> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> was
> >> causing some strange behavior in the Oozie unit tests:
> >>
> >> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> unit
> >> tests in a module. With YARN-1490 we saw that, regardless of test
> order,
> >> the last few tests would timeout waiting for an MR job to finish; on
> slower
> >> machines, the entire test suite would timeout. Through some digging, I
> >> found that we were getting a ton of "Connection refused" Exceptions on
> >> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> >>
> >> After a bunch of investigation, I found that the problem went away once
> >> YARN-1490 was removed. Though I couldn't figure out the exact problem.
> >> Even though this occurred in unit tests, it does make me concerned that
> it
> >> could indicate some bigger issue in a long-running real cluster (where
> >> everything isn't running on the same machine) that we haven't seen yet.
> >>
> >>
> >>
> >> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> >> wrote:
> >>
> >>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
> >> Will
> >>> pull it out of branch-2.3 if anyone objects.
> >>>
> >>>
> >>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> aagarwal@hortonworks.com
> >>>> wrote:
> >>>
> >>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
> >>>>
> >>>>
> >>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> >> aagarwal@hortonworks.com
> >>>>> wrote:
> >>>>
> >>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> >>>>>
> >>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> >>>>>
> >>>>>
> >>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> >> tucu@cloudera.com
> >>>>> wrote:
> >>>>>
> >>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> >> odd
> >>>>>> ways
> >>>>>> (to the point it seems un-deterministic).
> >>>>>>
> >>>>>> I'd say eiher YARN-1577 is fixed or we revert
> >>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> >>> reverts)
> >>>>>> from Hadoop 2.3 branch before doing the release.
> >>>>>>
> >>>>>>
> >>>>>> I've verified that after reverting those JIRAs things work fine with
> >>>>>> unmanaged AMs.
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
> >>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
> >>>>>>>
> >>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
> >>>> progress
> >>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
> >>>>>> objections?
> >>>>>>>
> >>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> >>>>>>>
> >>>>>>> Overall, I'll try get this out in next couple of days if we can
> >>> clear
> >>>>>> the
> >>>>>>> list.
> >>>>>>>
> >>>>>>> thanks,
> >>>>>>> Arun
> >>>>>>>
> >>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> >> are
> >>>> now
> >>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >>>>>>>>
> >>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> >>> out
> >>>>>> with
> >>>>>>> the YARN ones.
> >>>>>>>>
> >>>>>>>> thanks,
> >>>>>>>> Arun
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Arun C. Murthy
> >>>>>>> Hortonworks Inc.
> >>>>>>> http://hortonworks.com/
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>> entity to
> >>>>>>> which it is addressed and may contain information that is
> >>>> confidential,
> >>>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>>>> reader
> >>>>>>> of this message is not the intended recipient, you are hereby
> >>> notified
> >>>>>> that
> >>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>> forwarding of this communication is strictly prohibited. If you
> >> have
> >>>>>>> received this communication in error, please contact the sender
> >>>>>> immediately
> >>>>>>> and delete it from your system. Thank You.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Alejandro
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> >> entity
> >>> to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> >> reader
> >>>> of this message is not the intended recipient, you are hereby notified
> >>> that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> >>> immediately
> >>>> and delete it from your system. Thank You.
> >>>>
> >>>
> >>
> >
> >
> >
> > --
> > Alejandro
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Hi Vinod,
Nothing confidential,
* With umanaged AMs I'm seeing the trace I've posted a couple of days ago
in YARN-1577 (
https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853
).
* Also, Robert has been digging in Oozie testcases failing/getting suck
with several token renewer threads, this failures happened consistently at
different places around the same testcases (like some file descriptors
leaking out), reverting YARN-1490 fixes the problem. The potential issue
with this is that a long running client (oozie) my run into this situation
thus becoming unstable.
*Robert,* mind posting to YARN-1490 the jvm thread dump at the time of test
hanging?
After YARN-1493 & YARN-1490 we have a couple of JIRAs trying to fix issues
introduced by them, and we still didn't get them right.
Because this, the improvements driven by YARN-1493 & YARN-1490 seem that
require more work before being stable.
IMO, being conservative, we should do 2.3 without them and roll them with
2.4. If we want to do regular releases we will have to make this kind of
calls, else we will start dragging the releases.
Sounds like a plan?
Thanks.
On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli
<vi...@apache.org>wrote:
> Hey
>
> I am not against removing them from 2.3 if that is helpful for progress.
> But I want to understand what the issues are before we make that decision.
>
> There is the issue with unmanaged AM that is clearly known and I was
> thinking of coming to the past two days, but couldn't. What is this new
> issue that we (confidently?) pinned down to YARN-1490?
>
> Thanks
> +Vinod
>
> On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
>
> > Thanks Robert,
> >
> > All,
> >
> > So it seems that YARN-1493 and YARN-1490 are introducing serious
> > regressions.
> >
> > I would propose to revert them and the follow up JIRAs from the 2.3
> branch
> > and keep working on them on trunk/branch-2 until the are stable (I would
> > even prefer reverting them from branch-2 not to block a 2.4 if they are
> not
> > ready in time).
> >
> > As I've mentioned before, the list of JIRAs to revert were:
> >
> > YARN-1493
> > YARN-1490
> > YARN-1166
> > YARN-1041
> > YARN-1566
> >
> > Plus 2 additional JIRAs committed since my email on this issue 2 days
> ago:
> >
> > *YARN-1661
> > *YARN-1689 (not sure if this JIRA is related in functionality to the
> > previous ones but it is creating conflicts).
> >
> > I think we should hold on continuing work on top of something that is
> > broken until the broken stuff is fixed.
> >
> > Quoting Arun, "Committers - Henceforth, please use extreme caution while
> > committing to branch-2.3. Please commit *only* blockers to 2.3."
> >
> > YARN-1661 & YARN-1689 are not blockers.
> >
> > Unless there are objections, I'll revert all these JIRAs from branch-2.3
> > tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >
> > I'm inclined to revert them from branch-2 as well.
> >
> > Thoughts?
> >
> > Thanks.
> >
> >
> > On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
> >
> >> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> was
> >> causing some strange behavior in the Oozie unit tests:
> >>
> >> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> unit
> >> tests in a module. With YARN-1490 we saw that, regardless of test
> order,
> >> the last few tests would timeout waiting for an MR job to finish; on
> slower
> >> machines, the entire test suite would timeout. Through some digging, I
> >> found that we were getting a ton of "Connection refused" Exceptions on
> >> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> >>
> >> After a bunch of investigation, I found that the problem went away once
> >> YARN-1490 was removed. Though I couldn't figure out the exact problem.
> >> Even though this occurred in unit tests, it does make me concerned that
> it
> >> could indicate some bigger issue in a long-running real cluster (where
> >> everything isn't running on the same machine) that we haven't seen yet.
> >>
> >>
> >>
> >> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> >> wrote:
> >>
> >>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
> >> Will
> >>> pull it out of branch-2.3 if anyone objects.
> >>>
> >>>
> >>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> aagarwal@hortonworks.com
> >>>> wrote:
> >>>
> >>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
> >>>>
> >>>>
> >>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> >> aagarwal@hortonworks.com
> >>>>> wrote:
> >>>>
> >>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> >>>>>
> >>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> >>>>>
> >>>>>
> >>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> >> tucu@cloudera.com
> >>>>> wrote:
> >>>>>
> >>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> >> odd
> >>>>>> ways
> >>>>>> (to the point it seems un-deterministic).
> >>>>>>
> >>>>>> I'd say eiher YARN-1577 is fixed or we revert
> >>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> >>> reverts)
> >>>>>> from Hadoop 2.3 branch before doing the release.
> >>>>>>
> >>>>>>
> >>>>>> I've verified that after reverting those JIRAs things work fine with
> >>>>>> unmanaged AMs.
> >>>>>>
> >>>>>> Thanks.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
> >>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
> >>>>>>>
> >>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
> >>>> progress
> >>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
> >>>>>> objections?
> >>>>>>>
> >>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> >>>>>>>
> >>>>>>> Overall, I'll try get this out in next couple of days if we can
> >>> clear
> >>>>>> the
> >>>>>>> list.
> >>>>>>>
> >>>>>>> thanks,
> >>>>>>> Arun
> >>>>>>>
> >>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> >>>> wrote:
> >>>>>>>
> >>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> >> are
> >>>> now
> >>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >>>>>>>>
> >>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> >>> out
> >>>>>> with
> >>>>>>> the YARN ones.
> >>>>>>>>
> >>>>>>>> thanks,
> >>>>>>>> Arun
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> Arun C. Murthy
> >>>>>>> Hortonworks Inc.
> >>>>>>> http://hortonworks.com/
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> --
> >>>>>>> CONFIDENTIALITY NOTICE
> >>>>>>> NOTICE: This message is intended for the use of the individual or
> >>>>>> entity to
> >>>>>>> which it is addressed and may contain information that is
> >>>> confidential,
> >>>>>>> privileged and exempt from disclosure under applicable law. If the
> >>>>>> reader
> >>>>>>> of this message is not the intended recipient, you are hereby
> >>> notified
> >>>>>> that
> >>>>>>> any printing, copying, dissemination, distribution, disclosure or
> >>>>>>> forwarding of this communication is strictly prohibited. If you
> >> have
> >>>>>>> received this communication in error, please contact the sender
> >>>>>> immediately
> >>>>>>> and delete it from your system. Thank You.
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Alejandro
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>> --
> >>>> CONFIDENTIALITY NOTICE
> >>>> NOTICE: This message is intended for the use of the individual or
> >> entity
> >>> to
> >>>> which it is addressed and may contain information that is
> confidential,
> >>>> privileged and exempt from disclosure under applicable law. If the
> >> reader
> >>>> of this message is not the intended recipient, you are hereby notified
> >>> that
> >>>> any printing, copying, dissemination, distribution, disclosure or
> >>>> forwarding of this communication is strictly prohibited. If you have
> >>>> received this communication in error, please contact the sender
> >>> immediately
> >>>> and delete it from your system. Thank You.
> >>>>
> >>>
> >>
> >
> >
> >
> > --
> > Alejandro
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Hey
I am not against removing them from 2.3 if that is helpful for progress. But I want to understand what the issues are before we make that decision.
There is the issue with unmanaged AM that is clearly known and I was thinking of coming to the past two days, but couldn't. What is this new issue that we (confidently?) pinned down to YARN-1490?
Thanks
+Vinod
On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Thanks Robert,
>
> All,
>
> So it seems that YARN-1493 and YARN-1490 are introducing serious
> regressions.
>
> I would propose to revert them and the follow up JIRAs from the 2.3 branch
> and keep working on them on trunk/branch-2 until the are stable (I would
> even prefer reverting them from branch-2 not to block a 2.4 if they are not
> ready in time).
>
> As I've mentioned before, the list of JIRAs to revert were:
>
> YARN-1493
> YARN-1490
> YARN-1166
> YARN-1041
> YARN-1566
>
> Plus 2 additional JIRAs committed since my email on this issue 2 days ago:
>
> *YARN-1661
> *YARN-1689 (not sure if this JIRA is related in functionality to the
> previous ones but it is creating conflicts).
>
> I think we should hold on continuing work on top of something that is
> broken until the broken stuff is fixed.
>
> Quoting Arun, "Committers - Henceforth, please use extreme caution while
> committing to branch-2.3. Please commit *only* blockers to 2.3."
>
> YARN-1661 & YARN-1689 are not blockers.
>
> Unless there are objections, I'll revert all these JIRAs from branch-2.3
> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>
> I'm inclined to revert them from branch-2 as well.
>
> Thoughts?
>
> Thanks.
>
>
> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com> wrote:
>
>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it was
>> causing some strange behavior in the Oozie unit tests:
>>
>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all unit
>> tests in a module. With YARN-1490 we saw that, regardless of test order,
>> the last few tests would timeout waiting for an MR job to finish; on slower
>> machines, the entire test suite would timeout. Through some digging, I
>> found that we were getting a ton of "Connection refused" Exceptions on
>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>>
>> After a bunch of investigation, I found that the problem went away once
>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
>> Even though this occurred in unit tests, it does make me concerned that it
>> could indicate some bigger issue in a long-running real cluster (where
>> everything isn't running on the same machine) that we haven't seen yet.
>>
>>
>>
>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
>> wrote:
>>
>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
>> Will
>>> pull it out of branch-2.3 if anyone objects.
>>>
>>>
>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <aagarwal@hortonworks.com
>>>> wrote:
>>>
>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
>>>>
>>>>
>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
>> aagarwal@hortonworks.com
>>>>> wrote:
>>>>
>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>>>>>
>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>>>>>
>>>>>
>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
>> tucu@cloudera.com
>>>>> wrote:
>>>>>
>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
>> odd
>>>>>> ways
>>>>>> (to the point it seems un-deterministic).
>>>>>>
>>>>>> I'd say eiher YARN-1577 is fixed or we revert
>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
>>> reverts)
>>>>>> from Hadoop 2.3 branch before doing the release.
>>>>>>
>>>>>>
>>>>>> I've verified that after reverting those JIRAs things work fine with
>>>>>> unmanaged AMs.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
>>>
>>>>>> wrote:
>>>>>>
>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>>>>>>>
>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
>>>> progress
>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
>>>>>> objections?
>>>>>>>
>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>>>>>>>
>>>>>>> Overall, I'll try get this out in next couple of days if we can
>>> clear
>>>>>> the
>>>>>>> list.
>>>>>>>
>>>>>>> thanks,
>>>>>>> Arun
>>>>>>>
>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
>>>> wrote:
>>>>>>>
>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
>> are
>>>> now
>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>>>>>>>>
>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
>>> out
>>>>>> with
>>>>>>> the YARN ones.
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Arun
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Arun C. Murthy
>>>>>>> Hortonworks Inc.
>>>>>>> http://hortonworks.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby
>>> notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you
>> have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Alejandro
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>> entity
>>> to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the
>> reader
>>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender
>>> immediately
>>>> and delete it from your system. Thank You.
>>>>
>>>
>>
>
>
>
> --
> Alejandro
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Hey
I am not against removing them from 2.3 if that is helpful for progress. But I want to understand what the issues are before we make that decision.
There is the issue with unmanaged AM that is clearly known and I was thinking of coming to the past two days, but couldn't. What is this new issue that we (confidently?) pinned down to YARN-1490?
Thanks
+Vinod
On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Thanks Robert,
>
> All,
>
> So it seems that YARN-1493 and YARN-1490 are introducing serious
> regressions.
>
> I would propose to revert them and the follow up JIRAs from the 2.3 branch
> and keep working on them on trunk/branch-2 until the are stable (I would
> even prefer reverting them from branch-2 not to block a 2.4 if they are not
> ready in time).
>
> As I've mentioned before, the list of JIRAs to revert were:
>
> YARN-1493
> YARN-1490
> YARN-1166
> YARN-1041
> YARN-1566
>
> Plus 2 additional JIRAs committed since my email on this issue 2 days ago:
>
> *YARN-1661
> *YARN-1689 (not sure if this JIRA is related in functionality to the
> previous ones but it is creating conflicts).
>
> I think we should hold on continuing work on top of something that is
> broken until the broken stuff is fixed.
>
> Quoting Arun, "Committers - Henceforth, please use extreme caution while
> committing to branch-2.3. Please commit *only* blockers to 2.3."
>
> YARN-1661 & YARN-1689 are not blockers.
>
> Unless there are objections, I'll revert all these JIRAs from branch-2.3
> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>
> I'm inclined to revert them from branch-2 as well.
>
> Thoughts?
>
> Thanks.
>
>
> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com> wrote:
>
>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it was
>> causing some strange behavior in the Oozie unit tests:
>>
>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all unit
>> tests in a module. With YARN-1490 we saw that, regardless of test order,
>> the last few tests would timeout waiting for an MR job to finish; on slower
>> machines, the entire test suite would timeout. Through some digging, I
>> found that we were getting a ton of "Connection refused" Exceptions on
>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>>
>> After a bunch of investigation, I found that the problem went away once
>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
>> Even though this occurred in unit tests, it does make me concerned that it
>> could indicate some bigger issue in a long-running real cluster (where
>> everything isn't running on the same machine) that we haven't seen yet.
>>
>>
>>
>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
>> wrote:
>>
>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
>> Will
>>> pull it out of branch-2.3 if anyone objects.
>>>
>>>
>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <aagarwal@hortonworks.com
>>>> wrote:
>>>
>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
>>>>
>>>>
>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
>> aagarwal@hortonworks.com
>>>>> wrote:
>>>>
>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>>>>>
>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>>>>>
>>>>>
>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
>> tucu@cloudera.com
>>>>> wrote:
>>>>>
>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
>> odd
>>>>>> ways
>>>>>> (to the point it seems un-deterministic).
>>>>>>
>>>>>> I'd say eiher YARN-1577 is fixed or we revert
>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
>>> reverts)
>>>>>> from Hadoop 2.3 branch before doing the release.
>>>>>>
>>>>>>
>>>>>> I've verified that after reverting those JIRAs things work fine with
>>>>>> unmanaged AMs.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
>>>
>>>>>> wrote:
>>>>>>
>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>>>>>>>
>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
>>>> progress
>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
>>>>>> objections?
>>>>>>>
>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>>>>>>>
>>>>>>> Overall, I'll try get this out in next couple of days if we can
>>> clear
>>>>>> the
>>>>>>> list.
>>>>>>>
>>>>>>> thanks,
>>>>>>> Arun
>>>>>>>
>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
>>>> wrote:
>>>>>>>
>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
>> are
>>>> now
>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>>>>>>>>
>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
>>> out
>>>>>> with
>>>>>>> the YARN ones.
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Arun
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Arun C. Murthy
>>>>>>> Hortonworks Inc.
>>>>>>> http://hortonworks.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby
>>> notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you
>> have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Alejandro
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>> entity
>>> to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the
>> reader
>>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender
>>> immediately
>>>> and delete it from your system. Thank You.
>>>>
>>>
>>
>
>
>
> --
> Alejandro
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
yep, the idea is to pull all of them out from branch2.3. things go back to normal then.
thanks
Alejandro
(phone typing)
> On Feb 6, 2014, at 17:39, Zhijie Shen <zs...@hortonworks.com> wrote:
>
> Recently I brought 4 JIRAs to branch-2.3, which are MAPREDUCE-5743, YARN-1628,
> YARN-1661 and YARN-1689. Recall that we mark test failure fixes as blockers
> for pior releases as closing to release, thus I brought to branch-2.3
> MAPREDUCE-5743
> and YARN-1628 that are the fixes for the test failure on 2.3.0, but didn't
> marked them as blockers. Please let me know if I should do that.
>
> YARN-1661 is a fix for exit log of DS AppMaster, otherwise the exit log of
> it will always be failure, which sounds a critical issue to me. Feel free
> to pull it out if any objects.
>
> YARN-1689 is brought to branch-2.3 as YARN-1493 is still in this branch. It
> fixes one bug caused by YARN-1493. Those should be included or excluded
> together upon the decision.
>
> Thanks,
> Zhijie
>
>
>> On Thu, Feb 6, 2014 at 5:15 PM, Sandy Ryza <sa...@cloudera.com> wrote:
>>
>> +1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a
>> problem caused by YARN-1493 I think we can revert it in branch-2.3 as well.
>>
>> I think we should leave them in branch-2 for now. We can revert if 2.4 is
>> imminent and they're holding it up, but hopefully the issues they caused
>> will be fixed by then.
>>
>> -Sandy
>>
>>
>> On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>
>>> Thanks Robert,
>>>
>>> All,
>>>
>>> So it seems that YARN-1493 and YARN-1490 are introducing serious
>>> regressions.
>>>
>>> I would propose to revert them and the follow up JIRAs from the 2.3
>> branch
>>> and keep working on them on trunk/branch-2 until the are stable (I would
>>> even prefer reverting them from branch-2 not to block a 2.4 if they are
>> not
>>> ready in time).
>>>
>>> As I've mentioned before, the list of JIRAs to revert were:
>>>
>>> YARN-1493
>>> YARN-1490
>>> YARN-1166
>>> YARN-1041
>>> YARN-1566
>>>
>>> Plus 2 additional JIRAs committed since my email on this issue 2 days
>> ago:
>>>
>>> *YARN-1661
>>> *YARN-1689 (not sure if this JIRA is related in functionality to the
>>> previous ones but it is creating conflicts).
>>>
>>> I think we should hold on continuing work on top of something that is
>>> broken until the broken stuff is fixed.
>>>
>>> Quoting Arun, "Committers - Henceforth, please use extreme caution while
>>> committing to branch-2.3. Please commit *only* blockers to 2.3."
>>>
>>> YARN-1661 & YARN-1689 are not blockers.
>>>
>>> Unless there are objections, I'll revert all these JIRAs from branch-2.3
>>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>>>
>>> I'm inclined to revert them from branch-2 as well.
>>>
>>> Thoughts?
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
>>> wrote:
>>>
>>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
>>> was
>>>> causing some strange behavior in the Oozie unit tests:
>>>>
>>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
>>> unit
>>>> tests in a module. With YARN-1490 we saw that, regardless of test
>> order,
>>>> the last few tests would timeout waiting for an MR job to finish; on
>>> slower
>>>> machines, the entire test suite would timeout. Through some digging, I
>>>> found that we were getting a ton of "Connection refused" Exceptions on
>>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>>>>
>>>> After a bunch of investigation, I found that the problem went away once
>>>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
>>>> Even though this occurred in unit tests, it does make me concerned
>> that
>>> it
>>>> could indicate some bigger issue in a long-running real cluster (where
>>>> everything isn't running on the same machine) that we haven't seen yet.
>>>>
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
>>>> wrote:
>>>>
>>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
>> shortly.
>>>> Will
>>>>> pull it out of branch-2.3 if anyone objects.
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
>>> aagarwal@hortonworks.com
>>>>>> wrote:
>>>>>
>>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
>>>> aagarwal@hortonworks.com
>>>>>>> wrote:
>>>>>>
>>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>>>>>>>
>>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a
>> very
>>>> odd
>>>>>>>> ways
>>>>>>>> (to the point it seems un-deterministic).
>>>>>>>>
>>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
>>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
>>>>> reverts)
>>>>>>>> from Hadoop 2.3 branch before doing the release.
>>>>>>>>
>>>>>>>>
>>>>>>>> I've verified that after reverting those JIRAs things work fine
>>> with
>>>>>>>> unmanaged AMs.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
>>> acm@hortonworks.com
>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>>>>>>>>>
>>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
>>>>>> progress
>>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
>>>>>>>> objections?
>>>>>>>>>
>>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>>>>>>>>>
>>>>>>>>> Overall, I'll try get this out in next couple of days if we
>> can
>>>>> clear
>>>>>>>> the
>>>>>>>>> list.
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <
>> acm@hortonworks.com
>>>>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockerswe
>>>> are
>>>>>> now
>>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>>>>>>>>>>
>>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is
>>> helping
>>>>> out
>>>>>>>> with
>>>>>>>>> the YARN ones.
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual
>>> or
>>>>>>>> entity to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If
>>> the
>>>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure
>>> or
>>>>>>>>> forwarding of this communication is strictly prohibited. If
>> you
>>>> have
>>>>>>>>> received this communication in error, please contact the
>> sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Alejandro
>>>>>>
>>>>>> --
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>> to
>>>>>> which it is addressed and may contain information that is
>>> confidential,
>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>>>> of this message is not the intended recipient, you are hereby
>>> notified
>>>>> that
>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>> forwarding of this communication is strictly prohibited. If you
>> have
>>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>>> and delete it from your system. Thank You.
>>>
>>>
>>>
>>> --
>>> Alejandro
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
yep, the idea is to pull all of them out from branch2.3. things go back to normal then.
thanks
Alejandro
(phone typing)
> On Feb 6, 2014, at 17:39, Zhijie Shen <zs...@hortonworks.com> wrote:
>
> Recently I brought 4 JIRAs to branch-2.3, which are MAPREDUCE-5743, YARN-1628,
> YARN-1661 and YARN-1689. Recall that we mark test failure fixes as blockers
> for pior releases as closing to release, thus I brought to branch-2.3
> MAPREDUCE-5743
> and YARN-1628 that are the fixes for the test failure on 2.3.0, but didn't
> marked them as blockers. Please let me know if I should do that.
>
> YARN-1661 is a fix for exit log of DS AppMaster, otherwise the exit log of
> it will always be failure, which sounds a critical issue to me. Feel free
> to pull it out if any objects.
>
> YARN-1689 is brought to branch-2.3 as YARN-1493 is still in this branch. It
> fixes one bug caused by YARN-1493. Those should be included or excluded
> together upon the decision.
>
> Thanks,
> Zhijie
>
>
>> On Thu, Feb 6, 2014 at 5:15 PM, Sandy Ryza <sa...@cloudera.com> wrote:
>>
>> +1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a
>> problem caused by YARN-1493 I think we can revert it in branch-2.3 as well.
>>
>> I think we should leave them in branch-2 for now. We can revert if 2.4 is
>> imminent and they're holding it up, but hopefully the issues they caused
>> will be fixed by then.
>>
>> -Sandy
>>
>>
>> On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>
>>> Thanks Robert,
>>>
>>> All,
>>>
>>> So it seems that YARN-1493 and YARN-1490 are introducing serious
>>> regressions.
>>>
>>> I would propose to revert them and the follow up JIRAs from the 2.3
>> branch
>>> and keep working on them on trunk/branch-2 until the are stable (I would
>>> even prefer reverting them from branch-2 not to block a 2.4 if they are
>> not
>>> ready in time).
>>>
>>> As I've mentioned before, the list of JIRAs to revert were:
>>>
>>> YARN-1493
>>> YARN-1490
>>> YARN-1166
>>> YARN-1041
>>> YARN-1566
>>>
>>> Plus 2 additional JIRAs committed since my email on this issue 2 days
>> ago:
>>>
>>> *YARN-1661
>>> *YARN-1689 (not sure if this JIRA is related in functionality to the
>>> previous ones but it is creating conflicts).
>>>
>>> I think we should hold on continuing work on top of something that is
>>> broken until the broken stuff is fixed.
>>>
>>> Quoting Arun, "Committers - Henceforth, please use extreme caution while
>>> committing to branch-2.3. Please commit *only* blockers to 2.3."
>>>
>>> YARN-1661 & YARN-1689 are not blockers.
>>>
>>> Unless there are objections, I'll revert all these JIRAs from branch-2.3
>>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>>>
>>> I'm inclined to revert them from branch-2 as well.
>>>
>>> Thoughts?
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
>>> wrote:
>>>
>>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
>>> was
>>>> causing some strange behavior in the Oozie unit tests:
>>>>
>>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
>>> unit
>>>> tests in a module. With YARN-1490 we saw that, regardless of test
>> order,
>>>> the last few tests would timeout waiting for an MR job to finish; on
>>> slower
>>>> machines, the entire test suite would timeout. Through some digging, I
>>>> found that we were getting a ton of "Connection refused" Exceptions on
>>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>>>>
>>>> After a bunch of investigation, I found that the problem went away once
>>>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
>>>> Even though this occurred in unit tests, it does make me concerned
>> that
>>> it
>>>> could indicate some bigger issue in a long-running real cluster (where
>>>> everything isn't running on the same machine) that we haven't seen yet.
>>>>
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
>>>> wrote:
>>>>
>>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
>> shortly.
>>>> Will
>>>>> pull it out of branch-2.3 if anyone objects.
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
>>> aagarwal@hortonworks.com
>>>>>> wrote:
>>>>>
>>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
>>>> aagarwal@hortonworks.com
>>>>>>> wrote:
>>>>>>
>>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>>>>>>>
>>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a
>> very
>>>> odd
>>>>>>>> ways
>>>>>>>> (to the point it seems un-deterministic).
>>>>>>>>
>>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
>>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
>>>>> reverts)
>>>>>>>> from Hadoop 2.3 branch before doing the release.
>>>>>>>>
>>>>>>>>
>>>>>>>> I've verified that after reverting those JIRAs things work fine
>>> with
>>>>>>>> unmanaged AMs.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
>>> acm@hortonworks.com
>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>>>>>>>>>
>>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
>>>>>> progress
>>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
>>>>>>>> objections?
>>>>>>>>>
>>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>>>>>>>>>
>>>>>>>>> Overall, I'll try get this out in next couple of days if we
>> can
>>>>> clear
>>>>>>>> the
>>>>>>>>> list.
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <
>> acm@hortonworks.com
>>>>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockerswe
>>>> are
>>>>>> now
>>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>>>>>>>>>>
>>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is
>>> helping
>>>>> out
>>>>>>>> with
>>>>>>>>> the YARN ones.
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual
>>> or
>>>>>>>> entity to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If
>>> the
>>>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure
>>> or
>>>>>>>>> forwarding of this communication is strictly prohibited. If
>> you
>>>> have
>>>>>>>>> received this communication in error, please contact the
>> sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Alejandro
>>>>>>
>>>>>> --
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>> to
>>>>>> which it is addressed and may contain information that is
>>> confidential,
>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>>>> of this message is not the intended recipient, you are hereby
>>> notified
>>>>> that
>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>> forwarding of this communication is strictly prohibited. If you
>> have
>>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>>> and delete it from your system. Thank You.
>>>
>>>
>>>
>>> --
>>> Alejandro
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
yep, the idea is to pull all of them out from branch2.3. things go back to normal then.
thanks
Alejandro
(phone typing)
> On Feb 6, 2014, at 17:39, Zhijie Shen <zs...@hortonworks.com> wrote:
>
> Recently I brought 4 JIRAs to branch-2.3, which are MAPREDUCE-5743, YARN-1628,
> YARN-1661 and YARN-1689. Recall that we mark test failure fixes as blockers
> for pior releases as closing to release, thus I brought to branch-2.3
> MAPREDUCE-5743
> and YARN-1628 that are the fixes for the test failure on 2.3.0, but didn't
> marked them as blockers. Please let me know if I should do that.
>
> YARN-1661 is a fix for exit log of DS AppMaster, otherwise the exit log of
> it will always be failure, which sounds a critical issue to me. Feel free
> to pull it out if any objects.
>
> YARN-1689 is brought to branch-2.3 as YARN-1493 is still in this branch. It
> fixes one bug caused by YARN-1493. Those should be included or excluded
> together upon the decision.
>
> Thanks,
> Zhijie
>
>
>> On Thu, Feb 6, 2014 at 5:15 PM, Sandy Ryza <sa...@cloudera.com> wrote:
>>
>> +1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a
>> problem caused by YARN-1493 I think we can revert it in branch-2.3 as well.
>>
>> I think we should leave them in branch-2 for now. We can revert if 2.4 is
>> imminent and they're holding it up, but hopefully the issues they caused
>> will be fixed by then.
>>
>> -Sandy
>>
>>
>> On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>
>>> Thanks Robert,
>>>
>>> All,
>>>
>>> So it seems that YARN-1493 and YARN-1490 are introducing serious
>>> regressions.
>>>
>>> I would propose to revert them and the follow up JIRAs from the 2.3
>> branch
>>> and keep working on them on trunk/branch-2 until the are stable (I would
>>> even prefer reverting them from branch-2 not to block a 2.4 if they are
>> not
>>> ready in time).
>>>
>>> As I've mentioned before, the list of JIRAs to revert were:
>>>
>>> YARN-1493
>>> YARN-1490
>>> YARN-1166
>>> YARN-1041
>>> YARN-1566
>>>
>>> Plus 2 additional JIRAs committed since my email on this issue 2 days
>> ago:
>>>
>>> *YARN-1661
>>> *YARN-1689 (not sure if this JIRA is related in functionality to the
>>> previous ones but it is creating conflicts).
>>>
>>> I think we should hold on continuing work on top of something that is
>>> broken until the broken stuff is fixed.
>>>
>>> Quoting Arun, "Committers - Henceforth, please use extreme caution while
>>> committing to branch-2.3. Please commit *only* blockers to 2.3."
>>>
>>> YARN-1661 & YARN-1689 are not blockers.
>>>
>>> Unless there are objections, I'll revert all these JIRAs from branch-2.3
>>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>>>
>>> I'm inclined to revert them from branch-2 as well.
>>>
>>> Thoughts?
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
>>> wrote:
>>>
>>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
>>> was
>>>> causing some strange behavior in the Oozie unit tests:
>>>>
>>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
>>> unit
>>>> tests in a module. With YARN-1490 we saw that, regardless of test
>> order,
>>>> the last few tests would timeout waiting for an MR job to finish; on
>>> slower
>>>> machines, the entire test suite would timeout. Through some digging, I
>>>> found that we were getting a ton of "Connection refused" Exceptions on
>>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>>>>
>>>> After a bunch of investigation, I found that the problem went away once
>>>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
>>>> Even though this occurred in unit tests, it does make me concerned
>> that
>>> it
>>>> could indicate some bigger issue in a long-running real cluster (where
>>>> everything isn't running on the same machine) that we haven't seen yet.
>>>>
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
>>>> wrote:
>>>>
>>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
>> shortly.
>>>> Will
>>>>> pull it out of branch-2.3 if anyone objects.
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
>>> aagarwal@hortonworks.com
>>>>>> wrote:
>>>>>
>>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
>>>> aagarwal@hortonworks.com
>>>>>>> wrote:
>>>>>>
>>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>>>>>>>
>>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a
>> very
>>>> odd
>>>>>>>> ways
>>>>>>>> (to the point it seems un-deterministic).
>>>>>>>>
>>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
>>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
>>>>> reverts)
>>>>>>>> from Hadoop 2.3 branch before doing the release.
>>>>>>>>
>>>>>>>>
>>>>>>>> I've verified that after reverting those JIRAs things work fine
>>> with
>>>>>>>> unmanaged AMs.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
>>> acm@hortonworks.com
>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>>>>>>>>>
>>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
>>>>>> progress
>>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
>>>>>>>> objections?
>>>>>>>>>
>>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>>>>>>>>>
>>>>>>>>> Overall, I'll try get this out in next couple of days if we
>> can
>>>>> clear
>>>>>>>> the
>>>>>>>>> list.
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <
>> acm@hortonworks.com
>>>>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockerswe
>>>> are
>>>>>> now
>>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>>>>>>>>>>
>>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is
>>> helping
>>>>> out
>>>>>>>> with
>>>>>>>>> the YARN ones.
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual
>>> or
>>>>>>>> entity to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If
>>> the
>>>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure
>>> or
>>>>>>>>> forwarding of this communication is strictly prohibited. If
>> you
>>>> have
>>>>>>>>> received this communication in error, please contact the
>> sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Alejandro
>>>>>>
>>>>>> --
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>> to
>>>>>> which it is addressed and may contain information that is
>>> confidential,
>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>>>> of this message is not the intended recipient, you are hereby
>>> notified
>>>>> that
>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>> forwarding of this communication is strictly prohibited. If you
>> have
>>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>>> and delete it from your system. Thank You.
>>>
>>>
>>>
>>> --
>>> Alejandro
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
yep, the idea is to pull all of them out from branch2.3. things go back to normal then.
thanks
Alejandro
(phone typing)
> On Feb 6, 2014, at 17:39, Zhijie Shen <zs...@hortonworks.com> wrote:
>
> Recently I brought 4 JIRAs to branch-2.3, which are MAPREDUCE-5743, YARN-1628,
> YARN-1661 and YARN-1689. Recall that we mark test failure fixes as blockers
> for pior releases as closing to release, thus I brought to branch-2.3
> MAPREDUCE-5743
> and YARN-1628 that are the fixes for the test failure on 2.3.0, but didn't
> marked them as blockers. Please let me know if I should do that.
>
> YARN-1661 is a fix for exit log of DS AppMaster, otherwise the exit log of
> it will always be failure, which sounds a critical issue to me. Feel free
> to pull it out if any objects.
>
> YARN-1689 is brought to branch-2.3 as YARN-1493 is still in this branch. It
> fixes one bug caused by YARN-1493. Those should be included or excluded
> together upon the decision.
>
> Thanks,
> Zhijie
>
>
>> On Thu, Feb 6, 2014 at 5:15 PM, Sandy Ryza <sa...@cloudera.com> wrote:
>>
>> +1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a
>> problem caused by YARN-1493 I think we can revert it in branch-2.3 as well.
>>
>> I think we should leave them in branch-2 for now. We can revert if 2.4 is
>> imminent and they're holding it up, but hopefully the issues they caused
>> will be fixed by then.
>>
>> -Sandy
>>
>>
>> On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur <tucu@cloudera.com
>>> wrote:
>>
>>> Thanks Robert,
>>>
>>> All,
>>>
>>> So it seems that YARN-1493 and YARN-1490 are introducing serious
>>> regressions.
>>>
>>> I would propose to revert them and the follow up JIRAs from the 2.3
>> branch
>>> and keep working on them on trunk/branch-2 until the are stable (I would
>>> even prefer reverting them from branch-2 not to block a 2.4 if they are
>> not
>>> ready in time).
>>>
>>> As I've mentioned before, the list of JIRAs to revert were:
>>>
>>> YARN-1493
>>> YARN-1490
>>> YARN-1166
>>> YARN-1041
>>> YARN-1566
>>>
>>> Plus 2 additional JIRAs committed since my email on this issue 2 days
>> ago:
>>>
>>> *YARN-1661
>>> *YARN-1689 (not sure if this JIRA is related in functionality to the
>>> previous ones but it is creating conflicts).
>>>
>>> I think we should hold on continuing work on top of something that is
>>> broken until the broken stuff is fixed.
>>>
>>> Quoting Arun, "Committers - Henceforth, please use extreme caution while
>>> committing to branch-2.3. Please commit *only* blockers to 2.3."
>>>
>>> YARN-1661 & YARN-1689 are not blockers.
>>>
>>> Unless there are objections, I'll revert all these JIRAs from branch-2.3
>>> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>>>
>>> I'm inclined to revert them from branch-2 as well.
>>>
>>> Thoughts?
>>>
>>> Thanks.
>>>
>>>
>>> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
>>> wrote:
>>>
>>>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
>>> was
>>>> causing some strange behavior in the Oozie unit tests:
>>>>
>>>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all
>>> unit
>>>> tests in a module. With YARN-1490 we saw that, regardless of test
>> order,
>>>> the last few tests would timeout waiting for an MR job to finish; on
>>> slower
>>>> machines, the entire test suite would timeout. Through some digging, I
>>>> found that we were getting a ton of "Connection refused" Exceptions on
>>>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>>>>
>>>> After a bunch of investigation, I found that the problem went away once
>>>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
>>>> Even though this occurred in unit tests, it does make me concerned
>> that
>>> it
>>>> could indicate some bigger issue in a long-running real cluster (where
>>>> everything isn't running on the same machine) that we haven't seen yet.
>>>>
>>>>
>>>>
>>>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
>>>> wrote:
>>>>
>>>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
>> shortly.
>>>> Will
>>>>> pull it out of branch-2.3 if anyone objects.
>>>>>
>>>>>
>>>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
>>> aagarwal@hortonworks.com
>>>>>> wrote:
>>>>>
>>>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
>>>>>>
>>>>>>
>>>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
>>>> aagarwal@hortonworks.com
>>>>>>> wrote:
>>>>>>
>>>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>>>>>>>
>>>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
>>>> tucu@cloudera.com
>>>>>>> wrote:
>>>>>>>
>>>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a
>> very
>>>> odd
>>>>>>>> ways
>>>>>>>> (to the point it seems un-deterministic).
>>>>>>>>
>>>>>>>> I'd say eiher YARN-1577 is fixed or we revert
>>>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
>>>>> reverts)
>>>>>>>> from Hadoop 2.3 branch before doing the release.
>>>>>>>>
>>>>>>>>
>>>>>>>> I've verified that after reverting those JIRAs things work fine
>>> with
>>>>>>>> unmanaged AMs.
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
>>> acm@hortonworks.com
>>>>>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>>>>>>>>>
>>>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
>>>>>> progress
>>>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
>>>>>>>> objections?
>>>>>>>>>
>>>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>>>>>>>>>
>>>>>>>>> Overall, I'll try get this out in next couple of days if we
>> can
>>>>> clear
>>>>>>>> the
>>>>>>>>> list.
>>>>>>>>>
>>>>>>>>> thanks,
>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <
>> acm@hortonworks.com
>>>>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockerswe
>>>> are
>>>>>> now
>>>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>>>>>>>>>>
>>>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is
>>> helping
>>>>> out
>>>>>>>> with
>>>>>>>>> the YARN ones.
>>>>>>>>>>
>>>>>>>>>> thanks,
>>>>>>>>>> Arun
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Arun C. Murthy
>>>>>>>>> Hortonworks Inc.
>>>>>>>>> http://hortonworks.com/
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>>>> NOTICE: This message is intended for the use of the individual
>>> or
>>>>>>>> entity to
>>>>>>>>> which it is addressed and may contain information that is
>>>>>> confidential,
>>>>>>>>> privileged and exempt from disclosure under applicable law. If
>>> the
>>>>>>>> reader
>>>>>>>>> of this message is not the intended recipient, you are hereby
>>>>> notified
>>>>>>>> that
>>>>>>>>> any printing, copying, dissemination, distribution, disclosure
>>> or
>>>>>>>>> forwarding of this communication is strictly prohibited. If
>> you
>>>> have
>>>>>>>>> received this communication in error, please contact the
>> sender
>>>>>>>> immediately
>>>>>>>>> and delete it from your system. Thank You.
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Alejandro
>>>>>>
>>>>>> --
>>>>>> CONFIDENTIALITY NOTICE
>>>>>> NOTICE: This message is intended for the use of the individual or
>>>> entity
>>>>> to
>>>>>> which it is addressed and may contain information that is
>>> confidential,
>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>> reader
>>>>>> of this message is not the intended recipient, you are hereby
>>> notified
>>>>> that
>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>> forwarding of this communication is strictly prohibited. If you
>> have
>>>>>> received this communication in error, please contact the sender
>>>>> immediately
>>>>>> and delete it from your system. Thank You.
>>>
>>>
>>>
>>> --
>>> Alejandro
>
>
>
> --
> Zhijie Shen
> Hortonworks Inc.
> http://hortonworks.com/
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Zhijie Shen <zs...@hortonworks.com>.
Recently I brought 4 JIRAs to branch-2.3, which are MAPREDUCE-5743, YARN-1628,
YARN-1661 and YARN-1689. Recall that we mark test failure fixes as blockers
for pior releases as closing to release, thus I brought to branch-2.3
MAPREDUCE-5743
and YARN-1628 that are the fixes for the test failure on 2.3.0, but didn't
marked them as blockers. Please let me know if I should do that.
YARN-1661 is a fix for exit log of DS AppMaster, otherwise the exit log of
it will always be failure, which sounds a critical issue to me. Feel free
to pull it out if any objects.
YARN-1689 is brought to branch-2.3 as YARN-1493 is still in this branch. It
fixes one bug caused by YARN-1493. Those should be included or excluded
together upon the decision.
Thanks,
Zhijie
On Thu, Feb 6, 2014 at 5:15 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> +1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a
> problem caused by YARN-1493 I think we can revert it in branch-2.3 as well.
>
> I think we should leave them in branch-2 for now. We can revert if 2.4 is
> imminent and they're holding it up, but hopefully the issues they caused
> will be fixed by then.
>
> -Sandy
>
>
> On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Thanks Robert,
> >
> > All,
> >
> > So it seems that YARN-1493 and YARN-1490 are introducing serious
> > regressions.
> >
> > I would propose to revert them and the follow up JIRAs from the 2.3
> branch
> > and keep working on them on trunk/branch-2 until the are stable (I would
> > even prefer reverting them from branch-2 not to block a 2.4 if they are
> not
> > ready in time).
> >
> > As I've mentioned before, the list of JIRAs to revert were:
> >
> > YARN-1493
> > YARN-1490
> > YARN-1166
> > YARN-1041
> > YARN-1566
> >
> > Plus 2 additional JIRAs committed since my email on this issue 2 days
> ago:
> >
> > *YARN-1661
> > *YARN-1689 (not sure if this JIRA is related in functionality to the
> > previous ones but it is creating conflicts).
> >
> > I think we should hold on continuing work on top of something that is
> > broken until the broken stuff is fixed.
> >
> > Quoting Arun, "Committers - Henceforth, please use extreme caution while
> > committing to branch-2.3. Please commit *only* blockers to 2.3."
> >
> > YARN-1661 & YARN-1689 are not blockers.
> >
> > Unless there are objections, I'll revert all these JIRAs from branch-2.3
> > tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >
> > I'm inclined to revert them from branch-2 as well.
> >
> > Thoughts?
> >
> > Thanks.
> >
> >
> > On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> > wrote:
> >
> > > I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> > was
> > > causing some strange behavior in the Oozie unit tests:
> > >
> > > Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> > unit
> > > tests in a module. With YARN-1490 we saw that, regardless of test
> order,
> > > the last few tests would timeout waiting for an MR job to finish; on
> > slower
> > > machines, the entire test suite would timeout. Through some digging, I
> > > found that we were getting a ton of "Connection refused" Exceptions on
> > > LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> > >
> > > After a bunch of investigation, I found that the problem went away once
> > > YARN-1490 was removed. Though I couldn't figure out the exact problem.
> > > Even though this occurred in unit tests, it does make me concerned
> that
> > it
> > > could indicate some bigger issue in a long-running real cluster (where
> > > everything isn't running on the same machine) that we haven't seen yet.
> > >
> > >
> > >
> > > On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> > > wrote:
> > >
> > > > I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
> shortly.
> > > Will
> > > > pull it out of branch-2.3 if anyone objects.
> > > >
> > > >
> > > > On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> > aagarwal@hortonworks.com
> > > > >wrote:
> > > >
> > > > > Merged HADOOP-10273 to branch-2.3 as r1565456.
> > > > >
> > > > >
> > > > > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> > > aagarwal@hortonworks.com
> > > > > >wrote:
> > > > >
> > > > > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> > > > > >
> > > > > > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> > > tucu@cloudera.com
> > > > > >wrote:
> > > > > >
> > > > > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a
> very
> > > odd
> > > > > >> ways
> > > > > >> (to the point it seems un-deterministic).
> > > > > >>
> > > > > >> I'd say eiher YARN-1577 is fixed or we revert
> > > > > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> > > > reverts)
> > > > > >> from Hadoop 2.3 branch before doing the release.
> > > > > >>
> > > > > >>
> > > > > >> I've verified that after reverting those JIRAs things work fine
> > with
> > > > > >> unmanaged AMs.
> > > > > >>
> > > > > >> Thanks.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
> > acm@hortonworks.com
> > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> > > > > >> >
> > > > > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> > > > > progress
> > > > > >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> > > > > >> objections?
> > > > > >> >
> > > > > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> > > > > >> >
> > > > > >> > Overall, I'll try get this out in next couple of days if we
> can
> > > > clear
> > > > > >> the
> > > > > >> > list.
> > > > > >> >
> > > > > >> > thanks,
> > > > > >> > Arun
> > > > > >> >
> > > > > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockerswe
> > > are
> > > > > now
> > > > > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > > > > >> > >
> > > > > >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is
> > helping
> > > > out
> > > > > >> with
> > > > > >> > the YARN ones.
> > > > > >> > >
> > > > > >> > > thanks,
> > > > > >> > > Arun
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Arun C. Murthy
> > > > > >> > Hortonworks Inc.
> > > > > >> > http://hortonworks.com/
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > CONFIDENTIALITY NOTICE
> > > > > >> > NOTICE: This message is intended for the use of the individual
> > or
> > > > > >> entity to
> > > > > >> > which it is addressed and may contain information that is
> > > > > confidential,
> > > > > >> > privileged and exempt from disclosure under applicable law. If
> > the
> > > > > >> reader
> > > > > >> > of this message is not the intended recipient, you are hereby
> > > > notified
> > > > > >> that
> > > > > >> > any printing, copying, dissemination, distribution, disclosure
> > or
> > > > > >> > forwarding of this communication is strictly prohibited. If
> you
> > > have
> > > > > >> > received this communication in error, please contact the
> sender
> > > > > >> immediately
> > > > > >> > and delete it from your system. Thank You.
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Alejandro
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Zhijie Shen <zs...@hortonworks.com>.
Recently I brought 4 JIRAs to branch-2.3, which are MAPREDUCE-5743, YARN-1628,
YARN-1661 and YARN-1689. Recall that we mark test failure fixes as blockers
for pior releases as closing to release, thus I brought to branch-2.3
MAPREDUCE-5743
and YARN-1628 that are the fixes for the test failure on 2.3.0, but didn't
marked them as blockers. Please let me know if I should do that.
YARN-1661 is a fix for exit log of DS AppMaster, otherwise the exit log of
it will always be failure, which sounds a critical issue to me. Feel free
to pull it out if any objects.
YARN-1689 is brought to branch-2.3 as YARN-1493 is still in this branch. It
fixes one bug caused by YARN-1493. Those should be included or excluded
together upon the decision.
Thanks,
Zhijie
On Thu, Feb 6, 2014 at 5:15 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> +1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a
> problem caused by YARN-1493 I think we can revert it in branch-2.3 as well.
>
> I think we should leave them in branch-2 for now. We can revert if 2.4 is
> imminent and they're holding it up, but hopefully the issues they caused
> will be fixed by then.
>
> -Sandy
>
>
> On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Thanks Robert,
> >
> > All,
> >
> > So it seems that YARN-1493 and YARN-1490 are introducing serious
> > regressions.
> >
> > I would propose to revert them and the follow up JIRAs from the 2.3
> branch
> > and keep working on them on trunk/branch-2 until the are stable (I would
> > even prefer reverting them from branch-2 not to block a 2.4 if they are
> not
> > ready in time).
> >
> > As I've mentioned before, the list of JIRAs to revert were:
> >
> > YARN-1493
> > YARN-1490
> > YARN-1166
> > YARN-1041
> > YARN-1566
> >
> > Plus 2 additional JIRAs committed since my email on this issue 2 days
> ago:
> >
> > *YARN-1661
> > *YARN-1689 (not sure if this JIRA is related in functionality to the
> > previous ones but it is creating conflicts).
> >
> > I think we should hold on continuing work on top of something that is
> > broken until the broken stuff is fixed.
> >
> > Quoting Arun, "Committers - Henceforth, please use extreme caution while
> > committing to branch-2.3. Please commit *only* blockers to 2.3."
> >
> > YARN-1661 & YARN-1689 are not blockers.
> >
> > Unless there are objections, I'll revert all these JIRAs from branch-2.3
> > tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >
> > I'm inclined to revert them from branch-2 as well.
> >
> > Thoughts?
> >
> > Thanks.
> >
> >
> > On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> > wrote:
> >
> > > I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> > was
> > > causing some strange behavior in the Oozie unit tests:
> > >
> > > Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> > unit
> > > tests in a module. With YARN-1490 we saw that, regardless of test
> order,
> > > the last few tests would timeout waiting for an MR job to finish; on
> > slower
> > > machines, the entire test suite would timeout. Through some digging, I
> > > found that we were getting a ton of "Connection refused" Exceptions on
> > > LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> > >
> > > After a bunch of investigation, I found that the problem went away once
> > > YARN-1490 was removed. Though I couldn't figure out the exact problem.
> > > Even though this occurred in unit tests, it does make me concerned
> that
> > it
> > > could indicate some bigger issue in a long-running real cluster (where
> > > everything isn't running on the same machine) that we haven't seen yet.
> > >
> > >
> > >
> > > On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> > > wrote:
> > >
> > > > I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
> shortly.
> > > Will
> > > > pull it out of branch-2.3 if anyone objects.
> > > >
> > > >
> > > > On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> > aagarwal@hortonworks.com
> > > > >wrote:
> > > >
> > > > > Merged HADOOP-10273 to branch-2.3 as r1565456.
> > > > >
> > > > >
> > > > > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> > > aagarwal@hortonworks.com
> > > > > >wrote:
> > > > >
> > > > > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> > > > > >
> > > > > > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> > > tucu@cloudera.com
> > > > > >wrote:
> > > > > >
> > > > > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a
> very
> > > odd
> > > > > >> ways
> > > > > >> (to the point it seems un-deterministic).
> > > > > >>
> > > > > >> I'd say eiher YARN-1577 is fixed or we revert
> > > > > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> > > > reverts)
> > > > > >> from Hadoop 2.3 branch before doing the release.
> > > > > >>
> > > > > >>
> > > > > >> I've verified that after reverting those JIRAs things work fine
> > with
> > > > > >> unmanaged AMs.
> > > > > >>
> > > > > >> Thanks.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
> > acm@hortonworks.com
> > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> > > > > >> >
> > > > > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> > > > > progress
> > > > > >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> > > > > >> objections?
> > > > > >> >
> > > > > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> > > > > >> >
> > > > > >> > Overall, I'll try get this out in next couple of days if we
> can
> > > > clear
> > > > > >> the
> > > > > >> > list.
> > > > > >> >
> > > > > >> > thanks,
> > > > > >> > Arun
> > > > > >> >
> > > > > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockerswe
> > > are
> > > > > now
> > > > > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > > > > >> > >
> > > > > >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is
> > helping
> > > > out
> > > > > >> with
> > > > > >> > the YARN ones.
> > > > > >> > >
> > > > > >> > > thanks,
> > > > > >> > > Arun
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Arun C. Murthy
> > > > > >> > Hortonworks Inc.
> > > > > >> > http://hortonworks.com/
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > CONFIDENTIALITY NOTICE
> > > > > >> > NOTICE: This message is intended for the use of the individual
> > or
> > > > > >> entity to
> > > > > >> > which it is addressed and may contain information that is
> > > > > confidential,
> > > > > >> > privileged and exempt from disclosure under applicable law. If
> > the
> > > > > >> reader
> > > > > >> > of this message is not the intended recipient, you are hereby
> > > > notified
> > > > > >> that
> > > > > >> > any printing, copying, dissemination, distribution, disclosure
> > or
> > > > > >> > forwarding of this communication is strictly prohibited. If
> you
> > > have
> > > > > >> > received this communication in error, please contact the
> sender
> > > > > >> immediately
> > > > > >> > and delete it from your system. Thank You.
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Alejandro
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Zhijie Shen <zs...@hortonworks.com>.
Recently I brought 4 JIRAs to branch-2.3, which are MAPREDUCE-5743, YARN-1628,
YARN-1661 and YARN-1689. Recall that we mark test failure fixes as blockers
for pior releases as closing to release, thus I brought to branch-2.3
MAPREDUCE-5743
and YARN-1628 that are the fixes for the test failure on 2.3.0, but didn't
marked them as blockers. Please let me know if I should do that.
YARN-1661 is a fix for exit log of DS AppMaster, otherwise the exit log of
it will always be failure, which sounds a critical issue to me. Feel free
to pull it out if any objects.
YARN-1689 is brought to branch-2.3 as YARN-1493 is still in this branch. It
fixes one bug caused by YARN-1493. Those should be included or excluded
together upon the decision.
Thanks,
Zhijie
On Thu, Feb 6, 2014 at 5:15 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> +1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a
> problem caused by YARN-1493 I think we can revert it in branch-2.3 as well.
>
> I think we should leave them in branch-2 for now. We can revert if 2.4 is
> imminent and they're holding it up, but hopefully the issues they caused
> will be fixed by then.
>
> -Sandy
>
>
> On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Thanks Robert,
> >
> > All,
> >
> > So it seems that YARN-1493 and YARN-1490 are introducing serious
> > regressions.
> >
> > I would propose to revert them and the follow up JIRAs from the 2.3
> branch
> > and keep working on them on trunk/branch-2 until the are stable (I would
> > even prefer reverting them from branch-2 not to block a 2.4 if they are
> not
> > ready in time).
> >
> > As I've mentioned before, the list of JIRAs to revert were:
> >
> > YARN-1493
> > YARN-1490
> > YARN-1166
> > YARN-1041
> > YARN-1566
> >
> > Plus 2 additional JIRAs committed since my email on this issue 2 days
> ago:
> >
> > *YARN-1661
> > *YARN-1689 (not sure if this JIRA is related in functionality to the
> > previous ones but it is creating conflicts).
> >
> > I think we should hold on continuing work on top of something that is
> > broken until the broken stuff is fixed.
> >
> > Quoting Arun, "Committers - Henceforth, please use extreme caution while
> > committing to branch-2.3. Please commit *only* blockers to 2.3."
> >
> > YARN-1661 & YARN-1689 are not blockers.
> >
> > Unless there are objections, I'll revert all these JIRAs from branch-2.3
> > tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >
> > I'm inclined to revert them from branch-2 as well.
> >
> > Thoughts?
> >
> > Thanks.
> >
> >
> > On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> > wrote:
> >
> > > I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> > was
> > > causing some strange behavior in the Oozie unit tests:
> > >
> > > Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> > unit
> > > tests in a module. With YARN-1490 we saw that, regardless of test
> order,
> > > the last few tests would timeout waiting for an MR job to finish; on
> > slower
> > > machines, the entire test suite would timeout. Through some digging, I
> > > found that we were getting a ton of "Connection refused" Exceptions on
> > > LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> > >
> > > After a bunch of investigation, I found that the problem went away once
> > > YARN-1490 was removed. Though I couldn't figure out the exact problem.
> > > Even though this occurred in unit tests, it does make me concerned
> that
> > it
> > > could indicate some bigger issue in a long-running real cluster (where
> > > everything isn't running on the same machine) that we haven't seen yet.
> > >
> > >
> > >
> > > On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> > > wrote:
> > >
> > > > I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
> shortly.
> > > Will
> > > > pull it out of branch-2.3 if anyone objects.
> > > >
> > > >
> > > > On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> > aagarwal@hortonworks.com
> > > > >wrote:
> > > >
> > > > > Merged HADOOP-10273 to branch-2.3 as r1565456.
> > > > >
> > > > >
> > > > > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> > > aagarwal@hortonworks.com
> > > > > >wrote:
> > > > >
> > > > > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> > > > > >
> > > > > > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> > > tucu@cloudera.com
> > > > > >wrote:
> > > > > >
> > > > > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a
> very
> > > odd
> > > > > >> ways
> > > > > >> (to the point it seems un-deterministic).
> > > > > >>
> > > > > >> I'd say eiher YARN-1577 is fixed or we revert
> > > > > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> > > > reverts)
> > > > > >> from Hadoop 2.3 branch before doing the release.
> > > > > >>
> > > > > >>
> > > > > >> I've verified that after reverting those JIRAs things work fine
> > with
> > > > > >> unmanaged AMs.
> > > > > >>
> > > > > >> Thanks.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
> > acm@hortonworks.com
> > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> > > > > >> >
> > > > > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> > > > > progress
> > > > > >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> > > > > >> objections?
> > > > > >> >
> > > > > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> > > > > >> >
> > > > > >> > Overall, I'll try get this out in next couple of days if we
> can
> > > > clear
> > > > > >> the
> > > > > >> > list.
> > > > > >> >
> > > > > >> > thanks,
> > > > > >> > Arun
> > > > > >> >
> > > > > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockerswe
> > > are
> > > > > now
> > > > > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > > > > >> > >
> > > > > >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is
> > helping
> > > > out
> > > > > >> with
> > > > > >> > the YARN ones.
> > > > > >> > >
> > > > > >> > > thanks,
> > > > > >> > > Arun
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Arun C. Murthy
> > > > > >> > Hortonworks Inc.
> > > > > >> > http://hortonworks.com/
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > CONFIDENTIALITY NOTICE
> > > > > >> > NOTICE: This message is intended for the use of the individual
> > or
> > > > > >> entity to
> > > > > >> > which it is addressed and may contain information that is
> > > > > confidential,
> > > > > >> > privileged and exempt from disclosure under applicable law. If
> > the
> > > > > >> reader
> > > > > >> > of this message is not the intended recipient, you are hereby
> > > > notified
> > > > > >> that
> > > > > >> > any printing, copying, dissemination, distribution, disclosure
> > or
> > > > > >> > forwarding of this communication is strictly prohibited. If
> you
> > > have
> > > > > >> > received this communication in error, please contact the
> sender
> > > > > >> immediately
> > > > > >> > and delete it from your system. Thank You.
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Alejandro
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Zhijie Shen <zs...@hortonworks.com>.
Recently I brought 4 JIRAs to branch-2.3, which are MAPREDUCE-5743, YARN-1628,
YARN-1661 and YARN-1689. Recall that we mark test failure fixes as blockers
for pior releases as closing to release, thus I brought to branch-2.3
MAPREDUCE-5743
and YARN-1628 that are the fixes for the test failure on 2.3.0, but didn't
marked them as blockers. Please let me know if I should do that.
YARN-1661 is a fix for exit log of DS AppMaster, otherwise the exit log of
it will always be failure, which sounds a critical issue to me. Feel free
to pull it out if any objects.
YARN-1689 is brought to branch-2.3 as YARN-1493 is still in this branch. It
fixes one bug caused by YARN-1493. Those should be included or excluded
together upon the decision.
Thanks,
Zhijie
On Thu, Feb 6, 2014 at 5:15 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> +1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a
> problem caused by YARN-1493 I think we can revert it in branch-2.3 as well.
>
> I think we should leave them in branch-2 for now. We can revert if 2.4 is
> imminent and they're holding it up, but hopefully the issues they caused
> will be fixed by then.
>
> -Sandy
>
>
> On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
>
> > Thanks Robert,
> >
> > All,
> >
> > So it seems that YARN-1493 and YARN-1490 are introducing serious
> > regressions.
> >
> > I would propose to revert them and the follow up JIRAs from the 2.3
> branch
> > and keep working on them on trunk/branch-2 until the are stable (I would
> > even prefer reverting them from branch-2 not to block a 2.4 if they are
> not
> > ready in time).
> >
> > As I've mentioned before, the list of JIRAs to revert were:
> >
> > YARN-1493
> > YARN-1490
> > YARN-1166
> > YARN-1041
> > YARN-1566
> >
> > Plus 2 additional JIRAs committed since my email on this issue 2 days
> ago:
> >
> > *YARN-1661
> > *YARN-1689 (not sure if this JIRA is related in functionality to the
> > previous ones but it is creating conflicts).
> >
> > I think we should hold on continuing work on top of something that is
> > broken until the broken stuff is fixed.
> >
> > Quoting Arun, "Committers - Henceforth, please use extreme caution while
> > committing to branch-2.3. Please commit *only* blockers to 2.3."
> >
> > YARN-1661 & YARN-1689 are not blockers.
> >
> > Unless there are objections, I'll revert all these JIRAs from branch-2.3
> > tomorrow around noon and I'll update fixedVersion in the JIRAs.
> >
> > I'm inclined to revert them from branch-2 as well.
> >
> > Thoughts?
> >
> > Thanks.
> >
> >
> > On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> > wrote:
> >
> > > I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> > was
> > > causing some strange behavior in the Oozie unit tests:
> > >
> > > Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> > unit
> > > tests in a module. With YARN-1490 we saw that, regardless of test
> order,
> > > the last few tests would timeout waiting for an MR job to finish; on
> > slower
> > > machines, the entire test suite would timeout. Through some digging, I
> > > found that we were getting a ton of "Connection refused" Exceptions on
> > > LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> > >
> > > After a bunch of investigation, I found that the problem went away once
> > > YARN-1490 was removed. Though I couldn't figure out the exact problem.
> > > Even though this occurred in unit tests, it does make me concerned
> that
> > it
> > > could indicate some bigger issue in a long-running real cluster (where
> > > everything isn't running on the same machine) that we haven't seen yet.
> > >
> > >
> > >
> > > On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> > > wrote:
> > >
> > > > I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it
> shortly.
> > > Will
> > > > pull it out of branch-2.3 if anyone objects.
> > > >
> > > >
> > > > On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> > aagarwal@hortonworks.com
> > > > >wrote:
> > > >
> > > > > Merged HADOOP-10273 to branch-2.3 as r1565456.
> > > > >
> > > > >
> > > > > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> > > aagarwal@hortonworks.com
> > > > > >wrote:
> > > > >
> > > > > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> > > > > >
> > > > > > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> > > tucu@cloudera.com
> > > > > >wrote:
> > > > > >
> > > > > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a
> very
> > > odd
> > > > > >> ways
> > > > > >> (to the point it seems un-deterministic).
> > > > > >>
> > > > > >> I'd say eiher YARN-1577 is fixed or we revert
> > > > > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> > > > reverts)
> > > > > >> from Hadoop 2.3 branch before doing the release.
> > > > > >>
> > > > > >>
> > > > > >> I've verified that after reverting those JIRAs things work fine
> > with
> > > > > >> unmanaged AMs.
> > > > > >>
> > > > > >> Thanks.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
> > acm@hortonworks.com
> > > >
> > > > > >> wrote:
> > > > > >>
> > > > > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> > > > > >> >
> > > > > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> > > > > progress
> > > > > >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> > > > > >> objections?
> > > > > >> >
> > > > > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> > > > > >> >
> > > > > >> > Overall, I'll try get this out in next couple of days if we
> can
> > > > clear
> > > > > >> the
> > > > > >> > list.
> > > > > >> >
> > > > > >> > thanks,
> > > > > >> > Arun
> > > > > >> >
> > > > > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > > > wrote:
> > > > > >> >
> > > > > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockerswe
> > > are
> > > > > now
> > > > > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > > > > >> > >
> > > > > >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is
> > helping
> > > > out
> > > > > >> with
> > > > > >> > the YARN ones.
> > > > > >> > >
> > > > > >> > > thanks,
> > > > > >> > > Arun
> > > > > >> > >
> > > > > >> > >
> > > > > >> > >
> > > > > >> >
> > > > > >> > --
> > > > > >> > Arun C. Murthy
> > > > > >> > Hortonworks Inc.
> > > > > >> > http://hortonworks.com/
> > > > > >> >
> > > > > >> >
> > > > > >> >
> > > > > >> > --
> > > > > >> > CONFIDENTIALITY NOTICE
> > > > > >> > NOTICE: This message is intended for the use of the individual
> > or
> > > > > >> entity to
> > > > > >> > which it is addressed and may contain information that is
> > > > > confidential,
> > > > > >> > privileged and exempt from disclosure under applicable law. If
> > the
> > > > > >> reader
> > > > > >> > of this message is not the intended recipient, you are hereby
> > > > notified
> > > > > >> that
> > > > > >> > any printing, copying, dissemination, distribution, disclosure
> > or
> > > > > >> > forwarding of this communication is strictly prohibited. If
> you
> > > have
> > > > > >> > received this communication in error, please contact the
> sender
> > > > > >> immediately
> > > > > >> > and delete it from your system. Thank You.
> > > > > >> >
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> --
> > > > > >> Alejandro
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > > > --
> > > > > CONFIDENTIALITY NOTICE
> > > > > NOTICE: This message is intended for the use of the individual or
> > > entity
> > > > to
> > > > > which it is addressed and may contain information that is
> > confidential,
> > > > > privileged and exempt from disclosure under applicable law. If the
> > > reader
> > > > > of this message is not the intended recipient, you are hereby
> > notified
> > > > that
> > > > > any printing, copying, dissemination, distribution, disclosure or
> > > > > forwarding of this communication is strictly prohibited. If you
> have
> > > > > received this communication in error, please contact the sender
> > > > immediately
> > > > > and delete it from your system. Thank You.
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > Alejandro
> >
>
--
Zhijie Shen
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Sandy Ryza <sa...@cloudera.com>.
+1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a
problem caused by YARN-1493 I think we can revert it in branch-2.3 as well.
I think we should leave them in branch-2 for now. We can revert if 2.4 is
imminent and they're holding it up, but hopefully the issues they caused
will be fixed by then.
-Sandy
On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Thanks Robert,
>
> All,
>
> So it seems that YARN-1493 and YARN-1490 are introducing serious
> regressions.
>
> I would propose to revert them and the follow up JIRAs from the 2.3 branch
> and keep working on them on trunk/branch-2 until the are stable (I would
> even prefer reverting them from branch-2 not to block a 2.4 if they are not
> ready in time).
>
> As I've mentioned before, the list of JIRAs to revert were:
>
> YARN-1493
> YARN-1490
> YARN-1166
> YARN-1041
> YARN-1566
>
> Plus 2 additional JIRAs committed since my email on this issue 2 days ago:
>
> *YARN-1661
> *YARN-1689 (not sure if this JIRA is related in functionality to the
> previous ones but it is creating conflicts).
>
> I think we should hold on continuing work on top of something that is
> broken until the broken stuff is fixed.
>
> Quoting Arun, "Committers - Henceforth, please use extreme caution while
> committing to branch-2.3. Please commit *only* blockers to 2.3."
>
> YARN-1661 & YARN-1689 are not blockers.
>
> Unless there are objections, I'll revert all these JIRAs from branch-2.3
> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>
> I'm inclined to revert them from branch-2 as well.
>
> Thoughts?
>
> Thanks.
>
>
> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
> > I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> was
> > causing some strange behavior in the Oozie unit tests:
> >
> > Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> unit
> > tests in a module. With YARN-1490 we saw that, regardless of test order,
> > the last few tests would timeout waiting for an MR job to finish; on
> slower
> > machines, the entire test suite would timeout. Through some digging, I
> > found that we were getting a ton of "Connection refused" Exceptions on
> > LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> >
> > After a bunch of investigation, I found that the problem went away once
> > YARN-1490 was removed. Though I couldn't figure out the exact problem.
> > Even though this occurred in unit tests, it does make me concerned that
> it
> > could indicate some bigger issue in a long-running real cluster (where
> > everything isn't running on the same machine) that we haven't seen yet.
> >
> >
> >
> > On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> > wrote:
> >
> > > I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
> > Will
> > > pull it out of branch-2.3 if anyone objects.
> > >
> > >
> > > On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> aagarwal@hortonworks.com
> > > >wrote:
> > >
> > > > Merged HADOOP-10273 to branch-2.3 as r1565456.
> > > >
> > > >
> > > > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> > aagarwal@hortonworks.com
> > > > >wrote:
> > > >
> > > > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> > > > >
> > > > > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> > > > >
> > > > >
> > > > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > > >wrote:
> > > > >
> > > > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> > odd
> > > > >> ways
> > > > >> (to the point it seems un-deterministic).
> > > > >>
> > > > >> I'd say eiher YARN-1577 is fixed or we revert
> > > > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> > > reverts)
> > > > >> from Hadoop 2.3 branch before doing the release.
> > > > >>
> > > > >>
> > > > >> I've verified that after reverting those JIRAs things work fine
> with
> > > > >> unmanaged AMs.
> > > > >>
> > > > >> Thanks.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > > >> wrote:
> > > > >>
> > > > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> > > > >> >
> > > > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> > > > progress
> > > > >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> > > > >> objections?
> > > > >> >
> > > > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> > > > >> >
> > > > >> > Overall, I'll try get this out in next couple of days if we can
> > > clear
> > > > >> the
> > > > >> > list.
> > > > >> >
> > > > >> > thanks,
> > > > >> > Arun
> > > > >> >
> > > > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <acm@hortonworks.com
> >
> > > > wrote:
> > > > >> >
> > > > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> > are
> > > > now
> > > > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > > > >> > >
> > > > >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is
> helping
> > > out
> > > > >> with
> > > > >> > the YARN ones.
> > > > >> > >
> > > > >> > > thanks,
> > > > >> > > Arun
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> > --
> > > > >> > Arun C. Murthy
> > > > >> > Hortonworks Inc.
> > > > >> > http://hortonworks.com/
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > CONFIDENTIALITY NOTICE
> > > > >> > NOTICE: This message is intended for the use of the individual
> or
> > > > >> entity to
> > > > >> > which it is addressed and may contain information that is
> > > > confidential,
> > > > >> > privileged and exempt from disclosure under applicable law. If
> the
> > > > >> reader
> > > > >> > of this message is not the intended recipient, you are hereby
> > > notified
> > > > >> that
> > > > >> > any printing, copying, dissemination, distribution, disclosure
> or
> > > > >> > forwarding of this communication is strictly prohibited. If you
> > have
> > > > >> > received this communication in error, please contact the sender
> > > > >> immediately
> > > > >> > and delete it from your system. Thank You.
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Alejandro
> > > > >>
> > > > >
> > > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Re-swizzle 2.3
Posted by Sandy Ryza <sa...@cloudera.com>.
+1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a
problem caused by YARN-1493 I think we can revert it in branch-2.3 as well.
I think we should leave them in branch-2 for now. We can revert if 2.4 is
imminent and they're holding it up, but hopefully the issues they caused
will be fixed by then.
-Sandy
On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Thanks Robert,
>
> All,
>
> So it seems that YARN-1493 and YARN-1490 are introducing serious
> regressions.
>
> I would propose to revert them and the follow up JIRAs from the 2.3 branch
> and keep working on them on trunk/branch-2 until the are stable (I would
> even prefer reverting them from branch-2 not to block a 2.4 if they are not
> ready in time).
>
> As I've mentioned before, the list of JIRAs to revert were:
>
> YARN-1493
> YARN-1490
> YARN-1166
> YARN-1041
> YARN-1566
>
> Plus 2 additional JIRAs committed since my email on this issue 2 days ago:
>
> *YARN-1661
> *YARN-1689 (not sure if this JIRA is related in functionality to the
> previous ones but it is creating conflicts).
>
> I think we should hold on continuing work on top of something that is
> broken until the broken stuff is fixed.
>
> Quoting Arun, "Committers - Henceforth, please use extreme caution while
> committing to branch-2.3. Please commit *only* blockers to 2.3."
>
> YARN-1661 & YARN-1689 are not blockers.
>
> Unless there are objections, I'll revert all these JIRAs from branch-2.3
> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>
> I'm inclined to revert them from branch-2 as well.
>
> Thoughts?
>
> Thanks.
>
>
> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
> > I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> was
> > causing some strange behavior in the Oozie unit tests:
> >
> > Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> unit
> > tests in a module. With YARN-1490 we saw that, regardless of test order,
> > the last few tests would timeout waiting for an MR job to finish; on
> slower
> > machines, the entire test suite would timeout. Through some digging, I
> > found that we were getting a ton of "Connection refused" Exceptions on
> > LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> >
> > After a bunch of investigation, I found that the problem went away once
> > YARN-1490 was removed. Though I couldn't figure out the exact problem.
> > Even though this occurred in unit tests, it does make me concerned that
> it
> > could indicate some bigger issue in a long-running real cluster (where
> > everything isn't running on the same machine) that we haven't seen yet.
> >
> >
> >
> > On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> > wrote:
> >
> > > I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
> > Will
> > > pull it out of branch-2.3 if anyone objects.
> > >
> > >
> > > On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> aagarwal@hortonworks.com
> > > >wrote:
> > >
> > > > Merged HADOOP-10273 to branch-2.3 as r1565456.
> > > >
> > > >
> > > > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> > aagarwal@hortonworks.com
> > > > >wrote:
> > > >
> > > > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> > > > >
> > > > > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> > > > >
> > > > >
> > > > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > > >wrote:
> > > > >
> > > > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> > odd
> > > > >> ways
> > > > >> (to the point it seems un-deterministic).
> > > > >>
> > > > >> I'd say eiher YARN-1577 is fixed or we revert
> > > > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> > > reverts)
> > > > >> from Hadoop 2.3 branch before doing the release.
> > > > >>
> > > > >>
> > > > >> I've verified that after reverting those JIRAs things work fine
> with
> > > > >> unmanaged AMs.
> > > > >>
> > > > >> Thanks.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > > >> wrote:
> > > > >>
> > > > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> > > > >> >
> > > > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> > > > progress
> > > > >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> > > > >> objections?
> > > > >> >
> > > > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> > > > >> >
> > > > >> > Overall, I'll try get this out in next couple of days if we can
> > > clear
> > > > >> the
> > > > >> > list.
> > > > >> >
> > > > >> > thanks,
> > > > >> > Arun
> > > > >> >
> > > > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <acm@hortonworks.com
> >
> > > > wrote:
> > > > >> >
> > > > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> > are
> > > > now
> > > > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > > > >> > >
> > > > >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is
> helping
> > > out
> > > > >> with
> > > > >> > the YARN ones.
> > > > >> > >
> > > > >> > > thanks,
> > > > >> > > Arun
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> > --
> > > > >> > Arun C. Murthy
> > > > >> > Hortonworks Inc.
> > > > >> > http://hortonworks.com/
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > CONFIDENTIALITY NOTICE
> > > > >> > NOTICE: This message is intended for the use of the individual
> or
> > > > >> entity to
> > > > >> > which it is addressed and may contain information that is
> > > > confidential,
> > > > >> > privileged and exempt from disclosure under applicable law. If
> the
> > > > >> reader
> > > > >> > of this message is not the intended recipient, you are hereby
> > > notified
> > > > >> that
> > > > >> > any printing, copying, dissemination, distribution, disclosure
> or
> > > > >> > forwarding of this communication is strictly prohibited. If you
> > have
> > > > >> > received this communication in error, please contact the sender
> > > > >> immediately
> > > > >> > and delete it from your system. Thank You.
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Alejandro
> > > > >>
> > > > >
> > > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Re-swizzle 2.3
Posted by Sandy Ryza <sa...@cloudera.com>.
+1 to reverting those JIRAs from branch-2.3. As YARN-1689 is fixing a
problem caused by YARN-1493 I think we can revert it in branch-2.3 as well.
I think we should leave them in branch-2 for now. We can revert if 2.4 is
imminent and they're holding it up, but hopefully the issues they caused
will be fixed by then.
-Sandy
On Thu, Feb 6, 2014 at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> Thanks Robert,
>
> All,
>
> So it seems that YARN-1493 and YARN-1490 are introducing serious
> regressions.
>
> I would propose to revert them and the follow up JIRAs from the 2.3 branch
> and keep working on them on trunk/branch-2 until the are stable (I would
> even prefer reverting them from branch-2 not to block a 2.4 if they are not
> ready in time).
>
> As I've mentioned before, the list of JIRAs to revert were:
>
> YARN-1493
> YARN-1490
> YARN-1166
> YARN-1041
> YARN-1566
>
> Plus 2 additional JIRAs committed since my email on this issue 2 days ago:
>
> *YARN-1661
> *YARN-1689 (not sure if this JIRA is related in functionality to the
> previous ones but it is creating conflicts).
>
> I think we should hold on continuing work on top of something that is
> broken until the broken stuff is fixed.
>
> Quoting Arun, "Committers - Henceforth, please use extreme caution while
> committing to branch-2.3. Please commit *only* blockers to 2.3."
>
> YARN-1661 & YARN-1689 are not blockers.
>
> Unless there are objections, I'll revert all these JIRAs from branch-2.3
> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>
> I'm inclined to revert them from branch-2 as well.
>
> Thoughts?
>
> Thanks.
>
>
> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com>
> wrote:
>
> > I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it
> was
> > causing some strange behavior in the Oozie unit tests:
> >
> > Basically, we use a single MiniMRCluster and MiniDFSCluster across all
> unit
> > tests in a module. With YARN-1490 we saw that, regardless of test order,
> > the last few tests would timeout waiting for an MR job to finish; on
> slower
> > machines, the entire test suite would timeout. Through some digging, I
> > found that we were getting a ton of "Connection refused" Exceptions on
> > LeaseRenewer talking to the NN and a few on the AM talking to the RM.
> >
> > After a bunch of investigation, I found that the problem went away once
> > YARN-1490 was removed. Though I couldn't figure out the exact problem.
> > Even though this occurred in unit tests, it does make me concerned that
> it
> > could indicate some bigger issue in a long-running real cluster (where
> > everything isn't running on the same machine) that we haven't seen yet.
> >
> >
> >
> > On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> > wrote:
> >
> > > I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
> > Will
> > > pull it out of branch-2.3 if anyone objects.
> > >
> > >
> > > On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <
> aagarwal@hortonworks.com
> > > >wrote:
> > >
> > > > Merged HADOOP-10273 to branch-2.3 as r1565456.
> > > >
> > > >
> > > > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> > aagarwal@hortonworks.com
> > > > >wrote:
> > > >
> > > > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> > > > >
> > > > > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> > > > >
> > > > >
> > > > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> > tucu@cloudera.com
> > > > >wrote:
> > > > >
> > > > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> > odd
> > > > >> ways
> > > > >> (to the point it seems un-deterministic).
> > > > >>
> > > > >> I'd say eiher YARN-1577 is fixed or we revert
> > > > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> > > reverts)
> > > > >> from Hadoop 2.3 branch before doing the release.
> > > > >>
> > > > >>
> > > > >> I've verified that after reverting those JIRAs things work fine
> with
> > > > >> unmanaged AMs.
> > > > >>
> > > > >> Thanks.
> > > > >>
> > > > >>
> > > > >>
> > > > >>
> > > > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <
> acm@hortonworks.com
> > >
> > > > >> wrote:
> > > > >>
> > > > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> > > > >> >
> > > > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> > > > progress
> > > > >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> > > > >> objections?
> > > > >> >
> > > > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> > > > >> >
> > > > >> > Overall, I'll try get this out in next couple of days if we can
> > > clear
> > > > >> the
> > > > >> > list.
> > > > >> >
> > > > >> > thanks,
> > > > >> > Arun
> > > > >> >
> > > > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <acm@hortonworks.com
> >
> > > > wrote:
> > > > >> >
> > > > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> > are
> > > > now
> > > > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > > > >> > >
> > > > >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is
> helping
> > > out
> > > > >> with
> > > > >> > the YARN ones.
> > > > >> > >
> > > > >> > > thanks,
> > > > >> > > Arun
> > > > >> > >
> > > > >> > >
> > > > >> > >
> > > > >> >
> > > > >> > --
> > > > >> > Arun C. Murthy
> > > > >> > Hortonworks Inc.
> > > > >> > http://hortonworks.com/
> > > > >> >
> > > > >> >
> > > > >> >
> > > > >> > --
> > > > >> > CONFIDENTIALITY NOTICE
> > > > >> > NOTICE: This message is intended for the use of the individual
> or
> > > > >> entity to
> > > > >> > which it is addressed and may contain information that is
> > > > confidential,
> > > > >> > privileged and exempt from disclosure under applicable law. If
> the
> > > > >> reader
> > > > >> > of this message is not the intended recipient, you are hereby
> > > notified
> > > > >> that
> > > > >> > any printing, copying, dissemination, distribution, disclosure
> or
> > > > >> > forwarding of this communication is strictly prohibited. If you
> > have
> > > > >> > received this communication in error, please contact the sender
> > > > >> immediately
> > > > >> > and delete it from your system. Thank You.
> > > > >> >
> > > > >>
> > > > >>
> > > > >>
> > > > >> --
> > > > >> Alejandro
> > > > >>
> > > > >
> > > > >
> > > >
> > > > --
> > > > CONFIDENTIALITY NOTICE
> > > > NOTICE: This message is intended for the use of the individual or
> > entity
> > > to
> > > > which it is addressed and may contain information that is
> confidential,
> > > > privileged and exempt from disclosure under applicable law. If the
> > reader
> > > > of this message is not the intended recipient, you are hereby
> notified
> > > that
> > > > any printing, copying, dissemination, distribution, disclosure or
> > > > forwarding of this communication is strictly prohibited. If you have
> > > > received this communication in error, please contact the sender
> > > immediately
> > > > and delete it from your system. Thank You.
> > > >
> > >
> >
>
>
>
> --
> Alejandro
>
Re: Re-swizzle 2.3
Posted by Steve Loughran <st...@hortonworks.com>.
On 6 February 2014 17:07, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Thanks Robert,
>
> All,
>
>
>
> I'm inclined to revert them from branch-2 as well.
>
>
-1 to that; if there are issues we should be able to find and fix them soon
enough. Even if you aren't doing long-lived YARN services yet, even llama
benefits from this zero-container-loss on AM restart.
We do have Hoya using this (introspection code because the protobuf
structures are hidden away), means that you can kill the AM and HBase &
Accumulo clusters stay up in their YARN containers, the restarted AM gets
that list of containers (and any pending events), rebuilds its data
structures and carries on as before. Sweet!
https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hoya/yarn/appmaster/HoyaAppMaster.java#L551
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Hey
I am not against removing them from 2.3 if that is helpful for progress. But I want to understand what the issues are before we make that decision.
There is the issue with unmanaged AM that is clearly known and I was thinking of coming to the past two days, but couldn't. What is this new issue that we (confidently?) pinned down to YARN-1490?
Thanks
+Vinod
On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Thanks Robert,
>
> All,
>
> So it seems that YARN-1493 and YARN-1490 are introducing serious
> regressions.
>
> I would propose to revert them and the follow up JIRAs from the 2.3 branch
> and keep working on them on trunk/branch-2 until the are stable (I would
> even prefer reverting them from branch-2 not to block a 2.4 if they are not
> ready in time).
>
> As I've mentioned before, the list of JIRAs to revert were:
>
> YARN-1493
> YARN-1490
> YARN-1166
> YARN-1041
> YARN-1566
>
> Plus 2 additional JIRAs committed since my email on this issue 2 days ago:
>
> *YARN-1661
> *YARN-1689 (not sure if this JIRA is related in functionality to the
> previous ones but it is creating conflicts).
>
> I think we should hold on continuing work on top of something that is
> broken until the broken stuff is fixed.
>
> Quoting Arun, "Committers - Henceforth, please use extreme caution while
> committing to branch-2.3. Please commit *only* blockers to 2.3."
>
> YARN-1661 & YARN-1689 are not blockers.
>
> Unless there are objections, I'll revert all these JIRAs from branch-2.3
> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>
> I'm inclined to revert them from branch-2 as well.
>
> Thoughts?
>
> Thanks.
>
>
> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com> wrote:
>
>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it was
>> causing some strange behavior in the Oozie unit tests:
>>
>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all unit
>> tests in a module. With YARN-1490 we saw that, regardless of test order,
>> the last few tests would timeout waiting for an MR job to finish; on slower
>> machines, the entire test suite would timeout. Through some digging, I
>> found that we were getting a ton of "Connection refused" Exceptions on
>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>>
>> After a bunch of investigation, I found that the problem went away once
>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
>> Even though this occurred in unit tests, it does make me concerned that it
>> could indicate some bigger issue in a long-running real cluster (where
>> everything isn't running on the same machine) that we haven't seen yet.
>>
>>
>>
>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
>> wrote:
>>
>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
>> Will
>>> pull it out of branch-2.3 if anyone objects.
>>>
>>>
>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <aagarwal@hortonworks.com
>>>> wrote:
>>>
>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
>>>>
>>>>
>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
>> aagarwal@hortonworks.com
>>>>> wrote:
>>>>
>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>>>>>
>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>>>>>
>>>>>
>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
>> tucu@cloudera.com
>>>>> wrote:
>>>>>
>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
>> odd
>>>>>> ways
>>>>>> (to the point it seems un-deterministic).
>>>>>>
>>>>>> I'd say eiher YARN-1577 is fixed or we revert
>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
>>> reverts)
>>>>>> from Hadoop 2.3 branch before doing the release.
>>>>>>
>>>>>>
>>>>>> I've verified that after reverting those JIRAs things work fine with
>>>>>> unmanaged AMs.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
>>>
>>>>>> wrote:
>>>>>>
>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>>>>>>>
>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
>>>> progress
>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
>>>>>> objections?
>>>>>>>
>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>>>>>>>
>>>>>>> Overall, I'll try get this out in next couple of days if we can
>>> clear
>>>>>> the
>>>>>>> list.
>>>>>>>
>>>>>>> thanks,
>>>>>>> Arun
>>>>>>>
>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
>>>> wrote:
>>>>>>>
>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
>> are
>>>> now
>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>>>>>>>>
>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
>>> out
>>>>>> with
>>>>>>> the YARN ones.
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Arun
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Arun C. Murthy
>>>>>>> Hortonworks Inc.
>>>>>>> http://hortonworks.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby
>>> notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you
>> have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Alejandro
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>> entity
>>> to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the
>> reader
>>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender
>>> immediately
>>>> and delete it from your system. Thank You.
>>>>
>>>
>>
>
>
>
> --
> Alejandro
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Steve Loughran <st...@hortonworks.com>.
On 6 February 2014 17:07, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Thanks Robert,
>
> All,
>
>
>
> I'm inclined to revert them from branch-2 as well.
>
>
-1 to that; if there are issues we should be able to find and fix them soon
enough. Even if you aren't doing long-lived YARN services yet, even llama
benefits from this zero-container-loss on AM restart.
We do have Hoya using this (introspection code because the protobuf
structures are hidden away), means that you can kill the AM and HBase &
Accumulo clusters stay up in their YARN containers, the restarted AM gets
that list of containers (and any pending events), rebuilds its data
structures and carries on as before. Sweet!
https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hoya/yarn/appmaster/HoyaAppMaster.java#L551
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Steve Loughran <st...@hortonworks.com>.
On 6 February 2014 17:07, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Thanks Robert,
>
> All,
>
>
>
> I'm inclined to revert them from branch-2 as well.
>
>
-1 to that; if there are issues we should be able to find and fix them soon
enough. Even if you aren't doing long-lived YARN services yet, even llama
benefits from this zero-container-loss on AM restart.
We do have Hoya using this (introspection code because the protobuf
structures are hidden away), means that you can kill the AM and HBase &
Accumulo clusters stay up in their YARN containers, the restarted AM gets
that list of containers (and any pending events), rebuilds its data
structures and carries on as before. Sweet!
https://github.com/hortonworks/hoya/blob/develop/hoya-core/src/main/java/org/apache/hoya/yarn/appmaster/HoyaAppMaster.java#L551
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Hey
I am not against removing them from 2.3 if that is helpful for progress. But I want to understand what the issues are before we make that decision.
There is the issue with unmanaged AM that is clearly known and I was thinking of coming to the past two days, but couldn't. What is this new issue that we (confidently?) pinned down to YARN-1490?
Thanks
+Vinod
On Feb 6, 2014, at 5:07 PM, Alejandro Abdelnur <tu...@cloudera.com> wrote:
> Thanks Robert,
>
> All,
>
> So it seems that YARN-1493 and YARN-1490 are introducing serious
> regressions.
>
> I would propose to revert them and the follow up JIRAs from the 2.3 branch
> and keep working on them on trunk/branch-2 until the are stable (I would
> even prefer reverting them from branch-2 not to block a 2.4 if they are not
> ready in time).
>
> As I've mentioned before, the list of JIRAs to revert were:
>
> YARN-1493
> YARN-1490
> YARN-1166
> YARN-1041
> YARN-1566
>
> Plus 2 additional JIRAs committed since my email on this issue 2 days ago:
>
> *YARN-1661
> *YARN-1689 (not sure if this JIRA is related in functionality to the
> previous ones but it is creating conflicts).
>
> I think we should hold on continuing work on top of something that is
> broken until the broken stuff is fixed.
>
> Quoting Arun, "Committers - Henceforth, please use extreme caution while
> committing to branch-2.3. Please commit *only* blockers to 2.3."
>
> YARN-1661 & YARN-1689 are not blockers.
>
> Unless there are objections, I'll revert all these JIRAs from branch-2.3
> tomorrow around noon and I'll update fixedVersion in the JIRAs.
>
> I'm inclined to revert them from branch-2 as well.
>
> Thoughts?
>
> Thanks.
>
>
> On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com> wrote:
>
>> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it was
>> causing some strange behavior in the Oozie unit tests:
>>
>> Basically, we use a single MiniMRCluster and MiniDFSCluster across all unit
>> tests in a module. With YARN-1490 we saw that, regardless of test order,
>> the last few tests would timeout waiting for an MR job to finish; on slower
>> machines, the entire test suite would timeout. Through some digging, I
>> found that we were getting a ton of "Connection refused" Exceptions on
>> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>>
>> After a bunch of investigation, I found that the problem went away once
>> YARN-1490 was removed. Though I couldn't figure out the exact problem.
>> Even though this occurred in unit tests, it does make me concerned that it
>> could indicate some bigger issue in a long-running real cluster (where
>> everything isn't running on the same machine) that we haven't seen yet.
>>
>>
>>
>> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
>> wrote:
>>
>>> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
>> Will
>>> pull it out of branch-2.3 if anyone objects.
>>>
>>>
>>> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <aagarwal@hortonworks.com
>>>> wrote:
>>>
>>>> Merged HADOOP-10273 to branch-2.3 as r1565456.
>>>>
>>>>
>>>> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
>> aagarwal@hortonworks.com
>>>>> wrote:
>>>>
>>>>> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>>>>>
>>>>> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>>>>>
>>>>>
>>>>> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
>> tucu@cloudera.com
>>>>> wrote:
>>>>>
>>>>>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
>> odd
>>>>>> ways
>>>>>> (to the point it seems un-deterministic).
>>>>>>
>>>>>> I'd say eiher YARN-1577 is fixed or we revert
>>>>>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
>>> reverts)
>>>>>> from Hadoop 2.3 branch before doing the release.
>>>>>>
>>>>>>
>>>>>> I've verified that after reverting those JIRAs things work fine with
>>>>>> unmanaged AMs.
>>>>>>
>>>>>> Thanks.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
>>>
>>>>>> wrote:
>>>>>>
>>>>>>> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>>>>>>>
>>>>>>> Jian is away and I don't see YARN-1577 & YARN-1206 making much
>>>> progress
>>>>>>> till he is back; so I'm inclined to push both to 2.4 too. Any
>>>>>> objections?
>>>>>>>
>>>>>>> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>>>>>>>
>>>>>>> Overall, I'll try get this out in next couple of days if we can
>>> clear
>>>>>> the
>>>>>>> list.
>>>>>>>
>>>>>>> thanks,
>>>>>>> Arun
>>>>>>>
>>>>>>> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
>>>> wrote:
>>>>>>>
>>>>>>>> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
>> are
>>>> now
>>>>>>> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>>>>>>>>
>>>>>>>> Daryn (thanks!) has both the non-YARN covered. Vinod is helping
>>> out
>>>>>> with
>>>>>>> the YARN ones.
>>>>>>>>
>>>>>>>> thanks,
>>>>>>>> Arun
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Arun C. Murthy
>>>>>>> Hortonworks Inc.
>>>>>>> http://hortonworks.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> CONFIDENTIALITY NOTICE
>>>>>>> NOTICE: This message is intended for the use of the individual or
>>>>>> entity to
>>>>>>> which it is addressed and may contain information that is
>>>> confidential,
>>>>>>> privileged and exempt from disclosure under applicable law. If the
>>>>>> reader
>>>>>>> of this message is not the intended recipient, you are hereby
>>> notified
>>>>>> that
>>>>>>> any printing, copying, dissemination, distribution, disclosure or
>>>>>>> forwarding of this communication is strictly prohibited. If you
>> have
>>>>>>> received this communication in error, please contact the sender
>>>>>> immediately
>>>>>>> and delete it from your system. Thank You.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Alejandro
>>>>>>
>>>>>
>>>>>
>>>>
>>>> --
>>>> CONFIDENTIALITY NOTICE
>>>> NOTICE: This message is intended for the use of the individual or
>> entity
>>> to
>>>> which it is addressed and may contain information that is confidential,
>>>> privileged and exempt from disclosure under applicable law. If the
>> reader
>>>> of this message is not the intended recipient, you are hereby notified
>>> that
>>>> any printing, copying, dissemination, distribution, disclosure or
>>>> forwarding of this communication is strictly prohibited. If you have
>>>> received this communication in error, please contact the sender
>>> immediately
>>>> and delete it from your system. Thank You.
>>>>
>>>
>>
>
>
>
> --
> Alejandro
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Thanks Robert,
All,
So it seems that YARN-1493 and YARN-1490 are introducing serious
regressions.
I would propose to revert them and the follow up JIRAs from the 2.3 branch
and keep working on them on trunk/branch-2 until the are stable (I would
even prefer reverting them from branch-2 not to block a 2.4 if they are not
ready in time).
As I've mentioned before, the list of JIRAs to revert were:
YARN-1493
YARN-1490
YARN-1166
YARN-1041
YARN-1566
Plus 2 additional JIRAs committed since my email on this issue 2 days ago:
*YARN-1661
*YARN-1689 (not sure if this JIRA is related in functionality to the
previous ones but it is creating conflicts).
I think we should hold on continuing work on top of something that is
broken until the broken stuff is fixed.
Quoting Arun, "Committers - Henceforth, please use extreme caution while
committing to branch-2.3. Please commit *only* blockers to 2.3."
YARN-1661 & YARN-1689 are not blockers.
Unless there are objections, I'll revert all these JIRAs from branch-2.3
tomorrow around noon and I'll update fixedVersion in the JIRAs.
I'm inclined to revert them from branch-2 as well.
Thoughts?
Thanks.
On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com> wrote:
> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it was
> causing some strange behavior in the Oozie unit tests:
>
> Basically, we use a single MiniMRCluster and MiniDFSCluster across all unit
> tests in a module. With YARN-1490 we saw that, regardless of test order,
> the last few tests would timeout waiting for an MR job to finish; on slower
> machines, the entire test suite would timeout. Through some digging, I
> found that we were getting a ton of "Connection refused" Exceptions on
> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>
> After a bunch of investigation, I found that the problem went away once
> YARN-1490 was removed. Though I couldn't figure out the exact problem.
> Even though this occurred in unit tests, it does make me concerned that it
> could indicate some bigger issue in a long-running real cluster (where
> everything isn't running on the same machine) that we haven't seen yet.
>
>
>
> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
> > I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
> Will
> > pull it out of branch-2.3 if anyone objects.
> >
> >
> > On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <aagarwal@hortonworks.com
> > >wrote:
> >
> > > Merged HADOOP-10273 to branch-2.3 as r1565456.
> > >
> > >
> > > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> aagarwal@hortonworks.com
> > > >wrote:
> > >
> > > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> > > >
> > > > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> > > >
> > > >
> > > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > >wrote:
> > > >
> > > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> odd
> > > >> ways
> > > >> (to the point it seems un-deterministic).
> > > >>
> > > >> I'd say eiher YARN-1577 is fixed or we revert
> > > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> > reverts)
> > > >> from Hadoop 2.3 branch before doing the release.
> > > >>
> > > >>
> > > >> I've verified that after reverting those JIRAs things work fine with
> > > >> unmanaged AMs.
> > > >>
> > > >> Thanks.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
> >
> > > >> wrote:
> > > >>
> > > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> > > >> >
> > > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> > > progress
> > > >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> > > >> objections?
> > > >> >
> > > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> > > >> >
> > > >> > Overall, I'll try get this out in next couple of days if we can
> > clear
> > > >> the
> > > >> > list.
> > > >> >
> > > >> > thanks,
> > > >> > Arun
> > > >> >
> > > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >> >
> > > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> are
> > > now
> > > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > > >> > >
> > > >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> > out
> > > >> with
> > > >> > the YARN ones.
> > > >> > >
> > > >> > > thanks,
> > > >> > > Arun
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > Arun C. Murthy
> > > >> > Hortonworks Inc.
> > > >> > http://hortonworks.com/
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > CONFIDENTIALITY NOTICE
> > > >> > NOTICE: This message is intended for the use of the individual or
> > > >> entity to
> > > >> > which it is addressed and may contain information that is
> > > confidential,
> > > >> > privileged and exempt from disclosure under applicable law. If the
> > > >> reader
> > > >> > of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >> > any printing, copying, dissemination, distribution, disclosure or
> > > >> > forwarding of this communication is strictly prohibited. If you
> have
> > > >> > received this communication in error, please contact the sender
> > > >> immediately
> > > >> > and delete it from your system. Thank You.
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Alejandro
> > > >>
> > > >
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Thanks Robert,
All,
So it seems that YARN-1493 and YARN-1490 are introducing serious
regressions.
I would propose to revert them and the follow up JIRAs from the 2.3 branch
and keep working on them on trunk/branch-2 until the are stable (I would
even prefer reverting them from branch-2 not to block a 2.4 if they are not
ready in time).
As I've mentioned before, the list of JIRAs to revert were:
YARN-1493
YARN-1490
YARN-1166
YARN-1041
YARN-1566
Plus 2 additional JIRAs committed since my email on this issue 2 days ago:
*YARN-1661
*YARN-1689 (not sure if this JIRA is related in functionality to the
previous ones but it is creating conflicts).
I think we should hold on continuing work on top of something that is
broken until the broken stuff is fixed.
Quoting Arun, "Committers - Henceforth, please use extreme caution while
committing to branch-2.3. Please commit *only* blockers to 2.3."
YARN-1661 & YARN-1689 are not blockers.
Unless there are objections, I'll revert all these JIRAs from branch-2.3
tomorrow around noon and I'll update fixedVersion in the JIRAs.
I'm inclined to revert them from branch-2 as well.
Thoughts?
Thanks.
On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com> wrote:
> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it was
> causing some strange behavior in the Oozie unit tests:
>
> Basically, we use a single MiniMRCluster and MiniDFSCluster across all unit
> tests in a module. With YARN-1490 we saw that, regardless of test order,
> the last few tests would timeout waiting for an MR job to finish; on slower
> machines, the entire test suite would timeout. Through some digging, I
> found that we were getting a ton of "Connection refused" Exceptions on
> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>
> After a bunch of investigation, I found that the problem went away once
> YARN-1490 was removed. Though I couldn't figure out the exact problem.
> Even though this occurred in unit tests, it does make me concerned that it
> could indicate some bigger issue in a long-running real cluster (where
> everything isn't running on the same machine) that we haven't seen yet.
>
>
>
> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
> > I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
> Will
> > pull it out of branch-2.3 if anyone objects.
> >
> >
> > On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <aagarwal@hortonworks.com
> > >wrote:
> >
> > > Merged HADOOP-10273 to branch-2.3 as r1565456.
> > >
> > >
> > > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> aagarwal@hortonworks.com
> > > >wrote:
> > >
> > > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> > > >
> > > > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> > > >
> > > >
> > > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > >wrote:
> > > >
> > > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> odd
> > > >> ways
> > > >> (to the point it seems un-deterministic).
> > > >>
> > > >> I'd say eiher YARN-1577 is fixed or we revert
> > > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> > reverts)
> > > >> from Hadoop 2.3 branch before doing the release.
> > > >>
> > > >>
> > > >> I've verified that after reverting those JIRAs things work fine with
> > > >> unmanaged AMs.
> > > >>
> > > >> Thanks.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
> >
> > > >> wrote:
> > > >>
> > > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> > > >> >
> > > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> > > progress
> > > >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> > > >> objections?
> > > >> >
> > > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> > > >> >
> > > >> > Overall, I'll try get this out in next couple of days if we can
> > clear
> > > >> the
> > > >> > list.
> > > >> >
> > > >> > thanks,
> > > >> > Arun
> > > >> >
> > > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >> >
> > > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> are
> > > now
> > > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > > >> > >
> > > >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> > out
> > > >> with
> > > >> > the YARN ones.
> > > >> > >
> > > >> > > thanks,
> > > >> > > Arun
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > Arun C. Murthy
> > > >> > Hortonworks Inc.
> > > >> > http://hortonworks.com/
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > CONFIDENTIALITY NOTICE
> > > >> > NOTICE: This message is intended for the use of the individual or
> > > >> entity to
> > > >> > which it is addressed and may contain information that is
> > > confidential,
> > > >> > privileged and exempt from disclosure under applicable law. If the
> > > >> reader
> > > >> > of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >> > any printing, copying, dissemination, distribution, disclosure or
> > > >> > forwarding of this communication is strictly prohibited. If you
> have
> > > >> > received this communication in error, please contact the sender
> > > >> immediately
> > > >> > and delete it from your system. Thank You.
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Alejandro
> > > >>
> > > >
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Thanks Robert,
All,
So it seems that YARN-1493 and YARN-1490 are introducing serious
regressions.
I would propose to revert them and the follow up JIRAs from the 2.3 branch
and keep working on them on trunk/branch-2 until the are stable (I would
even prefer reverting them from branch-2 not to block a 2.4 if they are not
ready in time).
As I've mentioned before, the list of JIRAs to revert were:
YARN-1493
YARN-1490
YARN-1166
YARN-1041
YARN-1566
Plus 2 additional JIRAs committed since my email on this issue 2 days ago:
*YARN-1661
*YARN-1689 (not sure if this JIRA is related in functionality to the
previous ones but it is creating conflicts).
I think we should hold on continuing work on top of something that is
broken until the broken stuff is fixed.
Quoting Arun, "Committers - Henceforth, please use extreme caution while
committing to branch-2.3. Please commit *only* blockers to 2.3."
YARN-1661 & YARN-1689 are not blockers.
Unless there are objections, I'll revert all these JIRAs from branch-2.3
tomorrow around noon and I'll update fixedVersion in the JIRAs.
I'm inclined to revert them from branch-2 as well.
Thoughts?
Thanks.
On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com> wrote:
> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it was
> causing some strange behavior in the Oozie unit tests:
>
> Basically, we use a single MiniMRCluster and MiniDFSCluster across all unit
> tests in a module. With YARN-1490 we saw that, regardless of test order,
> the last few tests would timeout waiting for an MR job to finish; on slower
> machines, the entire test suite would timeout. Through some digging, I
> found that we were getting a ton of "Connection refused" Exceptions on
> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>
> After a bunch of investigation, I found that the problem went away once
> YARN-1490 was removed. Though I couldn't figure out the exact problem.
> Even though this occurred in unit tests, it does make me concerned that it
> could indicate some bigger issue in a long-running real cluster (where
> everything isn't running on the same machine) that we haven't seen yet.
>
>
>
> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
> > I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
> Will
> > pull it out of branch-2.3 if anyone objects.
> >
> >
> > On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <aagarwal@hortonworks.com
> > >wrote:
> >
> > > Merged HADOOP-10273 to branch-2.3 as r1565456.
> > >
> > >
> > > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> aagarwal@hortonworks.com
> > > >wrote:
> > >
> > > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> > > >
> > > > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> > > >
> > > >
> > > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > >wrote:
> > > >
> > > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> odd
> > > >> ways
> > > >> (to the point it seems un-deterministic).
> > > >>
> > > >> I'd say eiher YARN-1577 is fixed or we revert
> > > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> > reverts)
> > > >> from Hadoop 2.3 branch before doing the release.
> > > >>
> > > >>
> > > >> I've verified that after reverting those JIRAs things work fine with
> > > >> unmanaged AMs.
> > > >>
> > > >> Thanks.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
> >
> > > >> wrote:
> > > >>
> > > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> > > >> >
> > > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> > > progress
> > > >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> > > >> objections?
> > > >> >
> > > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> > > >> >
> > > >> > Overall, I'll try get this out in next couple of days if we can
> > clear
> > > >> the
> > > >> > list.
> > > >> >
> > > >> > thanks,
> > > >> > Arun
> > > >> >
> > > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >> >
> > > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> are
> > > now
> > > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > > >> > >
> > > >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> > out
> > > >> with
> > > >> > the YARN ones.
> > > >> > >
> > > >> > > thanks,
> > > >> > > Arun
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > Arun C. Murthy
> > > >> > Hortonworks Inc.
> > > >> > http://hortonworks.com/
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > CONFIDENTIALITY NOTICE
> > > >> > NOTICE: This message is intended for the use of the individual or
> > > >> entity to
> > > >> > which it is addressed and may contain information that is
> > > confidential,
> > > >> > privileged and exempt from disclosure under applicable law. If the
> > > >> reader
> > > >> > of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >> > any printing, copying, dissemination, distribution, disclosure or
> > > >> > forwarding of this communication is strictly prohibited. If you
> have
> > > >> > received this communication in error, please contact the sender
> > > >> immediately
> > > >> > and delete it from your system. Thank You.
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Alejandro
> > > >>
> > > >
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
Thanks Robert,
All,
So it seems that YARN-1493 and YARN-1490 are introducing serious
regressions.
I would propose to revert them and the follow up JIRAs from the 2.3 branch
and keep working on them on trunk/branch-2 until the are stable (I would
even prefer reverting them from branch-2 not to block a 2.4 if they are not
ready in time).
As I've mentioned before, the list of JIRAs to revert were:
YARN-1493
YARN-1490
YARN-1166
YARN-1041
YARN-1566
Plus 2 additional JIRAs committed since my email on this issue 2 days ago:
*YARN-1661
*YARN-1689 (not sure if this JIRA is related in functionality to the
previous ones but it is creating conflicts).
I think we should hold on continuing work on top of something that is
broken until the broken stuff is fixed.
Quoting Arun, "Committers - Henceforth, please use extreme caution while
committing to branch-2.3. Please commit *only* blockers to 2.3."
YARN-1661 & YARN-1689 are not blockers.
Unless there are objections, I'll revert all these JIRAs from branch-2.3
tomorrow around noon and I'll update fixedVersion in the JIRAs.
I'm inclined to revert them from branch-2 as well.
Thoughts?
Thanks.
On Thu, Feb 6, 2014 at 3:54 PM, Robert Kanter <rk...@cloudera.com> wrote:
> I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it was
> causing some strange behavior in the Oozie unit tests:
>
> Basically, we use a single MiniMRCluster and MiniDFSCluster across all unit
> tests in a module. With YARN-1490 we saw that, regardless of test order,
> the last few tests would timeout waiting for an MR job to finish; on slower
> machines, the entire test suite would timeout. Through some digging, I
> found that we were getting a ton of "Connection refused" Exceptions on
> LeaseRenewer talking to the NN and a few on the AM talking to the RM.
>
> After a bunch of investigation, I found that the problem went away once
> YARN-1490 was removed. Though I couldn't figure out the exact problem.
> Even though this occurred in unit tests, it does make me concerned that it
> could indicate some bigger issue in a long-running real cluster (where
> everything isn't running on the same machine) that we haven't seen yet.
>
>
>
> On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
> > I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly.
> Will
> > pull it out of branch-2.3 if anyone objects.
> >
> >
> > On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <aagarwal@hortonworks.com
> > >wrote:
> >
> > > Merged HADOOP-10273 to branch-2.3 as r1565456.
> > >
> > >
> > > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <
> aagarwal@hortonworks.com
> > > >wrote:
> > >
> > > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> > > >
> > > > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> > > >
> > > >
> > > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <
> tucu@cloudera.com
> > > >wrote:
> > > >
> > > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very
> odd
> > > >> ways
> > > >> (to the point it seems un-deterministic).
> > > >>
> > > >> I'd say eiher YARN-1577 is fixed or we revert
> > > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> > reverts)
> > > >> from Hadoop 2.3 branch before doing the release.
> > > >>
> > > >>
> > > >> I've verified that after reverting those JIRAs things work fine with
> > > >> unmanaged AMs.
> > > >>
> > > >> Thanks.
> > > >>
> > > >>
> > > >>
> > > >>
> > > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <acm@hortonworks.com
> >
> > > >> wrote:
> > > >>
> > > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> > > >> >
> > > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> > > progress
> > > >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> > > >> objections?
> > > >> >
> > > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> > > >> >
> > > >> > Overall, I'll try get this out in next couple of days if we can
> > clear
> > > >> the
> > > >> > list.
> > > >> >
> > > >> > thanks,
> > > >> > Arun
> > > >> >
> > > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> > > wrote:
> > > >> >
> > > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we
> are
> > > now
> > > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > > >> > >
> > > >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> > out
> > > >> with
> > > >> > the YARN ones.
> > > >> > >
> > > >> > > thanks,
> > > >> > > Arun
> > > >> > >
> > > >> > >
> > > >> > >
> > > >> >
> > > >> > --
> > > >> > Arun C. Murthy
> > > >> > Hortonworks Inc.
> > > >> > http://hortonworks.com/
> > > >> >
> > > >> >
> > > >> >
> > > >> > --
> > > >> > CONFIDENTIALITY NOTICE
> > > >> > NOTICE: This message is intended for the use of the individual or
> > > >> entity to
> > > >> > which it is addressed and may contain information that is
> > > confidential,
> > > >> > privileged and exempt from disclosure under applicable law. If the
> > > >> reader
> > > >> > of this message is not the intended recipient, you are hereby
> > notified
> > > >> that
> > > >> > any printing, copying, dissemination, distribution, disclosure or
> > > >> > forwarding of this communication is strictly prohibited. If you
> have
> > > >> > received this communication in error, please contact the sender
> > > >> immediately
> > > >> > and delete it from your system. Thank You.
> > > >> >
> > > >>
> > > >>
> > > >>
> > > >> --
> > > >> Alejandro
> > > >>
> > > >
> > > >
> > >
> > > --
> > > CONFIDENTIALITY NOTICE
> > > NOTICE: This message is intended for the use of the individual or
> entity
> > to
> > > which it is addressed and may contain information that is confidential,
> > > privileged and exempt from disclosure under applicable law. If the
> reader
> > > of this message is not the intended recipient, you are hereby notified
> > that
> > > any printing, copying, dissemination, distribution, disclosure or
> > > forwarding of this communication is strictly prohibited. If you have
> > > received this communication in error, please contact the sender
> > immediately
> > > and delete it from your system. Thank You.
> > >
> >
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Robert Kanter <rk...@cloudera.com>.
I think we should revert YARN-1490 from Hadoop 2.3 branch. I think it was
causing some strange behavior in the Oozie unit tests:
Basically, we use a single MiniMRCluster and MiniDFSCluster across all unit
tests in a module. With YARN-1490 we saw that, regardless of test order,
the last few tests would timeout waiting for an MR job to finish; on slower
machines, the entire test suite would timeout. Through some digging, I
found that we were getting a ton of "Connection refused" Exceptions on
LeaseRenewer talking to the NN and a few on the AM talking to the RM.
After a bunch of investigation, I found that the problem went away once
YARN-1490 was removed. Though I couldn't figure out the exact problem.
Even though this occurred in unit tests, it does make me concerned that it
could indicate some bigger issue in a long-running real cluster (where
everything isn't running on the same machine) that we haven't seen yet.
On Thu, Feb 6, 2014 at 3:06 PM, Karthik Kambatla <ka...@cloudera.com> wrote:
> I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly. Will
> pull it out of branch-2.3 if anyone objects.
>
>
> On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <aagarwal@hortonworks.com
> >wrote:
>
> > Merged HADOOP-10273 to branch-2.3 as r1565456.
> >
> >
> > On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <aagarwal@hortonworks.com
> > >wrote:
> >
> > > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> > >
> > > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> > >
> > >
> > > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <tucu@cloudera.com
> > >wrote:
> > >
> > >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very odd
> > >> ways
> > >> (to the point it seems un-deterministic).
> > >>
> > >> I'd say eiher YARN-1577 is fixed or we revert
> > >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean
> reverts)
> > >> from Hadoop 2.3 branch before doing the release.
> > >>
> > >>
> > >> I've verified that after reverting those JIRAs things work fine with
> > >> unmanaged AMs.
> > >>
> > >> Thanks.
> > >>
> > >>
> > >>
> > >>
> > >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <ac...@hortonworks.com>
> > >> wrote:
> > >>
> > >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> > >> >
> > >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> > progress
> > >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> > >> objections?
> > >> >
> > >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> > >> >
> > >> > Overall, I'll try get this out in next couple of days if we can
> clear
> > >> the
> > >> > list.
> > >> >
> > >> > thanks,
> > >> > Arun
> > >> >
> > >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> > wrote:
> > >> >
> > >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are
> > now
> > >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > >> > >
> > >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is helping
> out
> > >> with
> > >> > the YARN ones.
> > >> > >
> > >> > > thanks,
> > >> > > Arun
> > >> > >
> > >> > >
> > >> > >
> > >> >
> > >> > --
> > >> > Arun C. Murthy
> > >> > Hortonworks Inc.
> > >> > http://hortonworks.com/
> > >> >
> > >> >
> > >> >
> > >> > --
> > >> > CONFIDENTIALITY NOTICE
> > >> > NOTICE: This message is intended for the use of the individual or
> > >> entity to
> > >> > which it is addressed and may contain information that is
> > confidential,
> > >> > privileged and exempt from disclosure under applicable law. If the
> > >> reader
> > >> > of this message is not the intended recipient, you are hereby
> notified
> > >> that
> > >> > any printing, copying, dissemination, distribution, disclosure or
> > >> > forwarding of this communication is strictly prohibited. If you have
> > >> > received this communication in error, please contact the sender
> > >> immediately
> > >> > and delete it from your system. Thank You.
> > >> >
> > >>
> > >>
> > >>
> > >> --
> > >> Alejandro
> > >>
> > >
> > >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
Re: Re-swizzle 2.3
Posted by Karthik Kambatla <ka...@cloudera.com>.
I have marked MAPREDUCE-5744 a blocker for 2.3. Committing it shortly. Will
pull it out of branch-2.3 if anyone objects.
On Thu, Feb 6, 2014 at 2:04 PM, Arpit Agarwal <aa...@hortonworks.com>wrote:
> Merged HADOOP-10273 to branch-2.3 as r1565456.
>
>
> On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <aagarwal@hortonworks.com
> >wrote:
>
> > IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
> >
> > I will merge it to branch-2.3 tomorrow PST if no one disagrees.
> >
> >
> > On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <tucu@cloudera.com
> >wrote:
> >
> >> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very odd
> >> ways
> >> (to the point it seems un-deterministic).
> >>
> >> I'd say eiher YARN-1577 is fixed or we revert
> >> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean reverts)
> >> from Hadoop 2.3 branch before doing the release.
> >>
> >>
> >> I've verified that after reverting those JIRAs things work fine with
> >> unmanaged AMs.
> >>
> >> Thanks.
> >>
> >>
> >>
> >>
> >> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <ac...@hortonworks.com>
> >> wrote:
> >>
> >> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> >> >
> >> > Jian is away and I don't see YARN-1577 & YARN-1206 making much
> progress
> >> > till he is back; so I'm inclined to push both to 2.4 too. Any
> >> objections?
> >> >
> >> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> >> >
> >> > Overall, I'll try get this out in next couple of days if we can clear
> >> the
> >> > list.
> >> >
> >> > thanks,
> >> > Arun
> >> >
> >> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
> >> >
> >> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are
> now
> >> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >> > >
> >> > > Daryn (thanks!) has both the non-YARN covered. Vinod is helping out
> >> with
> >> > the YARN ones.
> >> > >
> >> > > thanks,
> >> > > Arun
> >> > >
> >> > >
> >> > >
> >> >
> >> > --
> >> > Arun C. Murthy
> >> > Hortonworks Inc.
> >> > http://hortonworks.com/
> >> >
> >> >
> >> >
> >> > --
> >> > CONFIDENTIALITY NOTICE
> >> > NOTICE: This message is intended for the use of the individual or
> >> entity to
> >> > which it is addressed and may contain information that is
> confidential,
> >> > privileged and exempt from disclosure under applicable law. If the
> >> reader
> >> > of this message is not the intended recipient, you are hereby notified
> >> that
> >> > any printing, copying, dissemination, distribution, disclosure or
> >> > forwarding of this communication is strictly prohibited. If you have
> >> > received this communication in error, please contact the sender
> >> immediately
> >> > and delete it from your system. Thank You.
> >> >
> >>
> >>
> >>
> >> --
> >> Alejandro
> >>
> >
> >
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Re-swizzle 2.3
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Merged HADOOP-10273 to branch-2.3 as r1565456.
On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <aa...@hortonworks.com>wrote:
> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>
> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>
>
> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very odd
>> ways
>> (to the point it seems un-deterministic).
>>
>> I'd say eiher YARN-1577 is fixed or we revert
>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean reverts)
>> from Hadoop 2.3 branch before doing the release.
>>
>>
>> I've verified that after reverting those JIRAs things work fine with
>> unmanaged AMs.
>>
>> Thanks.
>>
>>
>>
>>
>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <ac...@hortonworks.com>
>> wrote:
>>
>> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
>> >
>> > Jian is away and I don't see YARN-1577 & YARN-1206 making much progress
>> > till he is back; so I'm inclined to push both to 2.4 too. Any
>> objections?
>> >
>> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>> >
>> > Overall, I'll try get this out in next couple of days if we can clear
>> the
>> > list.
>> >
>> > thanks,
>> > Arun
>> >
>> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> >
>> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now
>> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>> > >
>> > > Daryn (thanks!) has both the non-YARN covered. Vinod is helping out
>> with
>> > the YARN ones.
>> > >
>> > > thanks,
>> > > Arun
>> > >
>> > >
>> > >
>> >
>> > --
>> > Arun C. Murthy
>> > Hortonworks Inc.
>> > http://hortonworks.com/
>> >
>> >
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or
>> entity to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>>
>>
>> --
>> Alejandro
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Arpit Agarwal <aa...@hortonworks.com>.
Merged HADOOP-10273 to branch-2.3 as r1565456.
On Wed, Feb 5, 2014 at 4:49 PM, Arpit Agarwal <aa...@hortonworks.com>wrote:
> IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
>
> I will merge it to branch-2.3 tomorrow PST if no one disagrees.
>
>
> On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
>
>> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very odd
>> ways
>> (to the point it seems un-deterministic).
>>
>> I'd say eiher YARN-1577 is fixed or we revert
>> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean reverts)
>> from Hadoop 2.3 branch before doing the release.
>>
>>
>> I've verified that after reverting those JIRAs things work fine with
>> unmanaged AMs.
>>
>> Thanks.
>>
>>
>>
>>
>> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <ac...@hortonworks.com>
>> wrote:
>>
>> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
>> >
>> > Jian is away and I don't see YARN-1577 & YARN-1206 making much progress
>> > till he is back; so I'm inclined to push both to 2.4 too. Any
>> objections?
>> >
>> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>> >
>> > Overall, I'll try get this out in next couple of days if we can clear
>> the
>> > list.
>> >
>> > thanks,
>> > Arun
>> >
>> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>> >
>> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now
>> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>> > >
>> > > Daryn (thanks!) has both the non-YARN covered. Vinod is helping out
>> with
>> > the YARN ones.
>> > >
>> > > thanks,
>> > > Arun
>> > >
>> > >
>> > >
>> >
>> > --
>> > Arun C. Murthy
>> > Hortonworks Inc.
>> > http://hortonworks.com/
>> >
>> >
>> >
>> > --
>> > CONFIDENTIALITY NOTICE
>> > NOTICE: This message is intended for the use of the individual or
>> entity to
>> > which it is addressed and may contain information that is confidential,
>> > privileged and exempt from disclosure under applicable law. If the
>> reader
>> > of this message is not the intended recipient, you are hereby notified
>> that
>> > any printing, copying, dissemination, distribution, disclosure or
>> > forwarding of this communication is strictly prohibited. If you have
>> > received this communication in error, please contact the sender
>> immediately
>> > and delete it from your system. Thank You.
>> >
>>
>>
>>
>> --
>> Alejandro
>>
>
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Arpit Agarwal <aa...@hortonworks.com>.
IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
I will merge it to branch-2.3 tomorrow PST if no one disagrees.
On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very odd ways
> (to the point it seems un-deterministic).
>
> I'd say eiher YARN-1577 is fixed or we revert
> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean reverts)
> from Hadoop 2.3 branch before doing the release.
>
>
> I've verified that after reverting those JIRAs things work fine with
> unmanaged AMs.
>
> Thanks.
>
>
>
>
> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
>
> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> >
> > Jian is away and I don't see YARN-1577 & YARN-1206 making much progress
> > till he is back; so I'm inclined to push both to 2.4 too. Any objections?
> >
> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> >
> > Overall, I'll try get this out in next couple of days if we can clear the
> > list.
> >
> > thanks,
> > Arun
> >
> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> >
> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now
> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > >
> > > Daryn (thanks!) has both the non-YARN covered. Vinod is helping out
> with
> > the YARN ones.
> > >
> > > thanks,
> > > Arun
> > >
> > >
> > >
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> Alejandro
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Arpit Agarwal <aa...@hortonworks.com>.
IMO HADOOP-10273 (Fix 'mvn site') should be included in 2.3.
I will merge it to branch-2.3 tomorrow PST if no one disagrees.
On Tue, Feb 4, 2014 at 5:03 PM, Alejandro Abdelnur <tu...@cloudera.com>wrote:
> IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very odd ways
> (to the point it seems un-deterministic).
>
> I'd say eiher YARN-1577 is fixed or we revert
> YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean reverts)
> from Hadoop 2.3 branch before doing the release.
>
>
> I've verified that after reverting those JIRAs things work fine with
> unmanaged AMs.
>
> Thanks.
>
>
>
>
> On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <ac...@hortonworks.com>
> wrote:
>
> > I punted YARN-1444 to 2.4 since it's a long-standing issue.
> >
> > Jian is away and I don't see YARN-1577 & YARN-1206 making much progress
> > till he is back; so I'm inclined to push both to 2.4 too. Any objections?
> >
> > Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
> >
> > Overall, I'll try get this out in next couple of days if we can clear the
> > list.
> >
> > thanks,
> > Arun
> >
> > On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> >
> > > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now
> > down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> > >
> > > Daryn (thanks!) has both the non-YARN covered. Vinod is helping out
> with
> > the YARN ones.
> > >
> > > thanks,
> > > Arun
> > >
> > >
> > >
> >
> > --
> > Arun C. Murthy
> > Hortonworks Inc.
> > http://hortonworks.com/
> >
> >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> >
>
>
>
> --
> Alejandro
>
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very odd ways
(to the point it seems un-deterministic).
I'd say eiher YARN-1577 is fixed or we revert
YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean reverts)
from Hadoop 2.3 branch before doing the release.
I've verified that after reverting those JIRAs things work fine with
unmanaged AMs.
Thanks.
On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>
> Jian is away and I don't see YARN-1577 & YARN-1206 making much progress
> till he is back; so I'm inclined to push both to 2.4 too. Any objections?
>
> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>
> Overall, I'll try get this out in next couple of days if we can clear the
> list.
>
> thanks,
> Arun
>
> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now
> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >
> > Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with
> the YARN ones.
> >
> > thanks,
> > Arun
> >
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very odd ways
(to the point it seems un-deterministic).
I'd say eiher YARN-1577 is fixed or we revert
YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean reverts)
from Hadoop 2.3 branch before doing the release.
I've verified that after reverting those JIRAs things work fine with
unmanaged AMs.
Thanks.
On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>
> Jian is away and I don't see YARN-1577 & YARN-1206 making much progress
> till he is back; so I'm inclined to push both to 2.4 too. Any objections?
>
> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>
> Overall, I'll try get this out in next couple of days if we can clear the
> list.
>
> thanks,
> Arun
>
> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now
> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >
> > Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with
> the YARN ones.
> >
> > thanks,
> > Arun
> >
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very odd ways
(to the point it seems un-deterministic).
I'd say eiher YARN-1577 is fixed or we revert
YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean reverts)
from Hadoop 2.3 branch before doing the release.
I've verified that after reverting those JIRAs things work fine with
unmanaged AMs.
Thanks.
On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>
> Jian is away and I don't see YARN-1577 & YARN-1206 making much progress
> till he is back; so I'm inclined to push both to 2.4 too. Any objections?
>
> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>
> Overall, I'll try get this out in next couple of days if we can clear the
> list.
>
> thanks,
> Arun
>
> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now
> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >
> > Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with
> the YARN ones.
> >
> > thanks,
> > Arun
> >
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Alejandro Abdelnur <tu...@cloudera.com>.
IMO YARN-1577 is a blocker, it is breaking unmanaged AMs in a very odd ways
(to the point it seems un-deterministic).
I'd say eiher YARN-1577 is fixed or we revert
YARN-1493/YARN-1490/YARN-1166/YARN-1041/YARN-1566 (almost clean reverts)
from Hadoop 2.3 branch before doing the release.
I've verified that after reverting those JIRAs things work fine with
unmanaged AMs.
Thanks.
On Tue, Feb 4, 2014 at 11:45 AM, Arun C Murthy <ac...@hortonworks.com> wrote:
> I punted YARN-1444 to 2.4 since it's a long-standing issue.
>
> Jian is away and I don't see YARN-1577 & YARN-1206 making much progress
> till he is back; so I'm inclined to push both to 2.4 too. Any objections?
>
> Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
>
> Overall, I'll try get this out in next couple of days if we can clear the
> list.
>
> thanks,
> Arun
>
> On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
>
> > An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now
> down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
> >
> > Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with
> the YARN ones.
> >
> > thanks,
> > Arun
> >
> >
> >
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
--
Alejandro
Re: Re-swizzle 2.3
Posted by Arun C Murthy <ac...@hortonworks.com>.
I punted YARN-1444 to 2.4 since it's a long-standing issue.
Jian is away and I don't see YARN-1577 & YARN-1206 making much progress till he is back; so I'm inclined to push both to 2.4 too. Any objections?
Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
Overall, I'll try get this out in next couple of days if we can clear the list.
thanks,
Arun
On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>
> Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with the YARN ones.
>
> thanks,
> Arun
>
>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Arun C Murthy <ac...@hortonworks.com>.
I punted YARN-1444 to 2.4 since it's a long-standing issue.
Jian is away and I don't see YARN-1577 & YARN-1206 making much progress till he is back; so I'm inclined to push both to 2.4 too. Any objections?
Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
Overall, I'll try get this out in next couple of days if we can clear the list.
thanks,
Arun
On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>
> Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with the YARN ones.
>
> thanks,
> Arun
>
>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Arun C Murthy <ac...@hortonworks.com>.
I punted YARN-1444 to 2.4 since it's a long-standing issue.
Jian is away and I don't see YARN-1577 & YARN-1206 making much progress till he is back; so I'm inclined to push both to 2.4 too. Any objections?
Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
Overall, I'll try get this out in next couple of days if we can clear the list.
thanks,
Arun
On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>
> Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with the YARN ones.
>
> thanks,
> Arun
>
>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Arun C Murthy <ac...@hortonworks.com>.
I punted YARN-1444 to 2.4 since it's a long-standing issue.
Jian is away and I don't see YARN-1577 & YARN-1206 making much progress till he is back; so I'm inclined to push both to 2.4 too. Any objections?
Looks like Daryn has both HADOOP-10301 & HDFS-4564 covered.
Overall, I'll try get this out in next couple of days if we can clear the list.
thanks,
Arun
On Feb 3, 2014, at 12:14 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
>
> Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with the YARN ones.
>
> thanks,
> Arun
>
>
>
--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Arun C Murthy <ac...@hortonworks.com>.
An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with the YARN ones.
thanks,
Arun
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Arun C Murthy <ac...@hortonworks.com>.
An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with the YARN ones.
thanks,
Arun
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Arun C Murthy <ac...@hortonworks.com>.
An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with the YARN ones.
thanks,
Arun
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Arun C Murthy <ac...@hortonworks.com>.
An update. Per https://s.apache.org/hadoop-2.3.0-blockers we are now down to 5 blockers: 1 Common, 1 HDFS, 3 YARN.
Daryn (thanks!) has both the non-YARN covered. Vinod is helping out with the YARN ones.
thanks,
Arun
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
That's right.
+Vinod
On Jan 31, 2014, at 5:25 PM, Andrew Wang <an...@cloudera.com> wrote:
> YARN-1673 IIUC relates to the AHS, so is actually only in branch-2 and not
> branch-2.3.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
That's right.
+Vinod
On Jan 31, 2014, at 5:25 PM, Andrew Wang <an...@cloudera.com> wrote:
> YARN-1673 IIUC relates to the AHS, so is actually only in branch-2 and not
> branch-2.3.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
That's right.
+Vinod
On Jan 31, 2014, at 5:25 PM, Andrew Wang <an...@cloudera.com> wrote:
> YARN-1673 IIUC relates to the AHS, so is actually only in branch-2 and not
> branch-2.3.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
That's right.
+Vinod
On Jan 31, 2014, at 5:25 PM, Andrew Wang <an...@cloudera.com> wrote:
> YARN-1673 IIUC relates to the AHS, so is actually only in branch-2 and not
> branch-2.3.
--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.
Re: Re-swizzle 2.3
Posted by Andrew Wang <an...@cloudera.com>.
Thanks for the link Arun, I went ahead and punted one HADOOP blocker, and
the remaining two HADOOP/HDFS looks like they're under active review.
Post-swizzle, it seems like most blockers for 2.4 would also apply to 2.3,
so I looked at that list too:
https://issues.apache.org/jira/issues/?filter=12326375&jql=project%20in%20(HADOOP%2C%20YARN%2C%20HDFS%2C%20MAPREDUCE)%20AND%20priority%20%3D%20Blocker%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20%22Target%20Version%2Fs%22%20%3D%20%222.4.0%22
YARN-1673 IIUC relates to the AHS, so is actually only in branch-2 and not
branch-2.3.
HADOOP-10048, Jason's comment says he's okay with it not being a blocker.
HDFS-5796 hasn't seen much action. Kihwal or Haohui, could you comment on
the importance/status? I don't have much context in this area.
Best,
Andrew
On Fri, Jan 31, 2014 at 4:29 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Thanks Vinod, appreciate it!
>
> I think we are very close.
>
> Here is a handy ref. to the list of blockers:
> http://s.apache.org/hadoop-2.3.0-blockers
>
> I'd appreciate if folks can help expedite these fixes, and, equally
> importantly bring up others they feel should be blockers for 2.3.0.
>
> thanks,
> Arun
>
> On Jan 30, 2014, at 12:42 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
> wrote:
>
> > That was quite some exercise, but I'm done with it now. Updated YARN's
> and MAPREDUCE's CHANGES.txt on trunk, branch-2 and branch-2.3. Let me know
> if you find some inaccuracies.
> >
> > Thanks,
> > +Vinod
> >
> > On Jan 29, 2014, at 10:49 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
> >
> >>
> >> Okay, I'll look at YARN and MR CHANGES.txt problems. Seems like they
> aren't addressed yet.
> >>
> >> +Vinod
> >>
> >>
> >> On Jan 29, 2014, at 3:24 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> >>
> >>> I just finished tuning up branch-2.3 and fixing up the HDFS and Common
> >>> CHANGES.txt in trunk, branch-2, and branch-2.3. I had to merge back a
> few
> >>> JIRAs committed between the swizzle and now where the fix version was
> 2.3
> >>> but weren't in branch-2.3.
> >>>
> >>> I think the only two HDFS and Common JIRAs that are marked for 2.4 are
> >>> these:
> >>>
> >>> HDFS-5842 Cannot create hftp filesystem when using a proxy user ugi
> and a
> >>> doAs on a secure cluster
> >>> HDFS-5781 Use an array to record the mapping between FSEditLogOpCode
> and
> >>> the corresponding byte value
> >>>
> >>> Jing, these both look safe to me if you want to merge them back, or I
> can
> >>> just do it.
> >>>
> >>> Thanks,
> >>> Andrew
> >>>
> >>> On Wed, Jan 29, 2014 at 1:21 PM, Doug Cutting <cu...@apache.org>
> wrote:
> >>>>
> >>>> On Wed, Jan 29, 2014 at 12:30 PM, Jason Lowe <jl...@yahoo-inc.com>
> wrote:
> >>>>> It is a bit concerning that the JIRA history showed that the target
> >>> version
> >>>>> was set at some point in the past but no record of it being cleared.
> >>>>
> >>>> Perhaps the version itself was renamed?
> >>>>
> >>>> Doug
> >>
> >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> > <signature.asc>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>
Re: Re-swizzle 2.3
Posted by Andrew Wang <an...@cloudera.com>.
Thanks for the link Arun, I went ahead and punted one HADOOP blocker, and
the remaining two HADOOP/HDFS looks like they're under active review.
Post-swizzle, it seems like most blockers for 2.4 would also apply to 2.3,
so I looked at that list too:
https://issues.apache.org/jira/issues/?filter=12326375&jql=project%20in%20(HADOOP%2C%20YARN%2C%20HDFS%2C%20MAPREDUCE)%20AND%20priority%20%3D%20Blocker%20AND%20status%20in%20(Open%2C%20%22In%20Progress%22%2C%20Reopened%2C%20%22Patch%20Available%22)%20AND%20%22Target%20Version%2Fs%22%20%3D%20%222.4.0%22
YARN-1673 IIUC relates to the AHS, so is actually only in branch-2 and not
branch-2.3.
HADOOP-10048, Jason's comment says he's okay with it not being a blocker.
HDFS-5796 hasn't seen much action. Kihwal or Haohui, could you comment on
the importance/status? I don't have much context in this area.
Best,
Andrew
On Fri, Jan 31, 2014 at 4:29 PM, Arun C Murthy <ac...@hortonworks.com> wrote:
> Thanks Vinod, appreciate it!
>
> I think we are very close.
>
> Here is a handy ref. to the list of blockers:
> http://s.apache.org/hadoop-2.3.0-blockers
>
> I'd appreciate if folks can help expedite these fixes, and, equally
> importantly bring up others they feel should be blockers for 2.3.0.
>
> thanks,
> Arun
>
> On Jan 30, 2014, at 12:42 PM, Vinod Kumar Vavilapalli <vi...@apache.org>
> wrote:
>
> > That was quite some exercise, but I'm done with it now. Updated YARN's
> and MAPREDUCE's CHANGES.txt on trunk, branch-2 and branch-2.3. Let me know
> if you find some inaccuracies.
> >
> > Thanks,
> > +Vinod
> >
> > On Jan 29, 2014, at 10:49 PM, Vinod Kumar Vavilapalli <
> vinodkv@apache.org> wrote:
> >
> >>
> >> Okay, I'll look at YARN and MR CHANGES.txt problems. Seems like they
> aren't addressed yet.
> >>
> >> +Vinod
> >>
> >>
> >> On Jan 29, 2014, at 3:24 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> >>
> >>> I just finished tuning up branch-2.3 and fixing up the HDFS and Common
> >>> CHANGES.txt in trunk, branch-2, and branch-2.3. I had to merge back a
> few
> >>> JIRAs committed between the swizzle and now where the fix version was
> 2.3
> >>> but weren't in branch-2.3.
> >>>
> >>> I think the only two HDFS and Common JIRAs that are marked for 2.4 are
> >>> these:
> >>>
> >>> HDFS-5842 Cannot create hftp filesystem when using a proxy user ugi
> and a
> >>> doAs on a secure cluster
> >>> HDFS-5781 Use an array to record the mapping between FSEditLogOpCode
> and
> >>> the corresponding byte value
> >>>
> >>> Jing, these both look safe to me if you want to merge them back, or I
> can
> >>> just do it.
> >>>
> >>> Thanks,
> >>> Andrew
> >>>
> >>> On Wed, Jan 29, 2014 at 1:21 PM, Doug Cutting <cu...@apache.org>
> wrote:
> >>>>
> >>>> On Wed, Jan 29, 2014 at 12:30 PM, Jason Lowe <jl...@yahoo-inc.com>
> wrote:
> >>>>> It is a bit concerning that the JIRA history showed that the target
> >>> version
> >>>>> was set at some point in the past but no record of it being cleared.
> >>>>
> >>>> Perhaps the version itself was renamed?
> >>>>
> >>>> Doug
> >>
> >
> >
> > --
> > CONFIDENTIALITY NOTICE
> > NOTICE: This message is intended for the use of the individual or entity
> to
> > which it is addressed and may contain information that is confidential,
> > privileged and exempt from disclosure under applicable law. If the reader
> > of this message is not the intended recipient, you are hereby notified
> that
> > any printing, copying, dissemination, distribution, disclosure or
> > forwarding of this communication is strictly prohibited. If you have
> > received this communication in error, please contact the sender
> immediately
> > and delete it from your system. Thank You.
> > <signature.asc>
>
> --
> Arun C. Murthy
> Hortonworks Inc.
> http://hortonworks.com/
>
>
>
> --
> CONFIDENTIALITY NOTICE
> NOTICE: This message is intended for the use of the individual or entity to
> which it is addressed and may contain information that is confidential,
> privileged and exempt from disclosure under applicable law. If the reader
> of this message is not the intended recipient, you are hereby notified that
> any printing, copying, dissemination, distribution, disclosure or
> forwarding of this communication is strictly prohibited. If you have
> received this communication in error, please contact the sender immediately
> and delete it from your system. Thank You.
>