You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Sangjin Lee <sj...@apache.org> on 2015/11/09 20:36:04 UTC
[YARN-2928] working on rebase with trunk
I am working on rebasing the YARN-2928 branch (timeline service v.2) with
trunk. So far it seems fairly straightforward, but I noted a few things
that require some investigation and discussion.
(1) TestDistributedShell failure (and timing out)
It appears that after the rebase the TestDistributedShell test is unhappy.
If I isolate the v2 tests only, I have a sporadic failure (missing the
application created event). If I run the test in its entirety, it times out
after 15 minutes. I suspect the failure is caused by YARN-4129
<https://issues.apache.org/jira/browse/YARN-4129>. I haven't tracked down
the cause of the timeout yet, though. This may or may not be related with
YARN-2859 <https://issues.apache.org/jira/browse/YARN-2859> that changed
the port that the AHS binds to, and the subsequent TestDistributedShell
test change (YARN-4326 <https://issues.apache.org/jira/browse/YARN-4326>).
We should file a JIRA to look into this once the rebase is done.
(2) reconciling with YARN-4183
<https://issues.apache.org/jira/browse/YARN-4183>
As Naga already pointed out on that JIRA, how we reconcile with the changes
made in YARN-4183 requires a discussion. It changed the config that drives
the RM's system metrics publisher from "timeline service enabled" to
"application history server enabled". I'm not entirely sure if that is the
right config to use for v.1, much less v.2. We should decide whether we
retain our version (continue to use "timeline service enabled"), and decide
how we enable v.1 and v.2 respectively. I think I'm going to preserve our
version in our branch for now, but we can certainly revisit it.
(3) reconciling with YARN-2556
<https://issues.apache.org/jira/browse/YARN-2556>
YARN-2556 was merged into trunk that delivers the timeline service
performance tests. Since we already merged the v.2 version of the tests in
our branch, I'm going to spend some time to make sure we eliminate code
duplication. I'll file a JIRA to clean this up after the rebase.
Please let me know if you have other feedback. Thanks!
Sangjin
Re: [YARN-2928] working on rebase with trunk
Posted by Sangjin Lee <sj...@apache.org>.
In order to unblock ourselves, for our branch I will push the result of the
rebase as a new branch named "feature-YARN-2928". The hope is that the
final resolution of the git rebase workflow would be close to something
like it and that it would be resolved before we have to rebase again.
Unless I hear objections soon, I'll push out that branch.
The implication is that any patches we generate for YARN-2928 from now on
would need to have the name such as "YARN-4053-feature-YARN-2928.001.patch".
Sangjin
On Mon, Nov 9, 2015 at 4:53 PM, Sangjin Lee <sj...@apache.org> wrote:
> I'm hitting a snag in git force push with the rebase (it's not allowing me
> to force push). I'll follow up with the infrastructure to see if we can
> swap branches. I'll update this email thread once it's all clear. Until
> then, please refrain from pulling and pushing branch YARN-2928.
>
> Thanks,
> Sangjin
>
> On Mon, Nov 9, 2015 at 3:33 PM, Naganarasimha G R (Naga) <
> garlanaganarasimha@huawei.com> wrote:
>
>> Hi Sangjin,
>>
>> Yes it would be fine you can go ahead,
>> Issue 1: i would take a look at it as it might be impacted by earlier
>> Jira YARN-4129. Once the merge is finished will take look into it.
>> Issue 2: "reconciling with YARN-4183": yes you are right it requires more
>> discussion and also some other approaches needs to be provided in V2 for
>> the scenarios mentioned by Jonathan Eagles
>>
>>
>>
>> Regards,
>>
>> + Naga
>>
>> ________________________________________
>> From: sjlee0@gmail.com [sjlee0@gmail.com] on behalf of Sangjin Lee [
>> sjlee@apache.org]
>> Sent: Tuesday, November 10, 2015 04:33
>> To: yarn-dev@hadoop.apache.org
>> Subject: Re: [YARN-2928] working on rebase with trunk
>>
>> I will go ahead and complete the rebase and force push based on what I
>> described above. It's likely you'll need to recreate your local YARN-2928
>> branch, and also rebase/adjust any private branch you may have had. FYI.
>> Thanks!
>>
>> Sangjin
>>
>> On Mon, Nov 9, 2015 at 11:36 AM, Sangjin Lee <sj...@apache.org> wrote:
>>
>> > I am working on rebasing the YARN-2928 branch (timeline service v.2)
>> with
>> > trunk. So far it seems fairly straightforward, but I noted a few things
>> > that require some investigation and discussion.
>> >
>> > (1) TestDistributedShell failure (and timing out)
>> > It appears that after the rebase the TestDistributedShell test is
>> unhappy.
>> > If I isolate the v2 tests only, I have a sporadic failure (missing the
>> > application created event). If I run the test in its entirety, it times
>> out
>> > after 15 minutes. I suspect the failure is caused by YARN-4129
>> > <https://issues.apache.org/jira/browse/YARN-4129>. I haven't tracked
>> down
>> > the cause of the timeout yet, though. This may or may not be related
>> with
>> > YARN-2859 <https://issues.apache.org/jira/browse/YARN-2859> that
>> changed
>> > the port that the AHS binds to, and the subsequent TestDistributedShell
>> > test change (YARN-4326 <https://issues.apache.org/jira/browse/YARN-4326
>> >).
>> > We should file a JIRA to look into this once the rebase is done.
>> >
>> > (2) reconciling with YARN-4183
>> > <https://issues.apache.org/jira/browse/YARN-4183>
>> > As Naga already pointed out on that JIRA, how we reconcile with the
>> > changes made in YARN-4183 requires a discussion. It changed the config
>> that
>> > drives the RM's system metrics publisher from "timeline service
>> enabled" to
>> > "application history server enabled". I'm not entirely sure if that is
>> the
>> > right config to use for v.1, much less v.2. We should decide whether we
>> > retain our version (continue to use "timeline service enabled"), and
>> decide
>> > how we enable v.1 and v.2 respectively. I think I'm going to preserve
>> our
>> > version in our branch for now, but we can certainly revisit it.
>> >
>> > (3) reconciling with YARN-2556
>> > <https://issues.apache.org/jira/browse/YARN-2556>
>> > YARN-2556 was merged into trunk that delivers the timeline service
>> > performance tests. Since we already merged the v.2 version of the tests
>> in
>> > our branch, I'm going to spend some time to make sure we eliminate code
>> > duplication. I'll file a JIRA to clean this up after the rebase.
>> >
>> > Please let me know if you have other feedback. Thanks!
>> >
>> > Sangjin
>> >
>>
>
>
Re: [YARN-2928] working on rebase with trunk
Posted by Karthik Kambatla <ka...@cloudera.com>.
Infra has recently disabled force-pushes for all branches. We should ask
for an exception for Hadoop. Let me start a common-dev thread.
On Mon, Nov 9, 2015 at 4:53 PM, Sangjin Lee <sj...@apache.org> wrote:
> I'm hitting a snag in git force push with the rebase (it's not allowing me
> to force push). I'll follow up with the infrastructure to see if we can
> swap branches. I'll update this email thread once it's all clear. Until
> then, please refrain from pulling and pushing branch YARN-2928.
>
> Thanks,
> Sangjin
>
> On Mon, Nov 9, 2015 at 3:33 PM, Naganarasimha G R (Naga) <
> garlanaganarasimha@huawei.com> wrote:
>
> > Hi Sangjin,
> >
> > Yes it would be fine you can go ahead,
> > Issue 1: i would take a look at it as it might be impacted by earlier
> > Jira YARN-4129. Once the merge is finished will take look into it.
> > Issue 2: "reconciling with YARN-4183": yes you are right it requires more
> > discussion and also some other approaches needs to be provided in V2 for
> > the scenarios mentioned by Jonathan Eagles
> >
> >
> >
> > Regards,
> >
> > + Naga
> >
> > ________________________________________
> > From: sjlee0@gmail.com [sjlee0@gmail.com] on behalf of Sangjin Lee [
> > sjlee@apache.org]
> > Sent: Tuesday, November 10, 2015 04:33
> > To: yarn-dev@hadoop.apache.org
> > Subject: Re: [YARN-2928] working on rebase with trunk
> >
> > I will go ahead and complete the rebase and force push based on what I
> > described above. It's likely you'll need to recreate your local YARN-2928
> > branch, and also rebase/adjust any private branch you may have had. FYI.
> > Thanks!
> >
> > Sangjin
> >
> > On Mon, Nov 9, 2015 at 11:36 AM, Sangjin Lee <sj...@apache.org> wrote:
> >
> > > I am working on rebasing the YARN-2928 branch (timeline service v.2)
> with
> > > trunk. So far it seems fairly straightforward, but I noted a few things
> > > that require some investigation and discussion.
> > >
> > > (1) TestDistributedShell failure (and timing out)
> > > It appears that after the rebase the TestDistributedShell test is
> > unhappy.
> > > If I isolate the v2 tests only, I have a sporadic failure (missing the
> > > application created event). If I run the test in its entirety, it times
> > out
> > > after 15 minutes. I suspect the failure is caused by YARN-4129
> > > <https://issues.apache.org/jira/browse/YARN-4129>. I haven't tracked
> > down
> > > the cause of the timeout yet, though. This may or may not be related
> with
> > > YARN-2859 <https://issues.apache.org/jira/browse/YARN-2859> that
> changed
> > > the port that the AHS binds to, and the subsequent TestDistributedShell
> > > test change (YARN-4326 <
> https://issues.apache.org/jira/browse/YARN-4326
> > >).
> > > We should file a JIRA to look into this once the rebase is done.
> > >
> > > (2) reconciling with YARN-4183
> > > <https://issues.apache.org/jira/browse/YARN-4183>
> > > As Naga already pointed out on that JIRA, how we reconcile with the
> > > changes made in YARN-4183 requires a discussion. It changed the config
> > that
> > > drives the RM's system metrics publisher from "timeline service
> enabled"
> > to
> > > "application history server enabled". I'm not entirely sure if that is
> > the
> > > right config to use for v.1, much less v.2. We should decide whether we
> > > retain our version (continue to use "timeline service enabled"), and
> > decide
> > > how we enable v.1 and v.2 respectively. I think I'm going to preserve
> our
> > > version in our branch for now, but we can certainly revisit it.
> > >
> > > (3) reconciling with YARN-2556
> > > <https://issues.apache.org/jira/browse/YARN-2556>
> > > YARN-2556 was merged into trunk that delivers the timeline service
> > > performance tests. Since we already merged the v.2 version of the tests
> > in
> > > our branch, I'm going to spend some time to make sure we eliminate code
> > > duplication. I'll file a JIRA to clean this up after the rebase.
> > >
> > > Please let me know if you have other feedback. Thanks!
> > >
> > > Sangjin
> > >
> >
>
Re: [YARN-2928] working on rebase with trunk
Posted by Varun Saxena <vs...@gmail.com>.
Yes, able to clone from this branch now.
Sorry for the trouble.
Not sure what the issue was at that time.
Thanks,
Varun Saxena.
On Thu, Nov 12, 2015 at 11:28 PM, Sangjin Lee <sj...@apache.org> wrote:
> The branch clearly exists in git-wip-us.apache.org:
> https://git-wip-us.apache.org/repos/asf?p=hadoop.git;a=heads
>
> Which git remote repo are you using?
>
> On Thu, Nov 12, 2015 at 8:08 AM, Varun Saxena <vs...@gmail.com>
> wrote:
>
> > Hi Sangjn,
> >
> > I am unable to clone the branch feature-YARN-2928.
> > It does not exist I guess.
> > I can find a branch YARN-2928-rebase which seems to have recent
> commits(for
> > instance, YARN-3840 and HDFS-9249).
> > Is that the one to use ?
> >
> > Thanks,
> > Varun Saxena.
> >
> >
> > On Thu, Nov 12, 2015 at 7:36 PM, Varun Saxena <vs...@gmail.com>
> > wrote:
> >
> > > Thanks a lot Sangjin !
> > >
> > > Varun Saxena.
> > >
> > > On Thu, Nov 12, 2015 at 6:52 AM, Sangjin Lee <sj...@apache.org> wrote:
> > >
> > >> I pushed the new branch (feature-YARN-2928). Please start using this
> > >> branch
> > >> from now on to see the timeline service v.2 work. If you're working
> on a
> > >> patch, please use the new branch name in the file name.
> > >>
> > >> Thanks,
> > >> Sangjin
> > >>
> > >> On Mon, Nov 9, 2015 at 4:53 PM, Sangjin Lee <sj...@apache.org> wrote:
> > >>
> > >> > I'm hitting a snag in git force push with the rebase (it's not
> > allowing
> > >> me
> > >> > to force push). I'll follow up with the infrastructure to see if we
> > can
> > >> > swap branches. I'll update this email thread once it's all clear.
> > Until
> > >> > then, please refrain from pulling and pushing branch YARN-2928.
> > >> >
> > >> > Thanks,
> > >> > Sangjin
> > >> >
> > >> > On Mon, Nov 9, 2015 at 3:33 PM, Naganarasimha G R (Naga) <
> > >> > garlanaganarasimha@huawei.com> wrote:
> > >> >
> > >> >> Hi Sangjin,
> > >> >>
> > >> >> Yes it would be fine you can go ahead,
> > >> >> Issue 1: i would take a look at it as it might be impacted by
> > earlier
> > >> >> Jira YARN-4129. Once the merge is finished will take look into it.
> > >> >> Issue 2: "reconciling with YARN-4183": yes you are right it
> requires
> > >> more
> > >> >> discussion and also some other approaches needs to be provided in
> V2
> > >> for
> > >> >> the scenarios mentioned by Jonathan Eagles
> > >> >>
> > >> >>
> > >> >>
> > >> >> Regards,
> > >> >>
> > >> >> + Naga
> > >> >>
> > >> >> ________________________________________
> > >> >> From: sjlee0@gmail.com [sjlee0@gmail.com] on behalf of Sangjin
> Lee [
> > >> >> sjlee@apache.org]
> > >> >> Sent: Tuesday, November 10, 2015 04:33
> > >> >> To: yarn-dev@hadoop.apache.org
> > >> >> Subject: Re: [YARN-2928] working on rebase with trunk
> > >> >>
> > >> >> I will go ahead and complete the rebase and force push based on
> what
> > I
> > >> >> described above. It's likely you'll need to recreate your local
> > >> YARN-2928
> > >> >> branch, and also rebase/adjust any private branch you may have had.
> > >> FYI.
> > >> >> Thanks!
> > >> >>
> > >> >> Sangjin
> > >> >>
> > >> >> On Mon, Nov 9, 2015 at 11:36 AM, Sangjin Lee <sj...@apache.org>
> > wrote:
> > >> >>
> > >> >> > I am working on rebasing the YARN-2928 branch (timeline service
> > v.2)
> > >> >> with
> > >> >> > trunk. So far it seems fairly straightforward, but I noted a few
> > >> things
> > >> >> > that require some investigation and discussion.
> > >> >> >
> > >> >> > (1) TestDistributedShell failure (and timing out)
> > >> >> > It appears that after the rebase the TestDistributedShell test is
> > >> >> unhappy.
> > >> >> > If I isolate the v2 tests only, I have a sporadic failure
> (missing
> > >> the
> > >> >> > application created event). If I run the test in its entirety, it
> > >> times
> > >> >> out
> > >> >> > after 15 minutes. I suspect the failure is caused by YARN-4129
> > >> >> > <https://issues.apache.org/jira/browse/YARN-4129>. I haven't
> > tracked
> > >> >> down
> > >> >> > the cause of the timeout yet, though. This may or may not be
> > related
> > >> >> with
> > >> >> > YARN-2859 <https://issues.apache.org/jira/browse/YARN-2859> that
> > >> >> changed
> > >> >> > the port that the AHS binds to, and the subsequent
> > >> TestDistributedShell
> > >> >> > test change (YARN-4326 <
> > >> https://issues.apache.org/jira/browse/YARN-4326
> > >> >> >).
> > >> >> > We should file a JIRA to look into this once the rebase is done.
> > >> >> >
> > >> >> > (2) reconciling with YARN-4183
> > >> >> > <https://issues.apache.org/jira/browse/YARN-4183>
> > >> >> > As Naga already pointed out on that JIRA, how we reconcile with
> the
> > >> >> > changes made in YARN-4183 requires a discussion. It changed the
> > >> config
> > >> >> that
> > >> >> > drives the RM's system metrics publisher from "timeline service
> > >> >> enabled" to
> > >> >> > "application history server enabled". I'm not entirely sure if
> that
> > >> is
> > >> >> the
> > >> >> > right config to use for v.1, much less v.2. We should decide
> > whether
> > >> we
> > >> >> > retain our version (continue to use "timeline service enabled"),
> > and
> > >> >> decide
> > >> >> > how we enable v.1 and v.2 respectively. I think I'm going to
> > preserve
> > >> >> our
> > >> >> > version in our branch for now, but we can certainly revisit it.
> > >> >> >
> > >> >> > (3) reconciling with YARN-2556
> > >> >> > <https://issues.apache.org/jira/browse/YARN-2556>
> > >> >> > YARN-2556 was merged into trunk that delivers the timeline
> service
> > >> >> > performance tests. Since we already merged the v.2 version of the
> > >> tests
> > >> >> in
> > >> >> > our branch, I'm going to spend some time to make sure we
> eliminate
> > >> code
> > >> >> > duplication. I'll file a JIRA to clean this up after the rebase.
> > >> >> >
> > >> >> > Please let me know if you have other feedback. Thanks!
> > >> >> >
> > >> >> > Sangjin
> > >> >> >
> > >> >>
> > >> >
> > >> >
> > >>
> > >
> > >
> >
>
Re: [YARN-2928] working on rebase with trunk
Posted by Sangjin Lee <sj...@apache.org>.
The branch clearly exists in git-wip-us.apache.org:
https://git-wip-us.apache.org/repos/asf?p=hadoop.git;a=heads
Which git remote repo are you using?
On Thu, Nov 12, 2015 at 8:08 AM, Varun Saxena <vs...@gmail.com>
wrote:
> Hi Sangjn,
>
> I am unable to clone the branch feature-YARN-2928.
> It does not exist I guess.
> I can find a branch YARN-2928-rebase which seems to have recent commits(for
> instance, YARN-3840 and HDFS-9249).
> Is that the one to use ?
>
> Thanks,
> Varun Saxena.
>
>
> On Thu, Nov 12, 2015 at 7:36 PM, Varun Saxena <vs...@gmail.com>
> wrote:
>
> > Thanks a lot Sangjin !
> >
> > Varun Saxena.
> >
> > On Thu, Nov 12, 2015 at 6:52 AM, Sangjin Lee <sj...@apache.org> wrote:
> >
> >> I pushed the new branch (feature-YARN-2928). Please start using this
> >> branch
> >> from now on to see the timeline service v.2 work. If you're working on a
> >> patch, please use the new branch name in the file name.
> >>
> >> Thanks,
> >> Sangjin
> >>
> >> On Mon, Nov 9, 2015 at 4:53 PM, Sangjin Lee <sj...@apache.org> wrote:
> >>
> >> > I'm hitting a snag in git force push with the rebase (it's not
> allowing
> >> me
> >> > to force push). I'll follow up with the infrastructure to see if we
> can
> >> > swap branches. I'll update this email thread once it's all clear.
> Until
> >> > then, please refrain from pulling and pushing branch YARN-2928.
> >> >
> >> > Thanks,
> >> > Sangjin
> >> >
> >> > On Mon, Nov 9, 2015 at 3:33 PM, Naganarasimha G R (Naga) <
> >> > garlanaganarasimha@huawei.com> wrote:
> >> >
> >> >> Hi Sangjin,
> >> >>
> >> >> Yes it would be fine you can go ahead,
> >> >> Issue 1: i would take a look at it as it might be impacted by
> earlier
> >> >> Jira YARN-4129. Once the merge is finished will take look into it.
> >> >> Issue 2: "reconciling with YARN-4183": yes you are right it requires
> >> more
> >> >> discussion and also some other approaches needs to be provided in V2
> >> for
> >> >> the scenarios mentioned by Jonathan Eagles
> >> >>
> >> >>
> >> >>
> >> >> Regards,
> >> >>
> >> >> + Naga
> >> >>
> >> >> ________________________________________
> >> >> From: sjlee0@gmail.com [sjlee0@gmail.com] on behalf of Sangjin Lee [
> >> >> sjlee@apache.org]
> >> >> Sent: Tuesday, November 10, 2015 04:33
> >> >> To: yarn-dev@hadoop.apache.org
> >> >> Subject: Re: [YARN-2928] working on rebase with trunk
> >> >>
> >> >> I will go ahead and complete the rebase and force push based on what
> I
> >> >> described above. It's likely you'll need to recreate your local
> >> YARN-2928
> >> >> branch, and also rebase/adjust any private branch you may have had.
> >> FYI.
> >> >> Thanks!
> >> >>
> >> >> Sangjin
> >> >>
> >> >> On Mon, Nov 9, 2015 at 11:36 AM, Sangjin Lee <sj...@apache.org>
> wrote:
> >> >>
> >> >> > I am working on rebasing the YARN-2928 branch (timeline service
> v.2)
> >> >> with
> >> >> > trunk. So far it seems fairly straightforward, but I noted a few
> >> things
> >> >> > that require some investigation and discussion.
> >> >> >
> >> >> > (1) TestDistributedShell failure (and timing out)
> >> >> > It appears that after the rebase the TestDistributedShell test is
> >> >> unhappy.
> >> >> > If I isolate the v2 tests only, I have a sporadic failure (missing
> >> the
> >> >> > application created event). If I run the test in its entirety, it
> >> times
> >> >> out
> >> >> > after 15 minutes. I suspect the failure is caused by YARN-4129
> >> >> > <https://issues.apache.org/jira/browse/YARN-4129>. I haven't
> tracked
> >> >> down
> >> >> > the cause of the timeout yet, though. This may or may not be
> related
> >> >> with
> >> >> > YARN-2859 <https://issues.apache.org/jira/browse/YARN-2859> that
> >> >> changed
> >> >> > the port that the AHS binds to, and the subsequent
> >> TestDistributedShell
> >> >> > test change (YARN-4326 <
> >> https://issues.apache.org/jira/browse/YARN-4326
> >> >> >).
> >> >> > We should file a JIRA to look into this once the rebase is done.
> >> >> >
> >> >> > (2) reconciling with YARN-4183
> >> >> > <https://issues.apache.org/jira/browse/YARN-4183>
> >> >> > As Naga already pointed out on that JIRA, how we reconcile with the
> >> >> > changes made in YARN-4183 requires a discussion. It changed the
> >> config
> >> >> that
> >> >> > drives the RM's system metrics publisher from "timeline service
> >> >> enabled" to
> >> >> > "application history server enabled". I'm not entirely sure if that
> >> is
> >> >> the
> >> >> > right config to use for v.1, much less v.2. We should decide
> whether
> >> we
> >> >> > retain our version (continue to use "timeline service enabled"),
> and
> >> >> decide
> >> >> > how we enable v.1 and v.2 respectively. I think I'm going to
> preserve
> >> >> our
> >> >> > version in our branch for now, but we can certainly revisit it.
> >> >> >
> >> >> > (3) reconciling with YARN-2556
> >> >> > <https://issues.apache.org/jira/browse/YARN-2556>
> >> >> > YARN-2556 was merged into trunk that delivers the timeline service
> >> >> > performance tests. Since we already merged the v.2 version of the
> >> tests
> >> >> in
> >> >> > our branch, I'm going to spend some time to make sure we eliminate
> >> code
> >> >> > duplication. I'll file a JIRA to clean this up after the rebase.
> >> >> >
> >> >> > Please let me know if you have other feedback. Thanks!
> >> >> >
> >> >> > Sangjin
> >> >> >
> >> >>
> >> >
> >> >
> >>
> >
> >
>
Re: [YARN-2928] working on rebase with trunk
Posted by Varun Saxena <vs...@gmail.com>.
Hi Sangjn,
I am unable to clone the branch feature-YARN-2928.
It does not exist I guess.
I can find a branch YARN-2928-rebase which seems to have recent commits(for
instance, YARN-3840 and HDFS-9249).
Is that the one to use ?
Thanks,
Varun Saxena.
On Thu, Nov 12, 2015 at 7:36 PM, Varun Saxena <vs...@gmail.com>
wrote:
> Thanks a lot Sangjin !
>
> Varun Saxena.
>
> On Thu, Nov 12, 2015 at 6:52 AM, Sangjin Lee <sj...@apache.org> wrote:
>
>> I pushed the new branch (feature-YARN-2928). Please start using this
>> branch
>> from now on to see the timeline service v.2 work. If you're working on a
>> patch, please use the new branch name in the file name.
>>
>> Thanks,
>> Sangjin
>>
>> On Mon, Nov 9, 2015 at 4:53 PM, Sangjin Lee <sj...@apache.org> wrote:
>>
>> > I'm hitting a snag in git force push with the rebase (it's not allowing
>> me
>> > to force push). I'll follow up with the infrastructure to see if we can
>> > swap branches. I'll update this email thread once it's all clear. Until
>> > then, please refrain from pulling and pushing branch YARN-2928.
>> >
>> > Thanks,
>> > Sangjin
>> >
>> > On Mon, Nov 9, 2015 at 3:33 PM, Naganarasimha G R (Naga) <
>> > garlanaganarasimha@huawei.com> wrote:
>> >
>> >> Hi Sangjin,
>> >>
>> >> Yes it would be fine you can go ahead,
>> >> Issue 1: i would take a look at it as it might be impacted by earlier
>> >> Jira YARN-4129. Once the merge is finished will take look into it.
>> >> Issue 2: "reconciling with YARN-4183": yes you are right it requires
>> more
>> >> discussion and also some other approaches needs to be provided in V2
>> for
>> >> the scenarios mentioned by Jonathan Eagles
>> >>
>> >>
>> >>
>> >> Regards,
>> >>
>> >> + Naga
>> >>
>> >> ________________________________________
>> >> From: sjlee0@gmail.com [sjlee0@gmail.com] on behalf of Sangjin Lee [
>> >> sjlee@apache.org]
>> >> Sent: Tuesday, November 10, 2015 04:33
>> >> To: yarn-dev@hadoop.apache.org
>> >> Subject: Re: [YARN-2928] working on rebase with trunk
>> >>
>> >> I will go ahead and complete the rebase and force push based on what I
>> >> described above. It's likely you'll need to recreate your local
>> YARN-2928
>> >> branch, and also rebase/adjust any private branch you may have had.
>> FYI.
>> >> Thanks!
>> >>
>> >> Sangjin
>> >>
>> >> On Mon, Nov 9, 2015 at 11:36 AM, Sangjin Lee <sj...@apache.org> wrote:
>> >>
>> >> > I am working on rebasing the YARN-2928 branch (timeline service v.2)
>> >> with
>> >> > trunk. So far it seems fairly straightforward, but I noted a few
>> things
>> >> > that require some investigation and discussion.
>> >> >
>> >> > (1) TestDistributedShell failure (and timing out)
>> >> > It appears that after the rebase the TestDistributedShell test is
>> >> unhappy.
>> >> > If I isolate the v2 tests only, I have a sporadic failure (missing
>> the
>> >> > application created event). If I run the test in its entirety, it
>> times
>> >> out
>> >> > after 15 minutes. I suspect the failure is caused by YARN-4129
>> >> > <https://issues.apache.org/jira/browse/YARN-4129>. I haven't tracked
>> >> down
>> >> > the cause of the timeout yet, though. This may or may not be related
>> >> with
>> >> > YARN-2859 <https://issues.apache.org/jira/browse/YARN-2859> that
>> >> changed
>> >> > the port that the AHS binds to, and the subsequent
>> TestDistributedShell
>> >> > test change (YARN-4326 <
>> https://issues.apache.org/jira/browse/YARN-4326
>> >> >).
>> >> > We should file a JIRA to look into this once the rebase is done.
>> >> >
>> >> > (2) reconciling with YARN-4183
>> >> > <https://issues.apache.org/jira/browse/YARN-4183>
>> >> > As Naga already pointed out on that JIRA, how we reconcile with the
>> >> > changes made in YARN-4183 requires a discussion. It changed the
>> config
>> >> that
>> >> > drives the RM's system metrics publisher from "timeline service
>> >> enabled" to
>> >> > "application history server enabled". I'm not entirely sure if that
>> is
>> >> the
>> >> > right config to use for v.1, much less v.2. We should decide whether
>> we
>> >> > retain our version (continue to use "timeline service enabled"), and
>> >> decide
>> >> > how we enable v.1 and v.2 respectively. I think I'm going to preserve
>> >> our
>> >> > version in our branch for now, but we can certainly revisit it.
>> >> >
>> >> > (3) reconciling with YARN-2556
>> >> > <https://issues.apache.org/jira/browse/YARN-2556>
>> >> > YARN-2556 was merged into trunk that delivers the timeline service
>> >> > performance tests. Since we already merged the v.2 version of the
>> tests
>> >> in
>> >> > our branch, I'm going to spend some time to make sure we eliminate
>> code
>> >> > duplication. I'll file a JIRA to clean this up after the rebase.
>> >> >
>> >> > Please let me know if you have other feedback. Thanks!
>> >> >
>> >> > Sangjin
>> >> >
>> >>
>> >
>> >
>>
>
>
Re: [YARN-2928] working on rebase with trunk
Posted by Varun Saxena <vs...@gmail.com>.
Thanks a lot Sangjin !
Varun Saxena.
On Thu, Nov 12, 2015 at 6:52 AM, Sangjin Lee <sj...@apache.org> wrote:
> I pushed the new branch (feature-YARN-2928). Please start using this branch
> from now on to see the timeline service v.2 work. If you're working on a
> patch, please use the new branch name in the file name.
>
> Thanks,
> Sangjin
>
> On Mon, Nov 9, 2015 at 4:53 PM, Sangjin Lee <sj...@apache.org> wrote:
>
> > I'm hitting a snag in git force push with the rebase (it's not allowing
> me
> > to force push). I'll follow up with the infrastructure to see if we can
> > swap branches. I'll update this email thread once it's all clear. Until
> > then, please refrain from pulling and pushing branch YARN-2928.
> >
> > Thanks,
> > Sangjin
> >
> > On Mon, Nov 9, 2015 at 3:33 PM, Naganarasimha G R (Naga) <
> > garlanaganarasimha@huawei.com> wrote:
> >
> >> Hi Sangjin,
> >>
> >> Yes it would be fine you can go ahead,
> >> Issue 1: i would take a look at it as it might be impacted by earlier
> >> Jira YARN-4129. Once the merge is finished will take look into it.
> >> Issue 2: "reconciling with YARN-4183": yes you are right it requires
> more
> >> discussion and also some other approaches needs to be provided in V2 for
> >> the scenarios mentioned by Jonathan Eagles
> >>
> >>
> >>
> >> Regards,
> >>
> >> + Naga
> >>
> >> ________________________________________
> >> From: sjlee0@gmail.com [sjlee0@gmail.com] on behalf of Sangjin Lee [
> >> sjlee@apache.org]
> >> Sent: Tuesday, November 10, 2015 04:33
> >> To: yarn-dev@hadoop.apache.org
> >> Subject: Re: [YARN-2928] working on rebase with trunk
> >>
> >> I will go ahead and complete the rebase and force push based on what I
> >> described above. It's likely you'll need to recreate your local
> YARN-2928
> >> branch, and also rebase/adjust any private branch you may have had. FYI.
> >> Thanks!
> >>
> >> Sangjin
> >>
> >> On Mon, Nov 9, 2015 at 11:36 AM, Sangjin Lee <sj...@apache.org> wrote:
> >>
> >> > I am working on rebasing the YARN-2928 branch (timeline service v.2)
> >> with
> >> > trunk. So far it seems fairly straightforward, but I noted a few
> things
> >> > that require some investigation and discussion.
> >> >
> >> > (1) TestDistributedShell failure (and timing out)
> >> > It appears that after the rebase the TestDistributedShell test is
> >> unhappy.
> >> > If I isolate the v2 tests only, I have a sporadic failure (missing the
> >> > application created event). If I run the test in its entirety, it
> times
> >> out
> >> > after 15 minutes. I suspect the failure is caused by YARN-4129
> >> > <https://issues.apache.org/jira/browse/YARN-4129>. I haven't tracked
> >> down
> >> > the cause of the timeout yet, though. This may or may not be related
> >> with
> >> > YARN-2859 <https://issues.apache.org/jira/browse/YARN-2859> that
> >> changed
> >> > the port that the AHS binds to, and the subsequent
> TestDistributedShell
> >> > test change (YARN-4326 <
> https://issues.apache.org/jira/browse/YARN-4326
> >> >).
> >> > We should file a JIRA to look into this once the rebase is done.
> >> >
> >> > (2) reconciling with YARN-4183
> >> > <https://issues.apache.org/jira/browse/YARN-4183>
> >> > As Naga already pointed out on that JIRA, how we reconcile with the
> >> > changes made in YARN-4183 requires a discussion. It changed the config
> >> that
> >> > drives the RM's system metrics publisher from "timeline service
> >> enabled" to
> >> > "application history server enabled". I'm not entirely sure if that is
> >> the
> >> > right config to use for v.1, much less v.2. We should decide whether
> we
> >> > retain our version (continue to use "timeline service enabled"), and
> >> decide
> >> > how we enable v.1 and v.2 respectively. I think I'm going to preserve
> >> our
> >> > version in our branch for now, but we can certainly revisit it.
> >> >
> >> > (3) reconciling with YARN-2556
> >> > <https://issues.apache.org/jira/browse/YARN-2556>
> >> > YARN-2556 was merged into trunk that delivers the timeline service
> >> > performance tests. Since we already merged the v.2 version of the
> tests
> >> in
> >> > our branch, I'm going to spend some time to make sure we eliminate
> code
> >> > duplication. I'll file a JIRA to clean this up after the rebase.
> >> >
> >> > Please let me know if you have other feedback. Thanks!
> >> >
> >> > Sangjin
> >> >
> >>
> >
> >
>
Re: [YARN-2928] working on rebase with trunk
Posted by Sangjin Lee <sj...@apache.org>.
I pushed the new branch (feature-YARN-2928). Please start using this branch
from now on to see the timeline service v.2 work. If you're working on a
patch, please use the new branch name in the file name.
Thanks,
Sangjin
On Mon, Nov 9, 2015 at 4:53 PM, Sangjin Lee <sj...@apache.org> wrote:
> I'm hitting a snag in git force push with the rebase (it's not allowing me
> to force push). I'll follow up with the infrastructure to see if we can
> swap branches. I'll update this email thread once it's all clear. Until
> then, please refrain from pulling and pushing branch YARN-2928.
>
> Thanks,
> Sangjin
>
> On Mon, Nov 9, 2015 at 3:33 PM, Naganarasimha G R (Naga) <
> garlanaganarasimha@huawei.com> wrote:
>
>> Hi Sangjin,
>>
>> Yes it would be fine you can go ahead,
>> Issue 1: i would take a look at it as it might be impacted by earlier
>> Jira YARN-4129. Once the merge is finished will take look into it.
>> Issue 2: "reconciling with YARN-4183": yes you are right it requires more
>> discussion and also some other approaches needs to be provided in V2 for
>> the scenarios mentioned by Jonathan Eagles
>>
>>
>>
>> Regards,
>>
>> + Naga
>>
>> ________________________________________
>> From: sjlee0@gmail.com [sjlee0@gmail.com] on behalf of Sangjin Lee [
>> sjlee@apache.org]
>> Sent: Tuesday, November 10, 2015 04:33
>> To: yarn-dev@hadoop.apache.org
>> Subject: Re: [YARN-2928] working on rebase with trunk
>>
>> I will go ahead and complete the rebase and force push based on what I
>> described above. It's likely you'll need to recreate your local YARN-2928
>> branch, and also rebase/adjust any private branch you may have had. FYI.
>> Thanks!
>>
>> Sangjin
>>
>> On Mon, Nov 9, 2015 at 11:36 AM, Sangjin Lee <sj...@apache.org> wrote:
>>
>> > I am working on rebasing the YARN-2928 branch (timeline service v.2)
>> with
>> > trunk. So far it seems fairly straightforward, but I noted a few things
>> > that require some investigation and discussion.
>> >
>> > (1) TestDistributedShell failure (and timing out)
>> > It appears that after the rebase the TestDistributedShell test is
>> unhappy.
>> > If I isolate the v2 tests only, I have a sporadic failure (missing the
>> > application created event). If I run the test in its entirety, it times
>> out
>> > after 15 minutes. I suspect the failure is caused by YARN-4129
>> > <https://issues.apache.org/jira/browse/YARN-4129>. I haven't tracked
>> down
>> > the cause of the timeout yet, though. This may or may not be related
>> with
>> > YARN-2859 <https://issues.apache.org/jira/browse/YARN-2859> that
>> changed
>> > the port that the AHS binds to, and the subsequent TestDistributedShell
>> > test change (YARN-4326 <https://issues.apache.org/jira/browse/YARN-4326
>> >).
>> > We should file a JIRA to look into this once the rebase is done.
>> >
>> > (2) reconciling with YARN-4183
>> > <https://issues.apache.org/jira/browse/YARN-4183>
>> > As Naga already pointed out on that JIRA, how we reconcile with the
>> > changes made in YARN-4183 requires a discussion. It changed the config
>> that
>> > drives the RM's system metrics publisher from "timeline service
>> enabled" to
>> > "application history server enabled". I'm not entirely sure if that is
>> the
>> > right config to use for v.1, much less v.2. We should decide whether we
>> > retain our version (continue to use "timeline service enabled"), and
>> decide
>> > how we enable v.1 and v.2 respectively. I think I'm going to preserve
>> our
>> > version in our branch for now, but we can certainly revisit it.
>> >
>> > (3) reconciling with YARN-2556
>> > <https://issues.apache.org/jira/browse/YARN-2556>
>> > YARN-2556 was merged into trunk that delivers the timeline service
>> > performance tests. Since we already merged the v.2 version of the tests
>> in
>> > our branch, I'm going to spend some time to make sure we eliminate code
>> > duplication. I'll file a JIRA to clean this up after the rebase.
>> >
>> > Please let me know if you have other feedback. Thanks!
>> >
>> > Sangjin
>> >
>>
>
>
Re: [YARN-2928] working on rebase with trunk
Posted by Sangjin Lee <sj...@apache.org>.
I'm hitting a snag in git force push with the rebase (it's not allowing me
to force push). I'll follow up with the infrastructure to see if we can
swap branches. I'll update this email thread once it's all clear. Until
then, please refrain from pulling and pushing branch YARN-2928.
Thanks,
Sangjin
On Mon, Nov 9, 2015 at 3:33 PM, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Sangjin,
>
> Yes it would be fine you can go ahead,
> Issue 1: i would take a look at it as it might be impacted by earlier
> Jira YARN-4129. Once the merge is finished will take look into it.
> Issue 2: "reconciling with YARN-4183": yes you are right it requires more
> discussion and also some other approaches needs to be provided in V2 for
> the scenarios mentioned by Jonathan Eagles
>
>
>
> Regards,
>
> + Naga
>
> ________________________________________
> From: sjlee0@gmail.com [sjlee0@gmail.com] on behalf of Sangjin Lee [
> sjlee@apache.org]
> Sent: Tuesday, November 10, 2015 04:33
> To: yarn-dev@hadoop.apache.org
> Subject: Re: [YARN-2928] working on rebase with trunk
>
> I will go ahead and complete the rebase and force push based on what I
> described above. It's likely you'll need to recreate your local YARN-2928
> branch, and also rebase/adjust any private branch you may have had. FYI.
> Thanks!
>
> Sangjin
>
> On Mon, Nov 9, 2015 at 11:36 AM, Sangjin Lee <sj...@apache.org> wrote:
>
> > I am working on rebasing the YARN-2928 branch (timeline service v.2) with
> > trunk. So far it seems fairly straightforward, but I noted a few things
> > that require some investigation and discussion.
> >
> > (1) TestDistributedShell failure (and timing out)
> > It appears that after the rebase the TestDistributedShell test is
> unhappy.
> > If I isolate the v2 tests only, I have a sporadic failure (missing the
> > application created event). If I run the test in its entirety, it times
> out
> > after 15 minutes. I suspect the failure is caused by YARN-4129
> > <https://issues.apache.org/jira/browse/YARN-4129>. I haven't tracked
> down
> > the cause of the timeout yet, though. This may or may not be related with
> > YARN-2859 <https://issues.apache.org/jira/browse/YARN-2859> that changed
> > the port that the AHS binds to, and the subsequent TestDistributedShell
> > test change (YARN-4326 <https://issues.apache.org/jira/browse/YARN-4326
> >).
> > We should file a JIRA to look into this once the rebase is done.
> >
> > (2) reconciling with YARN-4183
> > <https://issues.apache.org/jira/browse/YARN-4183>
> > As Naga already pointed out on that JIRA, how we reconcile with the
> > changes made in YARN-4183 requires a discussion. It changed the config
> that
> > drives the RM's system metrics publisher from "timeline service enabled"
> to
> > "application history server enabled". I'm not entirely sure if that is
> the
> > right config to use for v.1, much less v.2. We should decide whether we
> > retain our version (continue to use "timeline service enabled"), and
> decide
> > how we enable v.1 and v.2 respectively. I think I'm going to preserve our
> > version in our branch for now, but we can certainly revisit it.
> >
> > (3) reconciling with YARN-2556
> > <https://issues.apache.org/jira/browse/YARN-2556>
> > YARN-2556 was merged into trunk that delivers the timeline service
> > performance tests. Since we already merged the v.2 version of the tests
> in
> > our branch, I'm going to spend some time to make sure we eliminate code
> > duplication. I'll file a JIRA to clean this up after the rebase.
> >
> > Please let me know if you have other feedback. Thanks!
> >
> > Sangjin
> >
>
RE: [YARN-2928] working on rebase with trunk
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Sangjin,
Yes it would be fine you can go ahead,
Issue 1: i would take a look at it as it might be impacted by earlier Jira YARN-4129. Once the merge is finished will take look into it.
Issue 2: "reconciling with YARN-4183": yes you are right it requires more discussion and also some other approaches needs to be provided in V2 for the scenarios mentioned by Jonathan Eagles
Regards,
+ Naga
________________________________________
From: sjlee0@gmail.com [sjlee0@gmail.com] on behalf of Sangjin Lee [sjlee@apache.org]
Sent: Tuesday, November 10, 2015 04:33
To: yarn-dev@hadoop.apache.org
Subject: Re: [YARN-2928] working on rebase with trunk
I will go ahead and complete the rebase and force push based on what I
described above. It's likely you'll need to recreate your local YARN-2928
branch, and also rebase/adjust any private branch you may have had. FYI.
Thanks!
Sangjin
On Mon, Nov 9, 2015 at 11:36 AM, Sangjin Lee <sj...@apache.org> wrote:
> I am working on rebasing the YARN-2928 branch (timeline service v.2) with
> trunk. So far it seems fairly straightforward, but I noted a few things
> that require some investigation and discussion.
>
> (1) TestDistributedShell failure (and timing out)
> It appears that after the rebase the TestDistributedShell test is unhappy.
> If I isolate the v2 tests only, I have a sporadic failure (missing the
> application created event). If I run the test in its entirety, it times out
> after 15 minutes. I suspect the failure is caused by YARN-4129
> <https://issues.apache.org/jira/browse/YARN-4129>. I haven't tracked down
> the cause of the timeout yet, though. This may or may not be related with
> YARN-2859 <https://issues.apache.org/jira/browse/YARN-2859> that changed
> the port that the AHS binds to, and the subsequent TestDistributedShell
> test change (YARN-4326 <https://issues.apache.org/jira/browse/YARN-4326>).
> We should file a JIRA to look into this once the rebase is done.
>
> (2) reconciling with YARN-4183
> <https://issues.apache.org/jira/browse/YARN-4183>
> As Naga already pointed out on that JIRA, how we reconcile with the
> changes made in YARN-4183 requires a discussion. It changed the config that
> drives the RM's system metrics publisher from "timeline service enabled" to
> "application history server enabled". I'm not entirely sure if that is the
> right config to use for v.1, much less v.2. We should decide whether we
> retain our version (continue to use "timeline service enabled"), and decide
> how we enable v.1 and v.2 respectively. I think I'm going to preserve our
> version in our branch for now, but we can certainly revisit it.
>
> (3) reconciling with YARN-2556
> <https://issues.apache.org/jira/browse/YARN-2556>
> YARN-2556 was merged into trunk that delivers the timeline service
> performance tests. Since we already merged the v.2 version of the tests in
> our branch, I'm going to spend some time to make sure we eliminate code
> duplication. I'll file a JIRA to clean this up after the rebase.
>
> Please let me know if you have other feedback. Thanks!
>
> Sangjin
>
Re: [YARN-2928] working on rebase with trunk
Posted by Sangjin Lee <sj...@apache.org>.
I will go ahead and complete the rebase and force push based on what I
described above. It's likely you'll need to recreate your local YARN-2928
branch, and also rebase/adjust any private branch you may have had. FYI.
Thanks!
Sangjin
On Mon, Nov 9, 2015 at 11:36 AM, Sangjin Lee <sj...@apache.org> wrote:
> I am working on rebasing the YARN-2928 branch (timeline service v.2) with
> trunk. So far it seems fairly straightforward, but I noted a few things
> that require some investigation and discussion.
>
> (1) TestDistributedShell failure (and timing out)
> It appears that after the rebase the TestDistributedShell test is unhappy.
> If I isolate the v2 tests only, I have a sporadic failure (missing the
> application created event). If I run the test in its entirety, it times out
> after 15 minutes. I suspect the failure is caused by YARN-4129
> <https://issues.apache.org/jira/browse/YARN-4129>. I haven't tracked down
> the cause of the timeout yet, though. This may or may not be related with
> YARN-2859 <https://issues.apache.org/jira/browse/YARN-2859> that changed
> the port that the AHS binds to, and the subsequent TestDistributedShell
> test change (YARN-4326 <https://issues.apache.org/jira/browse/YARN-4326>).
> We should file a JIRA to look into this once the rebase is done.
>
> (2) reconciling with YARN-4183
> <https://issues.apache.org/jira/browse/YARN-4183>
> As Naga already pointed out on that JIRA, how we reconcile with the
> changes made in YARN-4183 requires a discussion. It changed the config that
> drives the RM's system metrics publisher from "timeline service enabled" to
> "application history server enabled". I'm not entirely sure if that is the
> right config to use for v.1, much less v.2. We should decide whether we
> retain our version (continue to use "timeline service enabled"), and decide
> how we enable v.1 and v.2 respectively. I think I'm going to preserve our
> version in our branch for now, but we can certainly revisit it.
>
> (3) reconciling with YARN-2556
> <https://issues.apache.org/jira/browse/YARN-2556>
> YARN-2556 was merged into trunk that delivers the timeline service
> performance tests. Since we already merged the v.2 version of the tests in
> our branch, I'm going to spend some time to make sure we eliminate code
> duplication. I'll file a JIRA to clean this up after the rebase.
>
> Please let me know if you have other feedback. Thanks!
>
> Sangjin
>