You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Vrushali Channapattan <vr...@gmail.com> on 2017/08/16 17:02:48 UTC
[DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Hi All
I’d like to open a discussion for merging Timeline Service v.2 (YARN-5355)
to trunk in a few weeks.
We have previously completed one merge onto trunk [1] and Timeline Service
v2 has been part of Hadoop release 3.0.0-alpha1.
Since then, we have been working on extending the capabilities of Timeline
Service v2 in a feature branch [2]. There are a few related issues pending
that are being actively worked upon & tested. As soon as they are resolved,
we plan on starting a merge vote within the next 2 weeks. The goal is to
get this in for Hadoop3 beta.
We have paid close attention to ensure that once disabled Timeline Service
v.2 does not impact existing functionality when disabled (by default).
At a high level, following are the key features that have been implemented
since 3.0.0-alpha1:
- Security (via Kerberos Authentication & delegation tokens) at the writer
[YARN-3053]
- Timeline server usability improvements [timeline-server
<https://issues.apache.org/jira/browse/YARN-5715?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20component%20%3D%20timelineserver%20AND%20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
]
- HBase specific improvements [atsv2-hbase
<https://issues.apache.org/jira/browse/YARN-6604?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
]
- REST API additions & improvements [timeline-reader
<https://issues.apache.org/jira/browse/YARN-5739?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20component%20in%20%28timelineclient%2C%20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
]
- Reader side simple authorization via whitelist [YARN-6820]
We would love to get your thoughts on these and more before we open a real
voting thread.
Special thanks to a team of folks who worked hard and contributed towards
this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
Thanks!
Vrushali
[1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
apache.org/msg23897.html
[2] feature branch YARN-5355 commits: https://github.com/
apache/hadoop/commits/YARN-5355
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
Thanks Rohith for that update! Sounds good. Let us know when that testing
is complete too.
On Sat, Aug 19, 2017 at 3:34 AM, Rohith Sharma K S <
rohithsharmaks@apache.org> wrote:
> Hi Sangjin
>
> Thanks for bringing this point.
> We did similar exercise with current YARN-5355 branch today. Tests are
> validated against default configuration and timeline server v.1.5 as well.
>
> All YARN major features such as RM HA/Restart/work-preserving-restart, NM
> restart, scheduling, sample MR/distributes shell jobs run are validated.
> Everything looks fine.
>
> This testing will also be done once YARN-5355 branch code freezed
> completely as well in couple of days.
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 18 August 2017 at 22:56, Sangjin Lee <sj...@apache.org> wrote:
>
> > Kudos to Vrushali and the team for getting ready for this large and
> > important feature! I know a huge team effort went into this. I look
> forward
> > to seeing this merged.
> >
> > I'd like to ask one piece of due diligence. Could you please inspect
> > rigorously to ensure that when disabled Timeline Service v.2 does not
> > impact other features in any way? We did a similar exercise when we had
> the
> > first drop, and it would be good to repeat that... Thanks!
> >
> > Sangjin
> >
> > On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > Great, thanks Vrushali! Sounds good to me.
> > >
> > > I have a few procedural release notes comments I'll put on YARN-5355,
> to
> > > make sure we advertise this to our users appropriately.
> > >
> > > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> > > vrushali.c@gmail.com> wrote:
> > >
> > > > Hi Andrew,
> > > >
> > > > Thanks for your response!
> > > >
> > > > There have been no changes to existing APIs since alpha1.
> > > >
> > > > We at Twitter have tested the feature to demonstrate it works at what
> > we
> > > > consider moderate scale but this did not include the security related
> > > > testing. The security testing is in progress at present by Timeline
> > > Service
> > > > V2 team in the community and we think we will have more details on
> this
> > > > very soon.
> > > >
> > > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what
> we
> > > > think of as "merge-blockers". The issues being targeted for merge are
> > in
> > > > [link1] below. There are about 59 jiras of which 56 are completed.
> > > >
> > > > We plan to make a new umbrella jira after the merge to trunk. We will
> > > then
> > > > create a new branch with the new jira name and move these open jiras
> > > under
> > > > YARN-5355 as subtasks of that new umbrella jira.
> > > >
> > > > thanks
> > > > Vrushali
> > > > [link1] https://issues.apache.org/jira/projects/YARN/versions/
> 12337991
> > > >
> > > >
> > > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <
> > andrew.wang@cloudera.com>
> > > > wrote:
> > > >
> > > >> Hi Vrushali,
> > > >>
> > > >> Glad to hear this major dev milestone is nearing completion!
> > > >>
> > > >> Repeating my request on other merge [DISCUSS] threads, could you
> > comment
> > > >> on testing and API stability of this merge? Our timeline for beta1
> is
> > > about
> > > >> a month out, so there's not much time to fix things beforehand.
> > > >>
> > > >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> > > most
> > > >> of these be moved out to a new umbrella? I'm wondering what needs to
> > be
> > > >> completed before sending the merge vote.
> > > >>
> > > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex
> > the
> > > >> beta1 release date for this feature than others. Hopefully that
> won't
> > be
> > > >> necessary though :)
> > > >>
> > > >> Best,
> > > >> Andrew
> > > >>
> > > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> > > >> vrushalic2016@gmail.com> wrote:
> > > >>
> > > >>> Looks like some of the hyperlinks appear messed up, my apologies,
> > > >>> resending
> > > >>> the same email with hopefully better looking content:
> > > >>>
> > > >>> Hi All,
> > > >>>
> > > >>> I'd like to open a discussion for merging Timeline Service v2
> > > (YARN-5355)
> > > >>> to trunk in a few weeks.
> > > >>>
> > > >>> We have previously completed one merge onto trunk [1] and Timeline
> > > >>> Service
> > > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >>>
> > > >>> Since then, we have been working on extending the capabilities of
> > > >>> Timeline
> > > >>> Service v2 in a feature branch [2]. There are a few related issues
> > > >>> pending
> > > >>> that are being actively worked upon and tested. As soon as they are
> > > >>> resolved, we plan on starting a merge vote within the next two
> weeks.
> > > The
> > > >>> goal is to get this into hadoop3 beta.
> > > >>>
> > > >>> We have paid close attention to ensure that once disabled Timeline
> > > >>> Service
> > > >>> v2 does not impact existing functionality when disabled (by
> default).
> > > >>>
> > > >>> At a high level, following are the key features that have been
> > > >>> implemented
> > > >>> since 3.0.0-alpha1:
> > > >>> - Security (via Kerberos Authentication & delegation tokens)
> > > [YARN-3053]
> > > >>> - Timeline server usability improvements [timeline-server
> > > >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > > >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> > > >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> > > >>> 2C%20created%20ASC>
> > > >>> ]
> > > >>> - HBase specific improvements [atsv2-hbase
> > > >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> > > >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> > > >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > > >>> ]
> > > >>> - REST API additions and improvements [timeline-reader
> > > >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > > >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> > > >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> > 20created%20ASC>
> > > >>> ]
> > > >>> - Reader side simple authorization via whitelist [YARN-6820]
> > > >>>
> > > >>> We would love to get your thoughts on this before we open a real
> > voting
> > > >>> thread.
> > > >>>
> > > >>> Special thanks to a team of folks who worked hard and contributed
> > > towards
> > > >>> this effort via patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >>>
> > > >>> Thanks
> > > >>> Vrushali
> > > >>> [1] Merge to trunk: http://www.mail-archive.com/
> > yarn-dev@hadoop.apache
> > > .
> > > >>> org/msg23897.html
> > > >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> > msg23897.html>
> > > >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> > > >>> ache/hadoop/commits/YARN-5355
> > > >>>
> > > >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> > > >>> vrushali.c@gmail.com> wrote:
> > > >>>
> > > >>> > Hi All
> > > >>> >
> > > >>> > I’d like to open a discussion for merging Timeline Service v.2
> > > >>> (YARN-5355)
> > > >>> > to trunk in a few weeks.
> > > >>> >
> > > >>> > We have previously completed one merge onto trunk [1] and
> Timeline
> > > >>> Service
> > > >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >>> >
> > > >>> > Since then, we have been working on extending the capabilities of
> > > >>> Timeline
> > > >>> > Service v2 in a feature branch [2]. There are a few related
> issues
> > > >>> pending
> > > >>> > that are being actively worked upon & tested. As soon as they are
> > > >>> resolved,
> > > >>> > we plan on starting a merge vote within the next 2 weeks. The
> goal
> > is
> > > >>> to
> > > >>> > get this in for Hadoop3 beta.
> > > >>> >
> > > >>> > We have paid close attention to ensure that once disabled
> Timeline
> > > >>> Service
> > > >>> > v.2 does not impact existing functionality when disabled (by
> > > default).
> > > >>> >
> > > >>> > At a high level, following are the key features that have been
> > > >>> implemented
> > > >>> > since 3.0.0-alpha1:
> > > >>> > - Security (via Kerberos Authentication & delegation tokens) at
> the
> > > >>> writer
> > > >>> > [YARN-3053]
> > > >>> > - Timeline server usability improvements [timeline-server
> > > >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > > >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > > >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > > >>> > ]
> > > >>> > - HBase specific improvements [atsv2-hbase
> > > >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> > > >>> ed%20DESC%2C%
> > > >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> > > >>> > ]
> > > >>> > - REST API additions & improvements [timeline-reader
> > > >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > > >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> > > >>> ty%20DESC%2C%
> > > >>> > 20created%20ASC>
> > > >>> > ]
> > > >>> > - Reader side simple authorization via whitelist [YARN-6820]
> > > >>> >
> > > >>> > We would love to get your thoughts on these and more before we
> > open a
> > > >>> real
> > > >>> > voting thread.
> > > >>> >
> > > >>> > Special thanks to a team of folks who worked hard and contributed
> > > >>> towards
> > > >>> > this effort with patches, reviews and guidance: Rohith Sharma K
> S,
> > > >>> Varun
> > > >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> > Joep
> > > >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >>> >
> > > >>> > Thanks!
> > > >>> > Vrushali
> > > >>> >
> > > >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> > > >>> > apache.org/msg23897.html
> > > >>> > [2] feature branch YARN-5355 commits: https://github.com/
> > > >>> > apache/hadoop/commits/YARN-5355
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
Thanks Rohith for that update! Sounds good. Let us know when that testing
is complete too.
On Sat, Aug 19, 2017 at 3:34 AM, Rohith Sharma K S <
rohithsharmaks@apache.org> wrote:
> Hi Sangjin
>
> Thanks for bringing this point.
> We did similar exercise with current YARN-5355 branch today. Tests are
> validated against default configuration and timeline server v.1.5 as well.
>
> All YARN major features such as RM HA/Restart/work-preserving-restart, NM
> restart, scheduling, sample MR/distributes shell jobs run are validated.
> Everything looks fine.
>
> This testing will also be done once YARN-5355 branch code freezed
> completely as well in couple of days.
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 18 August 2017 at 22:56, Sangjin Lee <sj...@apache.org> wrote:
>
> > Kudos to Vrushali and the team for getting ready for this large and
> > important feature! I know a huge team effort went into this. I look
> forward
> > to seeing this merged.
> >
> > I'd like to ask one piece of due diligence. Could you please inspect
> > rigorously to ensure that when disabled Timeline Service v.2 does not
> > impact other features in any way? We did a similar exercise when we had
> the
> > first drop, and it would be good to repeat that... Thanks!
> >
> > Sangjin
> >
> > On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > Great, thanks Vrushali! Sounds good to me.
> > >
> > > I have a few procedural release notes comments I'll put on YARN-5355,
> to
> > > make sure we advertise this to our users appropriately.
> > >
> > > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> > > vrushali.c@gmail.com> wrote:
> > >
> > > > Hi Andrew,
> > > >
> > > > Thanks for your response!
> > > >
> > > > There have been no changes to existing APIs since alpha1.
> > > >
> > > > We at Twitter have tested the feature to demonstrate it works at what
> > we
> > > > consider moderate scale but this did not include the security related
> > > > testing. The security testing is in progress at present by Timeline
> > > Service
> > > > V2 team in the community and we think we will have more details on
> this
> > > > very soon.
> > > >
> > > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what
> we
> > > > think of as "merge-blockers". The issues being targeted for merge are
> > in
> > > > [link1] below. There are about 59 jiras of which 56 are completed.
> > > >
> > > > We plan to make a new umbrella jira after the merge to trunk. We will
> > > then
> > > > create a new branch with the new jira name and move these open jiras
> > > under
> > > > YARN-5355 as subtasks of that new umbrella jira.
> > > >
> > > > thanks
> > > > Vrushali
> > > > [link1] https://issues.apache.org/jira/projects/YARN/versions/
> 12337991
> > > >
> > > >
> > > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <
> > andrew.wang@cloudera.com>
> > > > wrote:
> > > >
> > > >> Hi Vrushali,
> > > >>
> > > >> Glad to hear this major dev milestone is nearing completion!
> > > >>
> > > >> Repeating my request on other merge [DISCUSS] threads, could you
> > comment
> > > >> on testing and API stability of this merge? Our timeline for beta1
> is
> > > about
> > > >> a month out, so there's not much time to fix things beforehand.
> > > >>
> > > >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> > > most
> > > >> of these be moved out to a new umbrella? I'm wondering what needs to
> > be
> > > >> completed before sending the merge vote.
> > > >>
> > > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex
> > the
> > > >> beta1 release date for this feature than others. Hopefully that
> won't
> > be
> > > >> necessary though :)
> > > >>
> > > >> Best,
> > > >> Andrew
> > > >>
> > > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> > > >> vrushalic2016@gmail.com> wrote:
> > > >>
> > > >>> Looks like some of the hyperlinks appear messed up, my apologies,
> > > >>> resending
> > > >>> the same email with hopefully better looking content:
> > > >>>
> > > >>> Hi All,
> > > >>>
> > > >>> I'd like to open a discussion for merging Timeline Service v2
> > > (YARN-5355)
> > > >>> to trunk in a few weeks.
> > > >>>
> > > >>> We have previously completed one merge onto trunk [1] and Timeline
> > > >>> Service
> > > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >>>
> > > >>> Since then, we have been working on extending the capabilities of
> > > >>> Timeline
> > > >>> Service v2 in a feature branch [2]. There are a few related issues
> > > >>> pending
> > > >>> that are being actively worked upon and tested. As soon as they are
> > > >>> resolved, we plan on starting a merge vote within the next two
> weeks.
> > > The
> > > >>> goal is to get this into hadoop3 beta.
> > > >>>
> > > >>> We have paid close attention to ensure that once disabled Timeline
> > > >>> Service
> > > >>> v2 does not impact existing functionality when disabled (by
> default).
> > > >>>
> > > >>> At a high level, following are the key features that have been
> > > >>> implemented
> > > >>> since 3.0.0-alpha1:
> > > >>> - Security (via Kerberos Authentication & delegation tokens)
> > > [YARN-3053]
> > > >>> - Timeline server usability improvements [timeline-server
> > > >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > > >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> > > >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> > > >>> 2C%20created%20ASC>
> > > >>> ]
> > > >>> - HBase specific improvements [atsv2-hbase
> > > >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> > > >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> > > >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > > >>> ]
> > > >>> - REST API additions and improvements [timeline-reader
> > > >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > > >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> > > >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> > 20created%20ASC>
> > > >>> ]
> > > >>> - Reader side simple authorization via whitelist [YARN-6820]
> > > >>>
> > > >>> We would love to get your thoughts on this before we open a real
> > voting
> > > >>> thread.
> > > >>>
> > > >>> Special thanks to a team of folks who worked hard and contributed
> > > towards
> > > >>> this effort via patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >>>
> > > >>> Thanks
> > > >>> Vrushali
> > > >>> [1] Merge to trunk: http://www.mail-archive.com/
> > yarn-dev@hadoop.apache
> > > .
> > > >>> org/msg23897.html
> > > >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> > msg23897.html>
> > > >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> > > >>> ache/hadoop/commits/YARN-5355
> > > >>>
> > > >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> > > >>> vrushali.c@gmail.com> wrote:
> > > >>>
> > > >>> > Hi All
> > > >>> >
> > > >>> > I’d like to open a discussion for merging Timeline Service v.2
> > > >>> (YARN-5355)
> > > >>> > to trunk in a few weeks.
> > > >>> >
> > > >>> > We have previously completed one merge onto trunk [1] and
> Timeline
> > > >>> Service
> > > >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >>> >
> > > >>> > Since then, we have been working on extending the capabilities of
> > > >>> Timeline
> > > >>> > Service v2 in a feature branch [2]. There are a few related
> issues
> > > >>> pending
> > > >>> > that are being actively worked upon & tested. As soon as they are
> > > >>> resolved,
> > > >>> > we plan on starting a merge vote within the next 2 weeks. The
> goal
> > is
> > > >>> to
> > > >>> > get this in for Hadoop3 beta.
> > > >>> >
> > > >>> > We have paid close attention to ensure that once disabled
> Timeline
> > > >>> Service
> > > >>> > v.2 does not impact existing functionality when disabled (by
> > > default).
> > > >>> >
> > > >>> > At a high level, following are the key features that have been
> > > >>> implemented
> > > >>> > since 3.0.0-alpha1:
> > > >>> > - Security (via Kerberos Authentication & delegation tokens) at
> the
> > > >>> writer
> > > >>> > [YARN-3053]
> > > >>> > - Timeline server usability improvements [timeline-server
> > > >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > > >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > > >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > > >>> > ]
> > > >>> > - HBase specific improvements [atsv2-hbase
> > > >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> > > >>> ed%20DESC%2C%
> > > >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> > > >>> > ]
> > > >>> > - REST API additions & improvements [timeline-reader
> > > >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > > >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> > > >>> ty%20DESC%2C%
> > > >>> > 20created%20ASC>
> > > >>> > ]
> > > >>> > - Reader side simple authorization via whitelist [YARN-6820]
> > > >>> >
> > > >>> > We would love to get your thoughts on these and more before we
> > open a
> > > >>> real
> > > >>> > voting thread.
> > > >>> >
> > > >>> > Special thanks to a team of folks who worked hard and contributed
> > > >>> towards
> > > >>> > this effort with patches, reviews and guidance: Rohith Sharma K
> S,
> > > >>> Varun
> > > >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> > Joep
> > > >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >>> >
> > > >>> > Thanks!
> > > >>> > Vrushali
> > > >>> >
> > > >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> > > >>> > apache.org/msg23897.html
> > > >>> > [2] feature branch YARN-5355 commits: https://github.com/
> > > >>> > apache/hadoop/commits/YARN-5355
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
Thanks Rohith for that update! Sounds good. Let us know when that testing
is complete too.
On Sat, Aug 19, 2017 at 3:34 AM, Rohith Sharma K S <
rohithsharmaks@apache.org> wrote:
> Hi Sangjin
>
> Thanks for bringing this point.
> We did similar exercise with current YARN-5355 branch today. Tests are
> validated against default configuration and timeline server v.1.5 as well.
>
> All YARN major features such as RM HA/Restart/work-preserving-restart, NM
> restart, scheduling, sample MR/distributes shell jobs run are validated.
> Everything looks fine.
>
> This testing will also be done once YARN-5355 branch code freezed
> completely as well in couple of days.
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 18 August 2017 at 22:56, Sangjin Lee <sj...@apache.org> wrote:
>
> > Kudos to Vrushali and the team for getting ready for this large and
> > important feature! I know a huge team effort went into this. I look
> forward
> > to seeing this merged.
> >
> > I'd like to ask one piece of due diligence. Could you please inspect
> > rigorously to ensure that when disabled Timeline Service v.2 does not
> > impact other features in any way? We did a similar exercise when we had
> the
> > first drop, and it would be good to repeat that... Thanks!
> >
> > Sangjin
> >
> > On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > Great, thanks Vrushali! Sounds good to me.
> > >
> > > I have a few procedural release notes comments I'll put on YARN-5355,
> to
> > > make sure we advertise this to our users appropriately.
> > >
> > > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> > > vrushali.c@gmail.com> wrote:
> > >
> > > > Hi Andrew,
> > > >
> > > > Thanks for your response!
> > > >
> > > > There have been no changes to existing APIs since alpha1.
> > > >
> > > > We at Twitter have tested the feature to demonstrate it works at what
> > we
> > > > consider moderate scale but this did not include the security related
> > > > testing. The security testing is in progress at present by Timeline
> > > Service
> > > > V2 team in the community and we think we will have more details on
> this
> > > > very soon.
> > > >
> > > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what
> we
> > > > think of as "merge-blockers". The issues being targeted for merge are
> > in
> > > > [link1] below. There are about 59 jiras of which 56 are completed.
> > > >
> > > > We plan to make a new umbrella jira after the merge to trunk. We will
> > > then
> > > > create a new branch with the new jira name and move these open jiras
> > > under
> > > > YARN-5355 as subtasks of that new umbrella jira.
> > > >
> > > > thanks
> > > > Vrushali
> > > > [link1] https://issues.apache.org/jira/projects/YARN/versions/
> 12337991
> > > >
> > > >
> > > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <
> > andrew.wang@cloudera.com>
> > > > wrote:
> > > >
> > > >> Hi Vrushali,
> > > >>
> > > >> Glad to hear this major dev milestone is nearing completion!
> > > >>
> > > >> Repeating my request on other merge [DISCUSS] threads, could you
> > comment
> > > >> on testing and API stability of this merge? Our timeline for beta1
> is
> > > about
> > > >> a month out, so there's not much time to fix things beforehand.
> > > >>
> > > >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> > > most
> > > >> of these be moved out to a new umbrella? I'm wondering what needs to
> > be
> > > >> completed before sending the merge vote.
> > > >>
> > > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex
> > the
> > > >> beta1 release date for this feature than others. Hopefully that
> won't
> > be
> > > >> necessary though :)
> > > >>
> > > >> Best,
> > > >> Andrew
> > > >>
> > > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> > > >> vrushalic2016@gmail.com> wrote:
> > > >>
> > > >>> Looks like some of the hyperlinks appear messed up, my apologies,
> > > >>> resending
> > > >>> the same email with hopefully better looking content:
> > > >>>
> > > >>> Hi All,
> > > >>>
> > > >>> I'd like to open a discussion for merging Timeline Service v2
> > > (YARN-5355)
> > > >>> to trunk in a few weeks.
> > > >>>
> > > >>> We have previously completed one merge onto trunk [1] and Timeline
> > > >>> Service
> > > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >>>
> > > >>> Since then, we have been working on extending the capabilities of
> > > >>> Timeline
> > > >>> Service v2 in a feature branch [2]. There are a few related issues
> > > >>> pending
> > > >>> that are being actively worked upon and tested. As soon as they are
> > > >>> resolved, we plan on starting a merge vote within the next two
> weeks.
> > > The
> > > >>> goal is to get this into hadoop3 beta.
> > > >>>
> > > >>> We have paid close attention to ensure that once disabled Timeline
> > > >>> Service
> > > >>> v2 does not impact existing functionality when disabled (by
> default).
> > > >>>
> > > >>> At a high level, following are the key features that have been
> > > >>> implemented
> > > >>> since 3.0.0-alpha1:
> > > >>> - Security (via Kerberos Authentication & delegation tokens)
> > > [YARN-3053]
> > > >>> - Timeline server usability improvements [timeline-server
> > > >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > > >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> > > >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> > > >>> 2C%20created%20ASC>
> > > >>> ]
> > > >>> - HBase specific improvements [atsv2-hbase
> > > >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> > > >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> > > >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > > >>> ]
> > > >>> - REST API additions and improvements [timeline-reader
> > > >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > > >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> > > >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> > 20created%20ASC>
> > > >>> ]
> > > >>> - Reader side simple authorization via whitelist [YARN-6820]
> > > >>>
> > > >>> We would love to get your thoughts on this before we open a real
> > voting
> > > >>> thread.
> > > >>>
> > > >>> Special thanks to a team of folks who worked hard and contributed
> > > towards
> > > >>> this effort via patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >>>
> > > >>> Thanks
> > > >>> Vrushali
> > > >>> [1] Merge to trunk: http://www.mail-archive.com/
> > yarn-dev@hadoop.apache
> > > .
> > > >>> org/msg23897.html
> > > >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> > msg23897.html>
> > > >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> > > >>> ache/hadoop/commits/YARN-5355
> > > >>>
> > > >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> > > >>> vrushali.c@gmail.com> wrote:
> > > >>>
> > > >>> > Hi All
> > > >>> >
> > > >>> > I’d like to open a discussion for merging Timeline Service v.2
> > > >>> (YARN-5355)
> > > >>> > to trunk in a few weeks.
> > > >>> >
> > > >>> > We have previously completed one merge onto trunk [1] and
> Timeline
> > > >>> Service
> > > >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >>> >
> > > >>> > Since then, we have been working on extending the capabilities of
> > > >>> Timeline
> > > >>> > Service v2 in a feature branch [2]. There are a few related
> issues
> > > >>> pending
> > > >>> > that are being actively worked upon & tested. As soon as they are
> > > >>> resolved,
> > > >>> > we plan on starting a merge vote within the next 2 weeks. The
> goal
> > is
> > > >>> to
> > > >>> > get this in for Hadoop3 beta.
> > > >>> >
> > > >>> > We have paid close attention to ensure that once disabled
> Timeline
> > > >>> Service
> > > >>> > v.2 does not impact existing functionality when disabled (by
> > > default).
> > > >>> >
> > > >>> > At a high level, following are the key features that have been
> > > >>> implemented
> > > >>> > since 3.0.0-alpha1:
> > > >>> > - Security (via Kerberos Authentication & delegation tokens) at
> the
> > > >>> writer
> > > >>> > [YARN-3053]
> > > >>> > - Timeline server usability improvements [timeline-server
> > > >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > > >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > > >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > > >>> > ]
> > > >>> > - HBase specific improvements [atsv2-hbase
> > > >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> > > >>> ed%20DESC%2C%
> > > >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> > > >>> > ]
> > > >>> > - REST API additions & improvements [timeline-reader
> > > >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > > >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> > > >>> ty%20DESC%2C%
> > > >>> > 20created%20ASC>
> > > >>> > ]
> > > >>> > - Reader side simple authorization via whitelist [YARN-6820]
> > > >>> >
> > > >>> > We would love to get your thoughts on these and more before we
> > open a
> > > >>> real
> > > >>> > voting thread.
> > > >>> >
> > > >>> > Special thanks to a team of folks who worked hard and contributed
> > > >>> towards
> > > >>> > this effort with patches, reviews and guidance: Rohith Sharma K
> S,
> > > >>> Varun
> > > >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> > Joep
> > > >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >>> >
> > > >>> > Thanks!
> > > >>> > Vrushali
> > > >>> >
> > > >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> > > >>> > apache.org/msg23897.html
> > > >>> > [2] feature branch YARN-5355 commits: https://github.com/
> > > >>> > apache/hadoop/commits/YARN-5355
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
Thanks Rohith for that update! Sounds good. Let us know when that testing
is complete too.
On Sat, Aug 19, 2017 at 3:34 AM, Rohith Sharma K S <
rohithsharmaks@apache.org> wrote:
> Hi Sangjin
>
> Thanks for bringing this point.
> We did similar exercise with current YARN-5355 branch today. Tests are
> validated against default configuration and timeline server v.1.5 as well.
>
> All YARN major features such as RM HA/Restart/work-preserving-restart, NM
> restart, scheduling, sample MR/distributes shell jobs run are validated.
> Everything looks fine.
>
> This testing will also be done once YARN-5355 branch code freezed
> completely as well in couple of days.
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 18 August 2017 at 22:56, Sangjin Lee <sj...@apache.org> wrote:
>
> > Kudos to Vrushali and the team for getting ready for this large and
> > important feature! I know a huge team effort went into this. I look
> forward
> > to seeing this merged.
> >
> > I'd like to ask one piece of due diligence. Could you please inspect
> > rigorously to ensure that when disabled Timeline Service v.2 does not
> > impact other features in any way? We did a similar exercise when we had
> the
> > first drop, and it would be good to repeat that... Thanks!
> >
> > Sangjin
> >
> > On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> > > Great, thanks Vrushali! Sounds good to me.
> > >
> > > I have a few procedural release notes comments I'll put on YARN-5355,
> to
> > > make sure we advertise this to our users appropriately.
> > >
> > > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> > > vrushali.c@gmail.com> wrote:
> > >
> > > > Hi Andrew,
> > > >
> > > > Thanks for your response!
> > > >
> > > > There have been no changes to existing APIs since alpha1.
> > > >
> > > > We at Twitter have tested the feature to demonstrate it works at what
> > we
> > > > consider moderate scale but this did not include the security related
> > > > testing. The security testing is in progress at present by Timeline
> > > Service
> > > > V2 team in the community and we think we will have more details on
> this
> > > > very soon.
> > > >
> > > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what
> we
> > > > think of as "merge-blockers". The issues being targeted for merge are
> > in
> > > > [link1] below. There are about 59 jiras of which 56 are completed.
> > > >
> > > > We plan to make a new umbrella jira after the merge to trunk. We will
> > > then
> > > > create a new branch with the new jira name and move these open jiras
> > > under
> > > > YARN-5355 as subtasks of that new umbrella jira.
> > > >
> > > > thanks
> > > > Vrushali
> > > > [link1] https://issues.apache.org/jira/projects/YARN/versions/
> 12337991
> > > >
> > > >
> > > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <
> > andrew.wang@cloudera.com>
> > > > wrote:
> > > >
> > > >> Hi Vrushali,
> > > >>
> > > >> Glad to hear this major dev milestone is nearing completion!
> > > >>
> > > >> Repeating my request on other merge [DISCUSS] threads, could you
> > comment
> > > >> on testing and API stability of this merge? Our timeline for beta1
> is
> > > about
> > > >> a month out, so there's not much time to fix things beforehand.
> > > >>
> > > >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> > > most
> > > >> of these be moved out to a new umbrella? I'm wondering what needs to
> > be
> > > >> completed before sending the merge vote.
> > > >>
> > > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex
> > the
> > > >> beta1 release date for this feature than others. Hopefully that
> won't
> > be
> > > >> necessary though :)
> > > >>
> > > >> Best,
> > > >> Andrew
> > > >>
> > > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> > > >> vrushalic2016@gmail.com> wrote:
> > > >>
> > > >>> Looks like some of the hyperlinks appear messed up, my apologies,
> > > >>> resending
> > > >>> the same email with hopefully better looking content:
> > > >>>
> > > >>> Hi All,
> > > >>>
> > > >>> I'd like to open a discussion for merging Timeline Service v2
> > > (YARN-5355)
> > > >>> to trunk in a few weeks.
> > > >>>
> > > >>> We have previously completed one merge onto trunk [1] and Timeline
> > > >>> Service
> > > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >>>
> > > >>> Since then, we have been working on extending the capabilities of
> > > >>> Timeline
> > > >>> Service v2 in a feature branch [2]. There are a few related issues
> > > >>> pending
> > > >>> that are being actively worked upon and tested. As soon as they are
> > > >>> resolved, we plan on starting a merge vote within the next two
> weeks.
> > > The
> > > >>> goal is to get this into hadoop3 beta.
> > > >>>
> > > >>> We have paid close attention to ensure that once disabled Timeline
> > > >>> Service
> > > >>> v2 does not impact existing functionality when disabled (by
> default).
> > > >>>
> > > >>> At a high level, following are the key features that have been
> > > >>> implemented
> > > >>> since 3.0.0-alpha1:
> > > >>> - Security (via Kerberos Authentication & delegation tokens)
> > > [YARN-3053]
> > > >>> - Timeline server usability improvements [timeline-server
> > > >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > > >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> > > >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> > > >>> 2C%20created%20ASC>
> > > >>> ]
> > > >>> - HBase specific improvements [atsv2-hbase
> > > >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> > > >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> > > >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > > >>> ]
> > > >>> - REST API additions and improvements [timeline-reader
> > > >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> > > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > > >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> > > >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> > 20created%20ASC>
> > > >>> ]
> > > >>> - Reader side simple authorization via whitelist [YARN-6820]
> > > >>>
> > > >>> We would love to get your thoughts on this before we open a real
> > voting
> > > >>> thread.
> > > >>>
> > > >>> Special thanks to a team of folks who worked hard and contributed
> > > towards
> > > >>> this effort via patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >>>
> > > >>> Thanks
> > > >>> Vrushali
> > > >>> [1] Merge to trunk: http://www.mail-archive.com/
> > yarn-dev@hadoop.apache
> > > .
> > > >>> org/msg23897.html
> > > >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> > msg23897.html>
> > > >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> > > >>> ache/hadoop/commits/YARN-5355
> > > >>>
> > > >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>>
> > > >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> > > >>> vrushali.c@gmail.com> wrote:
> > > >>>
> > > >>> > Hi All
> > > >>> >
> > > >>> > I’d like to open a discussion for merging Timeline Service v.2
> > > >>> (YARN-5355)
> > > >>> > to trunk in a few weeks.
> > > >>> >
> > > >>> > We have previously completed one merge onto trunk [1] and
> Timeline
> > > >>> Service
> > > >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >>> >
> > > >>> > Since then, we have been working on extending the capabilities of
> > > >>> Timeline
> > > >>> > Service v2 in a feature branch [2]. There are a few related
> issues
> > > >>> pending
> > > >>> > that are being actively worked upon & tested. As soon as they are
> > > >>> resolved,
> > > >>> > we plan on starting a merge vote within the next 2 weeks. The
> goal
> > is
> > > >>> to
> > > >>> > get this in for Hadoop3 beta.
> > > >>> >
> > > >>> > We have paid close attention to ensure that once disabled
> Timeline
> > > >>> Service
> > > >>> > v.2 does not impact existing functionality when disabled (by
> > > default).
> > > >>> >
> > > >>> > At a high level, following are the key features that have been
> > > >>> implemented
> > > >>> > since 3.0.0-alpha1:
> > > >>> > - Security (via Kerberos Authentication & delegation tokens) at
> the
> > > >>> writer
> > > >>> > [YARN-3053]
> > > >>> > - Timeline server usability improvements [timeline-server
> > > >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > > >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > > >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > > >>> > ]
> > > >>> > - HBase specific improvements [atsv2-hbase
> > > >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> > > >>> ed%20DESC%2C%
> > > >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> > > >>> > ]
> > > >>> > - REST API additions & improvements [timeline-reader
> > > >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> > > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > > >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > > >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> > > >>> ty%20DESC%2C%
> > > >>> > 20created%20ASC>
> > > >>> > ]
> > > >>> > - Reader side simple authorization via whitelist [YARN-6820]
> > > >>> >
> > > >>> > We would love to get your thoughts on these and more before we
> > open a
> > > >>> real
> > > >>> > voting thread.
> > > >>> >
> > > >>> > Special thanks to a team of folks who worked hard and contributed
> > > >>> towards
> > > >>> > this effort with patches, reviews and guidance: Rohith Sharma K
> S,
> > > >>> Varun
> > > >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> > Joep
> > > >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >>> >
> > > >>> > Thanks!
> > > >>> > Vrushali
> > > >>> >
> > > >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> > > >>> > apache.org/msg23897.html
> > > >>> > [2] feature branch YARN-5355 commits: https://github.com/
> > > >>> > apache/hadoop/commits/YARN-5355
> > > >>> >
> > > >>>
> > > >>
> > > >>
> > > >
> > >
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Rohith Sharma K S <ro...@apache.org>.
Hi Sangjin
Thanks for bringing this point.
We did similar exercise with current YARN-5355 branch today. Tests are
validated against default configuration and timeline server v.1.5 as well.
All YARN major features such as RM HA/Restart/work-preserving-restart, NM
restart, scheduling, sample MR/distributes shell jobs run are validated.
Everything looks fine.
This testing will also be done once YARN-5355 branch code freezed
completely as well in couple of days.
Thanks & Regards
Rohith Sharma K S
On 18 August 2017 at 22:56, Sangjin Lee <sj...@apache.org> wrote:
> Kudos to Vrushali and the team for getting ready for this large and
> important feature! I know a huge team effort went into this. I look forward
> to seeing this merged.
>
> I'd like to ask one piece of due diligence. Could you please inspect
> rigorously to ensure that when disabled Timeline Service v.2 does not
> impact other features in any way? We did a similar exercise when we had the
> first drop, and it would be good to repeat that... Thanks!
>
> Sangjin
>
> On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Great, thanks Vrushali! Sounds good to me.
> >
> > I have a few procedural release notes comments I'll put on YARN-5355, to
> > make sure we advertise this to our users appropriately.
> >
> > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> > vrushali.c@gmail.com> wrote:
> >
> > > Hi Andrew,
> > >
> > > Thanks for your response!
> > >
> > > There have been no changes to existing APIs since alpha1.
> > >
> > > We at Twitter have tested the feature to demonstrate it works at what
> we
> > > consider moderate scale but this did not include the security related
> > > testing. The security testing is in progress at present by Timeline
> > Service
> > > V2 team in the community and we think we will have more details on this
> > > very soon.
> > >
> > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> > > think of as "merge-blockers". The issues being targeted for merge are
> in
> > > [link1] below. There are about 59 jiras of which 56 are completed.
> > >
> > > We plan to make a new umbrella jira after the merge to trunk. We will
> > then
> > > create a new branch with the new jira name and move these open jiras
> > under
> > > YARN-5355 as subtasks of that new umbrella jira.
> > >
> > > thanks
> > > Vrushali
> > > [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
> > >
> > >
> > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> > > wrote:
> > >
> > >> Hi Vrushali,
> > >>
> > >> Glad to hear this major dev milestone is nearing completion!
> > >>
> > >> Repeating my request on other merge [DISCUSS] threads, could you
> comment
> > >> on testing and API stability of this merge? Our timeline for beta1 is
> > about
> > >> a month out, so there's not much time to fix things beforehand.
> > >>
> > >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> > most
> > >> of these be moved out to a new umbrella? I'm wondering what needs to
> be
> > >> completed before sending the merge vote.
> > >>
> > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex
> the
> > >> beta1 release date for this feature than others. Hopefully that won't
> be
> > >> necessary though :)
> > >>
> > >> Best,
> > >> Andrew
> > >>
> > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> > >> vrushalic2016@gmail.com> wrote:
> > >>
> > >>> Looks like some of the hyperlinks appear messed up, my apologies,
> > >>> resending
> > >>> the same email with hopefully better looking content:
> > >>>
> > >>> Hi All,
> > >>>
> > >>> I'd like to open a discussion for merging Timeline Service v2
> > (YARN-5355)
> > >>> to trunk in a few weeks.
> > >>>
> > >>> We have previously completed one merge onto trunk [1] and Timeline
> > >>> Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> Service v2 in a feature branch [2]. There are a few related issues
> > >>> pending
> > >>> that are being actively worked upon and tested. As soon as they are
> > >>> resolved, we plan on starting a merge vote within the next two weeks.
> > The
> > >>> goal is to get this into hadoop3 beta.
> > >>>
> > >>> We have paid close attention to ensure that once disabled Timeline
> > >>> Service
> > >>> v2 does not impact existing functionality when disabled (by default).
> > >>>
> > >>> At a high level, following are the key features that have been
> > >>> implemented
> > >>> since 3.0.0-alpha1:
> > >>> - Security (via Kerberos Authentication & delegation tokens)
> > [YARN-3053]
> > >>> - Timeline server usability improvements [timeline-server
> > >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> > >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> > >>> 2C%20created%20ASC>
> > >>> ]
> > >>> - HBase specific improvements [atsv2-hbase
> > >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> > >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> > >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> ]
> > >>> - REST API additions and improvements [timeline-reader
> > >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> > >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> > >>> ]
> > >>> - Reader side simple authorization via whitelist [YARN-6820]
> > >>>
> > >>> We would love to get your thoughts on this before we open a real
> voting
> > >>> thread.
> > >>>
> > >>> Special thanks to a team of folks who worked hard and contributed
> > towards
> > >>> this effort via patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>>
> > >>> Thanks
> > >>> Vrushali
> > >>> [1] Merge to trunk: http://www.mail-archive.com/
> yarn-dev@hadoop.apache
> > .
> > >>> org/msg23897.html
> > >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg23897.html>
> > >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> > >>> ache/hadoop/commits/YARN-5355
> > >>>
> > >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> > >>> vrushali.c@gmail.com> wrote:
> > >>>
> > >>> > Hi All
> > >>> >
> > >>> > I’d like to open a discussion for merging Timeline Service v.2
> > >>> (YARN-5355)
> > >>> > to trunk in a few weeks.
> > >>> >
> > >>> > We have previously completed one merge onto trunk [1] and Timeline
> > >>> Service
> > >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>> >
> > >>> > Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> > Service v2 in a feature branch [2]. There are a few related issues
> > >>> pending
> > >>> > that are being actively worked upon & tested. As soon as they are
> > >>> resolved,
> > >>> > we plan on starting a merge vote within the next 2 weeks. The goal
> is
> > >>> to
> > >>> > get this in for Hadoop3 beta.
> > >>> >
> > >>> > We have paid close attention to ensure that once disabled Timeline
> > >>> Service
> > >>> > v.2 does not impact existing functionality when disabled (by
> > default).
> > >>> >
> > >>> > At a high level, following are the key features that have been
> > >>> implemented
> > >>> > since 3.0.0-alpha1:
> > >>> > - Security (via Kerberos Authentication & delegation tokens) at the
> > >>> writer
> > >>> > [YARN-3053]
> > >>> > - Timeline server usability improvements [timeline-server
> > >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> > ]
> > >>> > - HBase specific improvements [atsv2-hbase
> > >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> > >>> ed%20DESC%2C%
> > >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> > ]
> > >>> > - REST API additions & improvements [timeline-reader
> > >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> > >>> ty%20DESC%2C%
> > >>> > 20created%20ASC>
> > >>> > ]
> > >>> > - Reader side simple authorization via whitelist [YARN-6820]
> > >>> >
> > >>> > We would love to get your thoughts on these and more before we
> open a
> > >>> real
> > >>> > voting thread.
> > >>> >
> > >>> > Special thanks to a team of folks who worked hard and contributed
> > >>> towards
> > >>> > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > >>> Varun
> > >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>> >
> > >>> > Thanks!
> > >>> > Vrushali
> > >>> >
> > >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> > >>> > apache.org/msg23897.html
> > >>> > [2] feature branch YARN-5355 commits: https://github.com/
> > >>> > apache/hadoop/commits/YARN-5355
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Rohith Sharma K S <ro...@apache.org>.
Hi Sangjin
Thanks for bringing this point.
We did similar exercise with current YARN-5355 branch today. Tests are
validated against default configuration and timeline server v.1.5 as well.
All YARN major features such as RM HA/Restart/work-preserving-restart, NM
restart, scheduling, sample MR/distributes shell jobs run are validated.
Everything looks fine.
This testing will also be done once YARN-5355 branch code freezed
completely as well in couple of days.
Thanks & Regards
Rohith Sharma K S
On 18 August 2017 at 22:56, Sangjin Lee <sj...@apache.org> wrote:
> Kudos to Vrushali and the team for getting ready for this large and
> important feature! I know a huge team effort went into this. I look forward
> to seeing this merged.
>
> I'd like to ask one piece of due diligence. Could you please inspect
> rigorously to ensure that when disabled Timeline Service v.2 does not
> impact other features in any way? We did a similar exercise when we had the
> first drop, and it would be good to repeat that... Thanks!
>
> Sangjin
>
> On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Great, thanks Vrushali! Sounds good to me.
> >
> > I have a few procedural release notes comments I'll put on YARN-5355, to
> > make sure we advertise this to our users appropriately.
> >
> > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> > vrushali.c@gmail.com> wrote:
> >
> > > Hi Andrew,
> > >
> > > Thanks for your response!
> > >
> > > There have been no changes to existing APIs since alpha1.
> > >
> > > We at Twitter have tested the feature to demonstrate it works at what
> we
> > > consider moderate scale but this did not include the security related
> > > testing. The security testing is in progress at present by Timeline
> > Service
> > > V2 team in the community and we think we will have more details on this
> > > very soon.
> > >
> > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> > > think of as "merge-blockers". The issues being targeted for merge are
> in
> > > [link1] below. There are about 59 jiras of which 56 are completed.
> > >
> > > We plan to make a new umbrella jira after the merge to trunk. We will
> > then
> > > create a new branch with the new jira name and move these open jiras
> > under
> > > YARN-5355 as subtasks of that new umbrella jira.
> > >
> > > thanks
> > > Vrushali
> > > [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
> > >
> > >
> > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> > > wrote:
> > >
> > >> Hi Vrushali,
> > >>
> > >> Glad to hear this major dev milestone is nearing completion!
> > >>
> > >> Repeating my request on other merge [DISCUSS] threads, could you
> comment
> > >> on testing and API stability of this merge? Our timeline for beta1 is
> > about
> > >> a month out, so there's not much time to fix things beforehand.
> > >>
> > >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> > most
> > >> of these be moved out to a new umbrella? I'm wondering what needs to
> be
> > >> completed before sending the merge vote.
> > >>
> > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex
> the
> > >> beta1 release date for this feature than others. Hopefully that won't
> be
> > >> necessary though :)
> > >>
> > >> Best,
> > >> Andrew
> > >>
> > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> > >> vrushalic2016@gmail.com> wrote:
> > >>
> > >>> Looks like some of the hyperlinks appear messed up, my apologies,
> > >>> resending
> > >>> the same email with hopefully better looking content:
> > >>>
> > >>> Hi All,
> > >>>
> > >>> I'd like to open a discussion for merging Timeline Service v2
> > (YARN-5355)
> > >>> to trunk in a few weeks.
> > >>>
> > >>> We have previously completed one merge onto trunk [1] and Timeline
> > >>> Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> Service v2 in a feature branch [2]. There are a few related issues
> > >>> pending
> > >>> that are being actively worked upon and tested. As soon as they are
> > >>> resolved, we plan on starting a merge vote within the next two weeks.
> > The
> > >>> goal is to get this into hadoop3 beta.
> > >>>
> > >>> We have paid close attention to ensure that once disabled Timeline
> > >>> Service
> > >>> v2 does not impact existing functionality when disabled (by default).
> > >>>
> > >>> At a high level, following are the key features that have been
> > >>> implemented
> > >>> since 3.0.0-alpha1:
> > >>> - Security (via Kerberos Authentication & delegation tokens)
> > [YARN-3053]
> > >>> - Timeline server usability improvements [timeline-server
> > >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> > >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> > >>> 2C%20created%20ASC>
> > >>> ]
> > >>> - HBase specific improvements [atsv2-hbase
> > >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> > >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> > >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> ]
> > >>> - REST API additions and improvements [timeline-reader
> > >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> > >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> > >>> ]
> > >>> - Reader side simple authorization via whitelist [YARN-6820]
> > >>>
> > >>> We would love to get your thoughts on this before we open a real
> voting
> > >>> thread.
> > >>>
> > >>> Special thanks to a team of folks who worked hard and contributed
> > towards
> > >>> this effort via patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>>
> > >>> Thanks
> > >>> Vrushali
> > >>> [1] Merge to trunk: http://www.mail-archive.com/
> yarn-dev@hadoop.apache
> > .
> > >>> org/msg23897.html
> > >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg23897.html>
> > >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> > >>> ache/hadoop/commits/YARN-5355
> > >>>
> > >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> > >>> vrushali.c@gmail.com> wrote:
> > >>>
> > >>> > Hi All
> > >>> >
> > >>> > I’d like to open a discussion for merging Timeline Service v.2
> > >>> (YARN-5355)
> > >>> > to trunk in a few weeks.
> > >>> >
> > >>> > We have previously completed one merge onto trunk [1] and Timeline
> > >>> Service
> > >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>> >
> > >>> > Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> > Service v2 in a feature branch [2]. There are a few related issues
> > >>> pending
> > >>> > that are being actively worked upon & tested. As soon as they are
> > >>> resolved,
> > >>> > we plan on starting a merge vote within the next 2 weeks. The goal
> is
> > >>> to
> > >>> > get this in for Hadoop3 beta.
> > >>> >
> > >>> > We have paid close attention to ensure that once disabled Timeline
> > >>> Service
> > >>> > v.2 does not impact existing functionality when disabled (by
> > default).
> > >>> >
> > >>> > At a high level, following are the key features that have been
> > >>> implemented
> > >>> > since 3.0.0-alpha1:
> > >>> > - Security (via Kerberos Authentication & delegation tokens) at the
> > >>> writer
> > >>> > [YARN-3053]
> > >>> > - Timeline server usability improvements [timeline-server
> > >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> > ]
> > >>> > - HBase specific improvements [atsv2-hbase
> > >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> > >>> ed%20DESC%2C%
> > >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> > ]
> > >>> > - REST API additions & improvements [timeline-reader
> > >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> > >>> ty%20DESC%2C%
> > >>> > 20created%20ASC>
> > >>> > ]
> > >>> > - Reader side simple authorization via whitelist [YARN-6820]
> > >>> >
> > >>> > We would love to get your thoughts on these and more before we
> open a
> > >>> real
> > >>> > voting thread.
> > >>> >
> > >>> > Special thanks to a team of folks who worked hard and contributed
> > >>> towards
> > >>> > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > >>> Varun
> > >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>> >
> > >>> > Thanks!
> > >>> > Vrushali
> > >>> >
> > >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> > >>> > apache.org/msg23897.html
> > >>> > [2] feature branch YARN-5355 commits: https://github.com/
> > >>> > apache/hadoop/commits/YARN-5355
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Rohith Sharma K S <ro...@apache.org>.
Hi Sangjin
Thanks for bringing this point.
We did similar exercise with current YARN-5355 branch today. Tests are
validated against default configuration and timeline server v.1.5 as well.
All YARN major features such as RM HA/Restart/work-preserving-restart, NM
restart, scheduling, sample MR/distributes shell jobs run are validated.
Everything looks fine.
This testing will also be done once YARN-5355 branch code freezed
completely as well in couple of days.
Thanks & Regards
Rohith Sharma K S
On 18 August 2017 at 22:56, Sangjin Lee <sj...@apache.org> wrote:
> Kudos to Vrushali and the team for getting ready for this large and
> important feature! I know a huge team effort went into this. I look forward
> to seeing this merged.
>
> I'd like to ask one piece of due diligence. Could you please inspect
> rigorously to ensure that when disabled Timeline Service v.2 does not
> impact other features in any way? We did a similar exercise when we had the
> first drop, and it would be good to repeat that... Thanks!
>
> Sangjin
>
> On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Great, thanks Vrushali! Sounds good to me.
> >
> > I have a few procedural release notes comments I'll put on YARN-5355, to
> > make sure we advertise this to our users appropriately.
> >
> > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> > vrushali.c@gmail.com> wrote:
> >
> > > Hi Andrew,
> > >
> > > Thanks for your response!
> > >
> > > There have been no changes to existing APIs since alpha1.
> > >
> > > We at Twitter have tested the feature to demonstrate it works at what
> we
> > > consider moderate scale but this did not include the security related
> > > testing. The security testing is in progress at present by Timeline
> > Service
> > > V2 team in the community and we think we will have more details on this
> > > very soon.
> > >
> > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> > > think of as "merge-blockers". The issues being targeted for merge are
> in
> > > [link1] below. There are about 59 jiras of which 56 are completed.
> > >
> > > We plan to make a new umbrella jira after the merge to trunk. We will
> > then
> > > create a new branch with the new jira name and move these open jiras
> > under
> > > YARN-5355 as subtasks of that new umbrella jira.
> > >
> > > thanks
> > > Vrushali
> > > [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
> > >
> > >
> > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> > > wrote:
> > >
> > >> Hi Vrushali,
> > >>
> > >> Glad to hear this major dev milestone is nearing completion!
> > >>
> > >> Repeating my request on other merge [DISCUSS] threads, could you
> comment
> > >> on testing and API stability of this merge? Our timeline for beta1 is
> > about
> > >> a month out, so there's not much time to fix things beforehand.
> > >>
> > >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> > most
> > >> of these be moved out to a new umbrella? I'm wondering what needs to
> be
> > >> completed before sending the merge vote.
> > >>
> > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex
> the
> > >> beta1 release date for this feature than others. Hopefully that won't
> be
> > >> necessary though :)
> > >>
> > >> Best,
> > >> Andrew
> > >>
> > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> > >> vrushalic2016@gmail.com> wrote:
> > >>
> > >>> Looks like some of the hyperlinks appear messed up, my apologies,
> > >>> resending
> > >>> the same email with hopefully better looking content:
> > >>>
> > >>> Hi All,
> > >>>
> > >>> I'd like to open a discussion for merging Timeline Service v2
> > (YARN-5355)
> > >>> to trunk in a few weeks.
> > >>>
> > >>> We have previously completed one merge onto trunk [1] and Timeline
> > >>> Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> Service v2 in a feature branch [2]. There are a few related issues
> > >>> pending
> > >>> that are being actively worked upon and tested. As soon as they are
> > >>> resolved, we plan on starting a merge vote within the next two weeks.
> > The
> > >>> goal is to get this into hadoop3 beta.
> > >>>
> > >>> We have paid close attention to ensure that once disabled Timeline
> > >>> Service
> > >>> v2 does not impact existing functionality when disabled (by default).
> > >>>
> > >>> At a high level, following are the key features that have been
> > >>> implemented
> > >>> since 3.0.0-alpha1:
> > >>> - Security (via Kerberos Authentication & delegation tokens)
> > [YARN-3053]
> > >>> - Timeline server usability improvements [timeline-server
> > >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> > >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> > >>> 2C%20created%20ASC>
> > >>> ]
> > >>> - HBase specific improvements [atsv2-hbase
> > >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> > >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> > >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> ]
> > >>> - REST API additions and improvements [timeline-reader
> > >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> > >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> > >>> ]
> > >>> - Reader side simple authorization via whitelist [YARN-6820]
> > >>>
> > >>> We would love to get your thoughts on this before we open a real
> voting
> > >>> thread.
> > >>>
> > >>> Special thanks to a team of folks who worked hard and contributed
> > towards
> > >>> this effort via patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>>
> > >>> Thanks
> > >>> Vrushali
> > >>> [1] Merge to trunk: http://www.mail-archive.com/
> yarn-dev@hadoop.apache
> > .
> > >>> org/msg23897.html
> > >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg23897.html>
> > >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> > >>> ache/hadoop/commits/YARN-5355
> > >>>
> > >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> > >>> vrushali.c@gmail.com> wrote:
> > >>>
> > >>> > Hi All
> > >>> >
> > >>> > I’d like to open a discussion for merging Timeline Service v.2
> > >>> (YARN-5355)
> > >>> > to trunk in a few weeks.
> > >>> >
> > >>> > We have previously completed one merge onto trunk [1] and Timeline
> > >>> Service
> > >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>> >
> > >>> > Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> > Service v2 in a feature branch [2]. There are a few related issues
> > >>> pending
> > >>> > that are being actively worked upon & tested. As soon as they are
> > >>> resolved,
> > >>> > we plan on starting a merge vote within the next 2 weeks. The goal
> is
> > >>> to
> > >>> > get this in for Hadoop3 beta.
> > >>> >
> > >>> > We have paid close attention to ensure that once disabled Timeline
> > >>> Service
> > >>> > v.2 does not impact existing functionality when disabled (by
> > default).
> > >>> >
> > >>> > At a high level, following are the key features that have been
> > >>> implemented
> > >>> > since 3.0.0-alpha1:
> > >>> > - Security (via Kerberos Authentication & delegation tokens) at the
> > >>> writer
> > >>> > [YARN-3053]
> > >>> > - Timeline server usability improvements [timeline-server
> > >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> > ]
> > >>> > - HBase specific improvements [atsv2-hbase
> > >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> > >>> ed%20DESC%2C%
> > >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> > ]
> > >>> > - REST API additions & improvements [timeline-reader
> > >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> > >>> ty%20DESC%2C%
> > >>> > 20created%20ASC>
> > >>> > ]
> > >>> > - Reader side simple authorization via whitelist [YARN-6820]
> > >>> >
> > >>> > We would love to get your thoughts on these and more before we
> open a
> > >>> real
> > >>> > voting thread.
> > >>> >
> > >>> > Special thanks to a team of folks who worked hard and contributed
> > >>> towards
> > >>> > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > >>> Varun
> > >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>> >
> > >>> > Thanks!
> > >>> > Vrushali
> > >>> >
> > >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> > >>> > apache.org/msg23897.html
> > >>> > [2] feature branch YARN-5355 commits: https://github.com/
> > >>> > apache/hadoop/commits/YARN-5355
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Rohith Sharma K S <ro...@apache.org>.
Hi Sangjin
Thanks for bringing this point.
We did similar exercise with current YARN-5355 branch today. Tests are
validated against default configuration and timeline server v.1.5 as well.
All YARN major features such as RM HA/Restart/work-preserving-restart, NM
restart, scheduling, sample MR/distributes shell jobs run are validated.
Everything looks fine.
This testing will also be done once YARN-5355 branch code freezed
completely as well in couple of days.
Thanks & Regards
Rohith Sharma K S
On 18 August 2017 at 22:56, Sangjin Lee <sj...@apache.org> wrote:
> Kudos to Vrushali and the team for getting ready for this large and
> important feature! I know a huge team effort went into this. I look forward
> to seeing this merged.
>
> I'd like to ask one piece of due diligence. Could you please inspect
> rigorously to ensure that when disabled Timeline Service v.2 does not
> impact other features in any way? We did a similar exercise when we had the
> first drop, and it would be good to repeat that... Thanks!
>
> Sangjin
>
> On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
> > Great, thanks Vrushali! Sounds good to me.
> >
> > I have a few procedural release notes comments I'll put on YARN-5355, to
> > make sure we advertise this to our users appropriately.
> >
> > On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> > vrushali.c@gmail.com> wrote:
> >
> > > Hi Andrew,
> > >
> > > Thanks for your response!
> > >
> > > There have been no changes to existing APIs since alpha1.
> > >
> > > We at Twitter have tested the feature to demonstrate it works at what
> we
> > > consider moderate scale but this did not include the security related
> > > testing. The security testing is in progress at present by Timeline
> > Service
> > > V2 team in the community and we think we will have more details on this
> > > very soon.
> > >
> > > About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> > > think of as "merge-blockers". The issues being targeted for merge are
> in
> > > [link1] below. There are about 59 jiras of which 56 are completed.
> > >
> > > We plan to make a new umbrella jira after the merge to trunk. We will
> > then
> > > create a new branch with the new jira name and move these open jiras
> > under
> > > YARN-5355 as subtasks of that new umbrella jira.
> > >
> > > thanks
> > > Vrushali
> > > [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
> > >
> > >
> > > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <
> andrew.wang@cloudera.com>
> > > wrote:
> > >
> > >> Hi Vrushali,
> > >>
> > >> Glad to hear this major dev milestone is nearing completion!
> > >>
> > >> Repeating my request on other merge [DISCUSS] threads, could you
> comment
> > >> on testing and API stability of this merge? Our timeline for beta1 is
> > about
> > >> a month out, so there's not much time to fix things beforehand.
> > >>
> > >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> > most
> > >> of these be moved out to a new umbrella? I'm wondering what needs to
> be
> > >> completed before sending the merge vote.
> > >>
> > >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex
> the
> > >> beta1 release date for this feature than others. Hopefully that won't
> be
> > >> necessary though :)
> > >>
> > >> Best,
> > >> Andrew
> > >>
> > >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> > >> vrushalic2016@gmail.com> wrote:
> > >>
> > >>> Looks like some of the hyperlinks appear messed up, my apologies,
> > >>> resending
> > >>> the same email with hopefully better looking content:
> > >>>
> > >>> Hi All,
> > >>>
> > >>> I'd like to open a discussion for merging Timeline Service v2
> > (YARN-5355)
> > >>> to trunk in a few weeks.
> > >>>
> > >>> We have previously completed one merge onto trunk [1] and Timeline
> > >>> Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> Service v2 in a feature branch [2]. There are a few related issues
> > >>> pending
> > >>> that are being actively worked upon and tested. As soon as they are
> > >>> resolved, we plan on starting a merge vote within the next two weeks.
> > The
> > >>> goal is to get this into hadoop3 beta.
> > >>>
> > >>> We have paid close attention to ensure that once disabled Timeline
> > >>> Service
> > >>> v2 does not impact existing functionality when disabled (by default).
> > >>>
> > >>> At a high level, following are the key features that have been
> > >>> implemented
> > >>> since 3.0.0-alpha1:
> > >>> - Security (via Kerberos Authentication & delegation tokens)
> > [YARN-3053]
> > >>> - Timeline server usability improvements [timeline-server
> > >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> > >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> > >>> 2C%20created%20ASC>
> > >>> ]
> > >>> - HBase specific improvements [atsv2-hbase
> > >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> > >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> > >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> ]
> > >>> - REST API additions and improvements [timeline-reader
> > >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> > >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> > >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> > >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> > >>> ]
> > >>> - Reader side simple authorization via whitelist [YARN-6820]
> > >>>
> > >>> We would love to get your thoughts on this before we open a real
> voting
> > >>> thread.
> > >>>
> > >>> Special thanks to a team of folks who worked hard and contributed
> > towards
> > >>> this effort via patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>>
> > >>> Thanks
> > >>> Vrushali
> > >>> [1] Merge to trunk: http://www.mail-archive.com/
> yarn-dev@hadoop.apache
> > .
> > >>> org/msg23897.html
> > >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg23897.html>
> > >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> > >>> ache/hadoop/commits/YARN-5355
> > >>>
> > >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> > >>> vrushali.c@gmail.com> wrote:
> > >>>
> > >>> > Hi All
> > >>> >
> > >>> > I’d like to open a discussion for merging Timeline Service v.2
> > >>> (YARN-5355)
> > >>> > to trunk in a few weeks.
> > >>> >
> > >>> > We have previously completed one merge onto trunk [1] and Timeline
> > >>> Service
> > >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>> >
> > >>> > Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> > Service v2 in a feature branch [2]. There are a few related issues
> > >>> pending
> > >>> > that are being actively worked upon & tested. As soon as they are
> > >>> resolved,
> > >>> > we plan on starting a merge vote within the next 2 weeks. The goal
> is
> > >>> to
> > >>> > get this in for Hadoop3 beta.
> > >>> >
> > >>> > We have paid close attention to ensure that once disabled Timeline
> > >>> Service
> > >>> > v.2 does not impact existing functionality when disabled (by
> > default).
> > >>> >
> > >>> > At a high level, following are the key features that have been
> > >>> implemented
> > >>> > since 3.0.0-alpha1:
> > >>> > - Security (via Kerberos Authentication & delegation tokens) at the
> > >>> writer
> > >>> > [YARN-3053]
> > >>> > - Timeline server usability improvements [timeline-server
> > >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> > ]
> > >>> > - HBase specific improvements [atsv2-hbase
> > >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> > >>> ed%20DESC%2C%
> > >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > >>> > ]
> > >>> > - REST API additions & improvements [timeline-reader
> > >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> > >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> > >>> ty%20DESC%2C%
> > >>> > 20created%20ASC>
> > >>> > ]
> > >>> > - Reader side simple authorization via whitelist [YARN-6820]
> > >>> >
> > >>> > We would love to get your thoughts on these and more before we
> open a
> > >>> real
> > >>> > voting thread.
> > >>> >
> > >>> > Special thanks to a team of folks who worked hard and contributed
> > >>> towards
> > >>> > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > >>> Varun
> > >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>> >
> > >>> > Thanks!
> > >>> > Vrushali
> > >>> >
> > >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> > >>> > apache.org/msg23897.html
> > >>> > [2] feature branch YARN-5355 commits: https://github.com/
> > >>> > apache/hadoop/commits/YARN-5355
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
Kudos to Vrushali and the team for getting ready for this large and
important feature! I know a huge team effort went into this. I look forward
to seeing this merged.
I'd like to ask one piece of due diligence. Could you please inspect
rigorously to ensure that when disabled Timeline Service v.2 does not
impact other features in any way? We did a similar exercise when we had the
first drop, and it would be good to repeat that... Thanks!
Sangjin
On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <an...@cloudera.com>
wrote:
> Great, thanks Vrushali! Sounds good to me.
>
> I have a few procedural release notes comments I'll put on YARN-5355, to
> make sure we advertise this to our users appropriately.
>
> On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> vrushali.c@gmail.com> wrote:
>
> > Hi Andrew,
> >
> > Thanks for your response!
> >
> > There have been no changes to existing APIs since alpha1.
> >
> > We at Twitter have tested the feature to demonstrate it works at what we
> > consider moderate scale but this did not include the security related
> > testing. The security testing is in progress at present by Timeline
> Service
> > V2 team in the community and we think we will have more details on this
> > very soon.
> >
> > About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> > think of as "merge-blockers". The issues being targeted for merge are in
> > [link1] below. There are about 59 jiras of which 56 are completed.
> >
> > We plan to make a new umbrella jira after the merge to trunk. We will
> then
> > create a new branch with the new jira name and move these open jiras
> under
> > YARN-5355 as subtasks of that new umbrella jira.
> >
> > thanks
> > Vrushali
> > [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
> >
> >
> > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> >> Hi Vrushali,
> >>
> >> Glad to hear this major dev milestone is nearing completion!
> >>
> >> Repeating my request on other merge [DISCUSS] threads, could you comment
> >> on testing and API stability of this merge? Our timeline for beta1 is
> about
> >> a month out, so there's not much time to fix things beforehand.
> >>
> >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> most
> >> of these be moved out to a new umbrella? I'm wondering what needs to be
> >> completed before sending the merge vote.
> >>
> >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
> >> beta1 release date for this feature than others. Hopefully that won't be
> >> necessary though :)
> >>
> >> Best,
> >> Andrew
> >>
> >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> >> vrushalic2016@gmail.com> wrote:
> >>
> >>> Looks like some of the hyperlinks appear messed up, my apologies,
> >>> resending
> >>> the same email with hopefully better looking content:
> >>>
> >>> Hi All,
> >>>
> >>> I'd like to open a discussion for merging Timeline Service v2
> (YARN-5355)
> >>> to trunk in a few weeks.
> >>>
> >>> We have previously completed one merge onto trunk [1] and Timeline
> >>> Service
> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>>
> >>> Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> Service v2 in a feature branch [2]. There are a few related issues
> >>> pending
> >>> that are being actively worked upon and tested. As soon as they are
> >>> resolved, we plan on starting a merge vote within the next two weeks.
> The
> >>> goal is to get this into hadoop3 beta.
> >>>
> >>> We have paid close attention to ensure that once disabled Timeline
> >>> Service
> >>> v2 does not impact existing functionality when disabled (by default).
> >>>
> >>> At a high level, following are the key features that have been
> >>> implemented
> >>> since 3.0.0-alpha1:
> >>> - Security (via Kerberos Authentication & delegation tokens)
> [YARN-3053]
> >>> - Timeline server usability improvements [timeline-server
> >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> >>> 2C%20created%20ASC>
> >>> ]
> >>> - HBase specific improvements [atsv2-hbase
> >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> ]
> >>> - REST API additions and improvements [timeline-reader
> >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> ]
> >>> - Reader side simple authorization via whitelist [YARN-6820]
> >>>
> >>> We would love to get your thoughts on this before we open a real voting
> >>> thread.
> >>>
> >>> Special thanks to a team of folks who worked hard and contributed
> towards
> >>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>>
> >>> Thanks
> >>> Vrushali
> >>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache
> .
> >>> org/msg23897.html
> >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg23897.html>
> >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> >>> ache/hadoop/commits/YARN-5355
> >>>
> >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> >>> vrushali.c@gmail.com> wrote:
> >>>
> >>> > Hi All
> >>> >
> >>> > I’d like to open a discussion for merging Timeline Service v.2
> >>> (YARN-5355)
> >>> > to trunk in a few weeks.
> >>> >
> >>> > We have previously completed one merge onto trunk [1] and Timeline
> >>> Service
> >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >>> >
> >>> > Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> > Service v2 in a feature branch [2]. There are a few related issues
> >>> pending
> >>> > that are being actively worked upon & tested. As soon as they are
> >>> resolved,
> >>> > we plan on starting a merge vote within the next 2 weeks. The goal is
> >>> to
> >>> > get this in for Hadoop3 beta.
> >>> >
> >>> > We have paid close attention to ensure that once disabled Timeline
> >>> Service
> >>> > v.2 does not impact existing functionality when disabled (by
> default).
> >>> >
> >>> > At a high level, following are the key features that have been
> >>> implemented
> >>> > since 3.0.0-alpha1:
> >>> > - Security (via Kerberos Authentication & delegation tokens) at the
> >>> writer
> >>> > [YARN-3053]
> >>> > - Timeline server usability improvements [timeline-server
> >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> > ]
> >>> > - HBase specific improvements [atsv2-hbase
> >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> >>> ed%20DESC%2C%
> >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> > ]
> >>> > - REST API additions & improvements [timeline-reader
> >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> >>> ty%20DESC%2C%
> >>> > 20created%20ASC>
> >>> > ]
> >>> > - Reader side simple authorization via whitelist [YARN-6820]
> >>> >
> >>> > We would love to get your thoughts on these and more before we open a
> >>> real
> >>> > voting thread.
> >>> >
> >>> > Special thanks to a team of folks who worked hard and contributed
> >>> towards
> >>> > this effort with patches, reviews and guidance: Rohith Sharma K S,
> >>> Varun
> >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>> >
> >>> > Thanks!
> >>> > Vrushali
> >>> >
> >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> >>> > apache.org/msg23897.html
> >>> > [2] feature branch YARN-5355 commits: https://github.com/
> >>> > apache/hadoop/commits/YARN-5355
> >>> >
> >>>
> >>
> >>
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
Kudos to Vrushali and the team for getting ready for this large and
important feature! I know a huge team effort went into this. I look forward
to seeing this merged.
I'd like to ask one piece of due diligence. Could you please inspect
rigorously to ensure that when disabled Timeline Service v.2 does not
impact other features in any way? We did a similar exercise when we had the
first drop, and it would be good to repeat that... Thanks!
Sangjin
On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <an...@cloudera.com>
wrote:
> Great, thanks Vrushali! Sounds good to me.
>
> I have a few procedural release notes comments I'll put on YARN-5355, to
> make sure we advertise this to our users appropriately.
>
> On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> vrushali.c@gmail.com> wrote:
>
> > Hi Andrew,
> >
> > Thanks for your response!
> >
> > There have been no changes to existing APIs since alpha1.
> >
> > We at Twitter have tested the feature to demonstrate it works at what we
> > consider moderate scale but this did not include the security related
> > testing. The security testing is in progress at present by Timeline
> Service
> > V2 team in the community and we think we will have more details on this
> > very soon.
> >
> > About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> > think of as "merge-blockers". The issues being targeted for merge are in
> > [link1] below. There are about 59 jiras of which 56 are completed.
> >
> > We plan to make a new umbrella jira after the merge to trunk. We will
> then
> > create a new branch with the new jira name and move these open jiras
> under
> > YARN-5355 as subtasks of that new umbrella jira.
> >
> > thanks
> > Vrushali
> > [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
> >
> >
> > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> >> Hi Vrushali,
> >>
> >> Glad to hear this major dev milestone is nearing completion!
> >>
> >> Repeating my request on other merge [DISCUSS] threads, could you comment
> >> on testing and API stability of this merge? Our timeline for beta1 is
> about
> >> a month out, so there's not much time to fix things beforehand.
> >>
> >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> most
> >> of these be moved out to a new umbrella? I'm wondering what needs to be
> >> completed before sending the merge vote.
> >>
> >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
> >> beta1 release date for this feature than others. Hopefully that won't be
> >> necessary though :)
> >>
> >> Best,
> >> Andrew
> >>
> >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> >> vrushalic2016@gmail.com> wrote:
> >>
> >>> Looks like some of the hyperlinks appear messed up, my apologies,
> >>> resending
> >>> the same email with hopefully better looking content:
> >>>
> >>> Hi All,
> >>>
> >>> I'd like to open a discussion for merging Timeline Service v2
> (YARN-5355)
> >>> to trunk in a few weeks.
> >>>
> >>> We have previously completed one merge onto trunk [1] and Timeline
> >>> Service
> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>>
> >>> Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> Service v2 in a feature branch [2]. There are a few related issues
> >>> pending
> >>> that are being actively worked upon and tested. As soon as they are
> >>> resolved, we plan on starting a merge vote within the next two weeks.
> The
> >>> goal is to get this into hadoop3 beta.
> >>>
> >>> We have paid close attention to ensure that once disabled Timeline
> >>> Service
> >>> v2 does not impact existing functionality when disabled (by default).
> >>>
> >>> At a high level, following are the key features that have been
> >>> implemented
> >>> since 3.0.0-alpha1:
> >>> - Security (via Kerberos Authentication & delegation tokens)
> [YARN-3053]
> >>> - Timeline server usability improvements [timeline-server
> >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> >>> 2C%20created%20ASC>
> >>> ]
> >>> - HBase specific improvements [atsv2-hbase
> >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> ]
> >>> - REST API additions and improvements [timeline-reader
> >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> ]
> >>> - Reader side simple authorization via whitelist [YARN-6820]
> >>>
> >>> We would love to get your thoughts on this before we open a real voting
> >>> thread.
> >>>
> >>> Special thanks to a team of folks who worked hard and contributed
> towards
> >>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>>
> >>> Thanks
> >>> Vrushali
> >>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache
> .
> >>> org/msg23897.html
> >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg23897.html>
> >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> >>> ache/hadoop/commits/YARN-5355
> >>>
> >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> >>> vrushali.c@gmail.com> wrote:
> >>>
> >>> > Hi All
> >>> >
> >>> > I’d like to open a discussion for merging Timeline Service v.2
> >>> (YARN-5355)
> >>> > to trunk in a few weeks.
> >>> >
> >>> > We have previously completed one merge onto trunk [1] and Timeline
> >>> Service
> >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >>> >
> >>> > Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> > Service v2 in a feature branch [2]. There are a few related issues
> >>> pending
> >>> > that are being actively worked upon & tested. As soon as they are
> >>> resolved,
> >>> > we plan on starting a merge vote within the next 2 weeks. The goal is
> >>> to
> >>> > get this in for Hadoop3 beta.
> >>> >
> >>> > We have paid close attention to ensure that once disabled Timeline
> >>> Service
> >>> > v.2 does not impact existing functionality when disabled (by
> default).
> >>> >
> >>> > At a high level, following are the key features that have been
> >>> implemented
> >>> > since 3.0.0-alpha1:
> >>> > - Security (via Kerberos Authentication & delegation tokens) at the
> >>> writer
> >>> > [YARN-3053]
> >>> > - Timeline server usability improvements [timeline-server
> >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> > ]
> >>> > - HBase specific improvements [atsv2-hbase
> >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> >>> ed%20DESC%2C%
> >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> > ]
> >>> > - REST API additions & improvements [timeline-reader
> >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> >>> ty%20DESC%2C%
> >>> > 20created%20ASC>
> >>> > ]
> >>> > - Reader side simple authorization via whitelist [YARN-6820]
> >>> >
> >>> > We would love to get your thoughts on these and more before we open a
> >>> real
> >>> > voting thread.
> >>> >
> >>> > Special thanks to a team of folks who worked hard and contributed
> >>> towards
> >>> > this effort with patches, reviews and guidance: Rohith Sharma K S,
> >>> Varun
> >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>> >
> >>> > Thanks!
> >>> > Vrushali
> >>> >
> >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> >>> > apache.org/msg23897.html
> >>> > [2] feature branch YARN-5355 commits: https://github.com/
> >>> > apache/hadoop/commits/YARN-5355
> >>> >
> >>>
> >>
> >>
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
Kudos to Vrushali and the team for getting ready for this large and
important feature! I know a huge team effort went into this. I look forward
to seeing this merged.
I'd like to ask one piece of due diligence. Could you please inspect
rigorously to ensure that when disabled Timeline Service v.2 does not
impact other features in any way? We did a similar exercise when we had the
first drop, and it would be good to repeat that... Thanks!
Sangjin
On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <an...@cloudera.com>
wrote:
> Great, thanks Vrushali! Sounds good to me.
>
> I have a few procedural release notes comments I'll put on YARN-5355, to
> make sure we advertise this to our users appropriately.
>
> On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> vrushali.c@gmail.com> wrote:
>
> > Hi Andrew,
> >
> > Thanks for your response!
> >
> > There have been no changes to existing APIs since alpha1.
> >
> > We at Twitter have tested the feature to demonstrate it works at what we
> > consider moderate scale but this did not include the security related
> > testing. The security testing is in progress at present by Timeline
> Service
> > V2 team in the community and we think we will have more details on this
> > very soon.
> >
> > About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> > think of as "merge-blockers". The issues being targeted for merge are in
> > [link1] below. There are about 59 jiras of which 56 are completed.
> >
> > We plan to make a new umbrella jira after the merge to trunk. We will
> then
> > create a new branch with the new jira name and move these open jiras
> under
> > YARN-5355 as subtasks of that new umbrella jira.
> >
> > thanks
> > Vrushali
> > [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
> >
> >
> > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> >> Hi Vrushali,
> >>
> >> Glad to hear this major dev milestone is nearing completion!
> >>
> >> Repeating my request on other merge [DISCUSS] threads, could you comment
> >> on testing and API stability of this merge? Our timeline for beta1 is
> about
> >> a month out, so there's not much time to fix things beforehand.
> >>
> >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> most
> >> of these be moved out to a new umbrella? I'm wondering what needs to be
> >> completed before sending the merge vote.
> >>
> >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
> >> beta1 release date for this feature than others. Hopefully that won't be
> >> necessary though :)
> >>
> >> Best,
> >> Andrew
> >>
> >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> >> vrushalic2016@gmail.com> wrote:
> >>
> >>> Looks like some of the hyperlinks appear messed up, my apologies,
> >>> resending
> >>> the same email with hopefully better looking content:
> >>>
> >>> Hi All,
> >>>
> >>> I'd like to open a discussion for merging Timeline Service v2
> (YARN-5355)
> >>> to trunk in a few weeks.
> >>>
> >>> We have previously completed one merge onto trunk [1] and Timeline
> >>> Service
> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>>
> >>> Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> Service v2 in a feature branch [2]. There are a few related issues
> >>> pending
> >>> that are being actively worked upon and tested. As soon as they are
> >>> resolved, we plan on starting a merge vote within the next two weeks.
> The
> >>> goal is to get this into hadoop3 beta.
> >>>
> >>> We have paid close attention to ensure that once disabled Timeline
> >>> Service
> >>> v2 does not impact existing functionality when disabled (by default).
> >>>
> >>> At a high level, following are the key features that have been
> >>> implemented
> >>> since 3.0.0-alpha1:
> >>> - Security (via Kerberos Authentication & delegation tokens)
> [YARN-3053]
> >>> - Timeline server usability improvements [timeline-server
> >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> >>> 2C%20created%20ASC>
> >>> ]
> >>> - HBase specific improvements [atsv2-hbase
> >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> ]
> >>> - REST API additions and improvements [timeline-reader
> >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> ]
> >>> - Reader side simple authorization via whitelist [YARN-6820]
> >>>
> >>> We would love to get your thoughts on this before we open a real voting
> >>> thread.
> >>>
> >>> Special thanks to a team of folks who worked hard and contributed
> towards
> >>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>>
> >>> Thanks
> >>> Vrushali
> >>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache
> .
> >>> org/msg23897.html
> >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg23897.html>
> >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> >>> ache/hadoop/commits/YARN-5355
> >>>
> >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> >>> vrushali.c@gmail.com> wrote:
> >>>
> >>> > Hi All
> >>> >
> >>> > I’d like to open a discussion for merging Timeline Service v.2
> >>> (YARN-5355)
> >>> > to trunk in a few weeks.
> >>> >
> >>> > We have previously completed one merge onto trunk [1] and Timeline
> >>> Service
> >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >>> >
> >>> > Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> > Service v2 in a feature branch [2]. There are a few related issues
> >>> pending
> >>> > that are being actively worked upon & tested. As soon as they are
> >>> resolved,
> >>> > we plan on starting a merge vote within the next 2 weeks. The goal is
> >>> to
> >>> > get this in for Hadoop3 beta.
> >>> >
> >>> > We have paid close attention to ensure that once disabled Timeline
> >>> Service
> >>> > v.2 does not impact existing functionality when disabled (by
> default).
> >>> >
> >>> > At a high level, following are the key features that have been
> >>> implemented
> >>> > since 3.0.0-alpha1:
> >>> > - Security (via Kerberos Authentication & delegation tokens) at the
> >>> writer
> >>> > [YARN-3053]
> >>> > - Timeline server usability improvements [timeline-server
> >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> > ]
> >>> > - HBase specific improvements [atsv2-hbase
> >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> >>> ed%20DESC%2C%
> >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> > ]
> >>> > - REST API additions & improvements [timeline-reader
> >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> >>> ty%20DESC%2C%
> >>> > 20created%20ASC>
> >>> > ]
> >>> > - Reader side simple authorization via whitelist [YARN-6820]
> >>> >
> >>> > We would love to get your thoughts on these and more before we open a
> >>> real
> >>> > voting thread.
> >>> >
> >>> > Special thanks to a team of folks who worked hard and contributed
> >>> towards
> >>> > this effort with patches, reviews and guidance: Rohith Sharma K S,
> >>> Varun
> >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>> >
> >>> > Thanks!
> >>> > Vrushali
> >>> >
> >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> >>> > apache.org/msg23897.html
> >>> > [2] feature branch YARN-5355 commits: https://github.com/
> >>> > apache/hadoop/commits/YARN-5355
> >>> >
> >>>
> >>
> >>
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
Kudos to Vrushali and the team for getting ready for this large and
important feature! I know a huge team effort went into this. I look forward
to seeing this merged.
I'd like to ask one piece of due diligence. Could you please inspect
rigorously to ensure that when disabled Timeline Service v.2 does not
impact other features in any way? We did a similar exercise when we had the
first drop, and it would be good to repeat that... Thanks!
Sangjin
On Wed, Aug 16, 2017 at 11:44 AM, Andrew Wang <an...@cloudera.com>
wrote:
> Great, thanks Vrushali! Sounds good to me.
>
> I have a few procedural release notes comments I'll put on YARN-5355, to
> make sure we advertise this to our users appropriately.
>
> On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
> vrushali.c@gmail.com> wrote:
>
> > Hi Andrew,
> >
> > Thanks for your response!
> >
> > There have been no changes to existing APIs since alpha1.
> >
> > We at Twitter have tested the feature to demonstrate it works at what we
> > consider moderate scale but this did not include the security related
> > testing. The security testing is in progress at present by Timeline
> Service
> > V2 team in the community and we think we will have more details on this
> > very soon.
> >
> > About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> > think of as "merge-blockers". The issues being targeted for merge are in
> > [link1] below. There are about 59 jiras of which 56 are completed.
> >
> > We plan to make a new umbrella jira after the merge to trunk. We will
> then
> > create a new branch with the new jira name and move these open jiras
> under
> > YARN-5355 as subtasks of that new umbrella jira.
> >
> > thanks
> > Vrushali
> > [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
> >
> >
> > On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <an...@cloudera.com>
> > wrote:
> >
> >> Hi Vrushali,
> >>
> >> Glad to hear this major dev milestone is nearing completion!
> >>
> >> Repeating my request on other merge [DISCUSS] threads, could you comment
> >> on testing and API stability of this merge? Our timeline for beta1 is
> about
> >> a month out, so there's not much time to fix things beforehand.
> >>
> >> Looking at YARN-5355 there are also many unresolved subtasks. Should
> most
> >> of these be moved out to a new umbrella? I'm wondering what needs to be
> >> completed before sending the merge vote.
> >>
> >> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
> >> beta1 release date for this feature than others. Hopefully that won't be
> >> necessary though :)
> >>
> >> Best,
> >> Andrew
> >>
> >> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> >> vrushalic2016@gmail.com> wrote:
> >>
> >>> Looks like some of the hyperlinks appear messed up, my apologies,
> >>> resending
> >>> the same email with hopefully better looking content:
> >>>
> >>> Hi All,
> >>>
> >>> I'd like to open a discussion for merging Timeline Service v2
> (YARN-5355)
> >>> to trunk in a few weeks.
> >>>
> >>> We have previously completed one merge onto trunk [1] and Timeline
> >>> Service
> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>>
> >>> Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> Service v2 in a feature branch [2]. There are a few related issues
> >>> pending
> >>> that are being actively worked upon and tested. As soon as they are
> >>> resolved, we plan on starting a merge vote within the next two weeks.
> The
> >>> goal is to get this into hadoop3 beta.
> >>>
> >>> We have paid close attention to ensure that once disabled Timeline
> >>> Service
> >>> v2 does not impact existing functionality when disabled (by default).
> >>>
> >>> At a high level, following are the key features that have been
> >>> implemented
> >>> since 3.0.0-alpha1:
> >>> - Security (via Kerberos Authentication & delegation tokens)
> [YARN-3053]
> >>> - Timeline server usability improvements [timeline-server
> >>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> >>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
> >>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
> >>> 2C%20created%20ASC>
> >>> ]
> >>> - HBase specific improvements [atsv2-hbase
> >>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
> >>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> >>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> ]
> >>> - REST API additions and improvements [timeline-reader
> >>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
> >>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
> >>> omponent%20in%20(timelineclient%2C%20timelinereader)%
> >>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> ]
> >>> - Reader side simple authorization via whitelist [YARN-6820]
> >>>
> >>> We would love to get your thoughts on this before we open a real voting
> >>> thread.
> >>>
> >>> Special thanks to a team of folks who worked hard and contributed
> towards
> >>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>>
> >>> Thanks
> >>> Vrushali
> >>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache
> .
> >>> org/msg23897.html
> >>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg23897.html>
> >>> [2] feature branch YARN-5355 commits: https://github.com/ap
> >>> ache/hadoop/commits/YARN-5355
> >>>
> >>> <https://github.com/apache/hadoop/commits/YARN-5355>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> >>> vrushali.c@gmail.com> wrote:
> >>>
> >>> > Hi All
> >>> >
> >>> > I’d like to open a discussion for merging Timeline Service v.2
> >>> (YARN-5355)
> >>> > to trunk in a few weeks.
> >>> >
> >>> > We have previously completed one merge onto trunk [1] and Timeline
> >>> Service
> >>> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >>> >
> >>> > Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> > Service v2 in a feature branch [2]. There are a few related issues
> >>> pending
> >>> > that are being actively worked upon & tested. As soon as they are
> >>> resolved,
> >>> > we plan on starting a merge vote within the next 2 weeks. The goal is
> >>> to
> >>> > get this in for Hadoop3 beta.
> >>> >
> >>> > We have paid close attention to ensure that once disabled Timeline
> >>> Service
> >>> > v.2 does not impact existing functionality when disabled (by
> default).
> >>> >
> >>> > At a high level, following are the key features that have been
> >>> implemented
> >>> > since 3.0.0-alpha1:
> >>> > - Security (via Kerberos Authentication & delegation tokens) at the
> >>> writer
> >>> > [YARN-3053]
> >>> > - Timeline server usability improvements [timeline-server
> >>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> >>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> >>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> > ]
> >>> > - HBase specific improvements [atsv2-hbase
> >>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
> >>> ed%20DESC%2C%
> >>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> >>> > ]
> >>> > - REST API additions & improvements [timeline-reader
> >>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> >>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> >>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> >>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
> >>> ty%20DESC%2C%
> >>> > 20created%20ASC>
> >>> > ]
> >>> > - Reader side simple authorization via whitelist [YARN-6820]
> >>> >
> >>> > We would love to get your thoughts on these and more before we open a
> >>> real
> >>> > voting thread.
> >>> >
> >>> > Special thanks to a team of folks who worked hard and contributed
> >>> towards
> >>> > this effort with patches, reviews and guidance: Rohith Sharma K S,
> >>> Varun
> >>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>> >
> >>> > Thanks!
> >>> > Vrushali
> >>> >
> >>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> >>> > apache.org/msg23897.html
> >>> > [2] feature branch YARN-5355 commits: https://github.com/
> >>> > apache/hadoop/commits/YARN-5355
> >>> >
> >>>
> >>
> >>
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Andrew Wang <an...@cloudera.com>.
Great, thanks Vrushali! Sounds good to me.
I have a few procedural release notes comments I'll put on YARN-5355, to
make sure we advertise this to our users appropriately.
On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
vrushali.c@gmail.com> wrote:
> Hi Andrew,
>
> Thanks for your response!
>
> There have been no changes to existing APIs since alpha1.
>
> We at Twitter have tested the feature to demonstrate it works at what we
> consider moderate scale but this did not include the security related
> testing. The security testing is in progress at present by Timeline Service
> V2 team in the community and we think we will have more details on this
> very soon.
>
> About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> think of as "merge-blockers". The issues being targeted for merge are in
> [link1] below. There are about 59 jiras of which 56 are completed.
>
> We plan to make a new umbrella jira after the merge to trunk. We will then
> create a new branch with the new jira name and move these open jiras under
> YARN-5355 as subtasks of that new umbrella jira.
>
> thanks
> Vrushali
> [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
>
>
> On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
>> Hi Vrushali,
>>
>> Glad to hear this major dev milestone is nearing completion!
>>
>> Repeating my request on other merge [DISCUSS] threads, could you comment
>> on testing and API stability of this merge? Our timeline for beta1 is about
>> a month out, so there's not much time to fix things beforehand.
>>
>> Looking at YARN-5355 there are also many unresolved subtasks. Should most
>> of these be moved out to a new umbrella? I'm wondering what needs to be
>> completed before sending the merge vote.
>>
>> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
>> beta1 release date for this feature than others. Hopefully that won't be
>> necessary though :)
>>
>> Best,
>> Andrew
>>
>> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
>> vrushalic2016@gmail.com> wrote:
>>
>>> Looks like some of the hyperlinks appear messed up, my apologies,
>>> resending
>>> the same email with hopefully better looking content:
>>>
>>> Hi All,
>>>
>>> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
>>> to trunk in a few weeks.
>>>
>>> We have previously completed one merge onto trunk [1] and Timeline
>>> Service
>>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>
>>> Since then, we have been working on extending the capabilities of
>>> Timeline
>>> Service v2 in a feature branch [2]. There are a few related issues
>>> pending
>>> that are being actively worked upon and tested. As soon as they are
>>> resolved, we plan on starting a merge vote within the next two weeks. The
>>> goal is to get this into hadoop3 beta.
>>>
>>> We have paid close attention to ensure that once disabled Timeline
>>> Service
>>> v2 does not impact existing functionality when disabled (by default).
>>>
>>> At a high level, following are the key features that have been
>>> implemented
>>> since 3.0.0-alpha1:
>>> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
>>> - Timeline server usability improvements [timeline-server
>>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
>>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
>>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
>>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
>>> 2C%20created%20ASC>
>>> ]
>>> - HBase specific improvements [atsv2-hbase
>>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
>>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
>>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
>>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>>> ]
>>> - REST API additions and improvements [timeline-reader
>>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
>>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
>>> omponent%20in%20(timelineclient%2C%20timelinereader)%
>>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>>> ]
>>> - Reader side simple authorization via whitelist [YARN-6820]
>>>
>>> We would love to get your thoughts on this before we open a real voting
>>> thread.
>>>
>>> Special thanks to a team of folks who worked hard and contributed towards
>>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
>>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>>
>>> Thanks
>>> Vrushali
>>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
>>> org/msg23897.html
>>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg23897.html>
>>> [2] feature branch YARN-5355 commits: https://github.com/ap
>>> ache/hadoop/commits/YARN-5355
>>>
>>> <https://github.com/apache/hadoop/commits/YARN-5355>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
>>> vrushali.c@gmail.com> wrote:
>>>
>>> > Hi All
>>> >
>>> > I’d like to open a discussion for merging Timeline Service v.2
>>> (YARN-5355)
>>> > to trunk in a few weeks.
>>> >
>>> > We have previously completed one merge onto trunk [1] and Timeline
>>> Service
>>> > v2 has been part of Hadoop release 3.0.0-alpha1.
>>> >
>>> > Since then, we have been working on extending the capabilities of
>>> Timeline
>>> > Service v2 in a feature branch [2]. There are a few related issues
>>> pending
>>> > that are being actively worked upon & tested. As soon as they are
>>> resolved,
>>> > we plan on starting a merge vote within the next 2 weeks. The goal is
>>> to
>>> > get this in for Hadoop3 beta.
>>> >
>>> > We have paid close attention to ensure that once disabled Timeline
>>> Service
>>> > v.2 does not impact existing functionality when disabled (by default).
>>> >
>>> > At a high level, following are the key features that have been
>>> implemented
>>> > since 3.0.0-alpha1:
>>> > - Security (via Kerberos Authentication & delegation tokens) at the
>>> writer
>>> > [YARN-3053]
>>> > - Timeline server usability improvements [timeline-server
>>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
>>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
>>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
>>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>>> > ]
>>> > - HBase specific improvements [atsv2-hbase
>>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
>>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
>>> ed%20DESC%2C%
>>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>>> > ]
>>> > - REST API additions & improvements [timeline-reader
>>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
>>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
>>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
>>> ty%20DESC%2C%
>>> > 20created%20ASC>
>>> > ]
>>> > - Reader side simple authorization via whitelist [YARN-6820]
>>> >
>>> > We would love to get your thoughts on these and more before we open a
>>> real
>>> > voting thread.
>>> >
>>> > Special thanks to a team of folks who worked hard and contributed
>>> towards
>>> > this effort with patches, reviews and guidance: Rohith Sharma K S,
>>> Varun
>>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> >
>>> > Thanks!
>>> > Vrushali
>>> >
>>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
>>> > apache.org/msg23897.html
>>> > [2] feature branch YARN-5355 commits: https://github.com/
>>> > apache/hadoop/commits/YARN-5355
>>> >
>>>
>>
>>
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Andrew Wang <an...@cloudera.com>.
Great, thanks Vrushali! Sounds good to me.
I have a few procedural release notes comments I'll put on YARN-5355, to
make sure we advertise this to our users appropriately.
On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
vrushali.c@gmail.com> wrote:
> Hi Andrew,
>
> Thanks for your response!
>
> There have been no changes to existing APIs since alpha1.
>
> We at Twitter have tested the feature to demonstrate it works at what we
> consider moderate scale but this did not include the security related
> testing. The security testing is in progress at present by Timeline Service
> V2 team in the community and we think we will have more details on this
> very soon.
>
> About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> think of as "merge-blockers". The issues being targeted for merge are in
> [link1] below. There are about 59 jiras of which 56 are completed.
>
> We plan to make a new umbrella jira after the merge to trunk. We will then
> create a new branch with the new jira name and move these open jiras under
> YARN-5355 as subtasks of that new umbrella jira.
>
> thanks
> Vrushali
> [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
>
>
> On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
>> Hi Vrushali,
>>
>> Glad to hear this major dev milestone is nearing completion!
>>
>> Repeating my request on other merge [DISCUSS] threads, could you comment
>> on testing and API stability of this merge? Our timeline for beta1 is about
>> a month out, so there's not much time to fix things beforehand.
>>
>> Looking at YARN-5355 there are also many unresolved subtasks. Should most
>> of these be moved out to a new umbrella? I'm wondering what needs to be
>> completed before sending the merge vote.
>>
>> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
>> beta1 release date for this feature than others. Hopefully that won't be
>> necessary though :)
>>
>> Best,
>> Andrew
>>
>> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
>> vrushalic2016@gmail.com> wrote:
>>
>>> Looks like some of the hyperlinks appear messed up, my apologies,
>>> resending
>>> the same email with hopefully better looking content:
>>>
>>> Hi All,
>>>
>>> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
>>> to trunk in a few weeks.
>>>
>>> We have previously completed one merge onto trunk [1] and Timeline
>>> Service
>>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>
>>> Since then, we have been working on extending the capabilities of
>>> Timeline
>>> Service v2 in a feature branch [2]. There are a few related issues
>>> pending
>>> that are being actively worked upon and tested. As soon as they are
>>> resolved, we plan on starting a merge vote within the next two weeks. The
>>> goal is to get this into hadoop3 beta.
>>>
>>> We have paid close attention to ensure that once disabled Timeline
>>> Service
>>> v2 does not impact existing functionality when disabled (by default).
>>>
>>> At a high level, following are the key features that have been
>>> implemented
>>> since 3.0.0-alpha1:
>>> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
>>> - Timeline server usability improvements [timeline-server
>>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
>>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
>>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
>>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
>>> 2C%20created%20ASC>
>>> ]
>>> - HBase specific improvements [atsv2-hbase
>>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
>>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
>>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
>>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>>> ]
>>> - REST API additions and improvements [timeline-reader
>>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
>>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
>>> omponent%20in%20(timelineclient%2C%20timelinereader)%
>>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>>> ]
>>> - Reader side simple authorization via whitelist [YARN-6820]
>>>
>>> We would love to get your thoughts on this before we open a real voting
>>> thread.
>>>
>>> Special thanks to a team of folks who worked hard and contributed towards
>>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
>>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>>
>>> Thanks
>>> Vrushali
>>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
>>> org/msg23897.html
>>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg23897.html>
>>> [2] feature branch YARN-5355 commits: https://github.com/ap
>>> ache/hadoop/commits/YARN-5355
>>>
>>> <https://github.com/apache/hadoop/commits/YARN-5355>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
>>> vrushali.c@gmail.com> wrote:
>>>
>>> > Hi All
>>> >
>>> > I’d like to open a discussion for merging Timeline Service v.2
>>> (YARN-5355)
>>> > to trunk in a few weeks.
>>> >
>>> > We have previously completed one merge onto trunk [1] and Timeline
>>> Service
>>> > v2 has been part of Hadoop release 3.0.0-alpha1.
>>> >
>>> > Since then, we have been working on extending the capabilities of
>>> Timeline
>>> > Service v2 in a feature branch [2]. There are a few related issues
>>> pending
>>> > that are being actively worked upon & tested. As soon as they are
>>> resolved,
>>> > we plan on starting a merge vote within the next 2 weeks. The goal is
>>> to
>>> > get this in for Hadoop3 beta.
>>> >
>>> > We have paid close attention to ensure that once disabled Timeline
>>> Service
>>> > v.2 does not impact existing functionality when disabled (by default).
>>> >
>>> > At a high level, following are the key features that have been
>>> implemented
>>> > since 3.0.0-alpha1:
>>> > - Security (via Kerberos Authentication & delegation tokens) at the
>>> writer
>>> > [YARN-3053]
>>> > - Timeline server usability improvements [timeline-server
>>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
>>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
>>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
>>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>>> > ]
>>> > - HBase specific improvements [atsv2-hbase
>>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
>>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
>>> ed%20DESC%2C%
>>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>>> > ]
>>> > - REST API additions & improvements [timeline-reader
>>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
>>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
>>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
>>> ty%20DESC%2C%
>>> > 20created%20ASC>
>>> > ]
>>> > - Reader side simple authorization via whitelist [YARN-6820]
>>> >
>>> > We would love to get your thoughts on these and more before we open a
>>> real
>>> > voting thread.
>>> >
>>> > Special thanks to a team of folks who worked hard and contributed
>>> towards
>>> > this effort with patches, reviews and guidance: Rohith Sharma K S,
>>> Varun
>>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> >
>>> > Thanks!
>>> > Vrushali
>>> >
>>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
>>> > apache.org/msg23897.html
>>> > [2] feature branch YARN-5355 commits: https://github.com/
>>> > apache/hadoop/commits/YARN-5355
>>> >
>>>
>>
>>
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Andrew Wang <an...@cloudera.com>.
Great, thanks Vrushali! Sounds good to me.
I have a few procedural release notes comments I'll put on YARN-5355, to
make sure we advertise this to our users appropriately.
On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
vrushali.c@gmail.com> wrote:
> Hi Andrew,
>
> Thanks for your response!
>
> There have been no changes to existing APIs since alpha1.
>
> We at Twitter have tested the feature to demonstrate it works at what we
> consider moderate scale but this did not include the security related
> testing. The security testing is in progress at present by Timeline Service
> V2 team in the community and we think we will have more details on this
> very soon.
>
> About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> think of as "merge-blockers". The issues being targeted for merge are in
> [link1] below. There are about 59 jiras of which 56 are completed.
>
> We plan to make a new umbrella jira after the merge to trunk. We will then
> create a new branch with the new jira name and move these open jiras under
> YARN-5355 as subtasks of that new umbrella jira.
>
> thanks
> Vrushali
> [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
>
>
> On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
>> Hi Vrushali,
>>
>> Glad to hear this major dev milestone is nearing completion!
>>
>> Repeating my request on other merge [DISCUSS] threads, could you comment
>> on testing and API stability of this merge? Our timeline for beta1 is about
>> a month out, so there's not much time to fix things beforehand.
>>
>> Looking at YARN-5355 there are also many unresolved subtasks. Should most
>> of these be moved out to a new umbrella? I'm wondering what needs to be
>> completed before sending the merge vote.
>>
>> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
>> beta1 release date for this feature than others. Hopefully that won't be
>> necessary though :)
>>
>> Best,
>> Andrew
>>
>> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
>> vrushalic2016@gmail.com> wrote:
>>
>>> Looks like some of the hyperlinks appear messed up, my apologies,
>>> resending
>>> the same email with hopefully better looking content:
>>>
>>> Hi All,
>>>
>>> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
>>> to trunk in a few weeks.
>>>
>>> We have previously completed one merge onto trunk [1] and Timeline
>>> Service
>>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>
>>> Since then, we have been working on extending the capabilities of
>>> Timeline
>>> Service v2 in a feature branch [2]. There are a few related issues
>>> pending
>>> that are being actively worked upon and tested. As soon as they are
>>> resolved, we plan on starting a merge vote within the next two weeks. The
>>> goal is to get this into hadoop3 beta.
>>>
>>> We have paid close attention to ensure that once disabled Timeline
>>> Service
>>> v2 does not impact existing functionality when disabled (by default).
>>>
>>> At a high level, following are the key features that have been
>>> implemented
>>> since 3.0.0-alpha1:
>>> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
>>> - Timeline server usability improvements [timeline-server
>>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
>>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
>>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
>>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
>>> 2C%20created%20ASC>
>>> ]
>>> - HBase specific improvements [atsv2-hbase
>>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
>>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
>>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
>>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>>> ]
>>> - REST API additions and improvements [timeline-reader
>>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
>>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
>>> omponent%20in%20(timelineclient%2C%20timelinereader)%
>>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>>> ]
>>> - Reader side simple authorization via whitelist [YARN-6820]
>>>
>>> We would love to get your thoughts on this before we open a real voting
>>> thread.
>>>
>>> Special thanks to a team of folks who worked hard and contributed towards
>>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
>>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>>
>>> Thanks
>>> Vrushali
>>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
>>> org/msg23897.html
>>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg23897.html>
>>> [2] feature branch YARN-5355 commits: https://github.com/ap
>>> ache/hadoop/commits/YARN-5355
>>>
>>> <https://github.com/apache/hadoop/commits/YARN-5355>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
>>> vrushali.c@gmail.com> wrote:
>>>
>>> > Hi All
>>> >
>>> > I’d like to open a discussion for merging Timeline Service v.2
>>> (YARN-5355)
>>> > to trunk in a few weeks.
>>> >
>>> > We have previously completed one merge onto trunk [1] and Timeline
>>> Service
>>> > v2 has been part of Hadoop release 3.0.0-alpha1.
>>> >
>>> > Since then, we have been working on extending the capabilities of
>>> Timeline
>>> > Service v2 in a feature branch [2]. There are a few related issues
>>> pending
>>> > that are being actively worked upon & tested. As soon as they are
>>> resolved,
>>> > we plan on starting a merge vote within the next 2 weeks. The goal is
>>> to
>>> > get this in for Hadoop3 beta.
>>> >
>>> > We have paid close attention to ensure that once disabled Timeline
>>> Service
>>> > v.2 does not impact existing functionality when disabled (by default).
>>> >
>>> > At a high level, following are the key features that have been
>>> implemented
>>> > since 3.0.0-alpha1:
>>> > - Security (via Kerberos Authentication & delegation tokens) at the
>>> writer
>>> > [YARN-3053]
>>> > - Timeline server usability improvements [timeline-server
>>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
>>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
>>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
>>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>>> > ]
>>> > - HBase specific improvements [atsv2-hbase
>>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
>>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
>>> ed%20DESC%2C%
>>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>>> > ]
>>> > - REST API additions & improvements [timeline-reader
>>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
>>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
>>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
>>> ty%20DESC%2C%
>>> > 20created%20ASC>
>>> > ]
>>> > - Reader side simple authorization via whitelist [YARN-6820]
>>> >
>>> > We would love to get your thoughts on these and more before we open a
>>> real
>>> > voting thread.
>>> >
>>> > Special thanks to a team of folks who worked hard and contributed
>>> towards
>>> > this effort with patches, reviews and guidance: Rohith Sharma K S,
>>> Varun
>>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> >
>>> > Thanks!
>>> > Vrushali
>>> >
>>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
>>> > apache.org/msg23897.html
>>> > [2] feature branch YARN-5355 commits: https://github.com/
>>> > apache/hadoop/commits/YARN-5355
>>> >
>>>
>>
>>
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Andrew Wang <an...@cloudera.com>.
Great, thanks Vrushali! Sounds good to me.
I have a few procedural release notes comments I'll put on YARN-5355, to
make sure we advertise this to our users appropriately.
On Wed, Aug 16, 2017 at 11:32 AM, Vrushali Channapattan <
vrushali.c@gmail.com> wrote:
> Hi Andrew,
>
> Thanks for your response!
>
> There have been no changes to existing APIs since alpha1.
>
> We at Twitter have tested the feature to demonstrate it works at what we
> consider moderate scale but this did not include the security related
> testing. The security testing is in progress at present by Timeline Service
> V2 team in the community and we think we will have more details on this
> very soon.
>
> About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
> think of as "merge-blockers". The issues being targeted for merge are in
> [link1] below. There are about 59 jiras of which 56 are completed.
>
> We plan to make a new umbrella jira after the merge to trunk. We will then
> create a new branch with the new jira name and move these open jiras under
> YARN-5355 as subtasks of that new umbrella jira.
>
> thanks
> Vrushali
> [link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
>
>
> On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <an...@cloudera.com>
> wrote:
>
>> Hi Vrushali,
>>
>> Glad to hear this major dev milestone is nearing completion!
>>
>> Repeating my request on other merge [DISCUSS] threads, could you comment
>> on testing and API stability of this merge? Our timeline for beta1 is about
>> a month out, so there's not much time to fix things beforehand.
>>
>> Looking at YARN-5355 there are also many unresolved subtasks. Should most
>> of these be moved out to a new umbrella? I'm wondering what needs to be
>> completed before sending the merge vote.
>>
>> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
>> beta1 release date for this feature than others. Hopefully that won't be
>> necessary though :)
>>
>> Best,
>> Andrew
>>
>> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
>> vrushalic2016@gmail.com> wrote:
>>
>>> Looks like some of the hyperlinks appear messed up, my apologies,
>>> resending
>>> the same email with hopefully better looking content:
>>>
>>> Hi All,
>>>
>>> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
>>> to trunk in a few weeks.
>>>
>>> We have previously completed one merge onto trunk [1] and Timeline
>>> Service
>>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>
>>> Since then, we have been working on extending the capabilities of
>>> Timeline
>>> Service v2 in a feature branch [2]. There are a few related issues
>>> pending
>>> that are being actively worked upon and tested. As soon as they are
>>> resolved, we plan on starting a merge vote within the next two weeks. The
>>> goal is to get this into hadoop3 beta.
>>>
>>> We have paid close attention to ensure that once disabled Timeline
>>> Service
>>> v2 does not impact existing functionality when disabled (by default).
>>>
>>> At a high level, following are the key features that have been
>>> implemented
>>> since 3.0.0-alpha1:
>>> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
>>> - Timeline server usability improvements [timeline-server
>>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
>>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
>>> omponent%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv
>>> 2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%
>>> 2C%20created%20ASC>
>>> ]
>>> - HBase specific improvements [atsv2-hbase
>>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
>>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20l
>>> abels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
>>> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>>> ]
>>> - REST API additions and improvements [timeline-reader
>>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
>>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20c
>>> omponent%20in%20(timelineclient%2C%20timelinereader)%
>>> 20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>>> ]
>>> - Reader side simple authorization via whitelist [YARN-6820]
>>>
>>> We would love to get your thoughts on this before we open a real voting
>>> thread.
>>>
>>> Special thanks to a team of folks who worked hard and contributed towards
>>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
>>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>>
>>> Thanks
>>> Vrushali
>>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
>>> org/msg23897.html
>>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg23897.html>
>>> [2] feature branch YARN-5355 commits: https://github.com/ap
>>> ache/hadoop/commits/YARN-5355
>>>
>>> <https://github.com/apache/hadoop/commits/YARN-5355>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
>>> vrushali.c@gmail.com> wrote:
>>>
>>> > Hi All
>>> >
>>> > I’d like to open a discussion for merging Timeline Service v.2
>>> (YARN-5355)
>>> > to trunk in a few weeks.
>>> >
>>> > We have previously completed one merge onto trunk [1] and Timeline
>>> Service
>>> > v2 has been part of Hadoop release 3.0.0-alpha1.
>>> >
>>> > Since then, we have been working on extending the capabilities of
>>> Timeline
>>> > Service v2 in a feature branch [2]. There are a few related issues
>>> pending
>>> > that are being actively worked upon & tested. As soon as they are
>>> resolved,
>>> > we plan on starting a merge vote within the next 2 weeks. The goal is
>>> to
>>> > get this in for Hadoop3 beta.
>>> >
>>> > We have paid close attention to ensure that once disabled Timeline
>>> Service
>>> > v.2 does not impact existing functionality when disabled (by default).
>>> >
>>> > At a high level, following are the key features that have been
>>> implemented
>>> > since 3.0.0-alpha1:
>>> > - Security (via Kerberos Authentication & delegation tokens) at the
>>> writer
>>> > [YARN-3053]
>>> > - Timeline server usability improvements [timeline-server
>>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
>>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
>>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
>>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>>> > ]
>>> > - HBase specific improvements [atsv2-hbase
>>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
>>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
>>> ed%20DESC%2C%
>>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>>> > ]
>>> > - REST API additions & improvements [timeline-reader
>>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
>>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
>>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
>>> ty%20DESC%2C%
>>> > 20created%20ASC>
>>> > ]
>>> > - Reader side simple authorization via whitelist [YARN-6820]
>>> >
>>> > We would love to get your thoughts on these and more before we open a
>>> real
>>> > voting thread.
>>> >
>>> > Special thanks to a team of folks who worked hard and contributed
>>> towards
>>> > this effort with patches, reviews and guidance: Rohith Sharma K S,
>>> Varun
>>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> >
>>> > Thanks!
>>> > Vrushali
>>> >
>>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
>>> > apache.org/msg23897.html
>>> > [2] feature branch YARN-5355 commits: https://github.com/
>>> > apache/hadoop/commits/YARN-5355
>>> >
>>>
>>
>>
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Vrushali Channapattan <vr...@gmail.com>.
Hi Andrew,
Thanks for your response!
There have been no changes to existing APIs since alpha1.
We at Twitter have tested the feature to demonstrate it works at what we
consider moderate scale but this did not include the security related
testing. The security testing is in progress at present by Timeline Service
V2 team in the community and we think we will have more details on this
very soon.
About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
think of as "merge-blockers". The issues being targeted for merge are in
[link1] below. There are about 59 jiras of which 56 are completed.
We plan to make a new umbrella jira after the merge to trunk. We will then
create a new branch with the new jira name and move these open jiras under
YARN-5355 as subtasks of that new umbrella jira.
thanks
Vrushali
[link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <an...@cloudera.com>
wrote:
> Hi Vrushali,
>
> Glad to hear this major dev milestone is nearing completion!
>
> Repeating my request on other merge [DISCUSS] threads, could you comment
> on testing and API stability of this merge? Our timeline for beta1 is about
> a month out, so there's not much time to fix things beforehand.
>
> Looking at YARN-5355 there are also many unresolved subtasks. Should most
> of these be moved out to a new umbrella? I'm wondering what needs to be
> completed before sending the merge vote.
>
> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
> beta1 release date for this feature than others. Hopefully that won't be
> necessary though :)
>
> Best,
> Andrew
>
> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
>> Looks like some of the hyperlinks appear messed up, my apologies,
>> resending
>> the same email with hopefully better looking content:
>>
>> Hi All,
>>
>> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
>> to trunk in a few weeks.
>>
>> We have previously completed one merge onto trunk [1] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2]. There are a few related issues
>> pending
>> that are being actively worked upon and tested. As soon as they are
>> resolved, we plan on starting a merge vote within the next two weeks. The
>> goal is to get this into hadoop3 beta.
>>
>> We have paid close attention to ensure that once disabled Timeline
>> Service
>> v2 does not impact existing functionality when disabled (by default).
>>
>> At a high level, following are the key features that have been implemented
>> since 3.0.0-alpha1:
>> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
>> - Timeline server usability improvements [timeline-server
>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20component%20%3D%20timelineserver%20AND%20labels%20!%3D%
>> 20atsv2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%
>> 20DESC%2C%20created%20ASC>
>> ]
>> - HBase specific improvements [atsv2-hbase
>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%
>> 2C%20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>> ]
>> - REST API additions and improvements [timeline-reader
>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20component%20in%20(timelineclient%2C%20timelinere
>> ader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
>> 20created%20ASC>
>> ]
>> - Reader side simple authorization via whitelist [YARN-6820]
>>
>> We would love to get your thoughts on this before we open a real voting
>> thread.
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Thanks
>> Vrushali
>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
>> org/msg23897.html
>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg23897.html>
>> [2] feature branch YARN-5355 commits: https://github.com/ap
>> ache/hadoop/commits/YARN-5355
>>
>> <https://github.com/apache/hadoop/commits/YARN-5355>
>>
>>
>>
>>
>>
>>
>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
>> vrushali.c@gmail.com> wrote:
>>
>> > Hi All
>> >
>> > I’d like to open a discussion for merging Timeline Service v.2
>> (YARN-5355)
>> > to trunk in a few weeks.
>> >
>> > We have previously completed one merge onto trunk [1] and Timeline
>> Service
>> > v2 has been part of Hadoop release 3.0.0-alpha1.
>> >
>> > Since then, we have been working on extending the capabilities of
>> Timeline
>> > Service v2 in a feature branch [2]. There are a few related issues
>> pending
>> > that are being actively worked upon & tested. As soon as they are
>> resolved,
>> > we plan on starting a merge vote within the next 2 weeks. The goal is to
>> > get this in for Hadoop3 beta.
>> >
>> > We have paid close attention to ensure that once disabled Timeline
>> Service
>> > v.2 does not impact existing functionality when disabled (by default).
>> >
>> > At a high level, following are the key features that have been
>> implemented
>> > since 3.0.0-alpha1:
>> > - Security (via Kerberos Authentication & delegation tokens) at the
>> writer
>> > [YARN-3053]
>> > - Timeline server usability improvements [timeline-server
>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>> > ]
>> > - HBase specific improvements [atsv2-hbase
>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
>> ed%20DESC%2C%
>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>> > ]
>> > - REST API additions & improvements [timeline-reader
>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
>> ty%20DESC%2C%
>> > 20created%20ASC>
>> > ]
>> > - Reader side simple authorization via whitelist [YARN-6820]
>> >
>> > We would love to get your thoughts on these and more before we open a
>> real
>> > voting thread.
>> >
>> > Special thanks to a team of folks who worked hard and contributed
>> towards
>> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> >
>> > Thanks!
>> > Vrushali
>> >
>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
>> > apache.org/msg23897.html
>> > [2] feature branch YARN-5355 commits: https://github.com/
>> > apache/hadoop/commits/YARN-5355
>> >
>>
>
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Vrushali Channapattan <vr...@gmail.com>.
Hi Andrew,
Thanks for your response!
There have been no changes to existing APIs since alpha1.
We at Twitter have tested the feature to demonstrate it works at what we
consider moderate scale but this did not include the security related
testing. The security testing is in progress at present by Timeline Service
V2 team in the community and we think we will have more details on this
very soon.
About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
think of as "merge-blockers". The issues being targeted for merge are in
[link1] below. There are about 59 jiras of which 56 are completed.
We plan to make a new umbrella jira after the merge to trunk. We will then
create a new branch with the new jira name and move these open jiras under
YARN-5355 as subtasks of that new umbrella jira.
thanks
Vrushali
[link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <an...@cloudera.com>
wrote:
> Hi Vrushali,
>
> Glad to hear this major dev milestone is nearing completion!
>
> Repeating my request on other merge [DISCUSS] threads, could you comment
> on testing and API stability of this merge? Our timeline for beta1 is about
> a month out, so there's not much time to fix things beforehand.
>
> Looking at YARN-5355 there are also many unresolved subtasks. Should most
> of these be moved out to a new umbrella? I'm wondering what needs to be
> completed before sending the merge vote.
>
> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
> beta1 release date for this feature than others. Hopefully that won't be
> necessary though :)
>
> Best,
> Andrew
>
> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
>> Looks like some of the hyperlinks appear messed up, my apologies,
>> resending
>> the same email with hopefully better looking content:
>>
>> Hi All,
>>
>> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
>> to trunk in a few weeks.
>>
>> We have previously completed one merge onto trunk [1] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2]. There are a few related issues
>> pending
>> that are being actively worked upon and tested. As soon as they are
>> resolved, we plan on starting a merge vote within the next two weeks. The
>> goal is to get this into hadoop3 beta.
>>
>> We have paid close attention to ensure that once disabled Timeline
>> Service
>> v2 does not impact existing functionality when disabled (by default).
>>
>> At a high level, following are the key features that have been implemented
>> since 3.0.0-alpha1:
>> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
>> - Timeline server usability improvements [timeline-server
>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20component%20%3D%20timelineserver%20AND%20labels%20!%3D%
>> 20atsv2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%
>> 20DESC%2C%20created%20ASC>
>> ]
>> - HBase specific improvements [atsv2-hbase
>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%
>> 2C%20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>> ]
>> - REST API additions and improvements [timeline-reader
>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20component%20in%20(timelineclient%2C%20timelinere
>> ader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
>> 20created%20ASC>
>> ]
>> - Reader side simple authorization via whitelist [YARN-6820]
>>
>> We would love to get your thoughts on this before we open a real voting
>> thread.
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Thanks
>> Vrushali
>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
>> org/msg23897.html
>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg23897.html>
>> [2] feature branch YARN-5355 commits: https://github.com/ap
>> ache/hadoop/commits/YARN-5355
>>
>> <https://github.com/apache/hadoop/commits/YARN-5355>
>>
>>
>>
>>
>>
>>
>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
>> vrushali.c@gmail.com> wrote:
>>
>> > Hi All
>> >
>> > I’d like to open a discussion for merging Timeline Service v.2
>> (YARN-5355)
>> > to trunk in a few weeks.
>> >
>> > We have previously completed one merge onto trunk [1] and Timeline
>> Service
>> > v2 has been part of Hadoop release 3.0.0-alpha1.
>> >
>> > Since then, we have been working on extending the capabilities of
>> Timeline
>> > Service v2 in a feature branch [2]. There are a few related issues
>> pending
>> > that are being actively worked upon & tested. As soon as they are
>> resolved,
>> > we plan on starting a merge vote within the next 2 weeks. The goal is to
>> > get this in for Hadoop3 beta.
>> >
>> > We have paid close attention to ensure that once disabled Timeline
>> Service
>> > v.2 does not impact existing functionality when disabled (by default).
>> >
>> > At a high level, following are the key features that have been
>> implemented
>> > since 3.0.0-alpha1:
>> > - Security (via Kerberos Authentication & delegation tokens) at the
>> writer
>> > [YARN-3053]
>> > - Timeline server usability improvements [timeline-server
>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>> > ]
>> > - HBase specific improvements [atsv2-hbase
>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
>> ed%20DESC%2C%
>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>> > ]
>> > - REST API additions & improvements [timeline-reader
>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
>> ty%20DESC%2C%
>> > 20created%20ASC>
>> > ]
>> > - Reader side simple authorization via whitelist [YARN-6820]
>> >
>> > We would love to get your thoughts on these and more before we open a
>> real
>> > voting thread.
>> >
>> > Special thanks to a team of folks who worked hard and contributed
>> towards
>> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> >
>> > Thanks!
>> > Vrushali
>> >
>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
>> > apache.org/msg23897.html
>> > [2] feature branch YARN-5355 commits: https://github.com/
>> > apache/hadoop/commits/YARN-5355
>> >
>>
>
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Vrushali Channapattan <vr...@gmail.com>.
Hi Andrew,
Thanks for your response!
There have been no changes to existing APIs since alpha1.
We at Twitter have tested the feature to demonstrate it works at what we
consider moderate scale but this did not include the security related
testing. The security testing is in progress at present by Timeline Service
V2 team in the community and we think we will have more details on this
very soon.
About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
think of as "merge-blockers". The issues being targeted for merge are in
[link1] below. There are about 59 jiras of which 56 are completed.
We plan to make a new umbrella jira after the merge to trunk. We will then
create a new branch with the new jira name and move these open jiras under
YARN-5355 as subtasks of that new umbrella jira.
thanks
Vrushali
[link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <an...@cloudera.com>
wrote:
> Hi Vrushali,
>
> Glad to hear this major dev milestone is nearing completion!
>
> Repeating my request on other merge [DISCUSS] threads, could you comment
> on testing and API stability of this merge? Our timeline for beta1 is about
> a month out, so there's not much time to fix things beforehand.
>
> Looking at YARN-5355 there are also many unresolved subtasks. Should most
> of these be moved out to a new umbrella? I'm wondering what needs to be
> completed before sending the merge vote.
>
> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
> beta1 release date for this feature than others. Hopefully that won't be
> necessary though :)
>
> Best,
> Andrew
>
> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
>> Looks like some of the hyperlinks appear messed up, my apologies,
>> resending
>> the same email with hopefully better looking content:
>>
>> Hi All,
>>
>> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
>> to trunk in a few weeks.
>>
>> We have previously completed one merge onto trunk [1] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2]. There are a few related issues
>> pending
>> that are being actively worked upon and tested. As soon as they are
>> resolved, we plan on starting a merge vote within the next two weeks. The
>> goal is to get this into hadoop3 beta.
>>
>> We have paid close attention to ensure that once disabled Timeline
>> Service
>> v2 does not impact existing functionality when disabled (by default).
>>
>> At a high level, following are the key features that have been implemented
>> since 3.0.0-alpha1:
>> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
>> - Timeline server usability improvements [timeline-server
>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20component%20%3D%20timelineserver%20AND%20labels%20!%3D%
>> 20atsv2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%
>> 20DESC%2C%20created%20ASC>
>> ]
>> - HBase specific improvements [atsv2-hbase
>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%
>> 2C%20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>> ]
>> - REST API additions and improvements [timeline-reader
>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20component%20in%20(timelineclient%2C%20timelinere
>> ader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
>> 20created%20ASC>
>> ]
>> - Reader side simple authorization via whitelist [YARN-6820]
>>
>> We would love to get your thoughts on this before we open a real voting
>> thread.
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Thanks
>> Vrushali
>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
>> org/msg23897.html
>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg23897.html>
>> [2] feature branch YARN-5355 commits: https://github.com/ap
>> ache/hadoop/commits/YARN-5355
>>
>> <https://github.com/apache/hadoop/commits/YARN-5355>
>>
>>
>>
>>
>>
>>
>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
>> vrushali.c@gmail.com> wrote:
>>
>> > Hi All
>> >
>> > I’d like to open a discussion for merging Timeline Service v.2
>> (YARN-5355)
>> > to trunk in a few weeks.
>> >
>> > We have previously completed one merge onto trunk [1] and Timeline
>> Service
>> > v2 has been part of Hadoop release 3.0.0-alpha1.
>> >
>> > Since then, we have been working on extending the capabilities of
>> Timeline
>> > Service v2 in a feature branch [2]. There are a few related issues
>> pending
>> > that are being actively worked upon & tested. As soon as they are
>> resolved,
>> > we plan on starting a merge vote within the next 2 weeks. The goal is to
>> > get this in for Hadoop3 beta.
>> >
>> > We have paid close attention to ensure that once disabled Timeline
>> Service
>> > v.2 does not impact existing functionality when disabled (by default).
>> >
>> > At a high level, following are the key features that have been
>> implemented
>> > since 3.0.0-alpha1:
>> > - Security (via Kerberos Authentication & delegation tokens) at the
>> writer
>> > [YARN-3053]
>> > - Timeline server usability improvements [timeline-server
>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>> > ]
>> > - HBase specific improvements [atsv2-hbase
>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
>> ed%20DESC%2C%
>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>> > ]
>> > - REST API additions & improvements [timeline-reader
>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
>> ty%20DESC%2C%
>> > 20created%20ASC>
>> > ]
>> > - Reader side simple authorization via whitelist [YARN-6820]
>> >
>> > We would love to get your thoughts on these and more before we open a
>> real
>> > voting thread.
>> >
>> > Special thanks to a team of folks who worked hard and contributed
>> towards
>> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> >
>> > Thanks!
>> > Vrushali
>> >
>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
>> > apache.org/msg23897.html
>> > [2] feature branch YARN-5355 commits: https://github.com/
>> > apache/hadoop/commits/YARN-5355
>> >
>>
>
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Vrushali Channapattan <vr...@gmail.com>.
Hi Andrew,
Thanks for your response!
There have been no changes to existing APIs since alpha1.
We at Twitter have tested the feature to demonstrate it works at what we
consider moderate scale but this did not include the security related
testing. The security testing is in progress at present by Timeline Service
V2 team in the community and we think we will have more details on this
very soon.
About the jiras under YARN-5355: Only 3 of those sub-tasks are what we
think of as "merge-blockers". The issues being targeted for merge are in
[link1] below. There are about 59 jiras of which 56 are completed.
We plan to make a new umbrella jira after the merge to trunk. We will then
create a new branch with the new jira name and move these open jiras under
YARN-5355 as subtasks of that new umbrella jira.
thanks
Vrushali
[link1] https://issues.apache.org/jira/projects/YARN/versions/12337991
On Wed, Aug 16, 2017 at 10:47 AM, Andrew Wang <an...@cloudera.com>
wrote:
> Hi Vrushali,
>
> Glad to hear this major dev milestone is nearing completion!
>
> Repeating my request on other merge [DISCUSS] threads, could you comment
> on testing and API stability of this merge? Our timeline for beta1 is about
> a month out, so there's not much time to fix things beforehand.
>
> Looking at YARN-5355 there are also many unresolved subtasks. Should most
> of these be moved out to a new umbrella? I'm wondering what needs to be
> completed before sending the merge vote.
>
> Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
> beta1 release date for this feature than others. Hopefully that won't be
> necessary though :)
>
> Best,
> Andrew
>
> On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
>> Looks like some of the hyperlinks appear messed up, my apologies,
>> resending
>> the same email with hopefully better looking content:
>>
>> Hi All,
>>
>> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
>> to trunk in a few weeks.
>>
>> We have previously completed one merge onto trunk [1] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2]. There are a few related issues
>> pending
>> that are being actively worked upon and tested. As soon as they are
>> resolved, we plan on starting a merge vote within the next two weeks. The
>> goal is to get this into hadoop3 beta.
>>
>> We have paid close attention to ensure that once disabled Timeline
>> Service
>> v2 does not impact existing functionality when disabled (by default).
>>
>> At a high level, following are the key features that have been implemented
>> since 3.0.0-alpha1:
>> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
>> - Timeline server usability improvements [timeline-server
>> <https://issues.apache.org/jira/browse/YARN-5715?jql=project
>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20component%20%3D%20timelineserver%20AND%20labels%20!%3D%
>> 20atsv2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%
>> 20DESC%2C%20created%20ASC>
>> ]
>> - HBase specific improvements [atsv2-hbase
>> <https://issues.apache.org/jira/browse/YARN-6604?jql=project
>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%
>> 2C%20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>> ]
>> - REST API additions and improvements [timeline-reader
>> <https://issues.apache.org/jira/browse/YARN-5739?jql=project
>> %20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%
>> 20component%20in%20(timelineclient%2C%20timelinere
>> ader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
>> 20created%20ASC>
>> ]
>> - Reader side simple authorization via whitelist [YARN-6820]
>>
>> We would love to get your thoughts on this before we open a real voting
>> thread.
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Thanks
>> Vrushali
>> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
>> org/msg23897.html
>> <http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg23897.html>
>> [2] feature branch YARN-5355 commits: https://github.com/ap
>> ache/hadoop/commits/YARN-5355
>>
>> <https://github.com/apache/hadoop/commits/YARN-5355>
>>
>>
>>
>>
>>
>>
>> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
>> vrushali.c@gmail.com> wrote:
>>
>> > Hi All
>> >
>> > I’d like to open a discussion for merging Timeline Service v.2
>> (YARN-5355)
>> > to trunk in a few weeks.
>> >
>> > We have previously completed one merge onto trunk [1] and Timeline
>> Service
>> > v2 has been part of Hadoop release 3.0.0-alpha1.
>> >
>> > Since then, we have been working on extending the capabilities of
>> Timeline
>> > Service v2 in a feature branch [2]. There are a few related issues
>> pending
>> > that are being actively worked upon & tested. As soon as they are
>> resolved,
>> > we plan on starting a merge vote within the next 2 weeks. The goal is to
>> > get this in for Hadoop3 beta.
>> >
>> > We have paid close attention to ensure that once disabled Timeline
>> Service
>> > v.2 does not impact existing functionality when disabled (by default).
>> >
>> > At a high level, following are the key features that have been
>> implemented
>> > since 3.0.0-alpha1:
>> > - Security (via Kerberos Authentication & delegation tokens) at the
>> writer
>> > [YARN-3053]
>> > - Timeline server usability improvements [timeline-server
>> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
>> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
>> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
>> > ]
>> > - HBase specific improvements [atsv2-hbase
>> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updat
>> ed%20DESC%2C%
>> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
>> > ]
>> > - REST API additions & improvements [timeline-reader
>> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
>> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
>> > 5355%20AND%20component%20in%20%28timelineclient%2C%
>> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priori
>> ty%20DESC%2C%
>> > 20created%20ASC>
>> > ]
>> > - Reader side simple authorization via whitelist [YARN-6820]
>> >
>> > We would love to get your thoughts on these and more before we open a
>> real
>> > voting thread.
>> >
>> > Special thanks to a team of folks who worked hard and contributed
>> towards
>> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> >
>> > Thanks!
>> > Vrushali
>> >
>> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
>> > apache.org/msg23897.html
>> > [2] feature branch YARN-5355 commits: https://github.com/
>> > apache/hadoop/commits/YARN-5355
>> >
>>
>
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Andrew Wang <an...@cloudera.com>.
Hi Vrushali,
Glad to hear this major dev milestone is nearing completion!
Repeating my request on other merge [DISCUSS] threads, could you comment on
testing and API stability of this merge? Our timeline for beta1 is about a
month out, so there's not much time to fix things beforehand.
Looking at YARN-5355 there are also many unresolved subtasks. Should most
of these be moved out to a new umbrella? I'm wondering what needs to be
completed before sending the merge vote.
Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
beta1 release date for this feature than others. Hopefully that won't be
necessary though :)
Best,
Andrew
On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Looks like some of the hyperlinks appear messed up, my apologies, resending
> the same email with hopefully better looking content:
>
> Hi All,
>
> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
> to trunk in a few weeks.
>
> We have previously completed one merge onto trunk [1] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2]. There are a few related issues pending
> that are being actively worked upon and tested. As soon as they are
> resolved, we plan on starting a merge vote within the next two weeks. The
> goal is to get this into hadoop3 beta.
>
> We have paid close attention to ensure that once disabled Timeline Service
> v2 does not impact existing functionality when disabled (by default).
>
> At a high level, following are the key features that have been implemented
> since 3.0.0-alpha1:
> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
> - Timeline server usability improvements [timeline-server
> <https://issues.apache.org/jira/browse/YARN-5715?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20%3D%20timelineserver%20AND%
> 20labels%20!%3D%20atsv2-hbase%20ORDER%20BY%20updated%20ASC%
> 2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - HBase specific improvements [atsv2-hbase
> <https://issues.apache.org/jira/browse/YARN-6604?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - REST API additions and improvements [timeline-reader
> <https://issues.apache.org/jira/browse/YARN-5739?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20in%20(timelineclient%2C%
> 20timelinereader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> ]
> - Reader side simple authorization via whitelist [YARN-6820]
>
> We would love to get your thoughts on this before we open a real voting
> thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Thanks
> Vrushali
> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
> org/msg23897.html
> [2] feature branch YARN-5355 commits: https://github.com/ap
> ache/hadoop/commits/YARN-5355
> <https://github.com/apache/hadoop/commits/YARN-5355>
>
>
>
>
>
>
> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> vrushali.c@gmail.com> wrote:
>
> > Hi All
> >
> > I’d like to open a discussion for merging Timeline Service v.2
> (YARN-5355)
> > to trunk in a few weeks.
> >
> > We have previously completed one merge onto trunk [1] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2]. There are a few related issues
> pending
> > that are being actively worked upon & tested. As soon as they are
> resolved,
> > we plan on starting a merge vote within the next 2 weeks. The goal is to
> > get this in for Hadoop3 beta.
> >
> > We have paid close attention to ensure that once disabled Timeline
> Service
> > v.2 does not impact existing functionality when disabled (by default).
> >
> > At a high level, following are the key features that have been
> implemented
> > since 3.0.0-alpha1:
> > - Security (via Kerberos Authentication & delegation tokens) at the
> writer
> > [YARN-3053]
> > - Timeline server usability improvements [timeline-server
> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > ]
> > - HBase specific improvements [atsv2-hbase
> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%
> 20updated%20DESC%2C%
> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > ]
> > - REST API additions & improvements [timeline-reader
> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%
> 20priority%20DESC%2C%
> > 20created%20ASC>
> > ]
> > - Reader side simple authorization via whitelist [YARN-6820]
> >
> > We would love to get your thoughts on these and more before we open a
> real
> > voting thread.
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Thanks!
> > Vrushali
> >
> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> > apache.org/msg23897.html
> > [2] feature branch YARN-5355 commits: https://github.com/
> > apache/hadoop/commits/YARN-5355
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Andrew Wang <an...@cloudera.com>.
Hi Vrushali,
Glad to hear this major dev milestone is nearing completion!
Repeating my request on other merge [DISCUSS] threads, could you comment on
testing and API stability of this merge? Our timeline for beta1 is about a
month out, so there's not much time to fix things beforehand.
Looking at YARN-5355 there are also many unresolved subtasks. Should most
of these be moved out to a new umbrella? I'm wondering what needs to be
completed before sending the merge vote.
Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
beta1 release date for this feature than others. Hopefully that won't be
necessary though :)
Best,
Andrew
On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Looks like some of the hyperlinks appear messed up, my apologies, resending
> the same email with hopefully better looking content:
>
> Hi All,
>
> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
> to trunk in a few weeks.
>
> We have previously completed one merge onto trunk [1] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2]. There are a few related issues pending
> that are being actively worked upon and tested. As soon as they are
> resolved, we plan on starting a merge vote within the next two weeks. The
> goal is to get this into hadoop3 beta.
>
> We have paid close attention to ensure that once disabled Timeline Service
> v2 does not impact existing functionality when disabled (by default).
>
> At a high level, following are the key features that have been implemented
> since 3.0.0-alpha1:
> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
> - Timeline server usability improvements [timeline-server
> <https://issues.apache.org/jira/browse/YARN-5715?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20%3D%20timelineserver%20AND%
> 20labels%20!%3D%20atsv2-hbase%20ORDER%20BY%20updated%20ASC%
> 2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - HBase specific improvements [atsv2-hbase
> <https://issues.apache.org/jira/browse/YARN-6604?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - REST API additions and improvements [timeline-reader
> <https://issues.apache.org/jira/browse/YARN-5739?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20in%20(timelineclient%2C%
> 20timelinereader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> ]
> - Reader side simple authorization via whitelist [YARN-6820]
>
> We would love to get your thoughts on this before we open a real voting
> thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Thanks
> Vrushali
> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
> org/msg23897.html
> [2] feature branch YARN-5355 commits: https://github.com/ap
> ache/hadoop/commits/YARN-5355
> <https://github.com/apache/hadoop/commits/YARN-5355>
>
>
>
>
>
>
> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> vrushali.c@gmail.com> wrote:
>
> > Hi All
> >
> > I’d like to open a discussion for merging Timeline Service v.2
> (YARN-5355)
> > to trunk in a few weeks.
> >
> > We have previously completed one merge onto trunk [1] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2]. There are a few related issues
> pending
> > that are being actively worked upon & tested. As soon as they are
> resolved,
> > we plan on starting a merge vote within the next 2 weeks. The goal is to
> > get this in for Hadoop3 beta.
> >
> > We have paid close attention to ensure that once disabled Timeline
> Service
> > v.2 does not impact existing functionality when disabled (by default).
> >
> > At a high level, following are the key features that have been
> implemented
> > since 3.0.0-alpha1:
> > - Security (via Kerberos Authentication & delegation tokens) at the
> writer
> > [YARN-3053]
> > - Timeline server usability improvements [timeline-server
> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > ]
> > - HBase specific improvements [atsv2-hbase
> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%
> 20updated%20DESC%2C%
> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > ]
> > - REST API additions & improvements [timeline-reader
> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%
> 20priority%20DESC%2C%
> > 20created%20ASC>
> > ]
> > - Reader side simple authorization via whitelist [YARN-6820]
> >
> > We would love to get your thoughts on these and more before we open a
> real
> > voting thread.
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Thanks!
> > Vrushali
> >
> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> > apache.org/msg23897.html
> > [2] feature branch YARN-5355 commits: https://github.com/
> > apache/hadoop/commits/YARN-5355
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Andrew Wang <an...@cloudera.com>.
Hi Vrushali,
Glad to hear this major dev milestone is nearing completion!
Repeating my request on other merge [DISCUSS] threads, could you comment on
testing and API stability of this merge? Our timeline for beta1 is about a
month out, so there's not much time to fix things beforehand.
Looking at YARN-5355 there are also many unresolved subtasks. Should most
of these be moved out to a new umbrella? I'm wondering what needs to be
completed before sending the merge vote.
Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
beta1 release date for this feature than others. Hopefully that won't be
necessary though :)
Best,
Andrew
On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Looks like some of the hyperlinks appear messed up, my apologies, resending
> the same email with hopefully better looking content:
>
> Hi All,
>
> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
> to trunk in a few weeks.
>
> We have previously completed one merge onto trunk [1] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2]. There are a few related issues pending
> that are being actively worked upon and tested. As soon as they are
> resolved, we plan on starting a merge vote within the next two weeks. The
> goal is to get this into hadoop3 beta.
>
> We have paid close attention to ensure that once disabled Timeline Service
> v2 does not impact existing functionality when disabled (by default).
>
> At a high level, following are the key features that have been implemented
> since 3.0.0-alpha1:
> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
> - Timeline server usability improvements [timeline-server
> <https://issues.apache.org/jira/browse/YARN-5715?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20%3D%20timelineserver%20AND%
> 20labels%20!%3D%20atsv2-hbase%20ORDER%20BY%20updated%20ASC%
> 2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - HBase specific improvements [atsv2-hbase
> <https://issues.apache.org/jira/browse/YARN-6604?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - REST API additions and improvements [timeline-reader
> <https://issues.apache.org/jira/browse/YARN-5739?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20in%20(timelineclient%2C%
> 20timelinereader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> ]
> - Reader side simple authorization via whitelist [YARN-6820]
>
> We would love to get your thoughts on this before we open a real voting
> thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Thanks
> Vrushali
> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
> org/msg23897.html
> [2] feature branch YARN-5355 commits: https://github.com/ap
> ache/hadoop/commits/YARN-5355
> <https://github.com/apache/hadoop/commits/YARN-5355>
>
>
>
>
>
>
> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> vrushali.c@gmail.com> wrote:
>
> > Hi All
> >
> > I’d like to open a discussion for merging Timeline Service v.2
> (YARN-5355)
> > to trunk in a few weeks.
> >
> > We have previously completed one merge onto trunk [1] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2]. There are a few related issues
> pending
> > that are being actively worked upon & tested. As soon as they are
> resolved,
> > we plan on starting a merge vote within the next 2 weeks. The goal is to
> > get this in for Hadoop3 beta.
> >
> > We have paid close attention to ensure that once disabled Timeline
> Service
> > v.2 does not impact existing functionality when disabled (by default).
> >
> > At a high level, following are the key features that have been
> implemented
> > since 3.0.0-alpha1:
> > - Security (via Kerberos Authentication & delegation tokens) at the
> writer
> > [YARN-3053]
> > - Timeline server usability improvements [timeline-server
> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > ]
> > - HBase specific improvements [atsv2-hbase
> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%
> 20updated%20DESC%2C%
> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > ]
> > - REST API additions & improvements [timeline-reader
> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%
> 20priority%20DESC%2C%
> > 20created%20ASC>
> > ]
> > - Reader side simple authorization via whitelist [YARN-6820]
> >
> > We would love to get your thoughts on these and more before we open a
> real
> > voting thread.
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Thanks!
> > Vrushali
> >
> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> > apache.org/msg23897.html
> > [2] feature branch YARN-5355 commits: https://github.com/
> > apache/hadoop/commits/YARN-5355
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Andrew Wang <an...@cloudera.com>.
Hi Vrushali,
Glad to hear this major dev milestone is nearing completion!
Repeating my request on other merge [DISCUSS] threads, could you comment on
testing and API stability of this merge? Our timeline for beta1 is about a
month out, so there's not much time to fix things beforehand.
Looking at YARN-5355 there are also many unresolved subtasks. Should most
of these be moved out to a new umbrella? I'm wondering what needs to be
completed before sending the merge vote.
Given that TSv2 is committed for 3.0.0 GA, I'm more willing to flex the
beta1 release date for this feature than others. Hopefully that won't be
necessary though :)
Best,
Andrew
On Wed, Aug 16, 2017 at 10:26 AM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Looks like some of the hyperlinks appear messed up, my apologies, resending
> the same email with hopefully better looking content:
>
> Hi All,
>
> I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
> to trunk in a few weeks.
>
> We have previously completed one merge onto trunk [1] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2]. There are a few related issues pending
> that are being actively worked upon and tested. As soon as they are
> resolved, we plan on starting a merge vote within the next two weeks. The
> goal is to get this into hadoop3 beta.
>
> We have paid close attention to ensure that once disabled Timeline Service
> v2 does not impact existing functionality when disabled (by default).
>
> At a high level, following are the key features that have been implemented
> since 3.0.0-alpha1:
> - Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
> - Timeline server usability improvements [timeline-server
> <https://issues.apache.org/jira/browse/YARN-5715?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20%3D%20timelineserver%20AND%
> 20labels%20!%3D%20atsv2-hbase%20ORDER%20BY%20updated%20ASC%
> 2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - HBase specific improvements [atsv2-hbase
> <https://issues.apache.org/jira/browse/YARN-6604?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - REST API additions and improvements [timeline-reader
> <https://issues.apache.org/jira/browse/YARN-5739?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20in%20(timelineclient%2C%
> 20timelinereader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> ]
> - Reader side simple authorization via whitelist [YARN-6820]
>
> We would love to get your thoughts on this before we open a real voting
> thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Thanks
> Vrushali
> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
> org/msg23897.html
> [2] feature branch YARN-5355 commits: https://github.com/ap
> ache/hadoop/commits/YARN-5355
> <https://github.com/apache/hadoop/commits/YARN-5355>
>
>
>
>
>
>
> On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
> vrushali.c@gmail.com> wrote:
>
> > Hi All
> >
> > I’d like to open a discussion for merging Timeline Service v.2
> (YARN-5355)
> > to trunk in a few weeks.
> >
> > We have previously completed one merge onto trunk [1] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2]. There are a few related issues
> pending
> > that are being actively worked upon & tested. As soon as they are
> resolved,
> > we plan on starting a merge vote within the next 2 weeks. The goal is to
> > get this in for Hadoop3 beta.
> >
> > We have paid close attention to ensure that once disabled Timeline
> Service
> > v.2 does not impact existing functionality when disabled (by default).
> >
> > At a high level, following are the key features that have been
> implemented
> > since 3.0.0-alpha1:
> > - Security (via Kerberos Authentication & delegation tokens) at the
> writer
> > [YARN-3053]
> > - Timeline server usability improvements [timeline-server
> > <https://issues.apache.org/jira/browse/YARN-5715?jql=
> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20component%20%3D%20timelineserver%20AND%
> > 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> > 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> > ]
> > - HBase specific improvements [atsv2-hbase
> > <https://issues.apache.org/jira/browse/YARN-6604?jql=
> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%
> 20updated%20DESC%2C%
> > 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> > ]
> > - REST API additions & improvements [timeline-reader
> > <https://issues.apache.org/jira/browse/YARN-5739?jql=
> > project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> > 5355%20AND%20component%20in%20%28timelineclient%2C%
> > 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%
> 20priority%20DESC%2C%
> > 20created%20ASC>
> > ]
> > - Reader side simple authorization via whitelist [YARN-6820]
> >
> > We would love to get your thoughts on these and more before we open a
> real
> > voting thread.
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Thanks!
> > Vrushali
> >
> > [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> > apache.org/msg23897.html
> > [2] feature branch YARN-5355 commits: https://github.com/
> > apache/hadoop/commits/YARN-5355
> >
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Vrushali Channapattan <vr...@gmail.com>.
Looks like some of the hyperlinks appear messed up, my apologies, resending
the same email with hopefully better looking content:
Hi All,
I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
to trunk in a few weeks.
We have previously completed one merge onto trunk [1] and Timeline Service
v2 has been part of Hadoop release 3.0.0-alpha1.
Since then, we have been working on extending the capabilities of Timeline
Service v2 in a feature branch [2]. There are a few related issues pending
that are being actively worked upon and tested. As soon as they are
resolved, we plan on starting a merge vote within the next two weeks. The
goal is to get this into hadoop3 beta.
We have paid close attention to ensure that once disabled Timeline Service
v2 does not impact existing functionality when disabled (by default).
At a high level, following are the key features that have been implemented
since 3.0.0-alpha1:
- Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
- Timeline server usability improvements [timeline-server
<https://issues.apache.org/jira/browse/YARN-5715?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20component%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
]
- HBase specific improvements [atsv2-hbase
<https://issues.apache.org/jira/browse/YARN-6604?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
]
- REST API additions and improvements [timeline-reader
<https://issues.apache.org/jira/browse/YARN-5739?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20component%20in%20(timelineclient%2C%20timelinereader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
]
- Reader side simple authorization via whitelist [YARN-6820]
We would love to get your thoughts on this before we open a real voting
thread.
Special thanks to a team of folks who worked hard and contributed towards
this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
Thanks
Vrushali
[1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
org/msg23897.html
[2] feature branch YARN-5355 commits: https://github.com/ap
ache/hadoop/commits/YARN-5355
<https://github.com/apache/hadoop/commits/YARN-5355>
On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
vrushali.c@gmail.com> wrote:
> Hi All
>
> I’d like to open a discussion for merging Timeline Service v.2 (YARN-5355)
> to trunk in a few weeks.
>
> We have previously completed one merge onto trunk [1] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2]. There are a few related issues pending
> that are being actively worked upon & tested. As soon as they are resolved,
> we plan on starting a merge vote within the next 2 weeks. The goal is to
> get this in for Hadoop3 beta.
>
> We have paid close attention to ensure that once disabled Timeline Service
> v.2 does not impact existing functionality when disabled (by default).
>
> At a high level, following are the key features that have been implemented
> since 3.0.0-alpha1:
> - Security (via Kerberos Authentication & delegation tokens) at the writer
> [YARN-3053]
> - Timeline server usability improvements [timeline-server
> <https://issues.apache.org/jira/browse/YARN-5715?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20%3D%20timelineserver%20AND%
> 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - HBase specific improvements [atsv2-hbase
> <https://issues.apache.org/jira/browse/YARN-6604?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - REST API additions & improvements [timeline-reader
> <https://issues.apache.org/jira/browse/YARN-5739?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20in%20%28timelineclient%2C%
> 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> ]
> - Reader side simple authorization via whitelist [YARN-6820]
>
> We would love to get your thoughts on these and more before we open a real
> voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Thanks!
> Vrushali
>
> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> apache.org/msg23897.html
> [2] feature branch YARN-5355 commits: https://github.com/
> apache/hadoop/commits/YARN-5355
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Vrushali Channapattan <vr...@gmail.com>.
Looks like some of the hyperlinks appear messed up, my apologies, resending
the same email with hopefully better looking content:
Hi All,
I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
to trunk in a few weeks.
We have previously completed one merge onto trunk [1] and Timeline Service
v2 has been part of Hadoop release 3.0.0-alpha1.
Since then, we have been working on extending the capabilities of Timeline
Service v2 in a feature branch [2]. There are a few related issues pending
that are being actively worked upon and tested. As soon as they are
resolved, we plan on starting a merge vote within the next two weeks. The
goal is to get this into hadoop3 beta.
We have paid close attention to ensure that once disabled Timeline Service
v2 does not impact existing functionality when disabled (by default).
At a high level, following are the key features that have been implemented
since 3.0.0-alpha1:
- Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
- Timeline server usability improvements [timeline-server
<https://issues.apache.org/jira/browse/YARN-5715?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20component%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
]
- HBase specific improvements [atsv2-hbase
<https://issues.apache.org/jira/browse/YARN-6604?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
]
- REST API additions and improvements [timeline-reader
<https://issues.apache.org/jira/browse/YARN-5739?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20component%20in%20(timelineclient%2C%20timelinereader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
]
- Reader side simple authorization via whitelist [YARN-6820]
We would love to get your thoughts on this before we open a real voting
thread.
Special thanks to a team of folks who worked hard and contributed towards
this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
Thanks
Vrushali
[1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
org/msg23897.html
[2] feature branch YARN-5355 commits: https://github.com/ap
ache/hadoop/commits/YARN-5355
<https://github.com/apache/hadoop/commits/YARN-5355>
On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
vrushali.c@gmail.com> wrote:
> Hi All
>
> I’d like to open a discussion for merging Timeline Service v.2 (YARN-5355)
> to trunk in a few weeks.
>
> We have previously completed one merge onto trunk [1] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2]. There are a few related issues pending
> that are being actively worked upon & tested. As soon as they are resolved,
> we plan on starting a merge vote within the next 2 weeks. The goal is to
> get this in for Hadoop3 beta.
>
> We have paid close attention to ensure that once disabled Timeline Service
> v.2 does not impact existing functionality when disabled (by default).
>
> At a high level, following are the key features that have been implemented
> since 3.0.0-alpha1:
> - Security (via Kerberos Authentication & delegation tokens) at the writer
> [YARN-3053]
> - Timeline server usability improvements [timeline-server
> <https://issues.apache.org/jira/browse/YARN-5715?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20%3D%20timelineserver%20AND%
> 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - HBase specific improvements [atsv2-hbase
> <https://issues.apache.org/jira/browse/YARN-6604?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - REST API additions & improvements [timeline-reader
> <https://issues.apache.org/jira/browse/YARN-5739?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20in%20%28timelineclient%2C%
> 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> ]
> - Reader side simple authorization via whitelist [YARN-6820]
>
> We would love to get your thoughts on these and more before we open a real
> voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Thanks!
> Vrushali
>
> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> apache.org/msg23897.html
> [2] feature branch YARN-5355 commits: https://github.com/
> apache/hadoop/commits/YARN-5355
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Vrushali Channapattan <vr...@gmail.com>.
Looks like some of the hyperlinks appear messed up, my apologies, resending
the same email with hopefully better looking content:
Hi All,
I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
to trunk in a few weeks.
We have previously completed one merge onto trunk [1] and Timeline Service
v2 has been part of Hadoop release 3.0.0-alpha1.
Since then, we have been working on extending the capabilities of Timeline
Service v2 in a feature branch [2]. There are a few related issues pending
that are being actively worked upon and tested. As soon as they are
resolved, we plan on starting a merge vote within the next two weeks. The
goal is to get this into hadoop3 beta.
We have paid close attention to ensure that once disabled Timeline Service
v2 does not impact existing functionality when disabled (by default).
At a high level, following are the key features that have been implemented
since 3.0.0-alpha1:
- Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
- Timeline server usability improvements [timeline-server
<https://issues.apache.org/jira/browse/YARN-5715?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20component%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
]
- HBase specific improvements [atsv2-hbase
<https://issues.apache.org/jira/browse/YARN-6604?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
]
- REST API additions and improvements [timeline-reader
<https://issues.apache.org/jira/browse/YARN-5739?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20component%20in%20(timelineclient%2C%20timelinereader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
]
- Reader side simple authorization via whitelist [YARN-6820]
We would love to get your thoughts on this before we open a real voting
thread.
Special thanks to a team of folks who worked hard and contributed towards
this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
Thanks
Vrushali
[1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
org/msg23897.html
[2] feature branch YARN-5355 commits: https://github.com/ap
ache/hadoop/commits/YARN-5355
<https://github.com/apache/hadoop/commits/YARN-5355>
On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
vrushali.c@gmail.com> wrote:
> Hi All
>
> I’d like to open a discussion for merging Timeline Service v.2 (YARN-5355)
> to trunk in a few weeks.
>
> We have previously completed one merge onto trunk [1] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2]. There are a few related issues pending
> that are being actively worked upon & tested. As soon as they are resolved,
> we plan on starting a merge vote within the next 2 weeks. The goal is to
> get this in for Hadoop3 beta.
>
> We have paid close attention to ensure that once disabled Timeline Service
> v.2 does not impact existing functionality when disabled (by default).
>
> At a high level, following are the key features that have been implemented
> since 3.0.0-alpha1:
> - Security (via Kerberos Authentication & delegation tokens) at the writer
> [YARN-3053]
> - Timeline server usability improvements [timeline-server
> <https://issues.apache.org/jira/browse/YARN-5715?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20%3D%20timelineserver%20AND%
> 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - HBase specific improvements [atsv2-hbase
> <https://issues.apache.org/jira/browse/YARN-6604?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - REST API additions & improvements [timeline-reader
> <https://issues.apache.org/jira/browse/YARN-5739?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20in%20%28timelineclient%2C%
> 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> ]
> - Reader side simple authorization via whitelist [YARN-6820]
>
> We would love to get your thoughts on these and more before we open a real
> voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Thanks!
> Vrushali
>
> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> apache.org/msg23897.html
> [2] feature branch YARN-5355 commits: https://github.com/
> apache/hadoop/commits/YARN-5355
>
Re: [DISCUSS] Merging YARN-5355 (Timeline Service v.2) to trunk
Posted by Vrushali Channapattan <vr...@gmail.com>.
Looks like some of the hyperlinks appear messed up, my apologies, resending
the same email with hopefully better looking content:
Hi All,
I'd like to open a discussion for merging Timeline Service v2 (YARN-5355)
to trunk in a few weeks.
We have previously completed one merge onto trunk [1] and Timeline Service
v2 has been part of Hadoop release 3.0.0-alpha1.
Since then, we have been working on extending the capabilities of Timeline
Service v2 in a feature branch [2]. There are a few related issues pending
that are being actively worked upon and tested. As soon as they are
resolved, we plan on starting a merge vote within the next two weeks. The
goal is to get this into hadoop3 beta.
We have paid close attention to ensure that once disabled Timeline Service
v2 does not impact existing functionality when disabled (by default).
At a high level, following are the key features that have been implemented
since 3.0.0-alpha1:
- Security (via Kerberos Authentication & delegation tokens) [YARN-3053]
- Timeline server usability improvements [timeline-server
<https://issues.apache.org/jira/browse/YARN-5715?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20component%20%3D%20timelineserver%20AND%20labels%20!%3D%20atsv2-hbase%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
]
- HBase specific improvements [atsv2-hbase
<https://issues.apache.org/jira/browse/YARN-6604?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
]
- REST API additions and improvements [timeline-reader
<https://issues.apache.org/jira/browse/YARN-5739?jql=project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-5355%20AND%20component%20in%20(timelineclient%2C%20timelinereader)%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%20created%20ASC>
]
- Reader side simple authorization via whitelist [YARN-6820]
We would love to get your thoughts on this before we open a real voting
thread.
Special thanks to a team of folks who worked hard and contributed towards
this effort via patches, reviews and guidance: Rohith Sharma K S, Varun
Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
Thanks
Vrushali
[1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.apache.
org/msg23897.html
[2] feature branch YARN-5355 commits: https://github.com/ap
ache/hadoop/commits/YARN-5355
<https://github.com/apache/hadoop/commits/YARN-5355>
On Wed, Aug 16, 2017 at 10:02 AM, Vrushali Channapattan <
vrushali.c@gmail.com> wrote:
> Hi All
>
> I’d like to open a discussion for merging Timeline Service v.2 (YARN-5355)
> to trunk in a few weeks.
>
> We have previously completed one merge onto trunk [1] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2]. There are a few related issues pending
> that are being actively worked upon & tested. As soon as they are resolved,
> we plan on starting a merge vote within the next 2 weeks. The goal is to
> get this in for Hadoop3 beta.
>
> We have paid close attention to ensure that once disabled Timeline Service
> v.2 does not impact existing functionality when disabled (by default).
>
> At a high level, following are the key features that have been implemented
> since 3.0.0-alpha1:
> - Security (via Kerberos Authentication & delegation tokens) at the writer
> [YARN-3053]
> - Timeline server usability improvements [timeline-server
> <https://issues.apache.org/jira/browse/YARN-5715?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20%3D%20timelineserver%20AND%
> 20labels%20%21%3D%20atsv2-hbase%20ORDER%20BY%20updated%
> 20ASC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - HBase specific improvements [atsv2-hbase
> <https://issues.apache.org/jira/browse/YARN-6604?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20labels%20%3D%20atsv2-hbase%20ORDER%20BY%20updated%20DESC%2C%
> 20affectedVersion%20DESC%2C%20priority%20DESC%2C%20created%20ASC>
> ]
> - REST API additions & improvements [timeline-reader
> <https://issues.apache.org/jira/browse/YARN-5739?jql=
> project%20%3D%20YARN%20AND%20fixVersion%20%3D%20YARN-
> 5355%20AND%20component%20in%20%28timelineclient%2C%
> 20timelinereader%29%20ORDER%20BY%20updated%20ASC%2C%20priority%20DESC%2C%
> 20created%20ASC>
> ]
> - Reader side simple authorization via whitelist [YARN-6820]
>
> We would love to get your thoughts on these and more before we open a real
> voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Thanks!
> Vrushali
>
> [1] Merge to trunk: http://www.mail-archive.com/yarn-dev@hadoop.
> apache.org/msg23897.html
> [2] feature branch YARN-5355 commits: https://github.com/
> apache/hadoop/commits/YARN-5355
>