You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-dev@hadoop.apache.org by Vrushali Channapattan <vr...@gmail.com> on 2017/08/22 06:32:49 UTC
[VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Hi folks,
Per earlier discussion [1], I'd like to start a formal vote to merge
feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
run for 7 days, and will end August 29 11:00 PM PDT.
We have previously completed one merge onto trunk [3] and Timeline Service
v2 has been part of Hadoop release 3.0.0-alpha1.
Since then, we have been working on extending the capabilities of Timeline
Service v2 in a feature branch [2] for a while, and we are reasonably
confident that the state of the feature meets the criteria to be merged
onto trunk and we'd love folks to get their hands on it in a test capacity
and provide valuable feedback so that we can make it production-ready.
In a nutshell, Timeline Service v.2 delivers significant scalability and
usability improvements based on a new architecture. What we would like to
merge to trunk is termed "alpha 2" (milestone 2). The feature has a
complete end-to-end read/write flow with security and read level
authorization via whitelists. You should be able to start setting it up and
testing it.
At a high level, the following are the key features that have been
implemented since alpha1:
- Security via Kerberos Authentication and delegation tokens
- Read side simple authorization via whitelist
- Client configurable entity sort ordering
- Richer REST APIs for apps, app attempts, containers, fetching metrics by
timerange, pagination, sub-app entities
- Support for storing sub-application entities (entities that exist outside
the scope of an application)
- Configurable TTLs (time-to-live) for tables, configurable table prefixes,
configurable hbase cluster
- Flow level aggregations done as dynamic (table level) coprocessors
- Uses latest stable HBase release 1.2.6
There are a total of 82 subtasks that were completed as part of this effort.
We paid close attention to ensure that once disabled Timeline Service v.2
does not impact existing functionality when disabled (by default).
Special thanks to a team of folks who worked hard and contributed towards
this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
Regards,
Vrushali
[1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
[2] https://issues.apache.org/jira/browse/YARN-5355
[3] https://issues.apache.org/jira/browse/YARN-2928
[4] https://github.com/apache/hadoop/commits/YARN-5355
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali Channapattan <vr...@gmail.com>.
Hi Naga
Thank you for the kind words and appreciate your vote very much. Yes, among
everyone who has worked on contributing to the TSv2 code, there has been a
strong sense of camaraderie.
Let me try to address your questions one by one.
bq. Though i have not tested, just went through the jiras and features
added.
Thank you. Testing for TSv2 as such has been done at Twitter, Hortonworks,
Huawei and Cloudera.
bq. whether there was any other key out standing features left after phase
2 and their impacts ?
Yes, we have some features that we would like to build into TSv2 and we
have included those as part of "Current Status and Future Plans" in the
documentation. Some of the future plans include:
- Clustering of the readers
- Migration and compatibility with v.1
- Timeline collectors as separate instances from node managers
- Support for off-cluster clients
- More robust storage fault tolerance
- Better support for long-running apps
- Support for ACLs
- Offline (time-based periodic) aggregation for flows, users, and queues
for reporting and analysis
bq. it would interesting to share the plans (if available) of
upstream projects
like Tez, Spark to incorporate ATSv2.
We've had an initial discussion with some of the Tez community members a
few months back. Folks from Twitter, Yahoo!, Hortonworks and Huawei were
part of the discussion. It was a productive session and we came away with a
set of things to do right away, of which we have already added in features
like support for sub-application entities and simple whitelisting of users
so that Tez community can start experimenting with it. I believe some of
the Tez Yahoo! folks are in the very early stages of exploring TSv2.
Rohith Sharma from Hortonworks has done some more groundwork and has some
advanced thoughts around the TSv2-Tez integration considerations.
We also have had an initial discussion with the Federation team (Subru,
Carlo) a while back. Accordingly, we have already updated some aspects of
our schema to accommodate things like the application to cluster mappings.
We have an open jira for Federation integration (YARN-5357) which is not
included in this version.
We have not yet reached out to Spark folks but would love to do so.
thanks
Vrushali
On Wed, Aug 23, 2017 at 5:49 PM, Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> +1 (Binding), Great work guys, unwavering dedication and team effort. Kudos
> to the whole team.
> Though i have not tested, just went through the jiras and features added.
>
> Just wanted to know whether there was any other key out standing features
> left after phase 2 and their impacts ?
> And it would interesting to share the plans (if available) of upstream
> projects like Tez, Spark to incorporate ATSv2.
>
> Regards,
> + Naga
>
>
>
> On Thu, Aug 24, 2017 at 12:28 AM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
> > Hi All,
> >
> > Just to update.
> > Folks who may be interested in going through the documentation for the
> > feature can refer to current documentation attached in pdf format, on the
> > umbrella JIRA i.e. YARN-5355.
> >
> > Furthermore, we have run the YARN-5355 patch(with all the changes made in
> > the branch) against trunk and the build is almost green barring a few
> > checkstyle issues. The test failures which come up in the build are
> > outstanding issues on trunk.
> > Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo
> > mmentId=16138266&page=com.atlassian.jira.plugin.system.
> > issuetabpanels:comment-tabpanel#comment-16138266
> >
> > Thanks,
> > Varun Saxena.
> >
> > On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > vrushalic2016@gmail.com> wrote:
> >
> > > Hi folks,
> > >
> > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > will
> > > run for 7 days, and will end August 29 11:00 PM PDT.
> > >
> > > We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >
> > > Since then, we have been working on extending the capabilities of
> > Timeline
> > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > confident that the state of the feature meets the criteria to be merged
> > > onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > > and provide valuable feedback so that we can make it production-ready.
> > >
> > > In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> > > usability improvements based on a new architecture. What we would like
> to
> > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > complete end-to-end read/write flow with security and read level
> > > authorization via whitelists. You should be able to start setting it up
> > and
> > > testing it.
> > >
> > > At a high level, the following are the key features that have been
> > > implemented since alpha1:
> > > - Security via Kerberos Authentication and delegation tokens
> > > - Read side simple authorization via whitelist
> > > - Client configurable entity sort ordering
> > > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> > by
> > > timerange, pagination, sub-app entities
> > > - Support for storing sub-application entities (entities that exist
> > outside
> > > the scope of an application)
> > > - Configurable TTLs (time-to-live) for tables, configurable table
> > prefixes,
> > > configurable hbase cluster
> > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > - Uses latest stable HBase release 1.2.6
> > >
> > > There are a total of 82 subtasks that were completed as part of this
> > > effort.
> > >
> > > We paid close attention to ensure that once disabled Timeline Service
> v.2
> > > does not impact existing functionality when disabled (by default).
> > >
> > > Special thanks to a team of folks who worked hard and contributed
> towards
> > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >
> > > Regards,
> > > Vrushali
> > >
> > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27383.html
> > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali Channapattan <vr...@gmail.com>.
Hi Naga
Thank you for the kind words and appreciate your vote very much. Yes, among
everyone who has worked on contributing to the TSv2 code, there has been a
strong sense of camaraderie.
Let me try to address your questions one by one.
bq. Though i have not tested, just went through the jiras and features
added.
Thank you. Testing for TSv2 as such has been done at Twitter, Hortonworks,
Huawei and Cloudera.
bq. whether there was any other key out standing features left after phase
2 and their impacts ?
Yes, we have some features that we would like to build into TSv2 and we
have included those as part of "Current Status and Future Plans" in the
documentation. Some of the future plans include:
- Clustering of the readers
- Migration and compatibility with v.1
- Timeline collectors as separate instances from node managers
- Support for off-cluster clients
- More robust storage fault tolerance
- Better support for long-running apps
- Support for ACLs
- Offline (time-based periodic) aggregation for flows, users, and queues
for reporting and analysis
bq. it would interesting to share the plans (if available) of
upstream projects
like Tez, Spark to incorporate ATSv2.
We've had an initial discussion with some of the Tez community members a
few months back. Folks from Twitter, Yahoo!, Hortonworks and Huawei were
part of the discussion. It was a productive session and we came away with a
set of things to do right away, of which we have already added in features
like support for sub-application entities and simple whitelisting of users
so that Tez community can start experimenting with it. I believe some of
the Tez Yahoo! folks are in the very early stages of exploring TSv2.
Rohith Sharma from Hortonworks has done some more groundwork and has some
advanced thoughts around the TSv2-Tez integration considerations.
We also have had an initial discussion with the Federation team (Subru,
Carlo) a while back. Accordingly, we have already updated some aspects of
our schema to accommodate things like the application to cluster mappings.
We have an open jira for Federation integration (YARN-5357) which is not
included in this version.
We have not yet reached out to Spark folks but would love to do so.
thanks
Vrushali
On Wed, Aug 23, 2017 at 5:49 PM, Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> +1 (Binding), Great work guys, unwavering dedication and team effort. Kudos
> to the whole team.
> Though i have not tested, just went through the jiras and features added.
>
> Just wanted to know whether there was any other key out standing features
> left after phase 2 and their impacts ?
> And it would interesting to share the plans (if available) of upstream
> projects like Tez, Spark to incorporate ATSv2.
>
> Regards,
> + Naga
>
>
>
> On Thu, Aug 24, 2017 at 12:28 AM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
> > Hi All,
> >
> > Just to update.
> > Folks who may be interested in going through the documentation for the
> > feature can refer to current documentation attached in pdf format, on the
> > umbrella JIRA i.e. YARN-5355.
> >
> > Furthermore, we have run the YARN-5355 patch(with all the changes made in
> > the branch) against trunk and the build is almost green barring a few
> > checkstyle issues. The test failures which come up in the build are
> > outstanding issues on trunk.
> > Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo
> > mmentId=16138266&page=com.atlassian.jira.plugin.system.
> > issuetabpanels:comment-tabpanel#comment-16138266
> >
> > Thanks,
> > Varun Saxena.
> >
> > On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > vrushalic2016@gmail.com> wrote:
> >
> > > Hi folks,
> > >
> > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > will
> > > run for 7 days, and will end August 29 11:00 PM PDT.
> > >
> > > We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >
> > > Since then, we have been working on extending the capabilities of
> > Timeline
> > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > confident that the state of the feature meets the criteria to be merged
> > > onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > > and provide valuable feedback so that we can make it production-ready.
> > >
> > > In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> > > usability improvements based on a new architecture. What we would like
> to
> > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > complete end-to-end read/write flow with security and read level
> > > authorization via whitelists. You should be able to start setting it up
> > and
> > > testing it.
> > >
> > > At a high level, the following are the key features that have been
> > > implemented since alpha1:
> > > - Security via Kerberos Authentication and delegation tokens
> > > - Read side simple authorization via whitelist
> > > - Client configurable entity sort ordering
> > > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> > by
> > > timerange, pagination, sub-app entities
> > > - Support for storing sub-application entities (entities that exist
> > outside
> > > the scope of an application)
> > > - Configurable TTLs (time-to-live) for tables, configurable table
> > prefixes,
> > > configurable hbase cluster
> > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > - Uses latest stable HBase release 1.2.6
> > >
> > > There are a total of 82 subtasks that were completed as part of this
> > > effort.
> > >
> > > We paid close attention to ensure that once disabled Timeline Service
> v.2
> > > does not impact existing functionality when disabled (by default).
> > >
> > > Special thanks to a team of folks who worked hard and contributed
> towards
> > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >
> > > Regards,
> > > Vrushali
> > >
> > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27383.html
> > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali Channapattan <vr...@gmail.com>.
Hi Naga
Thank you for the kind words and appreciate your vote very much. Yes, among
everyone who has worked on contributing to the TSv2 code, there has been a
strong sense of camaraderie.
Let me try to address your questions one by one.
bq. Though i have not tested, just went through the jiras and features
added.
Thank you. Testing for TSv2 as such has been done at Twitter, Hortonworks,
Huawei and Cloudera.
bq. whether there was any other key out standing features left after phase
2 and their impacts ?
Yes, we have some features that we would like to build into TSv2 and we
have included those as part of "Current Status and Future Plans" in the
documentation. Some of the future plans include:
- Clustering of the readers
- Migration and compatibility with v.1
- Timeline collectors as separate instances from node managers
- Support for off-cluster clients
- More robust storage fault tolerance
- Better support for long-running apps
- Support for ACLs
- Offline (time-based periodic) aggregation for flows, users, and queues
for reporting and analysis
bq. it would interesting to share the plans (if available) of
upstream projects
like Tez, Spark to incorporate ATSv2.
We've had an initial discussion with some of the Tez community members a
few months back. Folks from Twitter, Yahoo!, Hortonworks and Huawei were
part of the discussion. It was a productive session and we came away with a
set of things to do right away, of which we have already added in features
like support for sub-application entities and simple whitelisting of users
so that Tez community can start experimenting with it. I believe some of
the Tez Yahoo! folks are in the very early stages of exploring TSv2.
Rohith Sharma from Hortonworks has done some more groundwork and has some
advanced thoughts around the TSv2-Tez integration considerations.
We also have had an initial discussion with the Federation team (Subru,
Carlo) a while back. Accordingly, we have already updated some aspects of
our schema to accommodate things like the application to cluster mappings.
We have an open jira for Federation integration (YARN-5357) which is not
included in this version.
We have not yet reached out to Spark folks but would love to do so.
thanks
Vrushali
On Wed, Aug 23, 2017 at 5:49 PM, Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> +1 (Binding), Great work guys, unwavering dedication and team effort. Kudos
> to the whole team.
> Though i have not tested, just went through the jiras and features added.
>
> Just wanted to know whether there was any other key out standing features
> left after phase 2 and their impacts ?
> And it would interesting to share the plans (if available) of upstream
> projects like Tez, Spark to incorporate ATSv2.
>
> Regards,
> + Naga
>
>
>
> On Thu, Aug 24, 2017 at 12:28 AM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
> > Hi All,
> >
> > Just to update.
> > Folks who may be interested in going through the documentation for the
> > feature can refer to current documentation attached in pdf format, on the
> > umbrella JIRA i.e. YARN-5355.
> >
> > Furthermore, we have run the YARN-5355 patch(with all the changes made in
> > the branch) against trunk and the build is almost green barring a few
> > checkstyle issues. The test failures which come up in the build are
> > outstanding issues on trunk.
> > Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo
> > mmentId=16138266&page=com.atlassian.jira.plugin.system.
> > issuetabpanels:comment-tabpanel#comment-16138266
> >
> > Thanks,
> > Varun Saxena.
> >
> > On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > vrushalic2016@gmail.com> wrote:
> >
> > > Hi folks,
> > >
> > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > will
> > > run for 7 days, and will end August 29 11:00 PM PDT.
> > >
> > > We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >
> > > Since then, we have been working on extending the capabilities of
> > Timeline
> > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > confident that the state of the feature meets the criteria to be merged
> > > onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > > and provide valuable feedback so that we can make it production-ready.
> > >
> > > In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> > > usability improvements based on a new architecture. What we would like
> to
> > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > complete end-to-end read/write flow with security and read level
> > > authorization via whitelists. You should be able to start setting it up
> > and
> > > testing it.
> > >
> > > At a high level, the following are the key features that have been
> > > implemented since alpha1:
> > > - Security via Kerberos Authentication and delegation tokens
> > > - Read side simple authorization via whitelist
> > > - Client configurable entity sort ordering
> > > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> > by
> > > timerange, pagination, sub-app entities
> > > - Support for storing sub-application entities (entities that exist
> > outside
> > > the scope of an application)
> > > - Configurable TTLs (time-to-live) for tables, configurable table
> > prefixes,
> > > configurable hbase cluster
> > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > - Uses latest stable HBase release 1.2.6
> > >
> > > There are a total of 82 subtasks that were completed as part of this
> > > effort.
> > >
> > > We paid close attention to ensure that once disabled Timeline Service
> v.2
> > > does not impact existing functionality when disabled (by default).
> > >
> > > Special thanks to a team of folks who worked hard and contributed
> towards
> > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >
> > > Regards,
> > > Vrushali
> > >
> > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27383.html
> > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali Channapattan <vr...@gmail.com>.
Hi Naga
Thank you for the kind words and appreciate your vote very much. Yes, among
everyone who has worked on contributing to the TSv2 code, there has been a
strong sense of camaraderie.
Let me try to address your questions one by one.
bq. Though i have not tested, just went through the jiras and features
added.
Thank you. Testing for TSv2 as such has been done at Twitter, Hortonworks,
Huawei and Cloudera.
bq. whether there was any other key out standing features left after phase
2 and their impacts ?
Yes, we have some features that we would like to build into TSv2 and we
have included those as part of "Current Status and Future Plans" in the
documentation. Some of the future plans include:
- Clustering of the readers
- Migration and compatibility with v.1
- Timeline collectors as separate instances from node managers
- Support for off-cluster clients
- More robust storage fault tolerance
- Better support for long-running apps
- Support for ACLs
- Offline (time-based periodic) aggregation for flows, users, and queues
for reporting and analysis
bq. it would interesting to share the plans (if available) of
upstream projects
like Tez, Spark to incorporate ATSv2.
We've had an initial discussion with some of the Tez community members a
few months back. Folks from Twitter, Yahoo!, Hortonworks and Huawei were
part of the discussion. It was a productive session and we came away with a
set of things to do right away, of which we have already added in features
like support for sub-application entities and simple whitelisting of users
so that Tez community can start experimenting with it. I believe some of
the Tez Yahoo! folks are in the very early stages of exploring TSv2.
Rohith Sharma from Hortonworks has done some more groundwork and has some
advanced thoughts around the TSv2-Tez integration considerations.
We also have had an initial discussion with the Federation team (Subru,
Carlo) a while back. Accordingly, we have already updated some aspects of
our schema to accommodate things like the application to cluster mappings.
We have an open jira for Federation integration (YARN-5357) which is not
included in this version.
We have not yet reached out to Spark folks but would love to do so.
thanks
Vrushali
On Wed, Aug 23, 2017 at 5:49 PM, Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> +1 (Binding), Great work guys, unwavering dedication and team effort. Kudos
> to the whole team.
> Though i have not tested, just went through the jiras and features added.
>
> Just wanted to know whether there was any other key out standing features
> left after phase 2 and their impacts ?
> And it would interesting to share the plans (if available) of upstream
> projects like Tez, Spark to incorporate ATSv2.
>
> Regards,
> + Naga
>
>
>
> On Thu, Aug 24, 2017 at 12:28 AM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
> > Hi All,
> >
> > Just to update.
> > Folks who may be interested in going through the documentation for the
> > feature can refer to current documentation attached in pdf format, on the
> > umbrella JIRA i.e. YARN-5355.
> >
> > Furthermore, we have run the YARN-5355 patch(with all the changes made in
> > the branch) against trunk and the build is almost green barring a few
> > checkstyle issues. The test failures which come up in the build are
> > outstanding issues on trunk.
> > Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo
> > mmentId=16138266&page=com.atlassian.jira.plugin.system.
> > issuetabpanels:comment-tabpanel#comment-16138266
> >
> > Thanks,
> > Varun Saxena.
> >
> > On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > vrushalic2016@gmail.com> wrote:
> >
> > > Hi folks,
> > >
> > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > will
> > > run for 7 days, and will end August 29 11:00 PM PDT.
> > >
> > > We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >
> > > Since then, we have been working on extending the capabilities of
> > Timeline
> > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > confident that the state of the feature meets the criteria to be merged
> > > onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > > and provide valuable feedback so that we can make it production-ready.
> > >
> > > In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> > > usability improvements based on a new architecture. What we would like
> to
> > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > complete end-to-end read/write flow with security and read level
> > > authorization via whitelists. You should be able to start setting it up
> > and
> > > testing it.
> > >
> > > At a high level, the following are the key features that have been
> > > implemented since alpha1:
> > > - Security via Kerberos Authentication and delegation tokens
> > > - Read side simple authorization via whitelist
> > > - Client configurable entity sort ordering
> > > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> > by
> > > timerange, pagination, sub-app entities
> > > - Support for storing sub-application entities (entities that exist
> > outside
> > > the scope of an application)
> > > - Configurable TTLs (time-to-live) for tables, configurable table
> > prefixes,
> > > configurable hbase cluster
> > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > - Uses latest stable HBase release 1.2.6
> > >
> > > There are a total of 82 subtasks that were completed as part of this
> > > effort.
> > >
> > > We paid close attention to ensure that once disabled Timeline Service
> v.2
> > > does not impact existing functionality when disabled (by default).
> > >
> > > Special thanks to a team of folks who worked hard and contributed
> towards
> > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >
> > > Regards,
> > > Vrushali
> > >
> > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27383.html
> > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Naganarasimha Garla <na...@apache.org>.
+1 (Binding), Great work guys, unwavering dedication and team effort. Kudos
to the whole team.
Though i have not tested, just went through the jiras and features added.
Just wanted to know whether there was any other key out standing features
left after phase 2 and their impacts ?
And it would interesting to share the plans (if available) of upstream
projects like Tez, Spark to incorporate ATSv2.
Regards,
+ Naga
On Thu, Aug 24, 2017 at 12:28 AM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> Hi All,
>
> Just to update.
> Folks who may be interested in going through the documentation for the
> feature can refer to current documentation attached in pdf format, on the
> umbrella JIRA i.e. YARN-5355.
>
> Furthermore, we have run the YARN-5355 patch(with all the changes made in
> the branch) against trunk and the build is almost green barring a few
> checkstyle issues. The test failures which come up in the build are
> outstanding issues on trunk.
> Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo
> mmentId=16138266&page=com.atlassian.jira.plugin.system.
> issuetabpanels:comment-tabpanel#comment-16138266
>
> Thanks,
> Varun Saxena.
>
> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> > effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Naganarasimha Garla <na...@apache.org>.
+1 (Binding), Great work guys, unwavering dedication and team effort. Kudos
to the whole team.
Though i have not tested, just went through the jiras and features added.
Just wanted to know whether there was any other key out standing features
left after phase 2 and their impacts ?
And it would interesting to share the plans (if available) of upstream
projects like Tez, Spark to incorporate ATSv2.
Regards,
+ Naga
On Thu, Aug 24, 2017 at 12:28 AM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> Hi All,
>
> Just to update.
> Folks who may be interested in going through the documentation for the
> feature can refer to current documentation attached in pdf format, on the
> umbrella JIRA i.e. YARN-5355.
>
> Furthermore, we have run the YARN-5355 patch(with all the changes made in
> the branch) against trunk and the build is almost green barring a few
> checkstyle issues. The test failures which come up in the build are
> outstanding issues on trunk.
> Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo
> mmentId=16138266&page=com.atlassian.jira.plugin.system.
> issuetabpanels:comment-tabpanel#comment-16138266
>
> Thanks,
> Varun Saxena.
>
> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> > effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Naganarasimha Garla <na...@apache.org>.
+1 (Binding), Great work guys, unwavering dedication and team effort. Kudos
to the whole team.
Though i have not tested, just went through the jiras and features added.
Just wanted to know whether there was any other key out standing features
left after phase 2 and their impacts ?
And it would interesting to share the plans (if available) of upstream
projects like Tez, Spark to incorporate ATSv2.
Regards,
+ Naga
On Thu, Aug 24, 2017 at 12:28 AM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> Hi All,
>
> Just to update.
> Folks who may be interested in going through the documentation for the
> feature can refer to current documentation attached in pdf format, on the
> umbrella JIRA i.e. YARN-5355.
>
> Furthermore, we have run the YARN-5355 patch(with all the changes made in
> the branch) against trunk and the build is almost green barring a few
> checkstyle issues. The test failures which come up in the build are
> outstanding issues on trunk.
> Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo
> mmentId=16138266&page=com.atlassian.jira.plugin.system.
> issuetabpanels:comment-tabpanel#comment-16138266
>
> Thanks,
> Varun Saxena.
>
> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> > effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Naganarasimha Garla <na...@apache.org>.
+1 (Binding), Great work guys, unwavering dedication and team effort. Kudos
to the whole team.
Though i have not tested, just went through the jiras and features added.
Just wanted to know whether there was any other key out standing features
left after phase 2 and their impacts ?
And it would interesting to share the plans (if available) of upstream
projects like Tez, Spark to incorporate ATSv2.
Regards,
+ Naga
On Thu, Aug 24, 2017 at 12:28 AM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> Hi All,
>
> Just to update.
> Folks who may be interested in going through the documentation for the
> feature can refer to current documentation attached in pdf format, on the
> umbrella JIRA i.e. YARN-5355.
>
> Furthermore, we have run the YARN-5355 patch(with all the changes made in
> the branch) against trunk and the build is almost green barring a few
> checkstyle issues. The test failures which come up in the build are
> outstanding issues on trunk.
> Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo
> mmentId=16138266&page=com.atlassian.jira.plugin.system.
> issuetabpanels:comment-tabpanel#comment-16138266
>
> Thanks,
> Varun Saxena.
>
> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> > effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Hi All,
Just to update.
Folks who may be interested in going through the documentation for the
feature can refer to current documentation attached in pdf format, on the
umbrella JIRA i.e. YARN-5355.
Furthermore, we have run the YARN-5355 patch(with all the changes made in
the branch) against trunk and the build is almost green barring a few
checkstyle issues. The test failures which come up in the build are
outstanding issues on trunk.
Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo
mmentId=16138266&page=com.atlassian.jira.plugin.system.
issuetabpanels:comment-tabpanel#comment-16138266
Thanks,
Varun Saxena.
On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Jason Lowe <jl...@oath.com.INVALID>.
+1 (binding)
I participated in the review for the reader authorization and verified that
ATSv2 has no significant impact when disabled. Looking forward to seeing
the next increment in functionality in a release. A big thank you to
everyone involved in this effort!
Jason
On Tue, Aug 22, 2017 at 1:32 AM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to
trunk
Posted by Jian He <jh...@hortonworks.com>.
+1 (binding)
I’ve participated in the review of ATSv2 security related patch.
JIan
> On Aug 28, 2017, at 1:39 AM, Rohith Sharma K S <ro...@apache.org> wrote:
>
> +1 (binding)
>
> Thank you very much for the great team work!
>
> Built from source and deployed in secured cluster. The below are the test
> result.
>
> Deployment :
> Standard hadoop security deployment authentication and authorization as
> well.
> Branch-2 Hadoop and Hbase security cluster.
> Branch-3 Hadoop security cluster. HBase client is pointing to Branch-2
> hbase cluster.
> All security configurations are set in-place.
> Each service is running with its own user. Say, HDFS is running with
> hdfs, YARN user is running with yarn, Hbase is running with hbase
> Smoke test user : test-user
>
> Test Cases :
>
> Authentication :
> Verify for all daemons start up successful : OK
> Run a MR job using test-user : OK
> Verify for REST API’s with in the scope of application : OK
> Verify for REST API’s newly added I.e outside scope of application : OK.
> RM Restart/ NM restart / RM_work-preserving restart has executed and
> verified for data : OK. (Entity validation is done, but not entity data
> validation!
> Token redistribution to AM, NM is verified.
>
> Authorization :
> 1 . Basic whitelisting of users to read has been validated. Works as
> expected!
>
> Disabling TSv2 configuration is also being tested.
>
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 22 August 2017 at 12:02, Vrushali Channapattan <vr...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
>> run for 7 days, and will end August 29 11:00 PM PDT.
>>
>> We have previously completed one merge onto trunk [3] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> confident that the state of the feature meets the criteria to be merged
>> onto trunk and we'd love folks to get their hands on it in a test capacity
>> and provide valuable feedback so that we can make it production-ready.
>>
>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>> usability improvements based on a new architecture. What we would like to
>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> complete end-to-end read/write flow with security and read level
>> authorization via whitelists. You should be able to start setting it up and
>> testing it.
>>
>> At a high level, the following are the key features that have been
>> implemented since alpha1:
>> - Security via Kerberos Authentication and delegation tokens
>> - Read side simple authorization via whitelist
>> - Client configurable entity sort ordering
>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>> timerange, pagination, sub-app entities
>> - Support for storing sub-application entities (entities that exist outside
>> the scope of an application)
>> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
>> configurable hbase cluster
>> - Flow level aggregations done as dynamic (table level) coprocessors
>> - Uses latest stable HBase release 1.2.6
>>
>> There are a total of 82 subtasks that were completed as part of this
>> effort.
>>
>> We paid close attention to ensure that once disabled Timeline Service v.2
>> does not impact existing functionality when disabled (by default).
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Regards,
>> Vrushali
>>
>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to
trunk
Posted by Jian He <jh...@hortonworks.com>.
+1 (binding)
I’ve participated in the review of ATSv2 security related patch.
JIan
> On Aug 28, 2017, at 1:39 AM, Rohith Sharma K S <ro...@apache.org> wrote:
>
> +1 (binding)
>
> Thank you very much for the great team work!
>
> Built from source and deployed in secured cluster. The below are the test
> result.
>
> Deployment :
> Standard hadoop security deployment authentication and authorization as
> well.
> Branch-2 Hadoop and Hbase security cluster.
> Branch-3 Hadoop security cluster. HBase client is pointing to Branch-2
> hbase cluster.
> All security configurations are set in-place.
> Each service is running with its own user. Say, HDFS is running with
> hdfs, YARN user is running with yarn, Hbase is running with hbase
> Smoke test user : test-user
>
> Test Cases :
>
> Authentication :
> Verify for all daemons start up successful : OK
> Run a MR job using test-user : OK
> Verify for REST API’s with in the scope of application : OK
> Verify for REST API’s newly added I.e outside scope of application : OK.
> RM Restart/ NM restart / RM_work-preserving restart has executed and
> verified for data : OK. (Entity validation is done, but not entity data
> validation!
> Token redistribution to AM, NM is verified.
>
> Authorization :
> 1 . Basic whitelisting of users to read has been validated. Works as
> expected!
>
> Disabling TSv2 configuration is also being tested.
>
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 22 August 2017 at 12:02, Vrushali Channapattan <vr...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
>> run for 7 days, and will end August 29 11:00 PM PDT.
>>
>> We have previously completed one merge onto trunk [3] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> confident that the state of the feature meets the criteria to be merged
>> onto trunk and we'd love folks to get their hands on it in a test capacity
>> and provide valuable feedback so that we can make it production-ready.
>>
>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>> usability improvements based on a new architecture. What we would like to
>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> complete end-to-end read/write flow with security and read level
>> authorization via whitelists. You should be able to start setting it up and
>> testing it.
>>
>> At a high level, the following are the key features that have been
>> implemented since alpha1:
>> - Security via Kerberos Authentication and delegation tokens
>> - Read side simple authorization via whitelist
>> - Client configurable entity sort ordering
>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>> timerange, pagination, sub-app entities
>> - Support for storing sub-application entities (entities that exist outside
>> the scope of an application)
>> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
>> configurable hbase cluster
>> - Flow level aggregations done as dynamic (table level) coprocessors
>> - Uses latest stable HBase release 1.2.6
>>
>> There are a total of 82 subtasks that were completed as part of this
>> effort.
>>
>> We paid close attention to ensure that once disabled Timeline Service v.2
>> does not impact existing functionality when disabled (by default).
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Regards,
>> Vrushali
>>
>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to
trunk
Posted by Jian He <jh...@hortonworks.com>.
+1 (binding)
I’ve participated in the review of ATSv2 security related patch.
JIan
> On Aug 28, 2017, at 1:39 AM, Rohith Sharma K S <ro...@apache.org> wrote:
>
> +1 (binding)
>
> Thank you very much for the great team work!
>
> Built from source and deployed in secured cluster. The below are the test
> result.
>
> Deployment :
> Standard hadoop security deployment authentication and authorization as
> well.
> Branch-2 Hadoop and Hbase security cluster.
> Branch-3 Hadoop security cluster. HBase client is pointing to Branch-2
> hbase cluster.
> All security configurations are set in-place.
> Each service is running with its own user. Say, HDFS is running with
> hdfs, YARN user is running with yarn, Hbase is running with hbase
> Smoke test user : test-user
>
> Test Cases :
>
> Authentication :
> Verify for all daemons start up successful : OK
> Run a MR job using test-user : OK
> Verify for REST API’s with in the scope of application : OK
> Verify for REST API’s newly added I.e outside scope of application : OK.
> RM Restart/ NM restart / RM_work-preserving restart has executed and
> verified for data : OK. (Entity validation is done, but not entity data
> validation!
> Token redistribution to AM, NM is verified.
>
> Authorization :
> 1 . Basic whitelisting of users to read has been validated. Works as
> expected!
>
> Disabling TSv2 configuration is also being tested.
>
>
> Thanks & Regards
> Rohith Sharma K S
>
> On 22 August 2017 at 12:02, Vrushali Channapattan <vr...@gmail.com>
> wrote:
>
>> Hi folks,
>>
>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
>> run for 7 days, and will end August 29 11:00 PM PDT.
>>
>> We have previously completed one merge onto trunk [3] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> confident that the state of the feature meets the criteria to be merged
>> onto trunk and we'd love folks to get their hands on it in a test capacity
>> and provide valuable feedback so that we can make it production-ready.
>>
>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>> usability improvements based on a new architecture. What we would like to
>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> complete end-to-end read/write flow with security and read level
>> authorization via whitelists. You should be able to start setting it up and
>> testing it.
>>
>> At a high level, the following are the key features that have been
>> implemented since alpha1:
>> - Security via Kerberos Authentication and delegation tokens
>> - Read side simple authorization via whitelist
>> - Client configurable entity sort ordering
>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>> timerange, pagination, sub-app entities
>> - Support for storing sub-application entities (entities that exist outside
>> the scope of an application)
>> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
>> configurable hbase cluster
>> - Flow level aggregations done as dynamic (table level) coprocessors
>> - Uses latest stable HBase release 1.2.6
>>
>> There are a total of 82 subtasks that were completed as part of this
>> effort.
>>
>> We paid close attention to ensure that once disabled Timeline Service v.2
>> does not impact existing functionality when disabled (by default).
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Regards,
>> Vrushali
>>
>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Rohith Sharma K S <ro...@apache.org>.
+1 (binding)
Thank you very much for the great team work!
Built from source and deployed in secured cluster. The below are the test
result.
Deployment :
Standard hadoop security deployment authentication and authorization as
well.
Branch-2 Hadoop and Hbase security cluster.
Branch-3 Hadoop security cluster. HBase client is pointing to Branch-2
hbase cluster.
All security configurations are set in-place.
Each service is running with its own user. Say, HDFS is running with
hdfs, YARN user is running with yarn, Hbase is running with hbase
Smoke test user : test-user
Test Cases :
Authentication :
Verify for all daemons start up successful : OK
Run a MR job using test-user : OK
Verify for REST API’s with in the scope of application : OK
Verify for REST API’s newly added I.e outside scope of application : OK.
RM Restart/ NM restart / RM_work-preserving restart has executed and
verified for data : OK. (Entity validation is done, but not entity data
validation!
Token redistribution to AM, NM is verified.
Authorization :
1 . Basic whitelisting of users to read has been validated. Works as
expected!
Disabling TSv2 configuration is also being tested.
Thanks & Regards
Rohith Sharma K S
On 22 August 2017 at 12:02, Vrushali Channapattan <vr...@gmail.com>
wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Hi All,
Please note that ATSv2 has been merged to branch-2.
Refer to https://github.com/apache/hadoop/commit/
54a694172d7162b2ff45d6e1780c0523cfda66e7
We will continue monitoring if everything works well post-merge.
Regards,
Varun Saxena.
On Mon, Oct 16, 2017 at 9:54 PM, Vrushali C <vr...@gmail.com> wrote:
> Timeline Service v2 should be landing on branch2 shortly.
>
> thanks
> Vrushali
>
> On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com>
> wrote:
>
>> Thanks everyone.
>>
>> It has been over a week (~9 days) since TSv2 has been merged to trunk
>> with no problems thus far. We are now thinking about merging timeline
>> service v2 to branch2 some time in the next few weeks.
>>
>> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
>> with our trunk based feature branch YARN-5355. Varun Saxena has been
>> diligently rebasing it to stay current with branch2.
>>
>> Currently we are in the process of testing it just like we did our due
>> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
>> branch2 code is a stable state to be merged.
>>
>> We will send out another email when we are ready to merge to branch2.
>> thanks
>> Vrushali
>>
>> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
>> wrote:
>>
>>> Good to see this merged. I have initiated a separate thread with a
>>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>>> to the 2.9 release thread as soon as we reach consensus.
>>>
>>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>>> wrote:
>>>
>>>> +1 to maintaining history.
>>>>
>>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>>> varun.saxena.apache@gmail.com> wrote:
>>>>
>>>> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
>>>> > Maintaining history I believe can be useful as it can make reverts
>>>> > easier if at all required.
>>>> > And can be an easy reference point to look at who had contributed what
>>>> > without having to go back to the branch.
>>>> >
>>>> > Regards,
>>>> > Varun Saxena.
>>>> >
>>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>>> think
>>>> > > that helps answer Steve's questions.
>>>> > >
>>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>>> merged
>>>> > to
>>>> > > trunk via "git merge --no-ff" .
>>>> > >
>>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge,
>>>> we
>>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>>> control
>>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>>> > >
>>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>>> Steve?
>>>> > >
>>>> > > thanks
>>>> > > Vrushali
>>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>>> 6afd9ef
>>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>>> pache.org%3E
>>>> > >
>>>> > >
>>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>>> wrote:
>>>> > >
>>>> > >> I recall this discussion about a couple of years ago:
>>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>>> > >> dev.hadoop.apache.org%3E
>>>> > >>
>>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>>> stevel@hortonworks.com
>>>> > >
>>>> > >> wrote:
>>>> > >>
>>>> > >>> I'd have assumed it would have gone in as one single patch,
>>>> rather than
>>>> > >>> a full history. I don't see why the trunk needs all the
>>>> evolutionary
>>>> > >>> history of a build.
>>>> > >>>
>>>> > >>> What should our policy/process be here?
>>>> > >>>
>>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>>> > >>>
>>>> > >>>
>>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> > >>> >
>>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>>> > >>> >
>>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>>> get on
>>>> > >>> with
>>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>>> > >>> >
>>>> > >>> > Regards
>>>> > >>> > Vrushali
>>>> > >>> >
>>>> > >>> >
>>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>>> > >>> >
>>>> > >>> >> +1 (binding).
>>>> > >>> >>
>>>> > >>> >> Kudos to all the team members for their great work!
>>>> > >>> >>
>>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>>> > >>> development
>>>> > >>> >> or review of most of the JIRAs'.
>>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>>> that
>>>> > >>> there
>>>> > >>> >> is no impact when ATSv2 is turned off.
>>>> > >>> >>
>>>> > >>> >> Regards,
>>>> > >>> >> Varun Saxena.
>>>> > >>> >>
>>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>>> > >>> >>
>>>> > >>> >>> Hi folks,
>>>> > >>> >>>
>>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>>> > merge
>>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>>> The
>>>> > >>> vote
>>>> > >>> >>> will
>>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>>> > >>> >>>
>>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>>> Timeline
>>>> > >>> Service
>>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>> > >>> >>>
>>>> > >>> >>> Since then, we have been working on extending the
>>>> capabilities of
>>>> > >>> Timeline
>>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>>> > reasonably
>>>> > >>> >>> confident that the state of the feature meets the criteria to
>>>> be
>>>> > >>> merged
>>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>>> test
>>>> > >>> capacity
>>>> > >>> >>> and provide valuable feedback so that we can make it
>>>> > >>> production-ready.
>>>> > >>> >>>
>>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>>> > scalability
>>>> > >>> and
>>>> > >>> >>> usability improvements based on a new architecture. What we
>>>> would
>>>> > >>> like to
>>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>>> has a
>>>> > >>> >>> complete end-to-end read/write flow with security and read
>>>> level
>>>> > >>> >>> authorization via whitelists. You should be able to start
>>>> setting
>>>> > it
>>>> > >>> up
>>>> > >>> >>> and
>>>> > >>> >>> testing it.
>>>> > >>> >>>
>>>> > >>> >>> At a high level, the following are the key features that have
>>>> been
>>>> > >>> >>> implemented since alpha1:
>>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>>> > >>> >>> - Read side simple authorization via whitelist
>>>> > >>> >>> - Client configurable entity sort ordering
>>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers,
>>>> fetching
>>>> > >>> metrics by
>>>> > >>> >>> timerange, pagination, sub-app entities
>>>> > >>> >>> - Support for storing sub-application entities (entities that
>>>> exist
>>>> > >>> >>> outside
>>>> > >>> >>> the scope of an application)
>>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>>> table
>>>> > >>> >>> prefixes,
>>>> > >>> >>> configurable hbase cluster
>>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>>> > coprocessors
>>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>>> > >>> >>>
>>>> > >>> >>> There are a total of 82 subtasks that were completed as part
>>>> of
>>>> > this
>>>> > >>> >>> effort.
>>>> > >>> >>>
>>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>>> > >>> Service v.2
>>>> > >>> >>> does not impact existing functionality when disabled (by
>>>> default).
>>>> > >>> >>>
>>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>>> contributed
>>>> > >>> towards
>>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>>> K S,
>>>> > >>> Varun
>>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>>> Vavilapalli,
>>>> > Joep
>>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal
>>>> Stack.
>>>> > >>> >>>
>>>> > >>> >>> Regards,
>>>> > >>> >>> Vrushali
>>>> > >>> >>>
>>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>>> rn-dev@hadoop.apache.org/msg27
>>>> > >>> 383.html
>>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>>> > >>> >>>
>>>> > >>> >>
>>>> > >>> >>
>>>> > >>>
>>>> > >>>
>>>> > >>> ------------------------------------------------------------
>>>> ---------
>>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>> > >>>
>>>> > >>>
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Hi All,
Please note that ATSv2 has been merged to branch-2.
Refer to https://github.com/apache/hadoop/commit/
54a694172d7162b2ff45d6e1780c0523cfda66e7
We will continue monitoring if everything works well post-merge.
Regards,
Varun Saxena.
On Mon, Oct 16, 2017 at 9:54 PM, Vrushali C <vr...@gmail.com> wrote:
> Timeline Service v2 should be landing on branch2 shortly.
>
> thanks
> Vrushali
>
> On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com>
> wrote:
>
>> Thanks everyone.
>>
>> It has been over a week (~9 days) since TSv2 has been merged to trunk
>> with no problems thus far. We are now thinking about merging timeline
>> service v2 to branch2 some time in the next few weeks.
>>
>> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
>> with our trunk based feature branch YARN-5355. Varun Saxena has been
>> diligently rebasing it to stay current with branch2.
>>
>> Currently we are in the process of testing it just like we did our due
>> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
>> branch2 code is a stable state to be merged.
>>
>> We will send out another email when we are ready to merge to branch2.
>> thanks
>> Vrushali
>>
>> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
>> wrote:
>>
>>> Good to see this merged. I have initiated a separate thread with a
>>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>>> to the 2.9 release thread as soon as we reach consensus.
>>>
>>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>>> wrote:
>>>
>>>> +1 to maintaining history.
>>>>
>>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>>> varun.saxena.apache@gmail.com> wrote:
>>>>
>>>> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
>>>> > Maintaining history I believe can be useful as it can make reverts
>>>> > easier if at all required.
>>>> > And can be an easy reference point to look at who had contributed what
>>>> > without having to go back to the branch.
>>>> >
>>>> > Regards,
>>>> > Varun Saxena.
>>>> >
>>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>>> think
>>>> > > that helps answer Steve's questions.
>>>> > >
>>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>>> merged
>>>> > to
>>>> > > trunk via "git merge --no-ff" .
>>>> > >
>>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge,
>>>> we
>>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>>> control
>>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>>> > >
>>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>>> Steve?
>>>> > >
>>>> > > thanks
>>>> > > Vrushali
>>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>>> 6afd9ef
>>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>>> pache.org%3E
>>>> > >
>>>> > >
>>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>>> wrote:
>>>> > >
>>>> > >> I recall this discussion about a couple of years ago:
>>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>>> > >> dev.hadoop.apache.org%3E
>>>> > >>
>>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>>> stevel@hortonworks.com
>>>> > >
>>>> > >> wrote:
>>>> > >>
>>>> > >>> I'd have assumed it would have gone in as one single patch,
>>>> rather than
>>>> > >>> a full history. I don't see why the trunk needs all the
>>>> evolutionary
>>>> > >>> history of a build.
>>>> > >>>
>>>> > >>> What should our policy/process be here?
>>>> > >>>
>>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>>> > >>>
>>>> > >>>
>>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> > >>> >
>>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>>> > >>> >
>>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>>> get on
>>>> > >>> with
>>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>>> > >>> >
>>>> > >>> > Regards
>>>> > >>> > Vrushali
>>>> > >>> >
>>>> > >>> >
>>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>>> > >>> >
>>>> > >>> >> +1 (binding).
>>>> > >>> >>
>>>> > >>> >> Kudos to all the team members for their great work!
>>>> > >>> >>
>>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>>> > >>> development
>>>> > >>> >> or review of most of the JIRAs'.
>>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>>> that
>>>> > >>> there
>>>> > >>> >> is no impact when ATSv2 is turned off.
>>>> > >>> >>
>>>> > >>> >> Regards,
>>>> > >>> >> Varun Saxena.
>>>> > >>> >>
>>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>>> > >>> >>
>>>> > >>> >>> Hi folks,
>>>> > >>> >>>
>>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>>> > merge
>>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>>> The
>>>> > >>> vote
>>>> > >>> >>> will
>>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>>> > >>> >>>
>>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>>> Timeline
>>>> > >>> Service
>>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>> > >>> >>>
>>>> > >>> >>> Since then, we have been working on extending the
>>>> capabilities of
>>>> > >>> Timeline
>>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>>> > reasonably
>>>> > >>> >>> confident that the state of the feature meets the criteria to
>>>> be
>>>> > >>> merged
>>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>>> test
>>>> > >>> capacity
>>>> > >>> >>> and provide valuable feedback so that we can make it
>>>> > >>> production-ready.
>>>> > >>> >>>
>>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>>> > scalability
>>>> > >>> and
>>>> > >>> >>> usability improvements based on a new architecture. What we
>>>> would
>>>> > >>> like to
>>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>>> has a
>>>> > >>> >>> complete end-to-end read/write flow with security and read
>>>> level
>>>> > >>> >>> authorization via whitelists. You should be able to start
>>>> setting
>>>> > it
>>>> > >>> up
>>>> > >>> >>> and
>>>> > >>> >>> testing it.
>>>> > >>> >>>
>>>> > >>> >>> At a high level, the following are the key features that have
>>>> been
>>>> > >>> >>> implemented since alpha1:
>>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>>> > >>> >>> - Read side simple authorization via whitelist
>>>> > >>> >>> - Client configurable entity sort ordering
>>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers,
>>>> fetching
>>>> > >>> metrics by
>>>> > >>> >>> timerange, pagination, sub-app entities
>>>> > >>> >>> - Support for storing sub-application entities (entities that
>>>> exist
>>>> > >>> >>> outside
>>>> > >>> >>> the scope of an application)
>>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>>> table
>>>> > >>> >>> prefixes,
>>>> > >>> >>> configurable hbase cluster
>>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>>> > coprocessors
>>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>>> > >>> >>>
>>>> > >>> >>> There are a total of 82 subtasks that were completed as part
>>>> of
>>>> > this
>>>> > >>> >>> effort.
>>>> > >>> >>>
>>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>>> > >>> Service v.2
>>>> > >>> >>> does not impact existing functionality when disabled (by
>>>> default).
>>>> > >>> >>>
>>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>>> contributed
>>>> > >>> towards
>>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>>> K S,
>>>> > >>> Varun
>>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>>> Vavilapalli,
>>>> > Joep
>>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal
>>>> Stack.
>>>> > >>> >>>
>>>> > >>> >>> Regards,
>>>> > >>> >>> Vrushali
>>>> > >>> >>>
>>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>>> rn-dev@hadoop.apache.org/msg27
>>>> > >>> 383.html
>>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>>> > >>> >>>
>>>> > >>> >>
>>>> > >>> >>
>>>> > >>>
>>>> > >>>
>>>> > >>> ------------------------------------------------------------
>>>> ---------
>>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>> > >>>
>>>> > >>>
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Hi All,
Please note that ATSv2 has been merged to branch-2.
Refer to https://github.com/apache/hadoop/commit/
54a694172d7162b2ff45d6e1780c0523cfda66e7
We will continue monitoring if everything works well post-merge.
Regards,
Varun Saxena.
On Mon, Oct 16, 2017 at 9:54 PM, Vrushali C <vr...@gmail.com> wrote:
> Timeline Service v2 should be landing on branch2 shortly.
>
> thanks
> Vrushali
>
> On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com>
> wrote:
>
>> Thanks everyone.
>>
>> It has been over a week (~9 days) since TSv2 has been merged to trunk
>> with no problems thus far. We are now thinking about merging timeline
>> service v2 to branch2 some time in the next few weeks.
>>
>> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
>> with our trunk based feature branch YARN-5355. Varun Saxena has been
>> diligently rebasing it to stay current with branch2.
>>
>> Currently we are in the process of testing it just like we did our due
>> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
>> branch2 code is a stable state to be merged.
>>
>> We will send out another email when we are ready to merge to branch2.
>> thanks
>> Vrushali
>>
>> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
>> wrote:
>>
>>> Good to see this merged. I have initiated a separate thread with a
>>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>>> to the 2.9 release thread as soon as we reach consensus.
>>>
>>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>>> wrote:
>>>
>>>> +1 to maintaining history.
>>>>
>>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>>> varun.saxena.apache@gmail.com> wrote:
>>>>
>>>> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
>>>> > Maintaining history I believe can be useful as it can make reverts
>>>> > easier if at all required.
>>>> > And can be an easy reference point to look at who had contributed what
>>>> > without having to go back to the branch.
>>>> >
>>>> > Regards,
>>>> > Varun Saxena.
>>>> >
>>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>>> think
>>>> > > that helps answer Steve's questions.
>>>> > >
>>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>>> merged
>>>> > to
>>>> > > trunk via "git merge --no-ff" .
>>>> > >
>>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge,
>>>> we
>>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>>> control
>>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>>> > >
>>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>>> Steve?
>>>> > >
>>>> > > thanks
>>>> > > Vrushali
>>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>>> 6afd9ef
>>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>>> pache.org%3E
>>>> > >
>>>> > >
>>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>>> wrote:
>>>> > >
>>>> > >> I recall this discussion about a couple of years ago:
>>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>>> > >> dev.hadoop.apache.org%3E
>>>> > >>
>>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>>> stevel@hortonworks.com
>>>> > >
>>>> > >> wrote:
>>>> > >>
>>>> > >>> I'd have assumed it would have gone in as one single patch,
>>>> rather than
>>>> > >>> a full history. I don't see why the trunk needs all the
>>>> evolutionary
>>>> > >>> history of a build.
>>>> > >>>
>>>> > >>> What should our policy/process be here?
>>>> > >>>
>>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>>> > >>>
>>>> > >>>
>>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> > >>> >
>>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>>> > >>> >
>>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>>> get on
>>>> > >>> with
>>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>>> > >>> >
>>>> > >>> > Regards
>>>> > >>> > Vrushali
>>>> > >>> >
>>>> > >>> >
>>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>>> > >>> >
>>>> > >>> >> +1 (binding).
>>>> > >>> >>
>>>> > >>> >> Kudos to all the team members for their great work!
>>>> > >>> >>
>>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>>> > >>> development
>>>> > >>> >> or review of most of the JIRAs'.
>>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>>> that
>>>> > >>> there
>>>> > >>> >> is no impact when ATSv2 is turned off.
>>>> > >>> >>
>>>> > >>> >> Regards,
>>>> > >>> >> Varun Saxena.
>>>> > >>> >>
>>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>>> > >>> >>
>>>> > >>> >>> Hi folks,
>>>> > >>> >>>
>>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>>> > merge
>>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>>> The
>>>> > >>> vote
>>>> > >>> >>> will
>>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>>> > >>> >>>
>>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>>> Timeline
>>>> > >>> Service
>>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>> > >>> >>>
>>>> > >>> >>> Since then, we have been working on extending the
>>>> capabilities of
>>>> > >>> Timeline
>>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>>> > reasonably
>>>> > >>> >>> confident that the state of the feature meets the criteria to
>>>> be
>>>> > >>> merged
>>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>>> test
>>>> > >>> capacity
>>>> > >>> >>> and provide valuable feedback so that we can make it
>>>> > >>> production-ready.
>>>> > >>> >>>
>>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>>> > scalability
>>>> > >>> and
>>>> > >>> >>> usability improvements based on a new architecture. What we
>>>> would
>>>> > >>> like to
>>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>>> has a
>>>> > >>> >>> complete end-to-end read/write flow with security and read
>>>> level
>>>> > >>> >>> authorization via whitelists. You should be able to start
>>>> setting
>>>> > it
>>>> > >>> up
>>>> > >>> >>> and
>>>> > >>> >>> testing it.
>>>> > >>> >>>
>>>> > >>> >>> At a high level, the following are the key features that have
>>>> been
>>>> > >>> >>> implemented since alpha1:
>>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>>> > >>> >>> - Read side simple authorization via whitelist
>>>> > >>> >>> - Client configurable entity sort ordering
>>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers,
>>>> fetching
>>>> > >>> metrics by
>>>> > >>> >>> timerange, pagination, sub-app entities
>>>> > >>> >>> - Support for storing sub-application entities (entities that
>>>> exist
>>>> > >>> >>> outside
>>>> > >>> >>> the scope of an application)
>>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>>> table
>>>> > >>> >>> prefixes,
>>>> > >>> >>> configurable hbase cluster
>>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>>> > coprocessors
>>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>>> > >>> >>>
>>>> > >>> >>> There are a total of 82 subtasks that were completed as part
>>>> of
>>>> > this
>>>> > >>> >>> effort.
>>>> > >>> >>>
>>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>>> > >>> Service v.2
>>>> > >>> >>> does not impact existing functionality when disabled (by
>>>> default).
>>>> > >>> >>>
>>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>>> contributed
>>>> > >>> towards
>>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>>> K S,
>>>> > >>> Varun
>>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>>> Vavilapalli,
>>>> > Joep
>>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal
>>>> Stack.
>>>> > >>> >>>
>>>> > >>> >>> Regards,
>>>> > >>> >>> Vrushali
>>>> > >>> >>>
>>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>>> rn-dev@hadoop.apache.org/msg27
>>>> > >>> 383.html
>>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>>> > >>> >>>
>>>> > >>> >>
>>>> > >>> >>
>>>> > >>>
>>>> > >>>
>>>> > >>> ------------------------------------------------------------
>>>> ---------
>>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>> > >>>
>>>> > >>>
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Hi All,
Please note that ATSv2 has been merged to branch-2.
Refer to https://github.com/apache/hadoop/commit/
54a694172d7162b2ff45d6e1780c0523cfda66e7
We will continue monitoring if everything works well post-merge.
Regards,
Varun Saxena.
On Mon, Oct 16, 2017 at 9:54 PM, Vrushali C <vr...@gmail.com> wrote:
> Timeline Service v2 should be landing on branch2 shortly.
>
> thanks
> Vrushali
>
> On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com>
> wrote:
>
>> Thanks everyone.
>>
>> It has been over a week (~9 days) since TSv2 has been merged to trunk
>> with no problems thus far. We are now thinking about merging timeline
>> service v2 to branch2 some time in the next few weeks.
>>
>> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
>> with our trunk based feature branch YARN-5355. Varun Saxena has been
>> diligently rebasing it to stay current with branch2.
>>
>> Currently we are in the process of testing it just like we did our due
>> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
>> branch2 code is a stable state to be merged.
>>
>> We will send out another email when we are ready to merge to branch2.
>> thanks
>> Vrushali
>>
>> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
>> wrote:
>>
>>> Good to see this merged. I have initiated a separate thread with a
>>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>>> to the 2.9 release thread as soon as we reach consensus.
>>>
>>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>>> wrote:
>>>
>>>> +1 to maintaining history.
>>>>
>>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>>> varun.saxena.apache@gmail.com> wrote:
>>>>
>>>> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
>>>> > Maintaining history I believe can be useful as it can make reverts
>>>> > easier if at all required.
>>>> > And can be an easy reference point to look at who had contributed what
>>>> > without having to go back to the branch.
>>>> >
>>>> > Regards,
>>>> > Varun Saxena.
>>>> >
>>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>>> think
>>>> > > that helps answer Steve's questions.
>>>> > >
>>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>>> merged
>>>> > to
>>>> > > trunk via "git merge --no-ff" .
>>>> > >
>>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge,
>>>> we
>>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>>> control
>>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>>> > >
>>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>>> Steve?
>>>> > >
>>>> > > thanks
>>>> > > Vrushali
>>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>>> 6afd9ef
>>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>>> pache.org%3E
>>>> > >
>>>> > >
>>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>>> wrote:
>>>> > >
>>>> > >> I recall this discussion about a couple of years ago:
>>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>>> > >> dev.hadoop.apache.org%3E
>>>> > >>
>>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>>> stevel@hortonworks.com
>>>> > >
>>>> > >> wrote:
>>>> > >>
>>>> > >>> I'd have assumed it would have gone in as one single patch,
>>>> rather than
>>>> > >>> a full history. I don't see why the trunk needs all the
>>>> evolutionary
>>>> > >>> history of a build.
>>>> > >>>
>>>> > >>> What should our policy/process be here?
>>>> > >>>
>>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>>> > >>>
>>>> > >>>
>>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> > >>> >
>>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>>> > >>> >
>>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>>> get on
>>>> > >>> with
>>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>>> > >>> >
>>>> > >>> > Regards
>>>> > >>> > Vrushali
>>>> > >>> >
>>>> > >>> >
>>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>>> > >>> >
>>>> > >>> >> +1 (binding).
>>>> > >>> >>
>>>> > >>> >> Kudos to all the team members for their great work!
>>>> > >>> >>
>>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>>> > >>> development
>>>> > >>> >> or review of most of the JIRAs'.
>>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>>> that
>>>> > >>> there
>>>> > >>> >> is no impact when ATSv2 is turned off.
>>>> > >>> >>
>>>> > >>> >> Regards,
>>>> > >>> >> Varun Saxena.
>>>> > >>> >>
>>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>>> > >>> >>
>>>> > >>> >>> Hi folks,
>>>> > >>> >>>
>>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>>> > merge
>>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>>> The
>>>> > >>> vote
>>>> > >>> >>> will
>>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>>> > >>> >>>
>>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>>> Timeline
>>>> > >>> Service
>>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>> > >>> >>>
>>>> > >>> >>> Since then, we have been working on extending the
>>>> capabilities of
>>>> > >>> Timeline
>>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>>> > reasonably
>>>> > >>> >>> confident that the state of the feature meets the criteria to
>>>> be
>>>> > >>> merged
>>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>>> test
>>>> > >>> capacity
>>>> > >>> >>> and provide valuable feedback so that we can make it
>>>> > >>> production-ready.
>>>> > >>> >>>
>>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>>> > scalability
>>>> > >>> and
>>>> > >>> >>> usability improvements based on a new architecture. What we
>>>> would
>>>> > >>> like to
>>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>>> has a
>>>> > >>> >>> complete end-to-end read/write flow with security and read
>>>> level
>>>> > >>> >>> authorization via whitelists. You should be able to start
>>>> setting
>>>> > it
>>>> > >>> up
>>>> > >>> >>> and
>>>> > >>> >>> testing it.
>>>> > >>> >>>
>>>> > >>> >>> At a high level, the following are the key features that have
>>>> been
>>>> > >>> >>> implemented since alpha1:
>>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>>> > >>> >>> - Read side simple authorization via whitelist
>>>> > >>> >>> - Client configurable entity sort ordering
>>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers,
>>>> fetching
>>>> > >>> metrics by
>>>> > >>> >>> timerange, pagination, sub-app entities
>>>> > >>> >>> - Support for storing sub-application entities (entities that
>>>> exist
>>>> > >>> >>> outside
>>>> > >>> >>> the scope of an application)
>>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>>> table
>>>> > >>> >>> prefixes,
>>>> > >>> >>> configurable hbase cluster
>>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>>> > coprocessors
>>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>>> > >>> >>>
>>>> > >>> >>> There are a total of 82 subtasks that were completed as part
>>>> of
>>>> > this
>>>> > >>> >>> effort.
>>>> > >>> >>>
>>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>>> > >>> Service v.2
>>>> > >>> >>> does not impact existing functionality when disabled (by
>>>> default).
>>>> > >>> >>>
>>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>>> contributed
>>>> > >>> towards
>>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>>> K S,
>>>> > >>> Varun
>>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>>> Vavilapalli,
>>>> > Joep
>>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal
>>>> Stack.
>>>> > >>> >>>
>>>> > >>> >>> Regards,
>>>> > >>> >>> Vrushali
>>>> > >>> >>>
>>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>>> rn-dev@hadoop.apache.org/msg27
>>>> > >>> 383.html
>>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>>> > >>> >>>
>>>> > >>> >>
>>>> > >>> >>
>>>> > >>>
>>>> > >>>
>>>> > >>> ------------------------------------------------------------
>>>> ---------
>>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>> > >>>
>>>> > >>>
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Timeline Service v2 should be landing on branch2 shortly.
thanks
Vrushali
On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com> wrote:
> Thanks everyone.
>
> It has been over a week (~9 days) since TSv2 has been merged to trunk with
> no problems thus far. We are now thinking about merging timeline service v2
> to branch2 some time in the next few weeks.
>
> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
> with our trunk based feature branch YARN-5355. Varun Saxena has been
> diligently rebasing it to stay current with branch2.
>
> Currently we are in the process of testing it just like we did our due
> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
> branch2 code is a stable state to be merged.
>
> We will send out another email when we are ready to merge to branch2.
> thanks
> Vrushali
>
> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
> wrote:
>
>> Good to see this merged. I have initiated a separate thread with a
>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>> to the 2.9 release thread as soon as we reach consensus.
>>
>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>> wrote:
>>
>>> +1 to maintaining history.
>>>
>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>> varun.saxena.apache@gmail.com> wrote:
>>>
>>> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
>>> > Maintaining history I believe can be useful as it can make reverts
>>> > easier if at all required.
>>> > And can be an easy reference point to look at who had contributed what
>>> > without having to go back to the branch.
>>> >
>>> > Regards,
>>> > Varun Saxena.
>>> >
>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> >
>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>> think
>>> > > that helps answer Steve's questions.
>>> > >
>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>> merged
>>> > to
>>> > > trunk via "git merge --no-ff" .
>>> > >
>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>> control
>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>> > >
>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>> Steve?
>>> > >
>>> > > thanks
>>> > > Vrushali
>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>> 6afd9ef
>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>> pache.org%3E
>>> > >
>>> > >
>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>> wrote:
>>> > >
>>> > >> I recall this discussion about a couple of years ago:
>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>> > >> dev.hadoop.apache.org%3E
>>> > >>
>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>> stevel@hortonworks.com
>>> > >
>>> > >> wrote:
>>> > >>
>>> > >>> I'd have assumed it would have gone in as one single patch, rather
>>> than
>>> > >>> a full history. I don't see why the trunk needs all the
>>> evolutionary
>>> > >>> history of a build.
>>> > >>>
>>> > >>> What should our policy/process be here?
>>> > >>>
>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>> > >>>
>>> > >>>
>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> > >>> >
>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>> > >>> >
>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>> get on
>>> > >>> with
>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>> > >>> >
>>> > >>> > Regards
>>> > >>> > Vrushali
>>> > >>> >
>>> > >>> >
>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>> > >>> >
>>> > >>> >> +1 (binding).
>>> > >>> >>
>>> > >>> >> Kudos to all the team members for their great work!
>>> > >>> >>
>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>> > >>> development
>>> > >>> >> or review of most of the JIRAs'.
>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>> that
>>> > >>> there
>>> > >>> >> is no impact when ATSv2 is turned off.
>>> > >>> >>
>>> > >>> >> Regards,
>>> > >>> >> Varun Saxena.
>>> > >>> >>
>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>> > >>> >>
>>> > >>> >>> Hi folks,
>>> > >>> >>>
>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>> > merge
>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>> The
>>> > >>> vote
>>> > >>> >>> will
>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> > >>> >>>
>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>> Timeline
>>> > >>> Service
>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> > >>> >>>
>>> > >>> >>> Since then, we have been working on extending the capabilities
>>> of
>>> > >>> Timeline
>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>> > reasonably
>>> > >>> >>> confident that the state of the feature meets the criteria to
>>> be
>>> > >>> merged
>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>> test
>>> > >>> capacity
>>> > >>> >>> and provide valuable feedback so that we can make it
>>> > >>> production-ready.
>>> > >>> >>>
>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>> > scalability
>>> > >>> and
>>> > >>> >>> usability improvements based on a new architecture. What we
>>> would
>>> > >>> like to
>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>> has a
>>> > >>> >>> complete end-to-end read/write flow with security and read
>>> level
>>> > >>> >>> authorization via whitelists. You should be able to start
>>> setting
>>> > it
>>> > >>> up
>>> > >>> >>> and
>>> > >>> >>> testing it.
>>> > >>> >>>
>>> > >>> >>> At a high level, the following are the key features that have
>>> been
>>> > >>> >>> implemented since alpha1:
>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>> > >>> >>> - Read side simple authorization via whitelist
>>> > >>> >>> - Client configurable entity sort ordering
>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>>> > >>> metrics by
>>> > >>> >>> timerange, pagination, sub-app entities
>>> > >>> >>> - Support for storing sub-application entities (entities that
>>> exist
>>> > >>> >>> outside
>>> > >>> >>> the scope of an application)
>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>> table
>>> > >>> >>> prefixes,
>>> > >>> >>> configurable hbase cluster
>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>> > coprocessors
>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>> > >>> >>>
>>> > >>> >>> There are a total of 82 subtasks that were completed as part of
>>> > this
>>> > >>> >>> effort.
>>> > >>> >>>
>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>> > >>> Service v.2
>>> > >>> >>> does not impact existing functionality when disabled (by
>>> default).
>>> > >>> >>>
>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>> contributed
>>> > >>> towards
>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>> K S,
>>> > >>> Varun
>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>> Vavilapalli,
>>> > Joep
>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> > >>> >>>
>>> > >>> >>> Regards,
>>> > >>> >>> Vrushali
>>> > >>> >>>
>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>> rn-dev@hadoop.apache.org/msg27
>>> > >>> 383.html
>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> > >>> >>>
>>> > >>> >>
>>> > >>> >>
>>> > >>>
>>> > >>>
>>> > >>> ------------------------------------------------------------
>>> ---------
>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>> > >>>
>>> > >>>
>>> > >>
>>> > >
>>> >
>>>
>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Timeline Service v2 should be landing on branch2 shortly.
thanks
Vrushali
On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com> wrote:
> Thanks everyone.
>
> It has been over a week (~9 days) since TSv2 has been merged to trunk with
> no problems thus far. We are now thinking about merging timeline service v2
> to branch2 some time in the next few weeks.
>
> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
> with our trunk based feature branch YARN-5355. Varun Saxena has been
> diligently rebasing it to stay current with branch2.
>
> Currently we are in the process of testing it just like we did our due
> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
> branch2 code is a stable state to be merged.
>
> We will send out another email when we are ready to merge to branch2.
> thanks
> Vrushali
>
> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
> wrote:
>
>> Good to see this merged. I have initiated a separate thread with a
>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>> to the 2.9 release thread as soon as we reach consensus.
>>
>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>> wrote:
>>
>>> +1 to maintaining history.
>>>
>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>> varun.saxena.apache@gmail.com> wrote:
>>>
>>> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
>>> > Maintaining history I believe can be useful as it can make reverts
>>> > easier if at all required.
>>> > And can be an easy reference point to look at who had contributed what
>>> > without having to go back to the branch.
>>> >
>>> > Regards,
>>> > Varun Saxena.
>>> >
>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> >
>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>> think
>>> > > that helps answer Steve's questions.
>>> > >
>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>> merged
>>> > to
>>> > > trunk via "git merge --no-ff" .
>>> > >
>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>> control
>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>> > >
>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>> Steve?
>>> > >
>>> > > thanks
>>> > > Vrushali
>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>> 6afd9ef
>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>> pache.org%3E
>>> > >
>>> > >
>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>> wrote:
>>> > >
>>> > >> I recall this discussion about a couple of years ago:
>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>> > >> dev.hadoop.apache.org%3E
>>> > >>
>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>> stevel@hortonworks.com
>>> > >
>>> > >> wrote:
>>> > >>
>>> > >>> I'd have assumed it would have gone in as one single patch, rather
>>> than
>>> > >>> a full history. I don't see why the trunk needs all the
>>> evolutionary
>>> > >>> history of a build.
>>> > >>>
>>> > >>> What should our policy/process be here?
>>> > >>>
>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>> > >>>
>>> > >>>
>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> > >>> >
>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>> > >>> >
>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>> get on
>>> > >>> with
>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>> > >>> >
>>> > >>> > Regards
>>> > >>> > Vrushali
>>> > >>> >
>>> > >>> >
>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>> > >>> >
>>> > >>> >> +1 (binding).
>>> > >>> >>
>>> > >>> >> Kudos to all the team members for their great work!
>>> > >>> >>
>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>> > >>> development
>>> > >>> >> or review of most of the JIRAs'.
>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>> that
>>> > >>> there
>>> > >>> >> is no impact when ATSv2 is turned off.
>>> > >>> >>
>>> > >>> >> Regards,
>>> > >>> >> Varun Saxena.
>>> > >>> >>
>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>> > >>> >>
>>> > >>> >>> Hi folks,
>>> > >>> >>>
>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>> > merge
>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>> The
>>> > >>> vote
>>> > >>> >>> will
>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> > >>> >>>
>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>> Timeline
>>> > >>> Service
>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> > >>> >>>
>>> > >>> >>> Since then, we have been working on extending the capabilities
>>> of
>>> > >>> Timeline
>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>> > reasonably
>>> > >>> >>> confident that the state of the feature meets the criteria to
>>> be
>>> > >>> merged
>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>> test
>>> > >>> capacity
>>> > >>> >>> and provide valuable feedback so that we can make it
>>> > >>> production-ready.
>>> > >>> >>>
>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>> > scalability
>>> > >>> and
>>> > >>> >>> usability improvements based on a new architecture. What we
>>> would
>>> > >>> like to
>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>> has a
>>> > >>> >>> complete end-to-end read/write flow with security and read
>>> level
>>> > >>> >>> authorization via whitelists. You should be able to start
>>> setting
>>> > it
>>> > >>> up
>>> > >>> >>> and
>>> > >>> >>> testing it.
>>> > >>> >>>
>>> > >>> >>> At a high level, the following are the key features that have
>>> been
>>> > >>> >>> implemented since alpha1:
>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>> > >>> >>> - Read side simple authorization via whitelist
>>> > >>> >>> - Client configurable entity sort ordering
>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>>> > >>> metrics by
>>> > >>> >>> timerange, pagination, sub-app entities
>>> > >>> >>> - Support for storing sub-application entities (entities that
>>> exist
>>> > >>> >>> outside
>>> > >>> >>> the scope of an application)
>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>> table
>>> > >>> >>> prefixes,
>>> > >>> >>> configurable hbase cluster
>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>> > coprocessors
>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>> > >>> >>>
>>> > >>> >>> There are a total of 82 subtasks that were completed as part of
>>> > this
>>> > >>> >>> effort.
>>> > >>> >>>
>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>> > >>> Service v.2
>>> > >>> >>> does not impact existing functionality when disabled (by
>>> default).
>>> > >>> >>>
>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>> contributed
>>> > >>> towards
>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>> K S,
>>> > >>> Varun
>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>> Vavilapalli,
>>> > Joep
>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> > >>> >>>
>>> > >>> >>> Regards,
>>> > >>> >>> Vrushali
>>> > >>> >>>
>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>> rn-dev@hadoop.apache.org/msg27
>>> > >>> 383.html
>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> > >>> >>>
>>> > >>> >>
>>> > >>> >>
>>> > >>>
>>> > >>>
>>> > >>> ------------------------------------------------------------
>>> ---------
>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>> > >>>
>>> > >>>
>>> > >>
>>> > >
>>> >
>>>
>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Timeline Service v2 should be landing on branch2 shortly.
thanks
Vrushali
On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com> wrote:
> Thanks everyone.
>
> It has been over a week (~9 days) since TSv2 has been merged to trunk with
> no problems thus far. We are now thinking about merging timeline service v2
> to branch2 some time in the next few weeks.
>
> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
> with our trunk based feature branch YARN-5355. Varun Saxena has been
> diligently rebasing it to stay current with branch2.
>
> Currently we are in the process of testing it just like we did our due
> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
> branch2 code is a stable state to be merged.
>
> We will send out another email when we are ready to merge to branch2.
> thanks
> Vrushali
>
> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
> wrote:
>
>> Good to see this merged. I have initiated a separate thread with a
>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>> to the 2.9 release thread as soon as we reach consensus.
>>
>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>> wrote:
>>
>>> +1 to maintaining history.
>>>
>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>> varun.saxena.apache@gmail.com> wrote:
>>>
>>> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
>>> > Maintaining history I believe can be useful as it can make reverts
>>> > easier if at all required.
>>> > And can be an easy reference point to look at who had contributed what
>>> > without having to go back to the branch.
>>> >
>>> > Regards,
>>> > Varun Saxena.
>>> >
>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> >
>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>> think
>>> > > that helps answer Steve's questions.
>>> > >
>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>> merged
>>> > to
>>> > > trunk via "git merge --no-ff" .
>>> > >
>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>> control
>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>> > >
>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>> Steve?
>>> > >
>>> > > thanks
>>> > > Vrushali
>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>> 6afd9ef
>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>> pache.org%3E
>>> > >
>>> > >
>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>> wrote:
>>> > >
>>> > >> I recall this discussion about a couple of years ago:
>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>> > >> dev.hadoop.apache.org%3E
>>> > >>
>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>> stevel@hortonworks.com
>>> > >
>>> > >> wrote:
>>> > >>
>>> > >>> I'd have assumed it would have gone in as one single patch, rather
>>> than
>>> > >>> a full history. I don't see why the trunk needs all the
>>> evolutionary
>>> > >>> history of a build.
>>> > >>>
>>> > >>> What should our policy/process be here?
>>> > >>>
>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>> > >>>
>>> > >>>
>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> > >>> >
>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>> > >>> >
>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>> get on
>>> > >>> with
>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>> > >>> >
>>> > >>> > Regards
>>> > >>> > Vrushali
>>> > >>> >
>>> > >>> >
>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>> > >>> >
>>> > >>> >> +1 (binding).
>>> > >>> >>
>>> > >>> >> Kudos to all the team members for their great work!
>>> > >>> >>
>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>> > >>> development
>>> > >>> >> or review of most of the JIRAs'.
>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>> that
>>> > >>> there
>>> > >>> >> is no impact when ATSv2 is turned off.
>>> > >>> >>
>>> > >>> >> Regards,
>>> > >>> >> Varun Saxena.
>>> > >>> >>
>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>> > >>> >>
>>> > >>> >>> Hi folks,
>>> > >>> >>>
>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>> > merge
>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>> The
>>> > >>> vote
>>> > >>> >>> will
>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> > >>> >>>
>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>> Timeline
>>> > >>> Service
>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> > >>> >>>
>>> > >>> >>> Since then, we have been working on extending the capabilities
>>> of
>>> > >>> Timeline
>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>> > reasonably
>>> > >>> >>> confident that the state of the feature meets the criteria to
>>> be
>>> > >>> merged
>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>> test
>>> > >>> capacity
>>> > >>> >>> and provide valuable feedback so that we can make it
>>> > >>> production-ready.
>>> > >>> >>>
>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>> > scalability
>>> > >>> and
>>> > >>> >>> usability improvements based on a new architecture. What we
>>> would
>>> > >>> like to
>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>> has a
>>> > >>> >>> complete end-to-end read/write flow with security and read
>>> level
>>> > >>> >>> authorization via whitelists. You should be able to start
>>> setting
>>> > it
>>> > >>> up
>>> > >>> >>> and
>>> > >>> >>> testing it.
>>> > >>> >>>
>>> > >>> >>> At a high level, the following are the key features that have
>>> been
>>> > >>> >>> implemented since alpha1:
>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>> > >>> >>> - Read side simple authorization via whitelist
>>> > >>> >>> - Client configurable entity sort ordering
>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>>> > >>> metrics by
>>> > >>> >>> timerange, pagination, sub-app entities
>>> > >>> >>> - Support for storing sub-application entities (entities that
>>> exist
>>> > >>> >>> outside
>>> > >>> >>> the scope of an application)
>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>> table
>>> > >>> >>> prefixes,
>>> > >>> >>> configurable hbase cluster
>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>> > coprocessors
>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>> > >>> >>>
>>> > >>> >>> There are a total of 82 subtasks that were completed as part of
>>> > this
>>> > >>> >>> effort.
>>> > >>> >>>
>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>> > >>> Service v.2
>>> > >>> >>> does not impact existing functionality when disabled (by
>>> default).
>>> > >>> >>>
>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>> contributed
>>> > >>> towards
>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>> K S,
>>> > >>> Varun
>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>> Vavilapalli,
>>> > Joep
>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> > >>> >>>
>>> > >>> >>> Regards,
>>> > >>> >>> Vrushali
>>> > >>> >>>
>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>> rn-dev@hadoop.apache.org/msg27
>>> > >>> 383.html
>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> > >>> >>>
>>> > >>> >>
>>> > >>> >>
>>> > >>>
>>> > >>>
>>> > >>> ------------------------------------------------------------
>>> ---------
>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>> > >>>
>>> > >>>
>>> > >>
>>> > >
>>> >
>>>
>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Timeline Service v2 should be landing on branch2 shortly.
thanks
Vrushali
On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com> wrote:
> Thanks everyone.
>
> It has been over a week (~9 days) since TSv2 has been merged to trunk with
> no problems thus far. We are now thinking about merging timeline service v2
> to branch2 some time in the next few weeks.
>
> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
> with our trunk based feature branch YARN-5355. Varun Saxena has been
> diligently rebasing it to stay current with branch2.
>
> Currently we are in the process of testing it just like we did our due
> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
> branch2 code is a stable state to be merged.
>
> We will send out another email when we are ready to merge to branch2.
> thanks
> Vrushali
>
> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
> wrote:
>
>> Good to see this merged. I have initiated a separate thread with a
>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>> to the 2.9 release thread as soon as we reach consensus.
>>
>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>> wrote:
>>
>>> +1 to maintaining history.
>>>
>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>> varun.saxena.apache@gmail.com> wrote:
>>>
>>> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
>>> > Maintaining history I believe can be useful as it can make reverts
>>> > easier if at all required.
>>> > And can be an easy reference point to look at who had contributed what
>>> > without having to go back to the branch.
>>> >
>>> > Regards,
>>> > Varun Saxena.
>>> >
>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> >
>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>> think
>>> > > that helps answer Steve's questions.
>>> > >
>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>> merged
>>> > to
>>> > > trunk via "git merge --no-ff" .
>>> > >
>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>> control
>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>> > >
>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>> Steve?
>>> > >
>>> > > thanks
>>> > > Vrushali
>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>> 6afd9ef
>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>> pache.org%3E
>>> > >
>>> > >
>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>> wrote:
>>> > >
>>> > >> I recall this discussion about a couple of years ago:
>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>> > >> dev.hadoop.apache.org%3E
>>> > >>
>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>> stevel@hortonworks.com
>>> > >
>>> > >> wrote:
>>> > >>
>>> > >>> I'd have assumed it would have gone in as one single patch, rather
>>> than
>>> > >>> a full history. I don't see why the trunk needs all the
>>> evolutionary
>>> > >>> history of a build.
>>> > >>>
>>> > >>> What should our policy/process be here?
>>> > >>>
>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>> > >>>
>>> > >>>
>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> > >>> >
>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>> > >>> >
>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>> get on
>>> > >>> with
>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>> > >>> >
>>> > >>> > Regards
>>> > >>> > Vrushali
>>> > >>> >
>>> > >>> >
>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>> > >>> >
>>> > >>> >> +1 (binding).
>>> > >>> >>
>>> > >>> >> Kudos to all the team members for their great work!
>>> > >>> >>
>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>> > >>> development
>>> > >>> >> or review of most of the JIRAs'.
>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>> that
>>> > >>> there
>>> > >>> >> is no impact when ATSv2 is turned off.
>>> > >>> >>
>>> > >>> >> Regards,
>>> > >>> >> Varun Saxena.
>>> > >>> >>
>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>> > >>> >>
>>> > >>> >>> Hi folks,
>>> > >>> >>>
>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>> > merge
>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>> The
>>> > >>> vote
>>> > >>> >>> will
>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> > >>> >>>
>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>> Timeline
>>> > >>> Service
>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> > >>> >>>
>>> > >>> >>> Since then, we have been working on extending the capabilities
>>> of
>>> > >>> Timeline
>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>> > reasonably
>>> > >>> >>> confident that the state of the feature meets the criteria to
>>> be
>>> > >>> merged
>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>> test
>>> > >>> capacity
>>> > >>> >>> and provide valuable feedback so that we can make it
>>> > >>> production-ready.
>>> > >>> >>>
>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>> > scalability
>>> > >>> and
>>> > >>> >>> usability improvements based on a new architecture. What we
>>> would
>>> > >>> like to
>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>> has a
>>> > >>> >>> complete end-to-end read/write flow with security and read
>>> level
>>> > >>> >>> authorization via whitelists. You should be able to start
>>> setting
>>> > it
>>> > >>> up
>>> > >>> >>> and
>>> > >>> >>> testing it.
>>> > >>> >>>
>>> > >>> >>> At a high level, the following are the key features that have
>>> been
>>> > >>> >>> implemented since alpha1:
>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>> > >>> >>> - Read side simple authorization via whitelist
>>> > >>> >>> - Client configurable entity sort ordering
>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>>> > >>> metrics by
>>> > >>> >>> timerange, pagination, sub-app entities
>>> > >>> >>> - Support for storing sub-application entities (entities that
>>> exist
>>> > >>> >>> outside
>>> > >>> >>> the scope of an application)
>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>> table
>>> > >>> >>> prefixes,
>>> > >>> >>> configurable hbase cluster
>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>> > coprocessors
>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>> > >>> >>>
>>> > >>> >>> There are a total of 82 subtasks that were completed as part of
>>> > this
>>> > >>> >>> effort.
>>> > >>> >>>
>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>> > >>> Service v.2
>>> > >>> >>> does not impact existing functionality when disabled (by
>>> default).
>>> > >>> >>>
>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>> contributed
>>> > >>> towards
>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>> K S,
>>> > >>> Varun
>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>> Vavilapalli,
>>> > Joep
>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> > >>> >>>
>>> > >>> >>> Regards,
>>> > >>> >>> Vrushali
>>> > >>> >>>
>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>> rn-dev@hadoop.apache.org/msg27
>>> > >>> 383.html
>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> > >>> >>>
>>> > >>> >>
>>> > >>> >>
>>> > >>>
>>> > >>>
>>> > >>> ------------------------------------------------------------
>>> ---------
>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>> > >>>
>>> > >>>
>>> > >>
>>> > >
>>> >
>>>
>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Thanks everyone.
It has been over a week (~9 days) since TSv2 has been merged to trunk with
no problems thus far. We are now thinking about merging timeline service v2
to branch2 some time in the next few weeks.
So far, we have been maintaining a branch2 based YARN-5355_branch2 along
with our trunk based feature branch YARN-5355. Varun Saxena has been
diligently rebasing it to stay current with branch2.
Currently we are in the process of testing it just like we did our due
diligence with the trunk based YARN-5355 branch and will ensure the TSv2
branch2 code is a stable state to be merged.
We will send out another email when we are ready to merge to branch2.
thanks
Vrushali
On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
wrote:
> Good to see this merged. I have initiated a separate thread with a smaller
> set of stakeholders to discuss inclusion in 2.9. We'll report back to the
> 2.9 release thread as soon as we reach consensus.
>
> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
> wrote:
>
>> +1 to maintaining history.
>>
>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>> varun.saxena.apache@gmail.com> wrote:
>>
>> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
>> > Maintaining history I believe can be useful as it can make reverts
>> > easier if at all required.
>> > And can be an easy reference point to look at who had contributed what
>> > without having to go back to the branch.
>> >
>> > Regards,
>> > Varun Saxena.
>> >
>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>> > wrote:
>> >
>> > > Thanks Sangjin for the link to the previous discussions on this! I
>> think
>> > > that helps answer Steve's questions.
>> > >
>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>> merged
>> > to
>> > > trunk via "git merge --no-ff" .
>> > >
>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>> > > chose to develop on a feature branch YARN-5355 so that we could
>> control
>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>> > >
>> > > Is the latest merge causing any conflicts or issues for s3guard,
>> Steve?
>> > >
>> > > thanks
>> > > Vrushali
>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>> 6afd9ef
>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org
>> %3E
>> > >
>> > >
>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>> wrote:
>> > >
>> > >> I recall this discussion about a couple of years ago:
>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>> > >> dev.hadoop.apache.org%3E
>> > >>
>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>> stevel@hortonworks.com
>> > >
>> > >> wrote:
>> > >>
>> > >>> I'd have assumed it would have gone in as one single patch, rather
>> than
>> > >>> a full history. I don't see why the trunk needs all the evolutionary
>> > >>> history of a build.
>> > >>>
>> > >>> What should our policy/process be here?
>> > >>>
>> > >>> I do currently plan to merge the s3guard in as one single squashed
>> > >>> patch; just getting HADOOP-14809 sorted first.
>> > >>>
>> > >>>
>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>> > wrote:
>> > >>> >
>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>> > >>> >
>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get
>> on
>> > >>> with
>> > >>> > the merge to trunk shortly. Thanks everyone!
>> > >>> >
>> > >>> > Regards
>> > >>> > Vrushali
>> > >>> >
>> > >>> >
>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>> > >>> > varun.saxena.apache@gmail.com> wrote:
>> > >>> >
>> > >>> >> +1 (binding).
>> > >>> >>
>> > >>> >> Kudos to all the team members for their great work!
>> > >>> >>
>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>> > >>> development
>> > >>> >> or review of most of the JIRAs'.
>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>> that
>> > >>> there
>> > >>> >> is no impact when ATSv2 is turned off.
>> > >>> >>
>> > >>> >> Regards,
>> > >>> >> Varun Saxena.
>> > >>> >>
>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> > >>> >> vrushalic2016@gmail.com> wrote:
>> > >>> >>
>> > >>> >>> Hi folks,
>> > >>> >>>
>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>> > merge
>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>> The
>> > >>> vote
>> > >>> >>> will
>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>> > >>> >>>
>> > >>> >>> We have previously completed one merge onto trunk [3] and
>> Timeline
>> > >>> Service
>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>> > >>> >>>
>> > >>> >>> Since then, we have been working on extending the capabilities
>> of
>> > >>> Timeline
>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>> > reasonably
>> > >>> >>> confident that the state of the feature meets the criteria to be
>> > >>> merged
>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>> test
>> > >>> capacity
>> > >>> >>> and provide valuable feedback so that we can make it
>> > >>> production-ready.
>> > >>> >>>
>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>> > scalability
>> > >>> and
>> > >>> >>> usability improvements based on a new architecture. What we
>> would
>> > >>> like to
>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>> has a
>> > >>> >>> complete end-to-end read/write flow with security and read level
>> > >>> >>> authorization via whitelists. You should be able to start
>> setting
>> > it
>> > >>> up
>> > >>> >>> and
>> > >>> >>> testing it.
>> > >>> >>>
>> > >>> >>> At a high level, the following are the key features that have
>> been
>> > >>> >>> implemented since alpha1:
>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>> > >>> >>> - Read side simple authorization via whitelist
>> > >>> >>> - Client configurable entity sort ordering
>> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>> > >>> metrics by
>> > >>> >>> timerange, pagination, sub-app entities
>> > >>> >>> - Support for storing sub-application entities (entities that
>> exist
>> > >>> >>> outside
>> > >>> >>> the scope of an application)
>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>> table
>> > >>> >>> prefixes,
>> > >>> >>> configurable hbase cluster
>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>> > coprocessors
>> > >>> >>> - Uses latest stable HBase release 1.2.6
>> > >>> >>>
>> > >>> >>> There are a total of 82 subtasks that were completed as part of
>> > this
>> > >>> >>> effort.
>> > >>> >>>
>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>> > >>> Service v.2
>> > >>> >>> does not impact existing functionality when disabled (by
>> default).
>> > >>> >>>
>> > >>> >>> Special thanks to a team of folks who worked hard and
>> contributed
>> > >>> towards
>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K
>> S,
>> > >>> Varun
>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
>> > Joep
>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> > >>> >>>
>> > >>> >>> Regards,
>> > >>> >>> Vrushali
>> > >>> >>>
>> > >>> >>> [1] http://www.mail-archive.com/ya
>> rn-dev@hadoop.apache.org/msg27
>> > >>> 383.html
>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>> > >>> >>>
>> > >>> >>
>> > >>> >>
>> > >>>
>> > >>>
>> > >>> ------------------------------------------------------------
>> ---------
>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>> > >>>
>> > >>>
>> > >>
>> > >
>> >
>>
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Thanks everyone.
It has been over a week (~9 days) since TSv2 has been merged to trunk with
no problems thus far. We are now thinking about merging timeline service v2
to branch2 some time in the next few weeks.
So far, we have been maintaining a branch2 based YARN-5355_branch2 along
with our trunk based feature branch YARN-5355. Varun Saxena has been
diligently rebasing it to stay current with branch2.
Currently we are in the process of testing it just like we did our due
diligence with the trunk based YARN-5355 branch and will ensure the TSv2
branch2 code is a stable state to be merged.
We will send out another email when we are ready to merge to branch2.
thanks
Vrushali
On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
wrote:
> Good to see this merged. I have initiated a separate thread with a smaller
> set of stakeholders to discuss inclusion in 2.9. We'll report back to the
> 2.9 release thread as soon as we reach consensus.
>
> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
> wrote:
>
>> +1 to maintaining history.
>>
>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>> varun.saxena.apache@gmail.com> wrote:
>>
>> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
>> > Maintaining history I believe can be useful as it can make reverts
>> > easier if at all required.
>> > And can be an easy reference point to look at who had contributed what
>> > without having to go back to the branch.
>> >
>> > Regards,
>> > Varun Saxena.
>> >
>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>> > wrote:
>> >
>> > > Thanks Sangjin for the link to the previous discussions on this! I
>> think
>> > > that helps answer Steve's questions.
>> > >
>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>> merged
>> > to
>> > > trunk via "git merge --no-ff" .
>> > >
>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>> > > chose to develop on a feature branch YARN-5355 so that we could
>> control
>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>> > >
>> > > Is the latest merge causing any conflicts or issues for s3guard,
>> Steve?
>> > >
>> > > thanks
>> > > Vrushali
>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>> 6afd9ef
>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org
>> %3E
>> > >
>> > >
>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>> wrote:
>> > >
>> > >> I recall this discussion about a couple of years ago:
>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>> > >> dev.hadoop.apache.org%3E
>> > >>
>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>> stevel@hortonworks.com
>> > >
>> > >> wrote:
>> > >>
>> > >>> I'd have assumed it would have gone in as one single patch, rather
>> than
>> > >>> a full history. I don't see why the trunk needs all the evolutionary
>> > >>> history of a build.
>> > >>>
>> > >>> What should our policy/process be here?
>> > >>>
>> > >>> I do currently plan to merge the s3guard in as one single squashed
>> > >>> patch; just getting HADOOP-14809 sorted first.
>> > >>>
>> > >>>
>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>> > wrote:
>> > >>> >
>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>> > >>> >
>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get
>> on
>> > >>> with
>> > >>> > the merge to trunk shortly. Thanks everyone!
>> > >>> >
>> > >>> > Regards
>> > >>> > Vrushali
>> > >>> >
>> > >>> >
>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>> > >>> > varun.saxena.apache@gmail.com> wrote:
>> > >>> >
>> > >>> >> +1 (binding).
>> > >>> >>
>> > >>> >> Kudos to all the team members for their great work!
>> > >>> >>
>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>> > >>> development
>> > >>> >> or review of most of the JIRAs'.
>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>> that
>> > >>> there
>> > >>> >> is no impact when ATSv2 is turned off.
>> > >>> >>
>> > >>> >> Regards,
>> > >>> >> Varun Saxena.
>> > >>> >>
>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> > >>> >> vrushalic2016@gmail.com> wrote:
>> > >>> >>
>> > >>> >>> Hi folks,
>> > >>> >>>
>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>> > merge
>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>> The
>> > >>> vote
>> > >>> >>> will
>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>> > >>> >>>
>> > >>> >>> We have previously completed one merge onto trunk [3] and
>> Timeline
>> > >>> Service
>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>> > >>> >>>
>> > >>> >>> Since then, we have been working on extending the capabilities
>> of
>> > >>> Timeline
>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>> > reasonably
>> > >>> >>> confident that the state of the feature meets the criteria to be
>> > >>> merged
>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>> test
>> > >>> capacity
>> > >>> >>> and provide valuable feedback so that we can make it
>> > >>> production-ready.
>> > >>> >>>
>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>> > scalability
>> > >>> and
>> > >>> >>> usability improvements based on a new architecture. What we
>> would
>> > >>> like to
>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>> has a
>> > >>> >>> complete end-to-end read/write flow with security and read level
>> > >>> >>> authorization via whitelists. You should be able to start
>> setting
>> > it
>> > >>> up
>> > >>> >>> and
>> > >>> >>> testing it.
>> > >>> >>>
>> > >>> >>> At a high level, the following are the key features that have
>> been
>> > >>> >>> implemented since alpha1:
>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>> > >>> >>> - Read side simple authorization via whitelist
>> > >>> >>> - Client configurable entity sort ordering
>> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>> > >>> metrics by
>> > >>> >>> timerange, pagination, sub-app entities
>> > >>> >>> - Support for storing sub-application entities (entities that
>> exist
>> > >>> >>> outside
>> > >>> >>> the scope of an application)
>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>> table
>> > >>> >>> prefixes,
>> > >>> >>> configurable hbase cluster
>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>> > coprocessors
>> > >>> >>> - Uses latest stable HBase release 1.2.6
>> > >>> >>>
>> > >>> >>> There are a total of 82 subtasks that were completed as part of
>> > this
>> > >>> >>> effort.
>> > >>> >>>
>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>> > >>> Service v.2
>> > >>> >>> does not impact existing functionality when disabled (by
>> default).
>> > >>> >>>
>> > >>> >>> Special thanks to a team of folks who worked hard and
>> contributed
>> > >>> towards
>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K
>> S,
>> > >>> Varun
>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
>> > Joep
>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> > >>> >>>
>> > >>> >>> Regards,
>> > >>> >>> Vrushali
>> > >>> >>>
>> > >>> >>> [1] http://www.mail-archive.com/ya
>> rn-dev@hadoop.apache.org/msg27
>> > >>> 383.html
>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>> > >>> >>>
>> > >>> >>
>> > >>> >>
>> > >>>
>> > >>>
>> > >>> ------------------------------------------------------------
>> ---------
>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>> > >>>
>> > >>>
>> > >>
>> > >
>> >
>>
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Thanks everyone.
It has been over a week (~9 days) since TSv2 has been merged to trunk with
no problems thus far. We are now thinking about merging timeline service v2
to branch2 some time in the next few weeks.
So far, we have been maintaining a branch2 based YARN-5355_branch2 along
with our trunk based feature branch YARN-5355. Varun Saxena has been
diligently rebasing it to stay current with branch2.
Currently we are in the process of testing it just like we did our due
diligence with the trunk based YARN-5355 branch and will ensure the TSv2
branch2 code is a stable state to be merged.
We will send out another email when we are ready to merge to branch2.
thanks
Vrushali
On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
wrote:
> Good to see this merged. I have initiated a separate thread with a smaller
> set of stakeholders to discuss inclusion in 2.9. We'll report back to the
> 2.9 release thread as soon as we reach consensus.
>
> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
> wrote:
>
>> +1 to maintaining history.
>>
>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>> varun.saxena.apache@gmail.com> wrote:
>>
>> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
>> > Maintaining history I believe can be useful as it can make reverts
>> > easier if at all required.
>> > And can be an easy reference point to look at who had contributed what
>> > without having to go back to the branch.
>> >
>> > Regards,
>> > Varun Saxena.
>> >
>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>> > wrote:
>> >
>> > > Thanks Sangjin for the link to the previous discussions on this! I
>> think
>> > > that helps answer Steve's questions.
>> > >
>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>> merged
>> > to
>> > > trunk via "git merge --no-ff" .
>> > >
>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>> > > chose to develop on a feature branch YARN-5355 so that we could
>> control
>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>> > >
>> > > Is the latest merge causing any conflicts or issues for s3guard,
>> Steve?
>> > >
>> > > thanks
>> > > Vrushali
>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>> 6afd9ef
>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org
>> %3E
>> > >
>> > >
>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>> wrote:
>> > >
>> > >> I recall this discussion about a couple of years ago:
>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>> > >> dev.hadoop.apache.org%3E
>> > >>
>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>> stevel@hortonworks.com
>> > >
>> > >> wrote:
>> > >>
>> > >>> I'd have assumed it would have gone in as one single patch, rather
>> than
>> > >>> a full history. I don't see why the trunk needs all the evolutionary
>> > >>> history of a build.
>> > >>>
>> > >>> What should our policy/process be here?
>> > >>>
>> > >>> I do currently plan to merge the s3guard in as one single squashed
>> > >>> patch; just getting HADOOP-14809 sorted first.
>> > >>>
>> > >>>
>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>> > wrote:
>> > >>> >
>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>> > >>> >
>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get
>> on
>> > >>> with
>> > >>> > the merge to trunk shortly. Thanks everyone!
>> > >>> >
>> > >>> > Regards
>> > >>> > Vrushali
>> > >>> >
>> > >>> >
>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>> > >>> > varun.saxena.apache@gmail.com> wrote:
>> > >>> >
>> > >>> >> +1 (binding).
>> > >>> >>
>> > >>> >> Kudos to all the team members for their great work!
>> > >>> >>
>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>> > >>> development
>> > >>> >> or review of most of the JIRAs'.
>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>> that
>> > >>> there
>> > >>> >> is no impact when ATSv2 is turned off.
>> > >>> >>
>> > >>> >> Regards,
>> > >>> >> Varun Saxena.
>> > >>> >>
>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> > >>> >> vrushalic2016@gmail.com> wrote:
>> > >>> >>
>> > >>> >>> Hi folks,
>> > >>> >>>
>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>> > merge
>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>> The
>> > >>> vote
>> > >>> >>> will
>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>> > >>> >>>
>> > >>> >>> We have previously completed one merge onto trunk [3] and
>> Timeline
>> > >>> Service
>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>> > >>> >>>
>> > >>> >>> Since then, we have been working on extending the capabilities
>> of
>> > >>> Timeline
>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>> > reasonably
>> > >>> >>> confident that the state of the feature meets the criteria to be
>> > >>> merged
>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>> test
>> > >>> capacity
>> > >>> >>> and provide valuable feedback so that we can make it
>> > >>> production-ready.
>> > >>> >>>
>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>> > scalability
>> > >>> and
>> > >>> >>> usability improvements based on a new architecture. What we
>> would
>> > >>> like to
>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>> has a
>> > >>> >>> complete end-to-end read/write flow with security and read level
>> > >>> >>> authorization via whitelists. You should be able to start
>> setting
>> > it
>> > >>> up
>> > >>> >>> and
>> > >>> >>> testing it.
>> > >>> >>>
>> > >>> >>> At a high level, the following are the key features that have
>> been
>> > >>> >>> implemented since alpha1:
>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>> > >>> >>> - Read side simple authorization via whitelist
>> > >>> >>> - Client configurable entity sort ordering
>> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>> > >>> metrics by
>> > >>> >>> timerange, pagination, sub-app entities
>> > >>> >>> - Support for storing sub-application entities (entities that
>> exist
>> > >>> >>> outside
>> > >>> >>> the scope of an application)
>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>> table
>> > >>> >>> prefixes,
>> > >>> >>> configurable hbase cluster
>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>> > coprocessors
>> > >>> >>> - Uses latest stable HBase release 1.2.6
>> > >>> >>>
>> > >>> >>> There are a total of 82 subtasks that were completed as part of
>> > this
>> > >>> >>> effort.
>> > >>> >>>
>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>> > >>> Service v.2
>> > >>> >>> does not impact existing functionality when disabled (by
>> default).
>> > >>> >>>
>> > >>> >>> Special thanks to a team of folks who worked hard and
>> contributed
>> > >>> towards
>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K
>> S,
>> > >>> Varun
>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
>> > Joep
>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> > >>> >>>
>> > >>> >>> Regards,
>> > >>> >>> Vrushali
>> > >>> >>>
>> > >>> >>> [1] http://www.mail-archive.com/ya
>> rn-dev@hadoop.apache.org/msg27
>> > >>> 383.html
>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>> > >>> >>>
>> > >>> >>
>> > >>> >>
>> > >>>
>> > >>>
>> > >>> ------------------------------------------------------------
>> ---------
>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>> > >>>
>> > >>>
>> > >>
>> > >
>> >
>>
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Thanks everyone.
It has been over a week (~9 days) since TSv2 has been merged to trunk with
no problems thus far. We are now thinking about merging timeline service v2
to branch2 some time in the next few weeks.
So far, we have been maintaining a branch2 based YARN-5355_branch2 along
with our trunk based feature branch YARN-5355. Varun Saxena has been
diligently rebasing it to stay current with branch2.
Currently we are in the process of testing it just like we did our due
diligence with the trunk based YARN-5355 branch and will ensure the TSv2
branch2 code is a stable state to be merged.
We will send out another email when we are ready to merge to branch2.
thanks
Vrushali
On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
wrote:
> Good to see this merged. I have initiated a separate thread with a smaller
> set of stakeholders to discuss inclusion in 2.9. We'll report back to the
> 2.9 release thread as soon as we reach consensus.
>
> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
> wrote:
>
>> +1 to maintaining history.
>>
>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>> varun.saxena.apache@gmail.com> wrote:
>>
>> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
>> > Maintaining history I believe can be useful as it can make reverts
>> > easier if at all required.
>> > And can be an easy reference point to look at who had contributed what
>> > without having to go back to the branch.
>> >
>> > Regards,
>> > Varun Saxena.
>> >
>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>> > wrote:
>> >
>> > > Thanks Sangjin for the link to the previous discussions on this! I
>> think
>> > > that helps answer Steve's questions.
>> > >
>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>> merged
>> > to
>> > > trunk via "git merge --no-ff" .
>> > >
>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>> > > chose to develop on a feature branch YARN-5355 so that we could
>> control
>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>> > >
>> > > Is the latest merge causing any conflicts or issues for s3guard,
>> Steve?
>> > >
>> > > thanks
>> > > Vrushali
>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>> 6afd9ef
>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org
>> %3E
>> > >
>> > >
>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>> wrote:
>> > >
>> > >> I recall this discussion about a couple of years ago:
>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>> > >> dev.hadoop.apache.org%3E
>> > >>
>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>> stevel@hortonworks.com
>> > >
>> > >> wrote:
>> > >>
>> > >>> I'd have assumed it would have gone in as one single patch, rather
>> than
>> > >>> a full history. I don't see why the trunk needs all the evolutionary
>> > >>> history of a build.
>> > >>>
>> > >>> What should our policy/process be here?
>> > >>>
>> > >>> I do currently plan to merge the s3guard in as one single squashed
>> > >>> patch; just getting HADOOP-14809 sorted first.
>> > >>>
>> > >>>
>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>> > wrote:
>> > >>> >
>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>> > >>> >
>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get
>> on
>> > >>> with
>> > >>> > the merge to trunk shortly. Thanks everyone!
>> > >>> >
>> > >>> > Regards
>> > >>> > Vrushali
>> > >>> >
>> > >>> >
>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>> > >>> > varun.saxena.apache@gmail.com> wrote:
>> > >>> >
>> > >>> >> +1 (binding).
>> > >>> >>
>> > >>> >> Kudos to all the team members for their great work!
>> > >>> >>
>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>> > >>> development
>> > >>> >> or review of most of the JIRAs'.
>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>> that
>> > >>> there
>> > >>> >> is no impact when ATSv2 is turned off.
>> > >>> >>
>> > >>> >> Regards,
>> > >>> >> Varun Saxena.
>> > >>> >>
>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> > >>> >> vrushalic2016@gmail.com> wrote:
>> > >>> >>
>> > >>> >>> Hi folks,
>> > >>> >>>
>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>> > merge
>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>> The
>> > >>> vote
>> > >>> >>> will
>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>> > >>> >>>
>> > >>> >>> We have previously completed one merge onto trunk [3] and
>> Timeline
>> > >>> Service
>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>> > >>> >>>
>> > >>> >>> Since then, we have been working on extending the capabilities
>> of
>> > >>> Timeline
>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>> > reasonably
>> > >>> >>> confident that the state of the feature meets the criteria to be
>> > >>> merged
>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>> test
>> > >>> capacity
>> > >>> >>> and provide valuable feedback so that we can make it
>> > >>> production-ready.
>> > >>> >>>
>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>> > scalability
>> > >>> and
>> > >>> >>> usability improvements based on a new architecture. What we
>> would
>> > >>> like to
>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>> has a
>> > >>> >>> complete end-to-end read/write flow with security and read level
>> > >>> >>> authorization via whitelists. You should be able to start
>> setting
>> > it
>> > >>> up
>> > >>> >>> and
>> > >>> >>> testing it.
>> > >>> >>>
>> > >>> >>> At a high level, the following are the key features that have
>> been
>> > >>> >>> implemented since alpha1:
>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>> > >>> >>> - Read side simple authorization via whitelist
>> > >>> >>> - Client configurable entity sort ordering
>> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>> > >>> metrics by
>> > >>> >>> timerange, pagination, sub-app entities
>> > >>> >>> - Support for storing sub-application entities (entities that
>> exist
>> > >>> >>> outside
>> > >>> >>> the scope of an application)
>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>> table
>> > >>> >>> prefixes,
>> > >>> >>> configurable hbase cluster
>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>> > coprocessors
>> > >>> >>> - Uses latest stable HBase release 1.2.6
>> > >>> >>>
>> > >>> >>> There are a total of 82 subtasks that were completed as part of
>> > this
>> > >>> >>> effort.
>> > >>> >>>
>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>> > >>> Service v.2
>> > >>> >>> does not impact existing functionality when disabled (by
>> default).
>> > >>> >>>
>> > >>> >>> Special thanks to a team of folks who worked hard and
>> contributed
>> > >>> towards
>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K
>> S,
>> > >>> Varun
>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
>> > Joep
>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> > >>> >>>
>> > >>> >>> Regards,
>> > >>> >>> Vrushali
>> > >>> >>>
>> > >>> >>> [1] http://www.mail-archive.com/ya
>> rn-dev@hadoop.apache.org/msg27
>> > >>> 383.html
>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>> > >>> >>>
>> > >>> >>
>> > >>> >>
>> > >>>
>> > >>>
>> > >>> ------------------------------------------------------------
>> ---------
>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>> > >>>
>> > >>>
>> > >>
>> > >
>> >
>>
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Subramaniam V K <su...@gmail.com>.
Good to see this merged. I have initiated a separate thread with a smaller
set of stakeholders to discuss inclusion in 2.9. We'll report back to the
2.9 release thread as soon as we reach consensus.
On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com> wrote:
> +1 to maintaining history.
>
> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
> > Maintaining history I believe can be useful as it can make reverts
> > easier if at all required.
> > And can be an easy reference point to look at who had contributed what
> > without having to go back to the branch.
> >
> > Regards,
> > Varun Saxena.
> >
> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
> > wrote:
> >
> > > Thanks Sangjin for the link to the previous discussions on this! I
> think
> > > that helps answer Steve's questions.
> > >
> > > As decided on that thread [1], YARN-5355 as a feature branch was merged
> > to
> > > trunk via "git merge --no-ff" .
> > >
> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
> > > chose to develop on a feature branch YARN-5355 so that we could control
> > > when changes went into trunk and didn't inadvertently disrupt trunk.
> > >
> > > Is the latest merge causing any conflicts or issues for s3guard, Steve?
> > >
> > > thanks
> > > Vrushali
> > > [1] https://lists.apache.org/thread.html/
> 43cd65c6b6c3c0e8ac2b3c76afd9ef
> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org
> %3E
> > >
> > >
> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
> > >
> > >> I recall this discussion about a couple of years ago:
> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
> > >> dev.hadoop.apache.org%3E
> > >>
> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
> stevel@hortonworks.com
> > >
> > >> wrote:
> > >>
> > >>> I'd have assumed it would have gone in as one single patch, rather
> than
> > >>> a full history. I don't see why the trunk needs all the evolutionary
> > >>> history of a build.
> > >>>
> > >>> What should our policy/process be here?
> > >>>
> > >>> I do currently plan to merge the s3guard in as one single squashed
> > >>> patch; just getting HADOOP-14809 sorted first.
> > >>>
> > >>>
> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
> > wrote:
> > >>> >
> > >>> > I'm adding my +1 (binding) to conclude the vote.
> > >>> >
> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get
> on
> > >>> with
> > >>> > the merge to trunk shortly. Thanks everyone!
> > >>> >
> > >>> > Regards
> > >>> > Vrushali
> > >>> >
> > >>> >
> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> > >>> > varun.saxena.apache@gmail.com> wrote:
> > >>> >
> > >>> >> +1 (binding).
> > >>> >>
> > >>> >> Kudos to all the team members for their great work!
> > >>> >>
> > >>> >> Being part of the ATSv2 team, I have been involved with either
> > >>> development
> > >>> >> or review of most of the JIRAs'.
> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
> that
> > >>> there
> > >>> >> is no impact when ATSv2 is turned off.
> > >>> >>
> > >>> >> Regards,
> > >>> >> Varun Saxena.
> > >>> >>
> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >>> >> vrushalic2016@gmail.com> wrote:
> > >>> >>
> > >>> >>> Hi folks,
> > >>> >>>
> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
> > merge
> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> > >>> vote
> > >>> >>> will
> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>> >>>
> > >>> >>> We have previously completed one merge onto trunk [3] and
> Timeline
> > >>> Service
> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>> >>>
> > >>> >>> Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
> > reasonably
> > >>> >>> confident that the state of the feature meets the criteria to be
> > >>> merged
> > >>> >>> onto trunk and we'd love folks to get their hands on it in a test
> > >>> capacity
> > >>> >>> and provide valuable feedback so that we can make it
> > >>> production-ready.
> > >>> >>>
> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
> > scalability
> > >>> and
> > >>> >>> usability improvements based on a new architecture. What we would
> > >>> like to
> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
> has a
> > >>> >>> complete end-to-end read/write flow with security and read level
> > >>> >>> authorization via whitelists. You should be able to start setting
> > it
> > >>> up
> > >>> >>> and
> > >>> >>> testing it.
> > >>> >>>
> > >>> >>> At a high level, the following are the key features that have
> been
> > >>> >>> implemented since alpha1:
> > >>> >>> - Security via Kerberos Authentication and delegation tokens
> > >>> >>> - Read side simple authorization via whitelist
> > >>> >>> - Client configurable entity sort ordering
> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> > >>> metrics by
> > >>> >>> timerange, pagination, sub-app entities
> > >>> >>> - Support for storing sub-application entities (entities that
> exist
> > >>> >>> outside
> > >>> >>> the scope of an application)
> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> > >>> >>> prefixes,
> > >>> >>> configurable hbase cluster
> > >>> >>> - Flow level aggregations done as dynamic (table level)
> > coprocessors
> > >>> >>> - Uses latest stable HBase release 1.2.6
> > >>> >>>
> > >>> >>> There are a total of 82 subtasks that were completed as part of
> > this
> > >>> >>> effort.
> > >>> >>>
> > >>> >>> We paid close attention to ensure that once disabled Timeline
> > >>> Service v.2
> > >>> >>> does not impact existing functionality when disabled (by
> default).
> > >>> >>>
> > >>> >>> Special thanks to a team of folks who worked hard and contributed
> > >>> towards
> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K
> S,
> > >>> Varun
> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> > Joep
> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>> >>>
> > >>> >>> Regards,
> > >>> >>> Vrushali
> > >>> >>>
> > >>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > >>> 383.html
> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >>> >>>
> > >>> >>
> > >>> >>
> > >>>
> > >>>
> > >>> ------------------------------------------------------------
> ---------
> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >>>
> > >>>
> > >>
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Subramaniam V K <su...@gmail.com>.
Good to see this merged. I have initiated a separate thread with a smaller
set of stakeholders to discuss inclusion in 2.9. We'll report back to the
2.9 release thread as soon as we reach consensus.
On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com> wrote:
> +1 to maintaining history.
>
> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
> > Maintaining history I believe can be useful as it can make reverts
> > easier if at all required.
> > And can be an easy reference point to look at who had contributed what
> > without having to go back to the branch.
> >
> > Regards,
> > Varun Saxena.
> >
> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
> > wrote:
> >
> > > Thanks Sangjin for the link to the previous discussions on this! I
> think
> > > that helps answer Steve's questions.
> > >
> > > As decided on that thread [1], YARN-5355 as a feature branch was merged
> > to
> > > trunk via "git merge --no-ff" .
> > >
> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
> > > chose to develop on a feature branch YARN-5355 so that we could control
> > > when changes went into trunk and didn't inadvertently disrupt trunk.
> > >
> > > Is the latest merge causing any conflicts or issues for s3guard, Steve?
> > >
> > > thanks
> > > Vrushali
> > > [1] https://lists.apache.org/thread.html/
> 43cd65c6b6c3c0e8ac2b3c76afd9ef
> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org
> %3E
> > >
> > >
> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
> > >
> > >> I recall this discussion about a couple of years ago:
> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
> > >> dev.hadoop.apache.org%3E
> > >>
> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
> stevel@hortonworks.com
> > >
> > >> wrote:
> > >>
> > >>> I'd have assumed it would have gone in as one single patch, rather
> than
> > >>> a full history. I don't see why the trunk needs all the evolutionary
> > >>> history of a build.
> > >>>
> > >>> What should our policy/process be here?
> > >>>
> > >>> I do currently plan to merge the s3guard in as one single squashed
> > >>> patch; just getting HADOOP-14809 sorted first.
> > >>>
> > >>>
> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
> > wrote:
> > >>> >
> > >>> > I'm adding my +1 (binding) to conclude the vote.
> > >>> >
> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get
> on
> > >>> with
> > >>> > the merge to trunk shortly. Thanks everyone!
> > >>> >
> > >>> > Regards
> > >>> > Vrushali
> > >>> >
> > >>> >
> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> > >>> > varun.saxena.apache@gmail.com> wrote:
> > >>> >
> > >>> >> +1 (binding).
> > >>> >>
> > >>> >> Kudos to all the team members for their great work!
> > >>> >>
> > >>> >> Being part of the ATSv2 team, I have been involved with either
> > >>> development
> > >>> >> or review of most of the JIRAs'.
> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
> that
> > >>> there
> > >>> >> is no impact when ATSv2 is turned off.
> > >>> >>
> > >>> >> Regards,
> > >>> >> Varun Saxena.
> > >>> >>
> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >>> >> vrushalic2016@gmail.com> wrote:
> > >>> >>
> > >>> >>> Hi folks,
> > >>> >>>
> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
> > merge
> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> > >>> vote
> > >>> >>> will
> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>> >>>
> > >>> >>> We have previously completed one merge onto trunk [3] and
> Timeline
> > >>> Service
> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>> >>>
> > >>> >>> Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
> > reasonably
> > >>> >>> confident that the state of the feature meets the criteria to be
> > >>> merged
> > >>> >>> onto trunk and we'd love folks to get their hands on it in a test
> > >>> capacity
> > >>> >>> and provide valuable feedback so that we can make it
> > >>> production-ready.
> > >>> >>>
> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
> > scalability
> > >>> and
> > >>> >>> usability improvements based on a new architecture. What we would
> > >>> like to
> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
> has a
> > >>> >>> complete end-to-end read/write flow with security and read level
> > >>> >>> authorization via whitelists. You should be able to start setting
> > it
> > >>> up
> > >>> >>> and
> > >>> >>> testing it.
> > >>> >>>
> > >>> >>> At a high level, the following are the key features that have
> been
> > >>> >>> implemented since alpha1:
> > >>> >>> - Security via Kerberos Authentication and delegation tokens
> > >>> >>> - Read side simple authorization via whitelist
> > >>> >>> - Client configurable entity sort ordering
> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> > >>> metrics by
> > >>> >>> timerange, pagination, sub-app entities
> > >>> >>> - Support for storing sub-application entities (entities that
> exist
> > >>> >>> outside
> > >>> >>> the scope of an application)
> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> > >>> >>> prefixes,
> > >>> >>> configurable hbase cluster
> > >>> >>> - Flow level aggregations done as dynamic (table level)
> > coprocessors
> > >>> >>> - Uses latest stable HBase release 1.2.6
> > >>> >>>
> > >>> >>> There are a total of 82 subtasks that were completed as part of
> > this
> > >>> >>> effort.
> > >>> >>>
> > >>> >>> We paid close attention to ensure that once disabled Timeline
> > >>> Service v.2
> > >>> >>> does not impact existing functionality when disabled (by
> default).
> > >>> >>>
> > >>> >>> Special thanks to a team of folks who worked hard and contributed
> > >>> towards
> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K
> S,
> > >>> Varun
> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> > Joep
> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>> >>>
> > >>> >>> Regards,
> > >>> >>> Vrushali
> > >>> >>>
> > >>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > >>> 383.html
> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >>> >>>
> > >>> >>
> > >>> >>
> > >>>
> > >>>
> > >>> ------------------------------------------------------------
> ---------
> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >>>
> > >>>
> > >>
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Subramaniam V K <su...@gmail.com>.
Good to see this merged. I have initiated a separate thread with a smaller
set of stakeholders to discuss inclusion in 2.9. We'll report back to the
2.9 release thread as soon as we reach consensus.
On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com> wrote:
> +1 to maintaining history.
>
> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
> > Maintaining history I believe can be useful as it can make reverts
> > easier if at all required.
> > And can be an easy reference point to look at who had contributed what
> > without having to go back to the branch.
> >
> > Regards,
> > Varun Saxena.
> >
> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
> > wrote:
> >
> > > Thanks Sangjin for the link to the previous discussions on this! I
> think
> > > that helps answer Steve's questions.
> > >
> > > As decided on that thread [1], YARN-5355 as a feature branch was merged
> > to
> > > trunk via "git merge --no-ff" .
> > >
> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
> > > chose to develop on a feature branch YARN-5355 so that we could control
> > > when changes went into trunk and didn't inadvertently disrupt trunk.
> > >
> > > Is the latest merge causing any conflicts or issues for s3guard, Steve?
> > >
> > > thanks
> > > Vrushali
> > > [1] https://lists.apache.org/thread.html/
> 43cd65c6b6c3c0e8ac2b3c76afd9ef
> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org
> %3E
> > >
> > >
> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
> > >
> > >> I recall this discussion about a couple of years ago:
> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
> > >> dev.hadoop.apache.org%3E
> > >>
> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
> stevel@hortonworks.com
> > >
> > >> wrote:
> > >>
> > >>> I'd have assumed it would have gone in as one single patch, rather
> than
> > >>> a full history. I don't see why the trunk needs all the evolutionary
> > >>> history of a build.
> > >>>
> > >>> What should our policy/process be here?
> > >>>
> > >>> I do currently plan to merge the s3guard in as one single squashed
> > >>> patch; just getting HADOOP-14809 sorted first.
> > >>>
> > >>>
> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
> > wrote:
> > >>> >
> > >>> > I'm adding my +1 (binding) to conclude the vote.
> > >>> >
> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get
> on
> > >>> with
> > >>> > the merge to trunk shortly. Thanks everyone!
> > >>> >
> > >>> > Regards
> > >>> > Vrushali
> > >>> >
> > >>> >
> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> > >>> > varun.saxena.apache@gmail.com> wrote:
> > >>> >
> > >>> >> +1 (binding).
> > >>> >>
> > >>> >> Kudos to all the team members for their great work!
> > >>> >>
> > >>> >> Being part of the ATSv2 team, I have been involved with either
> > >>> development
> > >>> >> or review of most of the JIRAs'.
> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
> that
> > >>> there
> > >>> >> is no impact when ATSv2 is turned off.
> > >>> >>
> > >>> >> Regards,
> > >>> >> Varun Saxena.
> > >>> >>
> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >>> >> vrushalic2016@gmail.com> wrote:
> > >>> >>
> > >>> >>> Hi folks,
> > >>> >>>
> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
> > merge
> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> > >>> vote
> > >>> >>> will
> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>> >>>
> > >>> >>> We have previously completed one merge onto trunk [3] and
> Timeline
> > >>> Service
> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>> >>>
> > >>> >>> Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
> > reasonably
> > >>> >>> confident that the state of the feature meets the criteria to be
> > >>> merged
> > >>> >>> onto trunk and we'd love folks to get their hands on it in a test
> > >>> capacity
> > >>> >>> and provide valuable feedback so that we can make it
> > >>> production-ready.
> > >>> >>>
> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
> > scalability
> > >>> and
> > >>> >>> usability improvements based on a new architecture. What we would
> > >>> like to
> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
> has a
> > >>> >>> complete end-to-end read/write flow with security and read level
> > >>> >>> authorization via whitelists. You should be able to start setting
> > it
> > >>> up
> > >>> >>> and
> > >>> >>> testing it.
> > >>> >>>
> > >>> >>> At a high level, the following are the key features that have
> been
> > >>> >>> implemented since alpha1:
> > >>> >>> - Security via Kerberos Authentication and delegation tokens
> > >>> >>> - Read side simple authorization via whitelist
> > >>> >>> - Client configurable entity sort ordering
> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> > >>> metrics by
> > >>> >>> timerange, pagination, sub-app entities
> > >>> >>> - Support for storing sub-application entities (entities that
> exist
> > >>> >>> outside
> > >>> >>> the scope of an application)
> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> > >>> >>> prefixes,
> > >>> >>> configurable hbase cluster
> > >>> >>> - Flow level aggregations done as dynamic (table level)
> > coprocessors
> > >>> >>> - Uses latest stable HBase release 1.2.6
> > >>> >>>
> > >>> >>> There are a total of 82 subtasks that were completed as part of
> > this
> > >>> >>> effort.
> > >>> >>>
> > >>> >>> We paid close attention to ensure that once disabled Timeline
> > >>> Service v.2
> > >>> >>> does not impact existing functionality when disabled (by
> default).
> > >>> >>>
> > >>> >>> Special thanks to a team of folks who worked hard and contributed
> > >>> towards
> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K
> S,
> > >>> Varun
> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> > Joep
> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>> >>>
> > >>> >>> Regards,
> > >>> >>> Vrushali
> > >>> >>>
> > >>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > >>> 383.html
> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >>> >>>
> > >>> >>
> > >>> >>
> > >>>
> > >>>
> > >>> ------------------------------------------------------------
> ---------
> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >>>
> > >>>
> > >>
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Subramaniam V K <su...@gmail.com>.
Good to see this merged. I have initiated a separate thread with a smaller
set of stakeholders to discuss inclusion in 2.9. We'll report back to the
2.9 release thread as soon as we reach consensus.
On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com> wrote:
> +1 to maintaining history.
>
> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
> > Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
> > Maintaining history I believe can be useful as it can make reverts
> > easier if at all required.
> > And can be an easy reference point to look at who had contributed what
> > without having to go back to the branch.
> >
> > Regards,
> > Varun Saxena.
> >
> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
> > wrote:
> >
> > > Thanks Sangjin for the link to the previous discussions on this! I
> think
> > > that helps answer Steve's questions.
> > >
> > > As decided on that thread [1], YARN-5355 as a feature branch was merged
> > to
> > > trunk via "git merge --no-ff" .
> > >
> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
> > > chose to develop on a feature branch YARN-5355 so that we could control
> > > when changes went into trunk and didn't inadvertently disrupt trunk.
> > >
> > > Is the latest merge causing any conflicts or issues for s3guard, Steve?
> > >
> > > thanks
> > > Vrushali
> > > [1] https://lists.apache.org/thread.html/
> 43cd65c6b6c3c0e8ac2b3c76afd9ef
> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org
> %3E
> > >
> > >
> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
> > >
> > >> I recall this discussion about a couple of years ago:
> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
> > >> dev.hadoop.apache.org%3E
> > >>
> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
> stevel@hortonworks.com
> > >
> > >> wrote:
> > >>
> > >>> I'd have assumed it would have gone in as one single patch, rather
> than
> > >>> a full history. I don't see why the trunk needs all the evolutionary
> > >>> history of a build.
> > >>>
> > >>> What should our policy/process be here?
> > >>>
> > >>> I do currently plan to merge the s3guard in as one single squashed
> > >>> patch; just getting HADOOP-14809 sorted first.
> > >>>
> > >>>
> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
> > wrote:
> > >>> >
> > >>> > I'm adding my +1 (binding) to conclude the vote.
> > >>> >
> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get
> on
> > >>> with
> > >>> > the merge to trunk shortly. Thanks everyone!
> > >>> >
> > >>> > Regards
> > >>> > Vrushali
> > >>> >
> > >>> >
> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> > >>> > varun.saxena.apache@gmail.com> wrote:
> > >>> >
> > >>> >> +1 (binding).
> > >>> >>
> > >>> >> Kudos to all the team members for their great work!
> > >>> >>
> > >>> >> Being part of the ATSv2 team, I have been involved with either
> > >>> development
> > >>> >> or review of most of the JIRAs'.
> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
> that
> > >>> there
> > >>> >> is no impact when ATSv2 is turned off.
> > >>> >>
> > >>> >> Regards,
> > >>> >> Varun Saxena.
> > >>> >>
> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >>> >> vrushalic2016@gmail.com> wrote:
> > >>> >>
> > >>> >>> Hi folks,
> > >>> >>>
> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
> > merge
> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> > >>> vote
> > >>> >>> will
> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>> >>>
> > >>> >>> We have previously completed one merge onto trunk [3] and
> Timeline
> > >>> Service
> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>> >>>
> > >>> >>> Since then, we have been working on extending the capabilities of
> > >>> Timeline
> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
> > reasonably
> > >>> >>> confident that the state of the feature meets the criteria to be
> > >>> merged
> > >>> >>> onto trunk and we'd love folks to get their hands on it in a test
> > >>> capacity
> > >>> >>> and provide valuable feedback so that we can make it
> > >>> production-ready.
> > >>> >>>
> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
> > scalability
> > >>> and
> > >>> >>> usability improvements based on a new architecture. What we would
> > >>> like to
> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
> has a
> > >>> >>> complete end-to-end read/write flow with security and read level
> > >>> >>> authorization via whitelists. You should be able to start setting
> > it
> > >>> up
> > >>> >>> and
> > >>> >>> testing it.
> > >>> >>>
> > >>> >>> At a high level, the following are the key features that have
> been
> > >>> >>> implemented since alpha1:
> > >>> >>> - Security via Kerberos Authentication and delegation tokens
> > >>> >>> - Read side simple authorization via whitelist
> > >>> >>> - Client configurable entity sort ordering
> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> > >>> metrics by
> > >>> >>> timerange, pagination, sub-app entities
> > >>> >>> - Support for storing sub-application entities (entities that
> exist
> > >>> >>> outside
> > >>> >>> the scope of an application)
> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> > >>> >>> prefixes,
> > >>> >>> configurable hbase cluster
> > >>> >>> - Flow level aggregations done as dynamic (table level)
> > coprocessors
> > >>> >>> - Uses latest stable HBase release 1.2.6
> > >>> >>>
> > >>> >>> There are a total of 82 subtasks that were completed as part of
> > this
> > >>> >>> effort.
> > >>> >>>
> > >>> >>> We paid close attention to ensure that once disabled Timeline
> > >>> Service v.2
> > >>> >>> does not impact existing functionality when disabled (by
> default).
> > >>> >>>
> > >>> >>> Special thanks to a team of folks who worked hard and contributed
> > >>> towards
> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K
> S,
> > >>> Varun
> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> > Joep
> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>> >>>
> > >>> >>> Regards,
> > >>> >>> Vrushali
> > >>> >>>
> > >>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > >>> 383.html
> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >>> >>>
> > >>> >>
> > >>> >>
> > >>>
> > >>>
> > >>> ------------------------------------------------------------
> ---------
> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> > >>>
> > >>>
> > >>
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Ravi Prakash <ra...@gmail.com>.
+1 to maintaining history.
On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
> Maintaining history I believe can be useful as it can make reverts
> easier if at all required.
> And can be an easy reference point to look at who had contributed what
> without having to go back to the branch.
>
> Regards,
> Varun Saxena.
>
> On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
> wrote:
>
> > Thanks Sangjin for the link to the previous discussions on this! I think
> > that helps answer Steve's questions.
> >
> > As decided on that thread [1], YARN-5355 as a feature branch was merged
> to
> > trunk via "git merge --no-ff" .
> >
> > Although trunk already had TSv2 code (alpha1) prior to this merge, we
> > chose to develop on a feature branch YARN-5355 so that we could control
> > when changes went into trunk and didn't inadvertently disrupt trunk.
> >
> > Is the latest merge causing any conflicts or issues for s3guard, Steve?
> >
> > thanks
> > Vrushali
> > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
> >
> >
> > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
> >
> >> I recall this discussion about a couple of years ago:
> >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
> >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
> >> dev.hadoop.apache.org%3E
> >>
> >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <stevel@hortonworks.com
> >
> >> wrote:
> >>
> >>> I'd have assumed it would have gone in as one single patch, rather than
> >>> a full history. I don't see why the trunk needs all the evolutionary
> >>> history of a build.
> >>>
> >>> What should our policy/process be here?
> >>>
> >>> I do currently plan to merge the s3guard in as one single squashed
> >>> patch; just getting HADOOP-14809 sorted first.
> >>>
> >>>
> >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
> wrote:
> >>> >
> >>> > I'm adding my +1 (binding) to conclude the vote.
> >>> >
> >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
> >>> with
> >>> > the merge to trunk shortly. Thanks everyone!
> >>> >
> >>> > Regards
> >>> > Vrushali
> >>> >
> >>> >
> >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> >>> > varun.saxena.apache@gmail.com> wrote:
> >>> >
> >>> >> +1 (binding).
> >>> >>
> >>> >> Kudos to all the team members for their great work!
> >>> >>
> >>> >> Being part of the ATSv2 team, I have been involved with either
> >>> development
> >>> >> or review of most of the JIRAs'.
> >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> >>> there
> >>> >> is no impact when ATSv2 is turned off.
> >>> >>
> >>> >> Regards,
> >>> >> Varun Saxena.
> >>> >>
> >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> >>> >> vrushalic2016@gmail.com> wrote:
> >>> >>
> >>> >>> Hi folks,
> >>> >>>
> >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
> merge
> >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> >>> vote
> >>> >>> will
> >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> >>> >>>
> >>> >>> We have previously completed one merge onto trunk [3] and Timeline
> >>> Service
> >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>> >>>
> >>> >>> Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> >>> Service v2 in a feature branch [2] for a while, and we are
> reasonably
> >>> >>> confident that the state of the feature meets the criteria to be
> >>> merged
> >>> >>> onto trunk and we'd love folks to get their hands on it in a test
> >>> capacity
> >>> >>> and provide valuable feedback so that we can make it
> >>> production-ready.
> >>> >>>
> >>> >>> In a nutshell, Timeline Service v.2 delivers significant
> scalability
> >>> and
> >>> >>> usability improvements based on a new architecture. What we would
> >>> like to
> >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> >>> >>> complete end-to-end read/write flow with security and read level
> >>> >>> authorization via whitelists. You should be able to start setting
> it
> >>> up
> >>> >>> and
> >>> >>> testing it.
> >>> >>>
> >>> >>> At a high level, the following are the key features that have been
> >>> >>> implemented since alpha1:
> >>> >>> - Security via Kerberos Authentication and delegation tokens
> >>> >>> - Read side simple authorization via whitelist
> >>> >>> - Client configurable entity sort ordering
> >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> >>> metrics by
> >>> >>> timerange, pagination, sub-app entities
> >>> >>> - Support for storing sub-application entities (entities that exist
> >>> >>> outside
> >>> >>> the scope of an application)
> >>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> >>> >>> prefixes,
> >>> >>> configurable hbase cluster
> >>> >>> - Flow level aggregations done as dynamic (table level)
> coprocessors
> >>> >>> - Uses latest stable HBase release 1.2.6
> >>> >>>
> >>> >>> There are a total of 82 subtasks that were completed as part of
> this
> >>> >>> effort.
> >>> >>>
> >>> >>> We paid close attention to ensure that once disabled Timeline
> >>> Service v.2
> >>> >>> does not impact existing functionality when disabled (by default).
> >>> >>>
> >>> >>> Special thanks to a team of folks who worked hard and contributed
> >>> towards
> >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> >>> Varun
> >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>> >>>
> >>> >>> Regards,
> >>> >>> Vrushali
> >>> >>>
> >>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> >>> 383.html
> >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> >>> >>>
> >>> >>
> >>> >>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >>>
> >>>
> >>
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Ravi Prakash <ra...@gmail.com>.
+1 to maintaining history.
On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
> Maintaining history I believe can be useful as it can make reverts
> easier if at all required.
> And can be an easy reference point to look at who had contributed what
> without having to go back to the branch.
>
> Regards,
> Varun Saxena.
>
> On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
> wrote:
>
> > Thanks Sangjin for the link to the previous discussions on this! I think
> > that helps answer Steve's questions.
> >
> > As decided on that thread [1], YARN-5355 as a feature branch was merged
> to
> > trunk via "git merge --no-ff" .
> >
> > Although trunk already had TSv2 code (alpha1) prior to this merge, we
> > chose to develop on a feature branch YARN-5355 so that we could control
> > when changes went into trunk and didn't inadvertently disrupt trunk.
> >
> > Is the latest merge causing any conflicts or issues for s3guard, Steve?
> >
> > thanks
> > Vrushali
> > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
> >
> >
> > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
> >
> >> I recall this discussion about a couple of years ago:
> >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
> >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
> >> dev.hadoop.apache.org%3E
> >>
> >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <stevel@hortonworks.com
> >
> >> wrote:
> >>
> >>> I'd have assumed it would have gone in as one single patch, rather than
> >>> a full history. I don't see why the trunk needs all the evolutionary
> >>> history of a build.
> >>>
> >>> What should our policy/process be here?
> >>>
> >>> I do currently plan to merge the s3guard in as one single squashed
> >>> patch; just getting HADOOP-14809 sorted first.
> >>>
> >>>
> >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
> wrote:
> >>> >
> >>> > I'm adding my +1 (binding) to conclude the vote.
> >>> >
> >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
> >>> with
> >>> > the merge to trunk shortly. Thanks everyone!
> >>> >
> >>> > Regards
> >>> > Vrushali
> >>> >
> >>> >
> >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> >>> > varun.saxena.apache@gmail.com> wrote:
> >>> >
> >>> >> +1 (binding).
> >>> >>
> >>> >> Kudos to all the team members for their great work!
> >>> >>
> >>> >> Being part of the ATSv2 team, I have been involved with either
> >>> development
> >>> >> or review of most of the JIRAs'.
> >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> >>> there
> >>> >> is no impact when ATSv2 is turned off.
> >>> >>
> >>> >> Regards,
> >>> >> Varun Saxena.
> >>> >>
> >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> >>> >> vrushalic2016@gmail.com> wrote:
> >>> >>
> >>> >>> Hi folks,
> >>> >>>
> >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
> merge
> >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> >>> vote
> >>> >>> will
> >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> >>> >>>
> >>> >>> We have previously completed one merge onto trunk [3] and Timeline
> >>> Service
> >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>> >>>
> >>> >>> Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> >>> Service v2 in a feature branch [2] for a while, and we are
> reasonably
> >>> >>> confident that the state of the feature meets the criteria to be
> >>> merged
> >>> >>> onto trunk and we'd love folks to get their hands on it in a test
> >>> capacity
> >>> >>> and provide valuable feedback so that we can make it
> >>> production-ready.
> >>> >>>
> >>> >>> In a nutshell, Timeline Service v.2 delivers significant
> scalability
> >>> and
> >>> >>> usability improvements based on a new architecture. What we would
> >>> like to
> >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> >>> >>> complete end-to-end read/write flow with security and read level
> >>> >>> authorization via whitelists. You should be able to start setting
> it
> >>> up
> >>> >>> and
> >>> >>> testing it.
> >>> >>>
> >>> >>> At a high level, the following are the key features that have been
> >>> >>> implemented since alpha1:
> >>> >>> - Security via Kerberos Authentication and delegation tokens
> >>> >>> - Read side simple authorization via whitelist
> >>> >>> - Client configurable entity sort ordering
> >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> >>> metrics by
> >>> >>> timerange, pagination, sub-app entities
> >>> >>> - Support for storing sub-application entities (entities that exist
> >>> >>> outside
> >>> >>> the scope of an application)
> >>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> >>> >>> prefixes,
> >>> >>> configurable hbase cluster
> >>> >>> - Flow level aggregations done as dynamic (table level)
> coprocessors
> >>> >>> - Uses latest stable HBase release 1.2.6
> >>> >>>
> >>> >>> There are a total of 82 subtasks that were completed as part of
> this
> >>> >>> effort.
> >>> >>>
> >>> >>> We paid close attention to ensure that once disabled Timeline
> >>> Service v.2
> >>> >>> does not impact existing functionality when disabled (by default).
> >>> >>>
> >>> >>> Special thanks to a team of folks who worked hard and contributed
> >>> towards
> >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> >>> Varun
> >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>> >>>
> >>> >>> Regards,
> >>> >>> Vrushali
> >>> >>>
> >>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> >>> 383.html
> >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> >>> >>>
> >>> >>
> >>> >>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >>>
> >>>
> >>
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Ravi Prakash <ra...@gmail.com>.
+1 to maintaining history.
On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
> Maintaining history I believe can be useful as it can make reverts
> easier if at all required.
> And can be an easy reference point to look at who had contributed what
> without having to go back to the branch.
>
> Regards,
> Varun Saxena.
>
> On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
> wrote:
>
> > Thanks Sangjin for the link to the previous discussions on this! I think
> > that helps answer Steve's questions.
> >
> > As decided on that thread [1], YARN-5355 as a feature branch was merged
> to
> > trunk via "git merge --no-ff" .
> >
> > Although trunk already had TSv2 code (alpha1) prior to this merge, we
> > chose to develop on a feature branch YARN-5355 so that we could control
> > when changes went into trunk and didn't inadvertently disrupt trunk.
> >
> > Is the latest merge causing any conflicts or issues for s3guard, Steve?
> >
> > thanks
> > Vrushali
> > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
> >
> >
> > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
> >
> >> I recall this discussion about a couple of years ago:
> >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
> >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
> >> dev.hadoop.apache.org%3E
> >>
> >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <stevel@hortonworks.com
> >
> >> wrote:
> >>
> >>> I'd have assumed it would have gone in as one single patch, rather than
> >>> a full history. I don't see why the trunk needs all the evolutionary
> >>> history of a build.
> >>>
> >>> What should our policy/process be here?
> >>>
> >>> I do currently plan to merge the s3guard in as one single squashed
> >>> patch; just getting HADOOP-14809 sorted first.
> >>>
> >>>
> >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
> wrote:
> >>> >
> >>> > I'm adding my +1 (binding) to conclude the vote.
> >>> >
> >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
> >>> with
> >>> > the merge to trunk shortly. Thanks everyone!
> >>> >
> >>> > Regards
> >>> > Vrushali
> >>> >
> >>> >
> >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> >>> > varun.saxena.apache@gmail.com> wrote:
> >>> >
> >>> >> +1 (binding).
> >>> >>
> >>> >> Kudos to all the team members for their great work!
> >>> >>
> >>> >> Being part of the ATSv2 team, I have been involved with either
> >>> development
> >>> >> or review of most of the JIRAs'.
> >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> >>> there
> >>> >> is no impact when ATSv2 is turned off.
> >>> >>
> >>> >> Regards,
> >>> >> Varun Saxena.
> >>> >>
> >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> >>> >> vrushalic2016@gmail.com> wrote:
> >>> >>
> >>> >>> Hi folks,
> >>> >>>
> >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
> merge
> >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> >>> vote
> >>> >>> will
> >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> >>> >>>
> >>> >>> We have previously completed one merge onto trunk [3] and Timeline
> >>> Service
> >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>> >>>
> >>> >>> Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> >>> Service v2 in a feature branch [2] for a while, and we are
> reasonably
> >>> >>> confident that the state of the feature meets the criteria to be
> >>> merged
> >>> >>> onto trunk and we'd love folks to get their hands on it in a test
> >>> capacity
> >>> >>> and provide valuable feedback so that we can make it
> >>> production-ready.
> >>> >>>
> >>> >>> In a nutshell, Timeline Service v.2 delivers significant
> scalability
> >>> and
> >>> >>> usability improvements based on a new architecture. What we would
> >>> like to
> >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> >>> >>> complete end-to-end read/write flow with security and read level
> >>> >>> authorization via whitelists. You should be able to start setting
> it
> >>> up
> >>> >>> and
> >>> >>> testing it.
> >>> >>>
> >>> >>> At a high level, the following are the key features that have been
> >>> >>> implemented since alpha1:
> >>> >>> - Security via Kerberos Authentication and delegation tokens
> >>> >>> - Read side simple authorization via whitelist
> >>> >>> - Client configurable entity sort ordering
> >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> >>> metrics by
> >>> >>> timerange, pagination, sub-app entities
> >>> >>> - Support for storing sub-application entities (entities that exist
> >>> >>> outside
> >>> >>> the scope of an application)
> >>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> >>> >>> prefixes,
> >>> >>> configurable hbase cluster
> >>> >>> - Flow level aggregations done as dynamic (table level)
> coprocessors
> >>> >>> - Uses latest stable HBase release 1.2.6
> >>> >>>
> >>> >>> There are a total of 82 subtasks that were completed as part of
> this
> >>> >>> effort.
> >>> >>>
> >>> >>> We paid close attention to ensure that once disabled Timeline
> >>> Service v.2
> >>> >>> does not impact existing functionality when disabled (by default).
> >>> >>>
> >>> >>> Special thanks to a team of folks who worked hard and contributed
> >>> towards
> >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> >>> Varun
> >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>> >>>
> >>> >>> Regards,
> >>> >>> Vrushali
> >>> >>>
> >>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> >>> 383.html
> >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> >>> >>>
> >>> >>
> >>> >>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >>>
> >>>
> >>
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Ravi Prakash <ra...@gmail.com>.
+1 to maintaining history.
On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
> Maintaining history I believe can be useful as it can make reverts
> easier if at all required.
> And can be an easy reference point to look at who had contributed what
> without having to go back to the branch.
>
> Regards,
> Varun Saxena.
>
> On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
> wrote:
>
> > Thanks Sangjin for the link to the previous discussions on this! I think
> > that helps answer Steve's questions.
> >
> > As decided on that thread [1], YARN-5355 as a feature branch was merged
> to
> > trunk via "git merge --no-ff" .
> >
> > Although trunk already had TSv2 code (alpha1) prior to this merge, we
> > chose to develop on a feature branch YARN-5355 so that we could control
> > when changes went into trunk and didn't inadvertently disrupt trunk.
> >
> > Is the latest merge causing any conflicts or issues for s3guard, Steve?
> >
> > thanks
> > Vrushali
> > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
> >
> >
> > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
> >
> >> I recall this discussion about a couple of years ago:
> >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
> >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
> >> dev.hadoop.apache.org%3E
> >>
> >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <stevel@hortonworks.com
> >
> >> wrote:
> >>
> >>> I'd have assumed it would have gone in as one single patch, rather than
> >>> a full history. I don't see why the trunk needs all the evolutionary
> >>> history of a build.
> >>>
> >>> What should our policy/process be here?
> >>>
> >>> I do currently plan to merge the s3guard in as one single squashed
> >>> patch; just getting HADOOP-14809 sorted first.
> >>>
> >>>
> >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
> wrote:
> >>> >
> >>> > I'm adding my +1 (binding) to conclude the vote.
> >>> >
> >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
> >>> with
> >>> > the merge to trunk shortly. Thanks everyone!
> >>> >
> >>> > Regards
> >>> > Vrushali
> >>> >
> >>> >
> >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> >>> > varun.saxena.apache@gmail.com> wrote:
> >>> >
> >>> >> +1 (binding).
> >>> >>
> >>> >> Kudos to all the team members for their great work!
> >>> >>
> >>> >> Being part of the ATSv2 team, I have been involved with either
> >>> development
> >>> >> or review of most of the JIRAs'.
> >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> >>> there
> >>> >> is no impact when ATSv2 is turned off.
> >>> >>
> >>> >> Regards,
> >>> >> Varun Saxena.
> >>> >>
> >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> >>> >> vrushalic2016@gmail.com> wrote:
> >>> >>
> >>> >>> Hi folks,
> >>> >>>
> >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
> merge
> >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> >>> vote
> >>> >>> will
> >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> >>> >>>
> >>> >>> We have previously completed one merge onto trunk [3] and Timeline
> >>> Service
> >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>> >>>
> >>> >>> Since then, we have been working on extending the capabilities of
> >>> Timeline
> >>> >>> Service v2 in a feature branch [2] for a while, and we are
> reasonably
> >>> >>> confident that the state of the feature meets the criteria to be
> >>> merged
> >>> >>> onto trunk and we'd love folks to get their hands on it in a test
> >>> capacity
> >>> >>> and provide valuable feedback so that we can make it
> >>> production-ready.
> >>> >>>
> >>> >>> In a nutshell, Timeline Service v.2 delivers significant
> scalability
> >>> and
> >>> >>> usability improvements based on a new architecture. What we would
> >>> like to
> >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> >>> >>> complete end-to-end read/write flow with security and read level
> >>> >>> authorization via whitelists. You should be able to start setting
> it
> >>> up
> >>> >>> and
> >>> >>> testing it.
> >>> >>>
> >>> >>> At a high level, the following are the key features that have been
> >>> >>> implemented since alpha1:
> >>> >>> - Security via Kerberos Authentication and delegation tokens
> >>> >>> - Read side simple authorization via whitelist
> >>> >>> - Client configurable entity sort ordering
> >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> >>> metrics by
> >>> >>> timerange, pagination, sub-app entities
> >>> >>> - Support for storing sub-application entities (entities that exist
> >>> >>> outside
> >>> >>> the scope of an application)
> >>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> >>> >>> prefixes,
> >>> >>> configurable hbase cluster
> >>> >>> - Flow level aggregations done as dynamic (table level)
> coprocessors
> >>> >>> - Uses latest stable HBase release 1.2.6
> >>> >>>
> >>> >>> There are a total of 82 subtasks that were completed as part of
> this
> >>> >>> effort.
> >>> >>>
> >>> >>> We paid close attention to ensure that once disabled Timeline
> >>> Service v.2
> >>> >>> does not impact existing functionality when disabled (by default).
> >>> >>>
> >>> >>> Special thanks to a team of folks who worked hard and contributed
> >>> towards
> >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> >>> Varun
> >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>> >>>
> >>> >>> Regards,
> >>> >>> Vrushali
> >>> >>>
> >>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> >>> 383.html
> >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> >>> >>>
> >>> >>
> >>> >>
> >>>
> >>>
> >>> ---------------------------------------------------------------------
> >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >>>
> >>>
> >>
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
Maintaining history I believe can be useful as it can make reverts
easier if at all required.
And can be an easy reference point to look at who had contributed what
without having to go back to the branch.
Regards,
Varun Saxena.
On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com> wrote:
> Thanks Sangjin for the link to the previous discussions on this! I think
> that helps answer Steve's questions.
>
> As decided on that thread [1], YARN-5355 as a feature branch was merged to
> trunk via "git merge --no-ff" .
>
> Although trunk already had TSv2 code (alpha1) prior to this merge, we
> chose to develop on a feature branch YARN-5355 so that we could control
> when changes went into trunk and didn't inadvertently disrupt trunk.
>
> Is the latest merge causing any conflicts or issues for s3guard, Steve?
>
> thanks
> Vrushali
> [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
>
> On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
>
>> I recall this discussion about a couple of years ago:
>> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>> dev.hadoop.apache.org%3E
>>
>> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>> I'd have assumed it would have gone in as one single patch, rather than
>>> a full history. I don't see why the trunk needs all the evolutionary
>>> history of a build.
>>>
>>> What should our policy/process be here?
>>>
>>> I do currently plan to merge the s3guard in as one single squashed
>>> patch; just getting HADOOP-14809 sorted first.
>>>
>>>
>>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
>>> >
>>> > I'm adding my +1 (binding) to conclude the vote.
>>> >
>>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
>>> with
>>> > the merge to trunk shortly. Thanks everyone!
>>> >
>>> > Regards
>>> > Vrushali
>>> >
>>> >
>>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>> > varun.saxena.apache@gmail.com> wrote:
>>> >
>>> >> +1 (binding).
>>> >>
>>> >> Kudos to all the team members for their great work!
>>> >>
>>> >> Being part of the ATSv2 team, I have been involved with either
>>> development
>>> >> or review of most of the JIRAs'.
>>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
>>> there
>>> >> is no impact when ATSv2 is turned off.
>>> >>
>>> >> Regards,
>>> >> Varun Saxena.
>>> >>
>>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>> >> vrushalic2016@gmail.com> wrote:
>>> >>
>>> >>> Hi folks,
>>> >>>
>>> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
>>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
>>> vote
>>> >>> will
>>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> >>>
>>> >>> We have previously completed one merge onto trunk [3] and Timeline
>>> Service
>>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> >>>
>>> >>> Since then, we have been working on extending the capabilities of
>>> Timeline
>>> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
>>> >>> confident that the state of the feature meets the criteria to be
>>> merged
>>> >>> onto trunk and we'd love folks to get their hands on it in a test
>>> capacity
>>> >>> and provide valuable feedback so that we can make it
>>> production-ready.
>>> >>>
>>> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
>>> and
>>> >>> usability improvements based on a new architecture. What we would
>>> like to
>>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>>> >>> complete end-to-end read/write flow with security and read level
>>> >>> authorization via whitelists. You should be able to start setting it
>>> up
>>> >>> and
>>> >>> testing it.
>>> >>>
>>> >>> At a high level, the following are the key features that have been
>>> >>> implemented since alpha1:
>>> >>> - Security via Kerberos Authentication and delegation tokens
>>> >>> - Read side simple authorization via whitelist
>>> >>> - Client configurable entity sort ordering
>>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>>> metrics by
>>> >>> timerange, pagination, sub-app entities
>>> >>> - Support for storing sub-application entities (entities that exist
>>> >>> outside
>>> >>> the scope of an application)
>>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
>>> >>> prefixes,
>>> >>> configurable hbase cluster
>>> >>> - Flow level aggregations done as dynamic (table level) coprocessors
>>> >>> - Uses latest stable HBase release 1.2.6
>>> >>>
>>> >>> There are a total of 82 subtasks that were completed as part of this
>>> >>> effort.
>>> >>>
>>> >>> We paid close attention to ensure that once disabled Timeline
>>> Service v.2
>>> >>> does not impact existing functionality when disabled (by default).
>>> >>>
>>> >>> Special thanks to a team of folks who worked hard and contributed
>>> towards
>>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
>>> Varun
>>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> >>>
>>> >>> Regards,
>>> >>> Vrushali
>>> >>>
>>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
>>> 383.html
>>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> >>>
>>> >>
>>> >>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>
>>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
Maintaining history I believe can be useful as it can make reverts
easier if at all required.
And can be an easy reference point to look at who had contributed what
without having to go back to the branch.
Regards,
Varun Saxena.
On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com> wrote:
> Thanks Sangjin for the link to the previous discussions on this! I think
> that helps answer Steve's questions.
>
> As decided on that thread [1], YARN-5355 as a feature branch was merged to
> trunk via "git merge --no-ff" .
>
> Although trunk already had TSv2 code (alpha1) prior to this merge, we
> chose to develop on a feature branch YARN-5355 so that we could control
> when changes went into trunk and didn't inadvertently disrupt trunk.
>
> Is the latest merge causing any conflicts or issues for s3guard, Steve?
>
> thanks
> Vrushali
> [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
>
> On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
>
>> I recall this discussion about a couple of years ago:
>> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>> dev.hadoop.apache.org%3E
>>
>> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>> I'd have assumed it would have gone in as one single patch, rather than
>>> a full history. I don't see why the trunk needs all the evolutionary
>>> history of a build.
>>>
>>> What should our policy/process be here?
>>>
>>> I do currently plan to merge the s3guard in as one single squashed
>>> patch; just getting HADOOP-14809 sorted first.
>>>
>>>
>>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
>>> >
>>> > I'm adding my +1 (binding) to conclude the vote.
>>> >
>>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
>>> with
>>> > the merge to trunk shortly. Thanks everyone!
>>> >
>>> > Regards
>>> > Vrushali
>>> >
>>> >
>>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>> > varun.saxena.apache@gmail.com> wrote:
>>> >
>>> >> +1 (binding).
>>> >>
>>> >> Kudos to all the team members for their great work!
>>> >>
>>> >> Being part of the ATSv2 team, I have been involved with either
>>> development
>>> >> or review of most of the JIRAs'.
>>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
>>> there
>>> >> is no impact when ATSv2 is turned off.
>>> >>
>>> >> Regards,
>>> >> Varun Saxena.
>>> >>
>>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>> >> vrushalic2016@gmail.com> wrote:
>>> >>
>>> >>> Hi folks,
>>> >>>
>>> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
>>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
>>> vote
>>> >>> will
>>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> >>>
>>> >>> We have previously completed one merge onto trunk [3] and Timeline
>>> Service
>>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> >>>
>>> >>> Since then, we have been working on extending the capabilities of
>>> Timeline
>>> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
>>> >>> confident that the state of the feature meets the criteria to be
>>> merged
>>> >>> onto trunk and we'd love folks to get their hands on it in a test
>>> capacity
>>> >>> and provide valuable feedback so that we can make it
>>> production-ready.
>>> >>>
>>> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
>>> and
>>> >>> usability improvements based on a new architecture. What we would
>>> like to
>>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>>> >>> complete end-to-end read/write flow with security and read level
>>> >>> authorization via whitelists. You should be able to start setting it
>>> up
>>> >>> and
>>> >>> testing it.
>>> >>>
>>> >>> At a high level, the following are the key features that have been
>>> >>> implemented since alpha1:
>>> >>> - Security via Kerberos Authentication and delegation tokens
>>> >>> - Read side simple authorization via whitelist
>>> >>> - Client configurable entity sort ordering
>>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>>> metrics by
>>> >>> timerange, pagination, sub-app entities
>>> >>> - Support for storing sub-application entities (entities that exist
>>> >>> outside
>>> >>> the scope of an application)
>>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
>>> >>> prefixes,
>>> >>> configurable hbase cluster
>>> >>> - Flow level aggregations done as dynamic (table level) coprocessors
>>> >>> - Uses latest stable HBase release 1.2.6
>>> >>>
>>> >>> There are a total of 82 subtasks that were completed as part of this
>>> >>> effort.
>>> >>>
>>> >>> We paid close attention to ensure that once disabled Timeline
>>> Service v.2
>>> >>> does not impact existing functionality when disabled (by default).
>>> >>>
>>> >>> Special thanks to a team of folks who worked hard and contributed
>>> towards
>>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
>>> Varun
>>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> >>>
>>> >>> Regards,
>>> >>> Vrushali
>>> >>>
>>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
>>> 383.html
>>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> >>>
>>> >>
>>> >>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>
>>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
Maintaining history I believe can be useful as it can make reverts
easier if at all required.
And can be an easy reference point to look at who had contributed what
without having to go back to the branch.
Regards,
Varun Saxena.
On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com> wrote:
> Thanks Sangjin for the link to the previous discussions on this! I think
> that helps answer Steve's questions.
>
> As decided on that thread [1], YARN-5355 as a feature branch was merged to
> trunk via "git merge --no-ff" .
>
> Although trunk already had TSv2 code (alpha1) prior to this merge, we
> chose to develop on a feature branch YARN-5355 so that we could control
> when changes went into trunk and didn't inadvertently disrupt trunk.
>
> Is the latest merge causing any conflicts or issues for s3guard, Steve?
>
> thanks
> Vrushali
> [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
>
> On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
>
>> I recall this discussion about a couple of years ago:
>> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>> dev.hadoop.apache.org%3E
>>
>> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>> I'd have assumed it would have gone in as one single patch, rather than
>>> a full history. I don't see why the trunk needs all the evolutionary
>>> history of a build.
>>>
>>> What should our policy/process be here?
>>>
>>> I do currently plan to merge the s3guard in as one single squashed
>>> patch; just getting HADOOP-14809 sorted first.
>>>
>>>
>>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
>>> >
>>> > I'm adding my +1 (binding) to conclude the vote.
>>> >
>>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
>>> with
>>> > the merge to trunk shortly. Thanks everyone!
>>> >
>>> > Regards
>>> > Vrushali
>>> >
>>> >
>>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>> > varun.saxena.apache@gmail.com> wrote:
>>> >
>>> >> +1 (binding).
>>> >>
>>> >> Kudos to all the team members for their great work!
>>> >>
>>> >> Being part of the ATSv2 team, I have been involved with either
>>> development
>>> >> or review of most of the JIRAs'.
>>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
>>> there
>>> >> is no impact when ATSv2 is turned off.
>>> >>
>>> >> Regards,
>>> >> Varun Saxena.
>>> >>
>>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>> >> vrushalic2016@gmail.com> wrote:
>>> >>
>>> >>> Hi folks,
>>> >>>
>>> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
>>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
>>> vote
>>> >>> will
>>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> >>>
>>> >>> We have previously completed one merge onto trunk [3] and Timeline
>>> Service
>>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> >>>
>>> >>> Since then, we have been working on extending the capabilities of
>>> Timeline
>>> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
>>> >>> confident that the state of the feature meets the criteria to be
>>> merged
>>> >>> onto trunk and we'd love folks to get their hands on it in a test
>>> capacity
>>> >>> and provide valuable feedback so that we can make it
>>> production-ready.
>>> >>>
>>> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
>>> and
>>> >>> usability improvements based on a new architecture. What we would
>>> like to
>>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>>> >>> complete end-to-end read/write flow with security and read level
>>> >>> authorization via whitelists. You should be able to start setting it
>>> up
>>> >>> and
>>> >>> testing it.
>>> >>>
>>> >>> At a high level, the following are the key features that have been
>>> >>> implemented since alpha1:
>>> >>> - Security via Kerberos Authentication and delegation tokens
>>> >>> - Read side simple authorization via whitelist
>>> >>> - Client configurable entity sort ordering
>>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>>> metrics by
>>> >>> timerange, pagination, sub-app entities
>>> >>> - Support for storing sub-application entities (entities that exist
>>> >>> outside
>>> >>> the scope of an application)
>>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
>>> >>> prefixes,
>>> >>> configurable hbase cluster
>>> >>> - Flow level aggregations done as dynamic (table level) coprocessors
>>> >>> - Uses latest stable HBase release 1.2.6
>>> >>>
>>> >>> There are a total of 82 subtasks that were completed as part of this
>>> >>> effort.
>>> >>>
>>> >>> We paid close attention to ensure that once disabled Timeline
>>> Service v.2
>>> >>> does not impact existing functionality when disabled (by default).
>>> >>>
>>> >>> Special thanks to a team of folks who worked hard and contributed
>>> towards
>>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
>>> Varun
>>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> >>>
>>> >>> Regards,
>>> >>> Vrushali
>>> >>>
>>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
>>> 383.html
>>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> >>>
>>> >>
>>> >>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>
>>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Yes, I had used "git merge --no-ff" while merging ATSv2 to trunk.
Maintaining history I believe can be useful as it can make reverts
easier if at all required.
And can be an easy reference point to look at who had contributed what
without having to go back to the branch.
Regards,
Varun Saxena.
On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com> wrote:
> Thanks Sangjin for the link to the previous discussions on this! I think
> that helps answer Steve's questions.
>
> As decided on that thread [1], YARN-5355 as a feature branch was merged to
> trunk via "git merge --no-ff" .
>
> Although trunk already had TSv2 code (alpha1) prior to this merge, we
> chose to develop on a feature branch YARN-5355 so that we could control
> when changes went into trunk and didn't inadvertently disrupt trunk.
>
> Is the latest merge causing any conflicts or issues for s3guard, Steve?
>
> thanks
> Vrushali
> [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
>
> On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
>
>> I recall this discussion about a couple of years ago:
>> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>> dev.hadoop.apache.org%3E
>>
>> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
>> wrote:
>>
>>> I'd have assumed it would have gone in as one single patch, rather than
>>> a full history. I don't see why the trunk needs all the evolutionary
>>> history of a build.
>>>
>>> What should our policy/process be here?
>>>
>>> I do currently plan to merge the s3guard in as one single squashed
>>> patch; just getting HADOOP-14809 sorted first.
>>>
>>>
>>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
>>> >
>>> > I'm adding my +1 (binding) to conclude the vote.
>>> >
>>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
>>> with
>>> > the merge to trunk shortly. Thanks everyone!
>>> >
>>> > Regards
>>> > Vrushali
>>> >
>>> >
>>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>> > varun.saxena.apache@gmail.com> wrote:
>>> >
>>> >> +1 (binding).
>>> >>
>>> >> Kudos to all the team members for their great work!
>>> >>
>>> >> Being part of the ATSv2 team, I have been involved with either
>>> development
>>> >> or review of most of the JIRAs'.
>>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
>>> there
>>> >> is no impact when ATSv2 is turned off.
>>> >>
>>> >> Regards,
>>> >> Varun Saxena.
>>> >>
>>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>> >> vrushalic2016@gmail.com> wrote:
>>> >>
>>> >>> Hi folks,
>>> >>>
>>> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
>>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
>>> vote
>>> >>> will
>>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> >>>
>>> >>> We have previously completed one merge onto trunk [3] and Timeline
>>> Service
>>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> >>>
>>> >>> Since then, we have been working on extending the capabilities of
>>> Timeline
>>> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
>>> >>> confident that the state of the feature meets the criteria to be
>>> merged
>>> >>> onto trunk and we'd love folks to get their hands on it in a test
>>> capacity
>>> >>> and provide valuable feedback so that we can make it
>>> production-ready.
>>> >>>
>>> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
>>> and
>>> >>> usability improvements based on a new architecture. What we would
>>> like to
>>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>>> >>> complete end-to-end read/write flow with security and read level
>>> >>> authorization via whitelists. You should be able to start setting it
>>> up
>>> >>> and
>>> >>> testing it.
>>> >>>
>>> >>> At a high level, the following are the key features that have been
>>> >>> implemented since alpha1:
>>> >>> - Security via Kerberos Authentication and delegation tokens
>>> >>> - Read side simple authorization via whitelist
>>> >>> - Client configurable entity sort ordering
>>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>>> metrics by
>>> >>> timerange, pagination, sub-app entities
>>> >>> - Support for storing sub-application entities (entities that exist
>>> >>> outside
>>> >>> the scope of an application)
>>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
>>> >>> prefixes,
>>> >>> configurable hbase cluster
>>> >>> - Flow level aggregations done as dynamic (table level) coprocessors
>>> >>> - Uses latest stable HBase release 1.2.6
>>> >>>
>>> >>> There are a total of 82 subtasks that were completed as part of this
>>> >>> effort.
>>> >>>
>>> >>> We paid close attention to ensure that once disabled Timeline
>>> Service v.2
>>> >>> does not impact existing functionality when disabled (by default).
>>> >>>
>>> >>> Special thanks to a team of folks who worked hard and contributed
>>> towards
>>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
>>> Varun
>>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> >>>
>>> >>> Regards,
>>> >>> Vrushali
>>> >>>
>>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
>>> 383.html
>>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> >>>
>>> >>
>>> >>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>
>>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Thanks Sangjin for the link to the previous discussions on this! I think
that helps answer Steve's questions.
As decided on that thread [1], YARN-5355 as a feature branch was merged to
trunk via "git merge --no-ff" .
Although trunk already had TSv2 code (alpha1) prior to this merge, we chose
to develop on a feature branch YARN-5355 so that we could control when
changes went into trunk and didn't inadvertently disrupt trunk.
Is the latest merge causing any conflicts or issues for s3guard, Steve?
thanks
Vrushali
[1]
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>> I'd have assumed it would have gone in as one single patch, rather than a
>> full history. I don't see why the trunk needs all the evolutionary history
>> of a build.
>>
>> What should our policy/process be here?
>>
>> I do currently plan to merge the s3guard in as one single squashed patch;
>> just getting HADOOP-14809 sorted first.
>>
>>
>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
>> >
>> > I'm adding my +1 (binding) to conclude the vote.
>> >
>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
>> with
>> > the merge to trunk shortly. Thanks everyone!
>> >
>> > Regards
>> > Vrushali
>> >
>> >
>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>> > varun.saxena.apache@gmail.com> wrote:
>> >
>> >> +1 (binding).
>> >>
>> >> Kudos to all the team members for their great work!
>> >>
>> >> Being part of the ATSv2 team, I have been involved with either
>> development
>> >> or review of most of the JIRAs'.
>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
>> there
>> >> is no impact when ATSv2 is turned off.
>> >>
>> >> Regards,
>> >> Varun Saxena.
>> >>
>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> >> vrushalic2016@gmail.com> wrote:
>> >>
>> >>> Hi folks,
>> >>>
>> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> >>> will
>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>> >>>
>> >>> We have previously completed one merge onto trunk [3] and Timeline
>> Service
>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>> >>>
>> >>> Since then, we have been working on extending the capabilities of
>> Timeline
>> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> >>> confident that the state of the feature meets the criteria to be
>> merged
>> >>> onto trunk and we'd love folks to get their hands on it in a test
>> capacity
>> >>> and provide valuable feedback so that we can make it production-ready.
>> >>>
>> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
>> and
>> >>> usability improvements based on a new architecture. What we would
>> like to
>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> >>> complete end-to-end read/write flow with security and read level
>> >>> authorization via whitelists. You should be able to start setting it
>> up
>> >>> and
>> >>> testing it.
>> >>>
>> >>> At a high level, the following are the key features that have been
>> >>> implemented since alpha1:
>> >>> - Security via Kerberos Authentication and delegation tokens
>> >>> - Read side simple authorization via whitelist
>> >>> - Client configurable entity sort ordering
>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>> metrics by
>> >>> timerange, pagination, sub-app entities
>> >>> - Support for storing sub-application entities (entities that exist
>> >>> outside
>> >>> the scope of an application)
>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
>> >>> prefixes,
>> >>> configurable hbase cluster
>> >>> - Flow level aggregations done as dynamic (table level) coprocessors
>> >>> - Uses latest stable HBase release 1.2.6
>> >>>
>> >>> There are a total of 82 subtasks that were completed as part of this
>> >>> effort.
>> >>>
>> >>> We paid close attention to ensure that once disabled Timeline Service
>> v.2
>> >>> does not impact existing functionality when disabled (by default).
>> >>>
>> >>> Special thanks to a team of folks who worked hard and contributed
>> towards
>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
>> Varun
>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> >>>
>> >>> Regards,
>> >>> Vrushali
>> >>>
>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
>> 383.html
>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>> >>>
>> >>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Colin McCabe <cm...@apache.org>.
The "git" way of doing things would be to rebase the feature branch on
master (trunk) and then commit the patch stack.
Squashing the entire feature into a 10 MB megapatch is the "svn" way of
doing things.
The svn workflow evolved because merging feature branches back to trunk
was really painful in svn. So people preferred just to basically do an
rsync from a checkout of the feature branch, edit that up a bit to make
sure they weren't overwriting something that happened in trunk, and then
do an "svn commit" that did not tie back to the feature branch.
P.S. Merges were so painful in SVN that some of the organizations I
worked for maintained spreadsheets of patches which had been merged to
each branch, since svn was of so little help with merging
P.P.S. svn eventually got slightly smarter about merges -- I think in
svn 1.6 or something?
Colin
On Wed, Aug 30, 2017, at 14:37, Sangjin Lee wrote:
> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> > I'd have assumed it would have gone in as one single patch, rather than a
> > full history. I don't see why the trunk needs all the evolutionary history
> > of a build.
> >
> > What should our policy/process be here?
> >
> > I do currently plan to merge the s3guard in as one single squashed patch;
> > just getting HADOOP-14809 sorted first.
> >
> >
> > > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
> > >
> > > I'm adding my +1 (binding) to conclude the vote.
> > >
> > > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > > the merge to trunk shortly. Thanks everyone!
> > >
> > > Regards
> > > Vrushali
> > >
> > >
> > > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> > > varun.saxena.apache@gmail.com> wrote:
> > >
> > >> +1 (binding).
> > >>
> > >> Kudos to all the team members for their great work!
> > >>
> > >> Being part of the ATSv2 team, I have been involved with either
> > development
> > >> or review of most of the JIRAs'.
> > >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> > there
> > >> is no impact when ATSv2 is turned off.
> > >>
> > >> Regards,
> > >> Varun Saxena.
> > >>
> > >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >> vrushalic2016@gmail.com> wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> > >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > >>> will
> > >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>>
> > >>> We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > Timeline
> > >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> > >>> confident that the state of the feature meets the criteria to be merged
> > >>> onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > >>> and provide valuable feedback so that we can make it production-ready.
> > >>>
> > >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > >>> usability improvements based on a new architecture. What we would like
> > to
> > >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > >>> complete end-to-end read/write flow with security and read level
> > >>> authorization via whitelists. You should be able to start setting it up
> > >>> and
> > >>> testing it.
> > >>>
> > >>> At a high level, the following are the key features that have been
> > >>> implemented since alpha1:
> > >>> - Security via Kerberos Authentication and delegation tokens
> > >>> - Read side simple authorization via whitelist
> > >>> - Client configurable entity sort ordering
> > >>> - Richer REST APIs for apps, app attempts, containers, fetching
> > metrics by
> > >>> timerange, pagination, sub-app entities
> > >>> - Support for storing sub-application entities (entities that exist
> > >>> outside
> > >>> the scope of an application)
> > >>> - Configurable TTLs (time-to-live) for tables, configurable table
> > >>> prefixes,
> > >>> configurable hbase cluster
> > >>> - Flow level aggregations done as dynamic (table level) coprocessors
> > >>> - Uses latest stable HBase release 1.2.6
> > >>>
> > >>> There are a total of 82 subtasks that were completed as part of this
> > >>> effort.
> > >>>
> > >>> We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > >>> does not impact existing functionality when disabled (by default).
> > >>>
> > >>> Special thanks to a team of folks who worked hard and contributed
> > towards
> > >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>>
> > >>> Regards,
> > >>> Vrushali
> > >>>
> > >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> > msg27383.html
> > >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> > >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> > >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >>>
> > >>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Colin McCabe <cm...@apache.org>.
The "git" way of doing things would be to rebase the feature branch on
master (trunk) and then commit the patch stack.
Squashing the entire feature into a 10 MB megapatch is the "svn" way of
doing things.
The svn workflow evolved because merging feature branches back to trunk
was really painful in svn. So people preferred just to basically do an
rsync from a checkout of the feature branch, edit that up a bit to make
sure they weren't overwriting something that happened in trunk, and then
do an "svn commit" that did not tie back to the feature branch.
P.S. Merges were so painful in SVN that some of the organizations I
worked for maintained spreadsheets of patches which had been merged to
each branch, since svn was of so little help with merging
P.P.S. svn eventually got slightly smarter about merges -- I think in
svn 1.6 or something?
Colin
On Wed, Aug 30, 2017, at 14:37, Sangjin Lee wrote:
> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> > I'd have assumed it would have gone in as one single patch, rather than a
> > full history. I don't see why the trunk needs all the evolutionary history
> > of a build.
> >
> > What should our policy/process be here?
> >
> > I do currently plan to merge the s3guard in as one single squashed patch;
> > just getting HADOOP-14809 sorted first.
> >
> >
> > > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
> > >
> > > I'm adding my +1 (binding) to conclude the vote.
> > >
> > > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > > the merge to trunk shortly. Thanks everyone!
> > >
> > > Regards
> > > Vrushali
> > >
> > >
> > > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> > > varun.saxena.apache@gmail.com> wrote:
> > >
> > >> +1 (binding).
> > >>
> > >> Kudos to all the team members for their great work!
> > >>
> > >> Being part of the ATSv2 team, I have been involved with either
> > development
> > >> or review of most of the JIRAs'.
> > >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> > there
> > >> is no impact when ATSv2 is turned off.
> > >>
> > >> Regards,
> > >> Varun Saxena.
> > >>
> > >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >> vrushalic2016@gmail.com> wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> > >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > >>> will
> > >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>>
> > >>> We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > Timeline
> > >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> > >>> confident that the state of the feature meets the criteria to be merged
> > >>> onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > >>> and provide valuable feedback so that we can make it production-ready.
> > >>>
> > >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > >>> usability improvements based on a new architecture. What we would like
> > to
> > >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > >>> complete end-to-end read/write flow with security and read level
> > >>> authorization via whitelists. You should be able to start setting it up
> > >>> and
> > >>> testing it.
> > >>>
> > >>> At a high level, the following are the key features that have been
> > >>> implemented since alpha1:
> > >>> - Security via Kerberos Authentication and delegation tokens
> > >>> - Read side simple authorization via whitelist
> > >>> - Client configurable entity sort ordering
> > >>> - Richer REST APIs for apps, app attempts, containers, fetching
> > metrics by
> > >>> timerange, pagination, sub-app entities
> > >>> - Support for storing sub-application entities (entities that exist
> > >>> outside
> > >>> the scope of an application)
> > >>> - Configurable TTLs (time-to-live) for tables, configurable table
> > >>> prefixes,
> > >>> configurable hbase cluster
> > >>> - Flow level aggregations done as dynamic (table level) coprocessors
> > >>> - Uses latest stable HBase release 1.2.6
> > >>>
> > >>> There are a total of 82 subtasks that were completed as part of this
> > >>> effort.
> > >>>
> > >>> We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > >>> does not impact existing functionality when disabled (by default).
> > >>>
> > >>> Special thanks to a team of folks who worked hard and contributed
> > towards
> > >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>>
> > >>> Regards,
> > >>> Vrushali
> > >>>
> > >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> > msg27383.html
> > >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> > >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> > >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >>>
> > >>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Thanks Sangjin for the link to the previous discussions on this! I think
that helps answer Steve's questions.
As decided on that thread [1], YARN-5355 as a feature branch was merged to
trunk via "git merge --no-ff" .
Although trunk already had TSv2 code (alpha1) prior to this merge, we chose
to develop on a feature branch YARN-5355 so that we could control when
changes went into trunk and didn't inadvertently disrupt trunk.
Is the latest merge causing any conflicts or issues for s3guard, Steve?
thanks
Vrushali
[1]
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>> I'd have assumed it would have gone in as one single patch, rather than a
>> full history. I don't see why the trunk needs all the evolutionary history
>> of a build.
>>
>> What should our policy/process be here?
>>
>> I do currently plan to merge the s3guard in as one single squashed patch;
>> just getting HADOOP-14809 sorted first.
>>
>>
>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
>> >
>> > I'm adding my +1 (binding) to conclude the vote.
>> >
>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
>> with
>> > the merge to trunk shortly. Thanks everyone!
>> >
>> > Regards
>> > Vrushali
>> >
>> >
>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>> > varun.saxena.apache@gmail.com> wrote:
>> >
>> >> +1 (binding).
>> >>
>> >> Kudos to all the team members for their great work!
>> >>
>> >> Being part of the ATSv2 team, I have been involved with either
>> development
>> >> or review of most of the JIRAs'.
>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
>> there
>> >> is no impact when ATSv2 is turned off.
>> >>
>> >> Regards,
>> >> Varun Saxena.
>> >>
>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> >> vrushalic2016@gmail.com> wrote:
>> >>
>> >>> Hi folks,
>> >>>
>> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> >>> will
>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>> >>>
>> >>> We have previously completed one merge onto trunk [3] and Timeline
>> Service
>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>> >>>
>> >>> Since then, we have been working on extending the capabilities of
>> Timeline
>> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> >>> confident that the state of the feature meets the criteria to be
>> merged
>> >>> onto trunk and we'd love folks to get their hands on it in a test
>> capacity
>> >>> and provide valuable feedback so that we can make it production-ready.
>> >>>
>> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
>> and
>> >>> usability improvements based on a new architecture. What we would
>> like to
>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> >>> complete end-to-end read/write flow with security and read level
>> >>> authorization via whitelists. You should be able to start setting it
>> up
>> >>> and
>> >>> testing it.
>> >>>
>> >>> At a high level, the following are the key features that have been
>> >>> implemented since alpha1:
>> >>> - Security via Kerberos Authentication and delegation tokens
>> >>> - Read side simple authorization via whitelist
>> >>> - Client configurable entity sort ordering
>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>> metrics by
>> >>> timerange, pagination, sub-app entities
>> >>> - Support for storing sub-application entities (entities that exist
>> >>> outside
>> >>> the scope of an application)
>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
>> >>> prefixes,
>> >>> configurable hbase cluster
>> >>> - Flow level aggregations done as dynamic (table level) coprocessors
>> >>> - Uses latest stable HBase release 1.2.6
>> >>>
>> >>> There are a total of 82 subtasks that were completed as part of this
>> >>> effort.
>> >>>
>> >>> We paid close attention to ensure that once disabled Timeline Service
>> v.2
>> >>> does not impact existing functionality when disabled (by default).
>> >>>
>> >>> Special thanks to a team of folks who worked hard and contributed
>> towards
>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
>> Varun
>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> >>>
>> >>> Regards,
>> >>> Vrushali
>> >>>
>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
>> 383.html
>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>> >>>
>> >>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Colin McCabe <cm...@apache.org>.
The "git" way of doing things would be to rebase the feature branch on
master (trunk) and then commit the patch stack.
Squashing the entire feature into a 10 MB megapatch is the "svn" way of
doing things.
The svn workflow evolved because merging feature branches back to trunk
was really painful in svn. So people preferred just to basically do an
rsync from a checkout of the feature branch, edit that up a bit to make
sure they weren't overwriting something that happened in trunk, and then
do an "svn commit" that did not tie back to the feature branch.
P.S. Merges were so painful in SVN that some of the organizations I
worked for maintained spreadsheets of patches which had been merged to
each branch, since svn was of so little help with merging
P.P.S. svn eventually got slightly smarter about merges -- I think in
svn 1.6 or something?
Colin
On Wed, Aug 30, 2017, at 14:37, Sangjin Lee wrote:
> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> > I'd have assumed it would have gone in as one single patch, rather than a
> > full history. I don't see why the trunk needs all the evolutionary history
> > of a build.
> >
> > What should our policy/process be here?
> >
> > I do currently plan to merge the s3guard in as one single squashed patch;
> > just getting HADOOP-14809 sorted first.
> >
> >
> > > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
> > >
> > > I'm adding my +1 (binding) to conclude the vote.
> > >
> > > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > > the merge to trunk shortly. Thanks everyone!
> > >
> > > Regards
> > > Vrushali
> > >
> > >
> > > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> > > varun.saxena.apache@gmail.com> wrote:
> > >
> > >> +1 (binding).
> > >>
> > >> Kudos to all the team members for their great work!
> > >>
> > >> Being part of the ATSv2 team, I have been involved with either
> > development
> > >> or review of most of the JIRAs'.
> > >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> > there
> > >> is no impact when ATSv2 is turned off.
> > >>
> > >> Regards,
> > >> Varun Saxena.
> > >>
> > >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >> vrushalic2016@gmail.com> wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> > >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > >>> will
> > >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>>
> > >>> We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > Timeline
> > >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> > >>> confident that the state of the feature meets the criteria to be merged
> > >>> onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > >>> and provide valuable feedback so that we can make it production-ready.
> > >>>
> > >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > >>> usability improvements based on a new architecture. What we would like
> > to
> > >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > >>> complete end-to-end read/write flow with security and read level
> > >>> authorization via whitelists. You should be able to start setting it up
> > >>> and
> > >>> testing it.
> > >>>
> > >>> At a high level, the following are the key features that have been
> > >>> implemented since alpha1:
> > >>> - Security via Kerberos Authentication and delegation tokens
> > >>> - Read side simple authorization via whitelist
> > >>> - Client configurable entity sort ordering
> > >>> - Richer REST APIs for apps, app attempts, containers, fetching
> > metrics by
> > >>> timerange, pagination, sub-app entities
> > >>> - Support for storing sub-application entities (entities that exist
> > >>> outside
> > >>> the scope of an application)
> > >>> - Configurable TTLs (time-to-live) for tables, configurable table
> > >>> prefixes,
> > >>> configurable hbase cluster
> > >>> - Flow level aggregations done as dynamic (table level) coprocessors
> > >>> - Uses latest stable HBase release 1.2.6
> > >>>
> > >>> There are a total of 82 subtasks that were completed as part of this
> > >>> effort.
> > >>>
> > >>> We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > >>> does not impact existing functionality when disabled (by default).
> > >>>
> > >>> Special thanks to a team of folks who worked hard and contributed
> > towards
> > >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>>
> > >>> Regards,
> > >>> Vrushali
> > >>>
> > >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> > msg27383.html
> > >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> > >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> > >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >>>
> > >>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Thanks Sangjin for the link to the previous discussions on this! I think
that helps answer Steve's questions.
As decided on that thread [1], YARN-5355 as a feature branch was merged to
trunk via "git merge --no-ff" .
Although trunk already had TSv2 code (alpha1) prior to this merge, we chose
to develop on a feature branch YARN-5355 so that we could control when
changes went into trunk and didn't inadvertently disrupt trunk.
Is the latest merge causing any conflicts or issues for s3guard, Steve?
thanks
Vrushali
[1]
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>> I'd have assumed it would have gone in as one single patch, rather than a
>> full history. I don't see why the trunk needs all the evolutionary history
>> of a build.
>>
>> What should our policy/process be here?
>>
>> I do currently plan to merge the s3guard in as one single squashed patch;
>> just getting HADOOP-14809 sorted first.
>>
>>
>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
>> >
>> > I'm adding my +1 (binding) to conclude the vote.
>> >
>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
>> with
>> > the merge to trunk shortly. Thanks everyone!
>> >
>> > Regards
>> > Vrushali
>> >
>> >
>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>> > varun.saxena.apache@gmail.com> wrote:
>> >
>> >> +1 (binding).
>> >>
>> >> Kudos to all the team members for their great work!
>> >>
>> >> Being part of the ATSv2 team, I have been involved with either
>> development
>> >> or review of most of the JIRAs'.
>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
>> there
>> >> is no impact when ATSv2 is turned off.
>> >>
>> >> Regards,
>> >> Varun Saxena.
>> >>
>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> >> vrushalic2016@gmail.com> wrote:
>> >>
>> >>> Hi folks,
>> >>>
>> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> >>> will
>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>> >>>
>> >>> We have previously completed one merge onto trunk [3] and Timeline
>> Service
>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>> >>>
>> >>> Since then, we have been working on extending the capabilities of
>> Timeline
>> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> >>> confident that the state of the feature meets the criteria to be
>> merged
>> >>> onto trunk and we'd love folks to get their hands on it in a test
>> capacity
>> >>> and provide valuable feedback so that we can make it production-ready.
>> >>>
>> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
>> and
>> >>> usability improvements based on a new architecture. What we would
>> like to
>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> >>> complete end-to-end read/write flow with security and read level
>> >>> authorization via whitelists. You should be able to start setting it
>> up
>> >>> and
>> >>> testing it.
>> >>>
>> >>> At a high level, the following are the key features that have been
>> >>> implemented since alpha1:
>> >>> - Security via Kerberos Authentication and delegation tokens
>> >>> - Read side simple authorization via whitelist
>> >>> - Client configurable entity sort ordering
>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>> metrics by
>> >>> timerange, pagination, sub-app entities
>> >>> - Support for storing sub-application entities (entities that exist
>> >>> outside
>> >>> the scope of an application)
>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
>> >>> prefixes,
>> >>> configurable hbase cluster
>> >>> - Flow level aggregations done as dynamic (table level) coprocessors
>> >>> - Uses latest stable HBase release 1.2.6
>> >>>
>> >>> There are a total of 82 subtasks that were completed as part of this
>> >>> effort.
>> >>>
>> >>> We paid close attention to ensure that once disabled Timeline Service
>> v.2
>> >>> does not impact existing functionality when disabled (by default).
>> >>>
>> >>> Special thanks to a team of folks who worked hard and contributed
>> towards
>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
>> Varun
>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> >>>
>> >>> Regards,
>> >>> Vrushali
>> >>>
>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
>> 383.html
>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>> >>>
>> >>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Colin McCabe <cm...@apache.org>.
The "git" way of doing things would be to rebase the feature branch on
master (trunk) and then commit the patch stack.
Squashing the entire feature into a 10 MB megapatch is the "svn" way of
doing things.
The svn workflow evolved because merging feature branches back to trunk
was really painful in svn. So people preferred just to basically do an
rsync from a checkout of the feature branch, edit that up a bit to make
sure they weren't overwriting something that happened in trunk, and then
do an "svn commit" that did not tie back to the feature branch.
P.S. Merges were so painful in SVN that some of the organizations I
worked for maintained spreadsheets of patches which had been merged to
each branch, since svn was of so little help with merging
P.P.S. svn eventually got slightly smarter about merges -- I think in
svn 1.6 or something?
Colin
On Wed, Aug 30, 2017, at 14:37, Sangjin Lee wrote:
> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
> > I'd have assumed it would have gone in as one single patch, rather than a
> > full history. I don't see why the trunk needs all the evolutionary history
> > of a build.
> >
> > What should our policy/process be here?
> >
> > I do currently plan to merge the s3guard in as one single squashed patch;
> > just getting HADOOP-14809 sorted first.
> >
> >
> > > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
> > >
> > > I'm adding my +1 (binding) to conclude the vote.
> > >
> > > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > > the merge to trunk shortly. Thanks everyone!
> > >
> > > Regards
> > > Vrushali
> > >
> > >
> > > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> > > varun.saxena.apache@gmail.com> wrote:
> > >
> > >> +1 (binding).
> > >>
> > >> Kudos to all the team members for their great work!
> > >>
> > >> Being part of the ATSv2 team, I have been involved with either
> > development
> > >> or review of most of the JIRAs'.
> > >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> > there
> > >> is no impact when ATSv2 is turned off.
> > >>
> > >> Regards,
> > >> Varun Saxena.
> > >>
> > >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >> vrushalic2016@gmail.com> wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> > >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > >>> will
> > >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>>
> > >>> We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > Timeline
> > >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> > >>> confident that the state of the feature meets the criteria to be merged
> > >>> onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > >>> and provide valuable feedback so that we can make it production-ready.
> > >>>
> > >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > >>> usability improvements based on a new architecture. What we would like
> > to
> > >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > >>> complete end-to-end read/write flow with security and read level
> > >>> authorization via whitelists. You should be able to start setting it up
> > >>> and
> > >>> testing it.
> > >>>
> > >>> At a high level, the following are the key features that have been
> > >>> implemented since alpha1:
> > >>> - Security via Kerberos Authentication and delegation tokens
> > >>> - Read side simple authorization via whitelist
> > >>> - Client configurable entity sort ordering
> > >>> - Richer REST APIs for apps, app attempts, containers, fetching
> > metrics by
> > >>> timerange, pagination, sub-app entities
> > >>> - Support for storing sub-application entities (entities that exist
> > >>> outside
> > >>> the scope of an application)
> > >>> - Configurable TTLs (time-to-live) for tables, configurable table
> > >>> prefixes,
> > >>> configurable hbase cluster
> > >>> - Flow level aggregations done as dynamic (table level) coprocessors
> > >>> - Uses latest stable HBase release 1.2.6
> > >>>
> > >>> There are a total of 82 subtasks that were completed as part of this
> > >>> effort.
> > >>>
> > >>> We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > >>> does not impact existing functionality when disabled (by default).
> > >>>
> > >>> Special thanks to a team of folks who worked hard and contributed
> > towards
> > >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >>>
> > >>> Regards,
> > >>> Vrushali
> > >>>
> > >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> > msg27383.html
> > >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> > >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> > >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >>>
> > >>
> > >>
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Thanks Sangjin for the link to the previous discussions on this! I think
that helps answer Steve's questions.
As decided on that thread [1], YARN-5355 as a feature branch was merged to
trunk via "git merge --no-ff" .
Although trunk already had TSv2 code (alpha1) prior to this merge, we chose
to develop on a feature branch YARN-5355 so that we could control when
changes went into trunk and didn't inadvertently disrupt trunk.
Is the latest merge causing any conflicts or issues for s3guard, Steve?
thanks
Vrushali
[1]
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org> wrote:
> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9ef
> f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
>
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
> wrote:
>
>> I'd have assumed it would have gone in as one single patch, rather than a
>> full history. I don't see why the trunk needs all the evolutionary history
>> of a build.
>>
>> What should our policy/process be here?
>>
>> I do currently plan to merge the s3guard in as one single squashed patch;
>> just getting HADOOP-14809 sorted first.
>>
>>
>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
>> >
>> > I'm adding my +1 (binding) to conclude the vote.
>> >
>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on
>> with
>> > the merge to trunk shortly. Thanks everyone!
>> >
>> > Regards
>> > Vrushali
>> >
>> >
>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>> > varun.saxena.apache@gmail.com> wrote:
>> >
>> >> +1 (binding).
>> >>
>> >> Kudos to all the team members for their great work!
>> >>
>> >> Being part of the ATSv2 team, I have been involved with either
>> development
>> >> or review of most of the JIRAs'.
>> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
>> there
>> >> is no impact when ATSv2 is turned off.
>> >>
>> >> Regards,
>> >> Varun Saxena.
>> >>
>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> >> vrushalic2016@gmail.com> wrote:
>> >>
>> >>> Hi folks,
>> >>>
>> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> >>> will
>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>> >>>
>> >>> We have previously completed one merge onto trunk [3] and Timeline
>> Service
>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>> >>>
>> >>> Since then, we have been working on extending the capabilities of
>> Timeline
>> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> >>> confident that the state of the feature meets the criteria to be
>> merged
>> >>> onto trunk and we'd love folks to get their hands on it in a test
>> capacity
>> >>> and provide valuable feedback so that we can make it production-ready.
>> >>>
>> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
>> and
>> >>> usability improvements based on a new architecture. What we would
>> like to
>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> >>> complete end-to-end read/write flow with security and read level
>> >>> authorization via whitelists. You should be able to start setting it
>> up
>> >>> and
>> >>> testing it.
>> >>>
>> >>> At a high level, the following are the key features that have been
>> >>> implemented since alpha1:
>> >>> - Security via Kerberos Authentication and delegation tokens
>> >>> - Read side simple authorization via whitelist
>> >>> - Client configurable entity sort ordering
>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>> metrics by
>> >>> timerange, pagination, sub-app entities
>> >>> - Support for storing sub-application entities (entities that exist
>> >>> outside
>> >>> the scope of an application)
>> >>> - Configurable TTLs (time-to-live) for tables, configurable table
>> >>> prefixes,
>> >>> configurable hbase cluster
>> >>> - Flow level aggregations done as dynamic (table level) coprocessors
>> >>> - Uses latest stable HBase release 1.2.6
>> >>>
>> >>> There are a total of 82 subtasks that were completed as part of this
>> >>> effort.
>> >>>
>> >>> We paid close attention to ensure that once disabled Timeline Service
>> v.2
>> >>> does not impact existing functionality when disabled (by default).
>> >>>
>> >>> Special thanks to a team of folks who worked hard and contributed
>> towards
>> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
>> Varun
>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> >>>
>> >>> Regards,
>> >>> Vrushali
>> >>>
>> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
>> 383.html
>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>> >>>
>> >>
>> >>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>
>>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
I recall this discussion about a couple of years ago:
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
wrote:
> I'd have assumed it would have gone in as one single patch, rather than a
> full history. I don't see why the trunk needs all the evolutionary history
> of a build.
>
> What should our policy/process be here?
>
> I do currently plan to merge the s3guard in as one single squashed patch;
> just getting HADOOP-14809 sorted first.
>
>
> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
> >
> > I'm adding my +1 (binding) to conclude the vote.
> >
> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > the merge to trunk shortly. Thanks everyone!
> >
> > Regards
> > Vrushali
> >
> >
> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> > varun.saxena.apache@gmail.com> wrote:
> >
> >> +1 (binding).
> >>
> >> Kudos to all the team members for their great work!
> >>
> >> Being part of the ATSv2 team, I have been involved with either
> development
> >> or review of most of the JIRAs'.
> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> there
> >> is no impact when ATSv2 is turned off.
> >>
> >> Regards,
> >> Varun Saxena.
> >>
> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> >> vrushalic2016@gmail.com> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> >>> will
> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> >>>
> >>> We have previously completed one merge onto trunk [3] and Timeline
> Service
> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>>
> >>> Since then, we have been working on extending the capabilities of
> Timeline
> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> >>> confident that the state of the feature meets the criteria to be merged
> >>> onto trunk and we'd love folks to get their hands on it in a test
> capacity
> >>> and provide valuable feedback so that we can make it production-ready.
> >>>
> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> >>> usability improvements based on a new architecture. What we would like
> to
> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> >>> complete end-to-end read/write flow with security and read level
> >>> authorization via whitelists. You should be able to start setting it up
> >>> and
> >>> testing it.
> >>>
> >>> At a high level, the following are the key features that have been
> >>> implemented since alpha1:
> >>> - Security via Kerberos Authentication and delegation tokens
> >>> - Read side simple authorization via whitelist
> >>> - Client configurable entity sort ordering
> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> metrics by
> >>> timerange, pagination, sub-app entities
> >>> - Support for storing sub-application entities (entities that exist
> >>> outside
> >>> the scope of an application)
> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> >>> prefixes,
> >>> configurable hbase cluster
> >>> - Flow level aggregations done as dynamic (table level) coprocessors
> >>> - Uses latest stable HBase release 1.2.6
> >>>
> >>> There are a total of 82 subtasks that were completed as part of this
> >>> effort.
> >>>
> >>> We paid close attention to ensure that once disabled Timeline Service
> v.2
> >>> does not impact existing functionality when disabled (by default).
> >>>
> >>> Special thanks to a team of folks who worked hard and contributed
> towards
> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>>
> >>> Regards,
> >>> Vrushali
> >>>
> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27383.html
> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> >>>
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
I recall this discussion about a couple of years ago:
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
wrote:
> I'd have assumed it would have gone in as one single patch, rather than a
> full history. I don't see why the trunk needs all the evolutionary history
> of a build.
>
> What should our policy/process be here?
>
> I do currently plan to merge the s3guard in as one single squashed patch;
> just getting HADOOP-14809 sorted first.
>
>
> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
> >
> > I'm adding my +1 (binding) to conclude the vote.
> >
> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > the merge to trunk shortly. Thanks everyone!
> >
> > Regards
> > Vrushali
> >
> >
> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> > varun.saxena.apache@gmail.com> wrote:
> >
> >> +1 (binding).
> >>
> >> Kudos to all the team members for their great work!
> >>
> >> Being part of the ATSv2 team, I have been involved with either
> development
> >> or review of most of the JIRAs'.
> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> there
> >> is no impact when ATSv2 is turned off.
> >>
> >> Regards,
> >> Varun Saxena.
> >>
> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> >> vrushalic2016@gmail.com> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> >>> will
> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> >>>
> >>> We have previously completed one merge onto trunk [3] and Timeline
> Service
> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>>
> >>> Since then, we have been working on extending the capabilities of
> Timeline
> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> >>> confident that the state of the feature meets the criteria to be merged
> >>> onto trunk and we'd love folks to get their hands on it in a test
> capacity
> >>> and provide valuable feedback so that we can make it production-ready.
> >>>
> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> >>> usability improvements based on a new architecture. What we would like
> to
> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> >>> complete end-to-end read/write flow with security and read level
> >>> authorization via whitelists. You should be able to start setting it up
> >>> and
> >>> testing it.
> >>>
> >>> At a high level, the following are the key features that have been
> >>> implemented since alpha1:
> >>> - Security via Kerberos Authentication and delegation tokens
> >>> - Read side simple authorization via whitelist
> >>> - Client configurable entity sort ordering
> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> metrics by
> >>> timerange, pagination, sub-app entities
> >>> - Support for storing sub-application entities (entities that exist
> >>> outside
> >>> the scope of an application)
> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> >>> prefixes,
> >>> configurable hbase cluster
> >>> - Flow level aggregations done as dynamic (table level) coprocessors
> >>> - Uses latest stable HBase release 1.2.6
> >>>
> >>> There are a total of 82 subtasks that were completed as part of this
> >>> effort.
> >>>
> >>> We paid close attention to ensure that once disabled Timeline Service
> v.2
> >>> does not impact existing functionality when disabled (by default).
> >>>
> >>> Special thanks to a team of folks who worked hard and contributed
> towards
> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>>
> >>> Regards,
> >>> Vrushali
> >>>
> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27383.html
> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> >>>
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
I recall this discussion about a couple of years ago:
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
wrote:
> I'd have assumed it would have gone in as one single patch, rather than a
> full history. I don't see why the trunk needs all the evolutionary history
> of a build.
>
> What should our policy/process be here?
>
> I do currently plan to merge the s3guard in as one single squashed patch;
> just getting HADOOP-14809 sorted first.
>
>
> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
> >
> > I'm adding my +1 (binding) to conclude the vote.
> >
> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > the merge to trunk shortly. Thanks everyone!
> >
> > Regards
> > Vrushali
> >
> >
> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> > varun.saxena.apache@gmail.com> wrote:
> >
> >> +1 (binding).
> >>
> >> Kudos to all the team members for their great work!
> >>
> >> Being part of the ATSv2 team, I have been involved with either
> development
> >> or review of most of the JIRAs'.
> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> there
> >> is no impact when ATSv2 is turned off.
> >>
> >> Regards,
> >> Varun Saxena.
> >>
> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> >> vrushalic2016@gmail.com> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> >>> will
> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> >>>
> >>> We have previously completed one merge onto trunk [3] and Timeline
> Service
> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>>
> >>> Since then, we have been working on extending the capabilities of
> Timeline
> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> >>> confident that the state of the feature meets the criteria to be merged
> >>> onto trunk and we'd love folks to get their hands on it in a test
> capacity
> >>> and provide valuable feedback so that we can make it production-ready.
> >>>
> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> >>> usability improvements based on a new architecture. What we would like
> to
> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> >>> complete end-to-end read/write flow with security and read level
> >>> authorization via whitelists. You should be able to start setting it up
> >>> and
> >>> testing it.
> >>>
> >>> At a high level, the following are the key features that have been
> >>> implemented since alpha1:
> >>> - Security via Kerberos Authentication and delegation tokens
> >>> - Read side simple authorization via whitelist
> >>> - Client configurable entity sort ordering
> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> metrics by
> >>> timerange, pagination, sub-app entities
> >>> - Support for storing sub-application entities (entities that exist
> >>> outside
> >>> the scope of an application)
> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> >>> prefixes,
> >>> configurable hbase cluster
> >>> - Flow level aggregations done as dynamic (table level) coprocessors
> >>> - Uses latest stable HBase release 1.2.6
> >>>
> >>> There are a total of 82 subtasks that were completed as part of this
> >>> effort.
> >>>
> >>> We paid close attention to ensure that once disabled Timeline Service
> v.2
> >>> does not impact existing functionality when disabled (by default).
> >>>
> >>> Special thanks to a team of folks who worked hard and contributed
> towards
> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>>
> >>> Regards,
> >>> Vrushali
> >>>
> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27383.html
> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> >>>
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
I recall this discussion about a couple of years ago:
https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <st...@hortonworks.com>
wrote:
> I'd have assumed it would have gone in as one single patch, rather than a
> full history. I don't see why the trunk needs all the evolutionary history
> of a build.
>
> What should our policy/process be here?
>
> I do currently plan to merge the s3guard in as one single squashed patch;
> just getting HADOOP-14809 sorted first.
>
>
> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
> >
> > I'm adding my +1 (binding) to conclude the vote.
> >
> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > the merge to trunk shortly. Thanks everyone!
> >
> > Regards
> > Vrushali
> >
> >
> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> > varun.saxena.apache@gmail.com> wrote:
> >
> >> +1 (binding).
> >>
> >> Kudos to all the team members for their great work!
> >>
> >> Being part of the ATSv2 team, I have been involved with either
> development
> >> or review of most of the JIRAs'.
> >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> there
> >> is no impact when ATSv2 is turned off.
> >>
> >> Regards,
> >> Varun Saxena.
> >>
> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> >> vrushalic2016@gmail.com> wrote:
> >>
> >>> Hi folks,
> >>>
> >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> >>> will
> >>> run for 7 days, and will end August 29 11:00 PM PDT.
> >>>
> >>> We have previously completed one merge onto trunk [3] and Timeline
> Service
> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> >>>
> >>> Since then, we have been working on extending the capabilities of
> Timeline
> >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> >>> confident that the state of the feature meets the criteria to be merged
> >>> onto trunk and we'd love folks to get their hands on it in a test
> capacity
> >>> and provide valuable feedback so that we can make it production-ready.
> >>>
> >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> >>> usability improvements based on a new architecture. What we would like
> to
> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> >>> complete end-to-end read/write flow with security and read level
> >>> authorization via whitelists. You should be able to start setting it up
> >>> and
> >>> testing it.
> >>>
> >>> At a high level, the following are the key features that have been
> >>> implemented since alpha1:
> >>> - Security via Kerberos Authentication and delegation tokens
> >>> - Read side simple authorization via whitelist
> >>> - Client configurable entity sort ordering
> >>> - Richer REST APIs for apps, app attempts, containers, fetching
> metrics by
> >>> timerange, pagination, sub-app entities
> >>> - Support for storing sub-application entities (entities that exist
> >>> outside
> >>> the scope of an application)
> >>> - Configurable TTLs (time-to-live) for tables, configurable table
> >>> prefixes,
> >>> configurable hbase cluster
> >>> - Flow level aggregations done as dynamic (table level) coprocessors
> >>> - Uses latest stable HBase release 1.2.6
> >>>
> >>> There are a total of 82 subtasks that were completed as part of this
> >>> effort.
> >>>
> >>> We paid close attention to ensure that once disabled Timeline Service
> v.2
> >>> does not impact existing functionality when disabled (by default).
> >>>
> >>> Special thanks to a team of folks who worked hard and contributed
> towards
> >>> this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >>>
> >>> Regards,
> >>> Vrushali
> >>>
> >>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27383.html
> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
> >>>
> >>
> >>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to
trunk
Posted by Steve Loughran <st...@hortonworks.com>.
I'd have assumed it would have gone in as one single patch, rather than a full history. I don't see why the trunk needs all the evolutionary history of a build.
What should our policy/process be here?
I do currently plan to merge the s3guard in as one single squashed patch; just getting HADOOP-14809 sorted first.
> On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
>
> I'm adding my +1 (binding) to conclude the vote.
>
> With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> the merge to trunk shortly. Thanks everyone!
>
> Regards
> Vrushali
>
>
> On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
>> +1 (binding).
>>
>> Kudos to all the team members for their great work!
>>
>> Being part of the ATSv2 team, I have been involved with either development
>> or review of most of the JIRAs'.
>> Tested ATSv2 in both secure and non-secure mode. Also verified that there
>> is no impact when ATSv2 is turned off.
>>
>> Regards,
>> Varun Saxena.
>>
>> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> vrushalic2016@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> Per earlier discussion [1], I'd like to start a formal vote to merge
>>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>>> will
>>> run for 7 days, and will end August 29 11:00 PM PDT.
>>>
>>> We have previously completed one merge onto trunk [3] and Timeline Service
>>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>
>>> Since then, we have been working on extending the capabilities of Timeline
>>> Service v2 in a feature branch [2] for a while, and we are reasonably
>>> confident that the state of the feature meets the criteria to be merged
>>> onto trunk and we'd love folks to get their hands on it in a test capacity
>>> and provide valuable feedback so that we can make it production-ready.
>>>
>>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>>> usability improvements based on a new architecture. What we would like to
>>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>>> complete end-to-end read/write flow with security and read level
>>> authorization via whitelists. You should be able to start setting it up
>>> and
>>> testing it.
>>>
>>> At a high level, the following are the key features that have been
>>> implemented since alpha1:
>>> - Security via Kerberos Authentication and delegation tokens
>>> - Read side simple authorization via whitelist
>>> - Client configurable entity sort ordering
>>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>>> timerange, pagination, sub-app entities
>>> - Support for storing sub-application entities (entities that exist
>>> outside
>>> the scope of an application)
>>> - Configurable TTLs (time-to-live) for tables, configurable table
>>> prefixes,
>>> configurable hbase cluster
>>> - Flow level aggregations done as dynamic (table level) coprocessors
>>> - Uses latest stable HBase release 1.2.6
>>>
>>> There are a total of 82 subtasks that were completed as part of this
>>> effort.
>>>
>>> We paid close attention to ensure that once disabled Timeline Service v.2
>>> does not impact existing functionality when disabled (by default).
>>>
>>> Special thanks to a team of folks who worked hard and contributed towards
>>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>>
>>> Regards,
>>> Vrushali
>>>
>>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>>
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to
trunk
Posted by Steve Loughran <st...@hortonworks.com>.
I'd have assumed it would have gone in as one single patch, rather than a full history. I don't see why the trunk needs all the evolutionary history of a build.
What should our policy/process be here?
I do currently plan to merge the s3guard in as one single squashed patch; just getting HADOOP-14809 sorted first.
> On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
>
> I'm adding my +1 (binding) to conclude the vote.
>
> With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> the merge to trunk shortly. Thanks everyone!
>
> Regards
> Vrushali
>
>
> On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
>> +1 (binding).
>>
>> Kudos to all the team members for their great work!
>>
>> Being part of the ATSv2 team, I have been involved with either development
>> or review of most of the JIRAs'.
>> Tested ATSv2 in both secure and non-secure mode. Also verified that there
>> is no impact when ATSv2 is turned off.
>>
>> Regards,
>> Varun Saxena.
>>
>> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> vrushalic2016@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> Per earlier discussion [1], I'd like to start a formal vote to merge
>>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>>> will
>>> run for 7 days, and will end August 29 11:00 PM PDT.
>>>
>>> We have previously completed one merge onto trunk [3] and Timeline Service
>>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>
>>> Since then, we have been working on extending the capabilities of Timeline
>>> Service v2 in a feature branch [2] for a while, and we are reasonably
>>> confident that the state of the feature meets the criteria to be merged
>>> onto trunk and we'd love folks to get their hands on it in a test capacity
>>> and provide valuable feedback so that we can make it production-ready.
>>>
>>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>>> usability improvements based on a new architecture. What we would like to
>>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>>> complete end-to-end read/write flow with security and read level
>>> authorization via whitelists. You should be able to start setting it up
>>> and
>>> testing it.
>>>
>>> At a high level, the following are the key features that have been
>>> implemented since alpha1:
>>> - Security via Kerberos Authentication and delegation tokens
>>> - Read side simple authorization via whitelist
>>> - Client configurable entity sort ordering
>>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>>> timerange, pagination, sub-app entities
>>> - Support for storing sub-application entities (entities that exist
>>> outside
>>> the scope of an application)
>>> - Configurable TTLs (time-to-live) for tables, configurable table
>>> prefixes,
>>> configurable hbase cluster
>>> - Flow level aggregations done as dynamic (table level) coprocessors
>>> - Uses latest stable HBase release 1.2.6
>>>
>>> There are a total of 82 subtasks that were completed as part of this
>>> effort.
>>>
>>> We paid close attention to ensure that once disabled Timeline Service v.2
>>> does not impact existing functionality when disabled (by default).
>>>
>>> Special thanks to a team of folks who worked hard and contributed towards
>>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>>
>>> Regards,
>>> Vrushali
>>>
>>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>>
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to
trunk
Posted by Steve Loughran <st...@hortonworks.com>.
I'd have assumed it would have gone in as one single patch, rather than a full history. I don't see why the trunk needs all the evolutionary history of a build.
What should our policy/process be here?
I do currently plan to merge the s3guard in as one single squashed patch; just getting HADOOP-14809 sorted first.
> On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
>
> I'm adding my +1 (binding) to conclude the vote.
>
> With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> the merge to trunk shortly. Thanks everyone!
>
> Regards
> Vrushali
>
>
> On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
>> +1 (binding).
>>
>> Kudos to all the team members for their great work!
>>
>> Being part of the ATSv2 team, I have been involved with either development
>> or review of most of the JIRAs'.
>> Tested ATSv2 in both secure and non-secure mode. Also verified that there
>> is no impact when ATSv2 is turned off.
>>
>> Regards,
>> Varun Saxena.
>>
>> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> vrushalic2016@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> Per earlier discussion [1], I'd like to start a formal vote to merge
>>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>>> will
>>> run for 7 days, and will end August 29 11:00 PM PDT.
>>>
>>> We have previously completed one merge onto trunk [3] and Timeline Service
>>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>
>>> Since then, we have been working on extending the capabilities of Timeline
>>> Service v2 in a feature branch [2] for a while, and we are reasonably
>>> confident that the state of the feature meets the criteria to be merged
>>> onto trunk and we'd love folks to get their hands on it in a test capacity
>>> and provide valuable feedback so that we can make it production-ready.
>>>
>>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>>> usability improvements based on a new architecture. What we would like to
>>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>>> complete end-to-end read/write flow with security and read level
>>> authorization via whitelists. You should be able to start setting it up
>>> and
>>> testing it.
>>>
>>> At a high level, the following are the key features that have been
>>> implemented since alpha1:
>>> - Security via Kerberos Authentication and delegation tokens
>>> - Read side simple authorization via whitelist
>>> - Client configurable entity sort ordering
>>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>>> timerange, pagination, sub-app entities
>>> - Support for storing sub-application entities (entities that exist
>>> outside
>>> the scope of an application)
>>> - Configurable TTLs (time-to-live) for tables, configurable table
>>> prefixes,
>>> configurable hbase cluster
>>> - Flow level aggregations done as dynamic (table level) coprocessors
>>> - Uses latest stable HBase release 1.2.6
>>>
>>> There are a total of 82 subtasks that were completed as part of this
>>> effort.
>>>
>>> We paid close attention to ensure that once disabled Timeline Service v.2
>>> does not impact existing functionality when disabled (by default).
>>>
>>> Special thanks to a team of folks who worked hard and contributed towards
>>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>>
>>> Regards,
>>> Vrushali
>>>
>>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>>
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to
trunk
Posted by Steve Loughran <st...@hortonworks.com>.
I'd have assumed it would have gone in as one single patch, rather than a full history. I don't see why the trunk needs all the evolutionary history of a build.
What should our policy/process be here?
I do currently plan to merge the s3guard in as one single squashed patch; just getting HADOOP-14809 sorted first.
> On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com> wrote:
>
> I'm adding my +1 (binding) to conclude the vote.
>
> With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> the merge to trunk shortly. Thanks everyone!
>
> Regards
> Vrushali
>
>
> On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
>> +1 (binding).
>>
>> Kudos to all the team members for their great work!
>>
>> Being part of the ATSv2 team, I have been involved with either development
>> or review of most of the JIRAs'.
>> Tested ATSv2 in both secure and non-secure mode. Also verified that there
>> is no impact when ATSv2 is turned off.
>>
>> Regards,
>> Varun Saxena.
>>
>> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> vrushalic2016@gmail.com> wrote:
>>
>>> Hi folks,
>>>
>>> Per earlier discussion [1], I'd like to start a formal vote to merge
>>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>>> will
>>> run for 7 days, and will end August 29 11:00 PM PDT.
>>>
>>> We have previously completed one merge onto trunk [3] and Timeline Service
>>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>
>>> Since then, we have been working on extending the capabilities of Timeline
>>> Service v2 in a feature branch [2] for a while, and we are reasonably
>>> confident that the state of the feature meets the criteria to be merged
>>> onto trunk and we'd love folks to get their hands on it in a test capacity
>>> and provide valuable feedback so that we can make it production-ready.
>>>
>>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>>> usability improvements based on a new architecture. What we would like to
>>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>>> complete end-to-end read/write flow with security and read level
>>> authorization via whitelists. You should be able to start setting it up
>>> and
>>> testing it.
>>>
>>> At a high level, the following are the key features that have been
>>> implemented since alpha1:
>>> - Security via Kerberos Authentication and delegation tokens
>>> - Read side simple authorization via whitelist
>>> - Client configurable entity sort ordering
>>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>>> timerange, pagination, sub-app entities
>>> - Support for storing sub-application entities (entities that exist
>>> outside
>>> the scope of an application)
>>> - Configurable TTLs (time-to-live) for tables, configurable table
>>> prefixes,
>>> configurable hbase cluster
>>> - Flow level aggregations done as dynamic (table level) coprocessors
>>> - Uses latest stable HBase release 1.2.6
>>>
>>> There are a total of 82 subtasks that were completed as part of this
>>> effort.
>>>
>>> We paid close attention to ensure that once disabled Timeline Service v.2
>>> does not impact existing functionality when disabled (by default).
>>>
>>> Special thanks to a team of folks who worked hard and contributed towards
>>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>>
>>> Regards,
>>> Vrushali
>>>
>>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>>
>>
>>
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
I'm adding my +1 (binding) to conclude the vote.
With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
the merge to trunk shortly. Thanks everyone!
Regards
Vrushali
On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> +1 (binding).
>
> Kudos to all the team members for their great work!
>
> Being part of the ATSv2 team, I have been involved with either development
> or review of most of the JIRAs'.
> Tested ATSv2 in both secure and non-secure mode. Also verified that there
> is no impact when ATSv2 is turned off.
>
> Regards,
> Varun Saxena.
>
> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
>> Hi folks,
>>
>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> will
>> run for 7 days, and will end August 29 11:00 PM PDT.
>>
>> We have previously completed one merge onto trunk [3] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> confident that the state of the feature meets the criteria to be merged
>> onto trunk and we'd love folks to get their hands on it in a test capacity
>> and provide valuable feedback so that we can make it production-ready.
>>
>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>> usability improvements based on a new architecture. What we would like to
>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> complete end-to-end read/write flow with security and read level
>> authorization via whitelists. You should be able to start setting it up
>> and
>> testing it.
>>
>> At a high level, the following are the key features that have been
>> implemented since alpha1:
>> - Security via Kerberos Authentication and delegation tokens
>> - Read side simple authorization via whitelist
>> - Client configurable entity sort ordering
>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>> timerange, pagination, sub-app entities
>> - Support for storing sub-application entities (entities that exist
>> outside
>> the scope of an application)
>> - Configurable TTLs (time-to-live) for tables, configurable table
>> prefixes,
>> configurable hbase cluster
>> - Flow level aggregations done as dynamic (table level) coprocessors
>> - Uses latest stable HBase release 1.2.6
>>
>> There are a total of 82 subtasks that were completed as part of this
>> effort.
>>
>> We paid close attention to ensure that once disabled Timeline Service v.2
>> does not impact existing functionality when disabled (by default).
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Regards,
>> Vrushali
>>
>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
I'm adding my +1 (binding) to conclude the vote.
With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
the merge to trunk shortly. Thanks everyone!
Regards
Vrushali
On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> +1 (binding).
>
> Kudos to all the team members for their great work!
>
> Being part of the ATSv2 team, I have been involved with either development
> or review of most of the JIRAs'.
> Tested ATSv2 in both secure and non-secure mode. Also verified that there
> is no impact when ATSv2 is turned off.
>
> Regards,
> Varun Saxena.
>
> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
>> Hi folks,
>>
>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> will
>> run for 7 days, and will end August 29 11:00 PM PDT.
>>
>> We have previously completed one merge onto trunk [3] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> confident that the state of the feature meets the criteria to be merged
>> onto trunk and we'd love folks to get their hands on it in a test capacity
>> and provide valuable feedback so that we can make it production-ready.
>>
>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>> usability improvements based on a new architecture. What we would like to
>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> complete end-to-end read/write flow with security and read level
>> authorization via whitelists. You should be able to start setting it up
>> and
>> testing it.
>>
>> At a high level, the following are the key features that have been
>> implemented since alpha1:
>> - Security via Kerberos Authentication and delegation tokens
>> - Read side simple authorization via whitelist
>> - Client configurable entity sort ordering
>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>> timerange, pagination, sub-app entities
>> - Support for storing sub-application entities (entities that exist
>> outside
>> the scope of an application)
>> - Configurable TTLs (time-to-live) for tables, configurable table
>> prefixes,
>> configurable hbase cluster
>> - Flow level aggregations done as dynamic (table level) coprocessors
>> - Uses latest stable HBase release 1.2.6
>>
>> There are a total of 82 subtasks that were completed as part of this
>> effort.
>>
>> We paid close attention to ensure that once disabled Timeline Service v.2
>> does not impact existing functionality when disabled (by default).
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Regards,
>> Vrushali
>>
>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
I'm adding my +1 (binding) to conclude the vote.
With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
the merge to trunk shortly. Thanks everyone!
Regards
Vrushali
On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> +1 (binding).
>
> Kudos to all the team members for their great work!
>
> Being part of the ATSv2 team, I have been involved with either development
> or review of most of the JIRAs'.
> Tested ATSv2 in both secure and non-secure mode. Also verified that there
> is no impact when ATSv2 is turned off.
>
> Regards,
> Varun Saxena.
>
> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
>> Hi folks,
>>
>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> will
>> run for 7 days, and will end August 29 11:00 PM PDT.
>>
>> We have previously completed one merge onto trunk [3] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> confident that the state of the feature meets the criteria to be merged
>> onto trunk and we'd love folks to get their hands on it in a test capacity
>> and provide valuable feedback so that we can make it production-ready.
>>
>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>> usability improvements based on a new architecture. What we would like to
>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> complete end-to-end read/write flow with security and read level
>> authorization via whitelists. You should be able to start setting it up
>> and
>> testing it.
>>
>> At a high level, the following are the key features that have been
>> implemented since alpha1:
>> - Security via Kerberos Authentication and delegation tokens
>> - Read side simple authorization via whitelist
>> - Client configurable entity sort ordering
>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>> timerange, pagination, sub-app entities
>> - Support for storing sub-application entities (entities that exist
>> outside
>> the scope of an application)
>> - Configurable TTLs (time-to-live) for tables, configurable table
>> prefixes,
>> configurable hbase cluster
>> - Flow level aggregations done as dynamic (table level) coprocessors
>> - Uses latest stable HBase release 1.2.6
>>
>> There are a total of 82 subtasks that were completed as part of this
>> effort.
>>
>> We paid close attention to ensure that once disabled Timeline Service v.2
>> does not impact existing functionality when disabled (by default).
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Regards,
>> Vrushali
>>
>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
I'm adding my +1 (binding) to conclude the vote.
With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
the merge to trunk shortly. Thanks everyone!
Regards
Vrushali
On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> +1 (binding).
>
> Kudos to all the team members for their great work!
>
> Being part of the ATSv2 team, I have been involved with either development
> or review of most of the JIRAs'.
> Tested ATSv2 in both secure and non-secure mode. Also verified that there
> is no impact when ATSv2 is turned off.
>
> Regards,
> Varun Saxena.
>
> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
>> Hi folks,
>>
>> Per earlier discussion [1], I'd like to start a formal vote to merge
>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
>> will
>> run for 7 days, and will end August 29 11:00 PM PDT.
>>
>> We have previously completed one merge onto trunk [3] and Timeline Service
>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>
>> Since then, we have been working on extending the capabilities of Timeline
>> Service v2 in a feature branch [2] for a while, and we are reasonably
>> confident that the state of the feature meets the criteria to be merged
>> onto trunk and we'd love folks to get their hands on it in a test capacity
>> and provide valuable feedback so that we can make it production-ready.
>>
>> In a nutshell, Timeline Service v.2 delivers significant scalability and
>> usability improvements based on a new architecture. What we would like to
>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
>> complete end-to-end read/write flow with security and read level
>> authorization via whitelists. You should be able to start setting it up
>> and
>> testing it.
>>
>> At a high level, the following are the key features that have been
>> implemented since alpha1:
>> - Security via Kerberos Authentication and delegation tokens
>> - Read side simple authorization via whitelist
>> - Client configurable entity sort ordering
>> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
>> timerange, pagination, sub-app entities
>> - Support for storing sub-application entities (entities that exist
>> outside
>> the scope of an application)
>> - Configurable TTLs (time-to-live) for tables, configurable table
>> prefixes,
>> configurable hbase cluster
>> - Flow level aggregations done as dynamic (table level) coprocessors
>> - Uses latest stable HBase release 1.2.6
>>
>> There are a total of 82 subtasks that were completed as part of this
>> effort.
>>
>> We paid close attention to ensure that once disabled Timeline Service v.2
>> does not impact existing functionality when disabled (by default).
>>
>> Special thanks to a team of folks who worked hard and contributed towards
>> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>
>> Regards,
>> Vrushali
>>
>> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
+1 (binding).
Kudos to all the team members for their great work!
Being part of the ATSv2 team, I have been involved with either development
or review of most of the JIRAs'.
Tested ATSv2 in both secure and non-secure mode. Also verified that there
is no impact when ATSv2 is turned off.
Regards,
Varun Saxena.
On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Rohith Sharma K S <ro...@apache.org>.
+1 (binding)
Thank you very much for the great team work!
Built from source and deployed in secured cluster. The below are the test
result.
Deployment :
Standard hadoop security deployment authentication and authorization as
well.
Branch-2 Hadoop and Hbase security cluster.
Branch-3 Hadoop security cluster. HBase client is pointing to Branch-2
hbase cluster.
All security configurations are set in-place.
Each service is running with its own user. Say, HDFS is running with
hdfs, YARN user is running with yarn, Hbase is running with hbase
Smoke test user : test-user
Test Cases :
Authentication :
Verify for all daemons start up successful : OK
Run a MR job using test-user : OK
Verify for REST API’s with in the scope of application : OK
Verify for REST API’s newly added I.e outside scope of application : OK.
RM Restart/ NM restart / RM_work-preserving restart has executed and
verified for data : OK. (Entity validation is done, but not entity data
validation!
Token redistribution to AM, NM is verified.
Authorization :
1 . Basic whitelisting of users to read has been validated. Works as
expected!
Disabling TSv2 configuration is also being tested.
Thanks & Regards
Rohith Sharma K S
On 22 August 2017 at 12:02, Vrushali Channapattan <vr...@gmail.com>
wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Rohith Sharma K S <ro...@apache.org>.
+1 (binding)
Thank you very much for the great team work!
Built from source and deployed in secured cluster. The below are the test
result.
Deployment :
Standard hadoop security deployment authentication and authorization as
well.
Branch-2 Hadoop and Hbase security cluster.
Branch-3 Hadoop security cluster. HBase client is pointing to Branch-2
hbase cluster.
All security configurations are set in-place.
Each service is running with its own user. Say, HDFS is running with
hdfs, YARN user is running with yarn, Hbase is running with hbase
Smoke test user : test-user
Test Cases :
Authentication :
Verify for all daemons start up successful : OK
Run a MR job using test-user : OK
Verify for REST API’s with in the scope of application : OK
Verify for REST API’s newly added I.e outside scope of application : OK.
RM Restart/ NM restart / RM_work-preserving restart has executed and
verified for data : OK. (Entity validation is done, but not entity data
validation!
Token redistribution to AM, NM is verified.
Authorization :
1 . Basic whitelisting of users to read has been validated. Works as
expected!
Disabling TSv2 configuration is also being tested.
Thanks & Regards
Rohith Sharma K S
On 22 August 2017 at 12:02, Vrushali Channapattan <vr...@gmail.com>
wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "J. Rottinghuis" <jr...@gmail.com>.
That makes sense to me, after merge to trunk and branch 2, it probably
makes sense to create a new jira and feature branch for atsv2 additional
feature dev.
Cheers,
Joep
On Fri, Aug 25, 2017 at 3:05 PM, Vrushali C <vr...@gmail.com> wrote:
> Hi Subru,
>
> Thanks for your vote and your response!
>
> Regarding your question about merging to branch2, I think, after the merge
> to trunk is done, it will be good to merge to branch-2 as soon as we can.
> A lot of testing with trunk has been done presently and it will be good to
> repeat the same with the YARN-5355_branch2 branch before we start working
> on newer features. That way trunk and branch-2 would be in a similar state
> with respect to timeline service v2 development.
>
> thanks
> Vrushali
>
>
>
>
>
>
>
> On Fri, Aug 25, 2017 at 2:28 PM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
> > Thanks Subru for voting.
> >
> >
> >> What are the timelines you are looking for getting this into branch-2?
> >
> > We haven't yet decided on it and were thinking of discussing this in
> > detail within the team after merge to trunk.
> > The timelines would depend on whether we release whatever we merge to
> > trunk in 2.9 or would we want to get in few other features which people
> > would like to see in 2.9
> > This would require some discussion with the stakeholders.
> > We were thinking of having a short discussion with you guys as well to
> > find out whether there are any further gaps in ATSv2 with respect to
> > federation support and if they can be filled before 2.9 release.
> >
> > Assuming 2.9 is targetted for October end, we would have to start a merge
> > vote in September end or October 1st week which leaves us with very
> little
> > time to take up large changes anyways.
> >
> > We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though
> which
> > we rebase with branch-2 regularly. So, if we decide to merge in branch-2
> > without any additional changes, we would be able to go for branch-2 merge
> > discussion and vote almost immediately.
> >
> > Regards,
> > Varun Saxena.
> >
> >
> >
> >
> > On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K <su...@gmail.com>
> > wrote:
> >
> >> +1 (binding).
> >>
> >> I have been following the effort and had few design discussions around
> the
> >> team especially about how it integrates with Federation. Overall I feel
> >> it's a welcome improvement to YARN.
> >>
> >> What are the timelines you are looking for getting this into branch-2?
> >>
> >> Thanks,
> >> Subru
> >>
> >> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
> >>
> >> > +1 (binding)
> >> >
> >> > I've built the current branch, and checked out a few basic areas
> >> including
> >> > documentation. Also perused the most recent changes that went in.
> >> >
> >> > Thanks much for the great team work! I look forward to seeing it in
> >> action.
> >> >
> >> > Regards,
> >> > Sangjin
> >> >
> >> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
> >> > wrote:
> >> >
> >> > > +1 from my side.
> >> > >
> >> > > More from the perspective of ensuring there is no impact of ATSv2
> >> when it
> >> > > is off (by default), I deployed the latest YARN-5355 bits into a few
> >> > > clusters and ran internal Smoke tests. The tests shows no impact
> when
> >> > ATSv2
> >> > > is off.
> >> > >
> >> > > Best,
> >> > > Haibo
> >> > >
> >> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> >> > >
> >> > > > Thank you very much Vrushali, Rohith, Varun and other folks who
> made
> >> > this
> >> > > > happen. Great work, really appreciate the same!!
> >> > > >
> >> > > > +1 (binding) from my side:
> >> > > >
> >> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> >> > > > # Accessed new YARN UI which shows various flows/flow activity
> etc.
> >> > Seems
> >> > > > fine.
> >> > > > # Based on code, looks like all apis are compatible.
> >> > > > # REST api docs looks fine as well, I guess we could improve that
> a
> >> bit
> >> > > > more post merge as well.
> >> > > > # Adding to additional thoughts which are discussed here, native
> >> > service
> >> > > > also could publish events to atsv2. I think that work is also
> >> happened
> >> > in
> >> > > > branch.
> >> > > >
> >> > > > Looking forward to a much wider adoption of ATSv2 with more
> >> projects.
> >> > > >
> >> > > > Thanks
> >> > > > Sunil
> >> > > >
> >> > > >
> >> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> >> > > > vrushalic2016@gmail.com> wrote:
> >> > > >
> >> > > > > Hi folks,
> >> > > > >
> >> > > > > Per earlier discussion [1], I'd like to start a formal vote to
> >> merge
> >> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
> The
> >> > vote
> >> > > > will
> >> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
> >> > > > >
> >> > > > > We have previously completed one merge onto trunk [3] and
> Timeline
> >> > > > Service
> >> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> >> > > > >
> >> > > > > Since then, we have been working on extending the capabilities
> of
> >> > > > Timeline
> >> > > > > Service v2 in a feature branch [2] for a while, and we are
> >> reasonably
> >> > > > > confident that the state of the feature meets the criteria to be
> >> > merged
> >> > > > > onto trunk and we'd love folks to get their hands on it in a
> test
> >> > > > capacity
> >> > > > > and provide valuable feedback so that we can make it
> >> > production-ready.
> >> > > > >
> >> > > > > In a nutshell, Timeline Service v.2 delivers significant
> >> scalability
> >> > > and
> >> > > > > usability improvements based on a new architecture. What we
> would
> >> > like
> >> > > to
> >> > > > > merge to trunk is termed "alpha 2" (milestone 2). The feature
> has
> >> a
> >> > > > > complete end-to-end read/write flow with security and read level
> >> > > > > authorization via whitelists. You should be able to start
> setting
> >> it
> >> > up
> >> > > > and
> >> > > > > testing it.
> >> > > > >
> >> > > > > At a high level, the following are the key features that have
> been
> >> > > > > implemented since alpha1:
> >> > > > > - Security via Kerberos Authentication and delegation tokens
> >> > > > > - Read side simple authorization via whitelist
> >> > > > > - Client configurable entity sort ordering
> >> > > > > - Richer REST APIs for apps, app attempts, containers, fetching
> >> > metrics
> >> > > > by
> >> > > > > timerange, pagination, sub-app entities
> >> > > > > - Support for storing sub-application entities (entities that
> >> exist
> >> > > > outside
> >> > > > > the scope of an application)
> >> > > > > - Configurable TTLs (time-to-live) for tables, configurable
> table
> >> > > > prefixes,
> >> > > > > configurable hbase cluster
> >> > > > > - Flow level aggregations done as dynamic (table level)
> >> coprocessors
> >> > > > > - Uses latest stable HBase release 1.2.6
> >> > > > >
> >> > > > > There are a total of 82 subtasks that were completed as part of
> >> this
> >> > > > > effort.
> >> > > > >
> >> > > > > We paid close attention to ensure that once disabled Timeline
> >> Service
> >> > > v.2
> >> > > > > does not impact existing functionality when disabled (by
> default).
> >> > > > >
> >> > > > > Special thanks to a team of folks who worked hard and
> contributed
> >> > > towards
> >> > > > > this effort with patches, reviews and guidance: Rohith Sharma K
> S,
> >> > > Varun
> >> > > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> >> Joep
> >> > > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >> > > > >
> >> > > > > Regards,
> >> > > > > Vrushali
> >> > > > >
> >> > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27
> >> > > 383.html
> >> > > > > [2] https://issues.apache.org/jira/browse/YARN-5355
> >> > > > > [3] https://issues.apache.org/jira/browse/YARN-2928
> >> > > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "J. Rottinghuis" <jr...@gmail.com>.
That makes sense to me, after merge to trunk and branch 2, it probably
makes sense to create a new jira and feature branch for atsv2 additional
feature dev.
Cheers,
Joep
On Fri, Aug 25, 2017 at 3:05 PM, Vrushali C <vr...@gmail.com> wrote:
> Hi Subru,
>
> Thanks for your vote and your response!
>
> Regarding your question about merging to branch2, I think, after the merge
> to trunk is done, it will be good to merge to branch-2 as soon as we can.
> A lot of testing with trunk has been done presently and it will be good to
> repeat the same with the YARN-5355_branch2 branch before we start working
> on newer features. That way trunk and branch-2 would be in a similar state
> with respect to timeline service v2 development.
>
> thanks
> Vrushali
>
>
>
>
>
>
>
> On Fri, Aug 25, 2017 at 2:28 PM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
> > Thanks Subru for voting.
> >
> >
> >> What are the timelines you are looking for getting this into branch-2?
> >
> > We haven't yet decided on it and were thinking of discussing this in
> > detail within the team after merge to trunk.
> > The timelines would depend on whether we release whatever we merge to
> > trunk in 2.9 or would we want to get in few other features which people
> > would like to see in 2.9
> > This would require some discussion with the stakeholders.
> > We were thinking of having a short discussion with you guys as well to
> > find out whether there are any further gaps in ATSv2 with respect to
> > federation support and if they can be filled before 2.9 release.
> >
> > Assuming 2.9 is targetted for October end, we would have to start a merge
> > vote in September end or October 1st week which leaves us with very
> little
> > time to take up large changes anyways.
> >
> > We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though
> which
> > we rebase with branch-2 regularly. So, if we decide to merge in branch-2
> > without any additional changes, we would be able to go for branch-2 merge
> > discussion and vote almost immediately.
> >
> > Regards,
> > Varun Saxena.
> >
> >
> >
> >
> > On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K <su...@gmail.com>
> > wrote:
> >
> >> +1 (binding).
> >>
> >> I have been following the effort and had few design discussions around
> the
> >> team especially about how it integrates with Federation. Overall I feel
> >> it's a welcome improvement to YARN.
> >>
> >> What are the timelines you are looking for getting this into branch-2?
> >>
> >> Thanks,
> >> Subru
> >>
> >> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
> >>
> >> > +1 (binding)
> >> >
> >> > I've built the current branch, and checked out a few basic areas
> >> including
> >> > documentation. Also perused the most recent changes that went in.
> >> >
> >> > Thanks much for the great team work! I look forward to seeing it in
> >> action.
> >> >
> >> > Regards,
> >> > Sangjin
> >> >
> >> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
> >> > wrote:
> >> >
> >> > > +1 from my side.
> >> > >
> >> > > More from the perspective of ensuring there is no impact of ATSv2
> >> when it
> >> > > is off (by default), I deployed the latest YARN-5355 bits into a few
> >> > > clusters and ran internal Smoke tests. The tests shows no impact
> when
> >> > ATSv2
> >> > > is off.
> >> > >
> >> > > Best,
> >> > > Haibo
> >> > >
> >> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> >> > >
> >> > > > Thank you very much Vrushali, Rohith, Varun and other folks who
> made
> >> > this
> >> > > > happen. Great work, really appreciate the same!!
> >> > > >
> >> > > > +1 (binding) from my side:
> >> > > >
> >> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> >> > > > # Accessed new YARN UI which shows various flows/flow activity
> etc.
> >> > Seems
> >> > > > fine.
> >> > > > # Based on code, looks like all apis are compatible.
> >> > > > # REST api docs looks fine as well, I guess we could improve that
> a
> >> bit
> >> > > > more post merge as well.
> >> > > > # Adding to additional thoughts which are discussed here, native
> >> > service
> >> > > > also could publish events to atsv2. I think that work is also
> >> happened
> >> > in
> >> > > > branch.
> >> > > >
> >> > > > Looking forward to a much wider adoption of ATSv2 with more
> >> projects.
> >> > > >
> >> > > > Thanks
> >> > > > Sunil
> >> > > >
> >> > > >
> >> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> >> > > > vrushalic2016@gmail.com> wrote:
> >> > > >
> >> > > > > Hi folks,
> >> > > > >
> >> > > > > Per earlier discussion [1], I'd like to start a formal vote to
> >> merge
> >> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
> The
> >> > vote
> >> > > > will
> >> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
> >> > > > >
> >> > > > > We have previously completed one merge onto trunk [3] and
> Timeline
> >> > > > Service
> >> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> >> > > > >
> >> > > > > Since then, we have been working on extending the capabilities
> of
> >> > > > Timeline
> >> > > > > Service v2 in a feature branch [2] for a while, and we are
> >> reasonably
> >> > > > > confident that the state of the feature meets the criteria to be
> >> > merged
> >> > > > > onto trunk and we'd love folks to get their hands on it in a
> test
> >> > > > capacity
> >> > > > > and provide valuable feedback so that we can make it
> >> > production-ready.
> >> > > > >
> >> > > > > In a nutshell, Timeline Service v.2 delivers significant
> >> scalability
> >> > > and
> >> > > > > usability improvements based on a new architecture. What we
> would
> >> > like
> >> > > to
> >> > > > > merge to trunk is termed "alpha 2" (milestone 2). The feature
> has
> >> a
> >> > > > > complete end-to-end read/write flow with security and read level
> >> > > > > authorization via whitelists. You should be able to start
> setting
> >> it
> >> > up
> >> > > > and
> >> > > > > testing it.
> >> > > > >
> >> > > > > At a high level, the following are the key features that have
> been
> >> > > > > implemented since alpha1:
> >> > > > > - Security via Kerberos Authentication and delegation tokens
> >> > > > > - Read side simple authorization via whitelist
> >> > > > > - Client configurable entity sort ordering
> >> > > > > - Richer REST APIs for apps, app attempts, containers, fetching
> >> > metrics
> >> > > > by
> >> > > > > timerange, pagination, sub-app entities
> >> > > > > - Support for storing sub-application entities (entities that
> >> exist
> >> > > > outside
> >> > > > > the scope of an application)
> >> > > > > - Configurable TTLs (time-to-live) for tables, configurable
> table
> >> > > > prefixes,
> >> > > > > configurable hbase cluster
> >> > > > > - Flow level aggregations done as dynamic (table level)
> >> coprocessors
> >> > > > > - Uses latest stable HBase release 1.2.6
> >> > > > >
> >> > > > > There are a total of 82 subtasks that were completed as part of
> >> this
> >> > > > > effort.
> >> > > > >
> >> > > > > We paid close attention to ensure that once disabled Timeline
> >> Service
> >> > > v.2
> >> > > > > does not impact existing functionality when disabled (by
> default).
> >> > > > >
> >> > > > > Special thanks to a team of folks who worked hard and
> contributed
> >> > > towards
> >> > > > > this effort with patches, reviews and guidance: Rohith Sharma K
> S,
> >> > > Varun
> >> > > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> >> Joep
> >> > > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >> > > > >
> >> > > > > Regards,
> >> > > > > Vrushali
> >> > > > >
> >> > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27
> >> > > 383.html
> >> > > > > [2] https://issues.apache.org/jira/browse/YARN-5355
> >> > > > > [3] https://issues.apache.org/jira/browse/YARN-2928
> >> > > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "J. Rottinghuis" <jr...@gmail.com>.
That makes sense to me, after merge to trunk and branch 2, it probably
makes sense to create a new jira and feature branch for atsv2 additional
feature dev.
Cheers,
Joep
On Fri, Aug 25, 2017 at 3:05 PM, Vrushali C <vr...@gmail.com> wrote:
> Hi Subru,
>
> Thanks for your vote and your response!
>
> Regarding your question about merging to branch2, I think, after the merge
> to trunk is done, it will be good to merge to branch-2 as soon as we can.
> A lot of testing with trunk has been done presently and it will be good to
> repeat the same with the YARN-5355_branch2 branch before we start working
> on newer features. That way trunk and branch-2 would be in a similar state
> with respect to timeline service v2 development.
>
> thanks
> Vrushali
>
>
>
>
>
>
>
> On Fri, Aug 25, 2017 at 2:28 PM, varunsaxena@apache.org <
> varun.saxena.apache@gmail.com> wrote:
>
> > Thanks Subru for voting.
> >
> >
> >> What are the timelines you are looking for getting this into branch-2?
> >
> > We haven't yet decided on it and were thinking of discussing this in
> > detail within the team after merge to trunk.
> > The timelines would depend on whether we release whatever we merge to
> > trunk in 2.9 or would we want to get in few other features which people
> > would like to see in 2.9
> > This would require some discussion with the stakeholders.
> > We were thinking of having a short discussion with you guys as well to
> > find out whether there are any further gaps in ATSv2 with respect to
> > federation support and if they can be filled before 2.9 release.
> >
> > Assuming 2.9 is targetted for October end, we would have to start a merge
> > vote in September end or October 1st week which leaves us with very
> little
> > time to take up large changes anyways.
> >
> > We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though
> which
> > we rebase with branch-2 regularly. So, if we decide to merge in branch-2
> > without any additional changes, we would be able to go for branch-2 merge
> > discussion and vote almost immediately.
> >
> > Regards,
> > Varun Saxena.
> >
> >
> >
> >
> > On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K <su...@gmail.com>
> > wrote:
> >
> >> +1 (binding).
> >>
> >> I have been following the effort and had few design discussions around
> the
> >> team especially about how it integrates with Federation. Overall I feel
> >> it's a welcome improvement to YARN.
> >>
> >> What are the timelines you are looking for getting this into branch-2?
> >>
> >> Thanks,
> >> Subru
> >>
> >> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
> >>
> >> > +1 (binding)
> >> >
> >> > I've built the current branch, and checked out a few basic areas
> >> including
> >> > documentation. Also perused the most recent changes that went in.
> >> >
> >> > Thanks much for the great team work! I look forward to seeing it in
> >> action.
> >> >
> >> > Regards,
> >> > Sangjin
> >> >
> >> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
> >> > wrote:
> >> >
> >> > > +1 from my side.
> >> > >
> >> > > More from the perspective of ensuring there is no impact of ATSv2
> >> when it
> >> > > is off (by default), I deployed the latest YARN-5355 bits into a few
> >> > > clusters and ran internal Smoke tests. The tests shows no impact
> when
> >> > ATSv2
> >> > > is off.
> >> > >
> >> > > Best,
> >> > > Haibo
> >> > >
> >> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> >> > >
> >> > > > Thank you very much Vrushali, Rohith, Varun and other folks who
> made
> >> > this
> >> > > > happen. Great work, really appreciate the same!!
> >> > > >
> >> > > > +1 (binding) from my side:
> >> > > >
> >> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> >> > > > # Accessed new YARN UI which shows various flows/flow activity
> etc.
> >> > Seems
> >> > > > fine.
> >> > > > # Based on code, looks like all apis are compatible.
> >> > > > # REST api docs looks fine as well, I guess we could improve that
> a
> >> bit
> >> > > > more post merge as well.
> >> > > > # Adding to additional thoughts which are discussed here, native
> >> > service
> >> > > > also could publish events to atsv2. I think that work is also
> >> happened
> >> > in
> >> > > > branch.
> >> > > >
> >> > > > Looking forward to a much wider adoption of ATSv2 with more
> >> projects.
> >> > > >
> >> > > > Thanks
> >> > > > Sunil
> >> > > >
> >> > > >
> >> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> >> > > > vrushalic2016@gmail.com> wrote:
> >> > > >
> >> > > > > Hi folks,
> >> > > > >
> >> > > > > Per earlier discussion [1], I'd like to start a formal vote to
> >> merge
> >> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
> The
> >> > vote
> >> > > > will
> >> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
> >> > > > >
> >> > > > > We have previously completed one merge onto trunk [3] and
> Timeline
> >> > > > Service
> >> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> >> > > > >
> >> > > > > Since then, we have been working on extending the capabilities
> of
> >> > > > Timeline
> >> > > > > Service v2 in a feature branch [2] for a while, and we are
> >> reasonably
> >> > > > > confident that the state of the feature meets the criteria to be
> >> > merged
> >> > > > > onto trunk and we'd love folks to get their hands on it in a
> test
> >> > > > capacity
> >> > > > > and provide valuable feedback so that we can make it
> >> > production-ready.
> >> > > > >
> >> > > > > In a nutshell, Timeline Service v.2 delivers significant
> >> scalability
> >> > > and
> >> > > > > usability improvements based on a new architecture. What we
> would
> >> > like
> >> > > to
> >> > > > > merge to trunk is termed "alpha 2" (milestone 2). The feature
> has
> >> a
> >> > > > > complete end-to-end read/write flow with security and read level
> >> > > > > authorization via whitelists. You should be able to start
> setting
> >> it
> >> > up
> >> > > > and
> >> > > > > testing it.
> >> > > > >
> >> > > > > At a high level, the following are the key features that have
> been
> >> > > > > implemented since alpha1:
> >> > > > > - Security via Kerberos Authentication and delegation tokens
> >> > > > > - Read side simple authorization via whitelist
> >> > > > > - Client configurable entity sort ordering
> >> > > > > - Richer REST APIs for apps, app attempts, containers, fetching
> >> > metrics
> >> > > > by
> >> > > > > timerange, pagination, sub-app entities
> >> > > > > - Support for storing sub-application entities (entities that
> >> exist
> >> > > > outside
> >> > > > > the scope of an application)
> >> > > > > - Configurable TTLs (time-to-live) for tables, configurable
> table
> >> > > > prefixes,
> >> > > > > configurable hbase cluster
> >> > > > > - Flow level aggregations done as dynamic (table level)
> >> coprocessors
> >> > > > > - Uses latest stable HBase release 1.2.6
> >> > > > >
> >> > > > > There are a total of 82 subtasks that were completed as part of
> >> this
> >> > > > > effort.
> >> > > > >
> >> > > > > We paid close attention to ensure that once disabled Timeline
> >> Service
> >> > > v.2
> >> > > > > does not impact existing functionality when disabled (by
> default).
> >> > > > >
> >> > > > > Special thanks to a team of folks who worked hard and
> contributed
> >> > > towards
> >> > > > > this effort with patches, reviews and guidance: Rohith Sharma K
> S,
> >> > > Varun
> >> > > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> >> Joep
> >> > > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >> > > > >
> >> > > > > Regards,
> >> > > > > Vrushali
> >> > > > >
> >> > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/
> msg27
> >> > > 383.html
> >> > > > > [2] https://issues.apache.org/jira/browse/YARN-5355
> >> > > > > [3] https://issues.apache.org/jira/browse/YARN-2928
> >> > > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Hi Subru,
Thanks for your vote and your response!
Regarding your question about merging to branch2, I think, after the merge
to trunk is done, it will be good to merge to branch-2 as soon as we can.
A lot of testing with trunk has been done presently and it will be good to
repeat the same with the YARN-5355_branch2 branch before we start working
on newer features. That way trunk and branch-2 would be in a similar state
with respect to timeline service v2 development.
thanks
Vrushali
On Fri, Aug 25, 2017 at 2:28 PM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> Thanks Subru for voting.
>
>
>> What are the timelines you are looking for getting this into branch-2?
>
> We haven't yet decided on it and were thinking of discussing this in
> detail within the team after merge to trunk.
> The timelines would depend on whether we release whatever we merge to
> trunk in 2.9 or would we want to get in few other features which people
> would like to see in 2.9
> This would require some discussion with the stakeholders.
> We were thinking of having a short discussion with you guys as well to
> find out whether there are any further gaps in ATSv2 with respect to
> federation support and if they can be filled before 2.9 release.
>
> Assuming 2.9 is targetted for October end, we would have to start a merge
> vote in September end or October 1st week which leaves us with very little
> time to take up large changes anyways.
>
> We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though which
> we rebase with branch-2 regularly. So, if we decide to merge in branch-2
> without any additional changes, we would be able to go for branch-2 merge
> discussion and vote almost immediately.
>
> Regards,
> Varun Saxena.
>
>
>
>
> On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K <su...@gmail.com>
> wrote:
>
>> +1 (binding).
>>
>> I have been following the effort and had few design discussions around the
>> team especially about how it integrates with Federation. Overall I feel
>> it's a welcome improvement to YARN.
>>
>> What are the timelines you are looking for getting this into branch-2?
>>
>> Thanks,
>> Subru
>>
>> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
>>
>> > +1 (binding)
>> >
>> > I've built the current branch, and checked out a few basic areas
>> including
>> > documentation. Also perused the most recent changes that went in.
>> >
>> > Thanks much for the great team work! I look forward to seeing it in
>> action.
>> >
>> > Regards,
>> > Sangjin
>> >
>> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
>> > wrote:
>> >
>> > > +1 from my side.
>> > >
>> > > More from the perspective of ensuring there is no impact of ATSv2
>> when it
>> > > is off (by default), I deployed the latest YARN-5355 bits into a few
>> > > clusters and ran internal Smoke tests. The tests shows no impact when
>> > ATSv2
>> > > is off.
>> > >
>> > > Best,
>> > > Haibo
>> > >
>> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
>> > >
>> > > > Thank you very much Vrushali, Rohith, Varun and other folks who made
>> > this
>> > > > happen. Great work, really appreciate the same!!
>> > > >
>> > > > +1 (binding) from my side:
>> > > >
>> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
>> > > > # Accessed new YARN UI which shows various flows/flow activity etc.
>> > Seems
>> > > > fine.
>> > > > # Based on code, looks like all apis are compatible.
>> > > > # REST api docs looks fine as well, I guess we could improve that a
>> bit
>> > > > more post merge as well.
>> > > > # Adding to additional thoughts which are discussed here, native
>> > service
>> > > > also could publish events to atsv2. I think that work is also
>> happened
>> > in
>> > > > branch.
>> > > >
>> > > > Looking forward to a much wider adoption of ATSv2 with more
>> projects.
>> > > >
>> > > > Thanks
>> > > > Sunil
>> > > >
>> > > >
>> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
>> > > > vrushalic2016@gmail.com> wrote:
>> > > >
>> > > > > Hi folks,
>> > > > >
>> > > > > Per earlier discussion [1], I'd like to start a formal vote to
>> merge
>> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
>> > vote
>> > > > will
>> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
>> > > > >
>> > > > > We have previously completed one merge onto trunk [3] and Timeline
>> > > > Service
>> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
>> > > > >
>> > > > > Since then, we have been working on extending the capabilities of
>> > > > Timeline
>> > > > > Service v2 in a feature branch [2] for a while, and we are
>> reasonably
>> > > > > confident that the state of the feature meets the criteria to be
>> > merged
>> > > > > onto trunk and we'd love folks to get their hands on it in a test
>> > > > capacity
>> > > > > and provide valuable feedback so that we can make it
>> > production-ready.
>> > > > >
>> > > > > In a nutshell, Timeline Service v.2 delivers significant
>> scalability
>> > > and
>> > > > > usability improvements based on a new architecture. What we would
>> > like
>> > > to
>> > > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has
>> a
>> > > > > complete end-to-end read/write flow with security and read level
>> > > > > authorization via whitelists. You should be able to start setting
>> it
>> > up
>> > > > and
>> > > > > testing it.
>> > > > >
>> > > > > At a high level, the following are the key features that have been
>> > > > > implemented since alpha1:
>> > > > > - Security via Kerberos Authentication and delegation tokens
>> > > > > - Read side simple authorization via whitelist
>> > > > > - Client configurable entity sort ordering
>> > > > > - Richer REST APIs for apps, app attempts, containers, fetching
>> > metrics
>> > > > by
>> > > > > timerange, pagination, sub-app entities
>> > > > > - Support for storing sub-application entities (entities that
>> exist
>> > > > outside
>> > > > > the scope of an application)
>> > > > > - Configurable TTLs (time-to-live) for tables, configurable table
>> > > > prefixes,
>> > > > > configurable hbase cluster
>> > > > > - Flow level aggregations done as dynamic (table level)
>> coprocessors
>> > > > > - Uses latest stable HBase release 1.2.6
>> > > > >
>> > > > > There are a total of 82 subtasks that were completed as part of
>> this
>> > > > > effort.
>> > > > >
>> > > > > We paid close attention to ensure that once disabled Timeline
>> Service
>> > > v.2
>> > > > > does not impact existing functionality when disabled (by default).
>> > > > >
>> > > > > Special thanks to a team of folks who worked hard and contributed
>> > > towards
>> > > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
>> > > Varun
>> > > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
>> Joep
>> > > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> > > > >
>> > > > > Regards,
>> > > > > Vrushali
>> > > > >
>> > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
>> > > 383.html
>> > > > > [2] https://issues.apache.org/jira/browse/YARN-5355
>> > > > > [3] https://issues.apache.org/jira/browse/YARN-2928
>> > > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Hi Subru,
Thanks for your vote and your response!
Regarding your question about merging to branch2, I think, after the merge
to trunk is done, it will be good to merge to branch-2 as soon as we can.
A lot of testing with trunk has been done presently and it will be good to
repeat the same with the YARN-5355_branch2 branch before we start working
on newer features. That way trunk and branch-2 would be in a similar state
with respect to timeline service v2 development.
thanks
Vrushali
On Fri, Aug 25, 2017 at 2:28 PM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> Thanks Subru for voting.
>
>
>> What are the timelines you are looking for getting this into branch-2?
>
> We haven't yet decided on it and were thinking of discussing this in
> detail within the team after merge to trunk.
> The timelines would depend on whether we release whatever we merge to
> trunk in 2.9 or would we want to get in few other features which people
> would like to see in 2.9
> This would require some discussion with the stakeholders.
> We were thinking of having a short discussion with you guys as well to
> find out whether there are any further gaps in ATSv2 with respect to
> federation support and if they can be filled before 2.9 release.
>
> Assuming 2.9 is targetted for October end, we would have to start a merge
> vote in September end or October 1st week which leaves us with very little
> time to take up large changes anyways.
>
> We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though which
> we rebase with branch-2 regularly. So, if we decide to merge in branch-2
> without any additional changes, we would be able to go for branch-2 merge
> discussion and vote almost immediately.
>
> Regards,
> Varun Saxena.
>
>
>
>
> On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K <su...@gmail.com>
> wrote:
>
>> +1 (binding).
>>
>> I have been following the effort and had few design discussions around the
>> team especially about how it integrates with Federation. Overall I feel
>> it's a welcome improvement to YARN.
>>
>> What are the timelines you are looking for getting this into branch-2?
>>
>> Thanks,
>> Subru
>>
>> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
>>
>> > +1 (binding)
>> >
>> > I've built the current branch, and checked out a few basic areas
>> including
>> > documentation. Also perused the most recent changes that went in.
>> >
>> > Thanks much for the great team work! I look forward to seeing it in
>> action.
>> >
>> > Regards,
>> > Sangjin
>> >
>> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
>> > wrote:
>> >
>> > > +1 from my side.
>> > >
>> > > More from the perspective of ensuring there is no impact of ATSv2
>> when it
>> > > is off (by default), I deployed the latest YARN-5355 bits into a few
>> > > clusters and ran internal Smoke tests. The tests shows no impact when
>> > ATSv2
>> > > is off.
>> > >
>> > > Best,
>> > > Haibo
>> > >
>> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
>> > >
>> > > > Thank you very much Vrushali, Rohith, Varun and other folks who made
>> > this
>> > > > happen. Great work, really appreciate the same!!
>> > > >
>> > > > +1 (binding) from my side:
>> > > >
>> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
>> > > > # Accessed new YARN UI which shows various flows/flow activity etc.
>> > Seems
>> > > > fine.
>> > > > # Based on code, looks like all apis are compatible.
>> > > > # REST api docs looks fine as well, I guess we could improve that a
>> bit
>> > > > more post merge as well.
>> > > > # Adding to additional thoughts which are discussed here, native
>> > service
>> > > > also could publish events to atsv2. I think that work is also
>> happened
>> > in
>> > > > branch.
>> > > >
>> > > > Looking forward to a much wider adoption of ATSv2 with more
>> projects.
>> > > >
>> > > > Thanks
>> > > > Sunil
>> > > >
>> > > >
>> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
>> > > > vrushalic2016@gmail.com> wrote:
>> > > >
>> > > > > Hi folks,
>> > > > >
>> > > > > Per earlier discussion [1], I'd like to start a formal vote to
>> merge
>> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
>> > vote
>> > > > will
>> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
>> > > > >
>> > > > > We have previously completed one merge onto trunk [3] and Timeline
>> > > > Service
>> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
>> > > > >
>> > > > > Since then, we have been working on extending the capabilities of
>> > > > Timeline
>> > > > > Service v2 in a feature branch [2] for a while, and we are
>> reasonably
>> > > > > confident that the state of the feature meets the criteria to be
>> > merged
>> > > > > onto trunk and we'd love folks to get their hands on it in a test
>> > > > capacity
>> > > > > and provide valuable feedback so that we can make it
>> > production-ready.
>> > > > >
>> > > > > In a nutshell, Timeline Service v.2 delivers significant
>> scalability
>> > > and
>> > > > > usability improvements based on a new architecture. What we would
>> > like
>> > > to
>> > > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has
>> a
>> > > > > complete end-to-end read/write flow with security and read level
>> > > > > authorization via whitelists. You should be able to start setting
>> it
>> > up
>> > > > and
>> > > > > testing it.
>> > > > >
>> > > > > At a high level, the following are the key features that have been
>> > > > > implemented since alpha1:
>> > > > > - Security via Kerberos Authentication and delegation tokens
>> > > > > - Read side simple authorization via whitelist
>> > > > > - Client configurable entity sort ordering
>> > > > > - Richer REST APIs for apps, app attempts, containers, fetching
>> > metrics
>> > > > by
>> > > > > timerange, pagination, sub-app entities
>> > > > > - Support for storing sub-application entities (entities that
>> exist
>> > > > outside
>> > > > > the scope of an application)
>> > > > > - Configurable TTLs (time-to-live) for tables, configurable table
>> > > > prefixes,
>> > > > > configurable hbase cluster
>> > > > > - Flow level aggregations done as dynamic (table level)
>> coprocessors
>> > > > > - Uses latest stable HBase release 1.2.6
>> > > > >
>> > > > > There are a total of 82 subtasks that were completed as part of
>> this
>> > > > > effort.
>> > > > >
>> > > > > We paid close attention to ensure that once disabled Timeline
>> Service
>> > > v.2
>> > > > > does not impact existing functionality when disabled (by default).
>> > > > >
>> > > > > Special thanks to a team of folks who worked hard and contributed
>> > > towards
>> > > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
>> > > Varun
>> > > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
>> Joep
>> > > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> > > > >
>> > > > > Regards,
>> > > > > Vrushali
>> > > > >
>> > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
>> > > 383.html
>> > > > > [2] https://issues.apache.org/jira/browse/YARN-5355
>> > > > > [3] https://issues.apache.org/jira/browse/YARN-2928
>> > > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Hi Subru,
Thanks for your vote and your response!
Regarding your question about merging to branch2, I think, after the merge
to trunk is done, it will be good to merge to branch-2 as soon as we can.
A lot of testing with trunk has been done presently and it will be good to
repeat the same with the YARN-5355_branch2 branch before we start working
on newer features. That way trunk and branch-2 would be in a similar state
with respect to timeline service v2 development.
thanks
Vrushali
On Fri, Aug 25, 2017 at 2:28 PM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> Thanks Subru for voting.
>
>
>> What are the timelines you are looking for getting this into branch-2?
>
> We haven't yet decided on it and were thinking of discussing this in
> detail within the team after merge to trunk.
> The timelines would depend on whether we release whatever we merge to
> trunk in 2.9 or would we want to get in few other features which people
> would like to see in 2.9
> This would require some discussion with the stakeholders.
> We were thinking of having a short discussion with you guys as well to
> find out whether there are any further gaps in ATSv2 with respect to
> federation support and if they can be filled before 2.9 release.
>
> Assuming 2.9 is targetted for October end, we would have to start a merge
> vote in September end or October 1st week which leaves us with very little
> time to take up large changes anyways.
>
> We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though which
> we rebase with branch-2 regularly. So, if we decide to merge in branch-2
> without any additional changes, we would be able to go for branch-2 merge
> discussion and vote almost immediately.
>
> Regards,
> Varun Saxena.
>
>
>
>
> On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K <su...@gmail.com>
> wrote:
>
>> +1 (binding).
>>
>> I have been following the effort and had few design discussions around the
>> team especially about how it integrates with Federation. Overall I feel
>> it's a welcome improvement to YARN.
>>
>> What are the timelines you are looking for getting this into branch-2?
>>
>> Thanks,
>> Subru
>>
>> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
>>
>> > +1 (binding)
>> >
>> > I've built the current branch, and checked out a few basic areas
>> including
>> > documentation. Also perused the most recent changes that went in.
>> >
>> > Thanks much for the great team work! I look forward to seeing it in
>> action.
>> >
>> > Regards,
>> > Sangjin
>> >
>> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
>> > wrote:
>> >
>> > > +1 from my side.
>> > >
>> > > More from the perspective of ensuring there is no impact of ATSv2
>> when it
>> > > is off (by default), I deployed the latest YARN-5355 bits into a few
>> > > clusters and ran internal Smoke tests. The tests shows no impact when
>> > ATSv2
>> > > is off.
>> > >
>> > > Best,
>> > > Haibo
>> > >
>> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
>> > >
>> > > > Thank you very much Vrushali, Rohith, Varun and other folks who made
>> > this
>> > > > happen. Great work, really appreciate the same!!
>> > > >
>> > > > +1 (binding) from my side:
>> > > >
>> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
>> > > > # Accessed new YARN UI which shows various flows/flow activity etc.
>> > Seems
>> > > > fine.
>> > > > # Based on code, looks like all apis are compatible.
>> > > > # REST api docs looks fine as well, I guess we could improve that a
>> bit
>> > > > more post merge as well.
>> > > > # Adding to additional thoughts which are discussed here, native
>> > service
>> > > > also could publish events to atsv2. I think that work is also
>> happened
>> > in
>> > > > branch.
>> > > >
>> > > > Looking forward to a much wider adoption of ATSv2 with more
>> projects.
>> > > >
>> > > > Thanks
>> > > > Sunil
>> > > >
>> > > >
>> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
>> > > > vrushalic2016@gmail.com> wrote:
>> > > >
>> > > > > Hi folks,
>> > > > >
>> > > > > Per earlier discussion [1], I'd like to start a formal vote to
>> merge
>> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
>> > vote
>> > > > will
>> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
>> > > > >
>> > > > > We have previously completed one merge onto trunk [3] and Timeline
>> > > > Service
>> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
>> > > > >
>> > > > > Since then, we have been working on extending the capabilities of
>> > > > Timeline
>> > > > > Service v2 in a feature branch [2] for a while, and we are
>> reasonably
>> > > > > confident that the state of the feature meets the criteria to be
>> > merged
>> > > > > onto trunk and we'd love folks to get their hands on it in a test
>> > > > capacity
>> > > > > and provide valuable feedback so that we can make it
>> > production-ready.
>> > > > >
>> > > > > In a nutshell, Timeline Service v.2 delivers significant
>> scalability
>> > > and
>> > > > > usability improvements based on a new architecture. What we would
>> > like
>> > > to
>> > > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has
>> a
>> > > > > complete end-to-end read/write flow with security and read level
>> > > > > authorization via whitelists. You should be able to start setting
>> it
>> > up
>> > > > and
>> > > > > testing it.
>> > > > >
>> > > > > At a high level, the following are the key features that have been
>> > > > > implemented since alpha1:
>> > > > > - Security via Kerberos Authentication and delegation tokens
>> > > > > - Read side simple authorization via whitelist
>> > > > > - Client configurable entity sort ordering
>> > > > > - Richer REST APIs for apps, app attempts, containers, fetching
>> > metrics
>> > > > by
>> > > > > timerange, pagination, sub-app entities
>> > > > > - Support for storing sub-application entities (entities that
>> exist
>> > > > outside
>> > > > > the scope of an application)
>> > > > > - Configurable TTLs (time-to-live) for tables, configurable table
>> > > > prefixes,
>> > > > > configurable hbase cluster
>> > > > > - Flow level aggregations done as dynamic (table level)
>> coprocessors
>> > > > > - Uses latest stable HBase release 1.2.6
>> > > > >
>> > > > > There are a total of 82 subtasks that were completed as part of
>> this
>> > > > > effort.
>> > > > >
>> > > > > We paid close attention to ensure that once disabled Timeline
>> Service
>> > > v.2
>> > > > > does not impact existing functionality when disabled (by default).
>> > > > >
>> > > > > Special thanks to a team of folks who worked hard and contributed
>> > > towards
>> > > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
>> > > Varun
>> > > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
>> Joep
>> > > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> > > > >
>> > > > > Regards,
>> > > > > Vrushali
>> > > > >
>> > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
>> > > 383.html
>> > > > > [2] https://issues.apache.org/jira/browse/YARN-5355
>> > > > > [3] https://issues.apache.org/jira/browse/YARN-2928
>> > > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Vrushali C <vr...@gmail.com>.
Hi Subru,
Thanks for your vote and your response!
Regarding your question about merging to branch2, I think, after the merge
to trunk is done, it will be good to merge to branch-2 as soon as we can.
A lot of testing with trunk has been done presently and it will be good to
repeat the same with the YARN-5355_branch2 branch before we start working
on newer features. That way trunk and branch-2 would be in a similar state
with respect to timeline service v2 development.
thanks
Vrushali
On Fri, Aug 25, 2017 at 2:28 PM, varunsaxena@apache.org <
varun.saxena.apache@gmail.com> wrote:
> Thanks Subru for voting.
>
>
>> What are the timelines you are looking for getting this into branch-2?
>
> We haven't yet decided on it and were thinking of discussing this in
> detail within the team after merge to trunk.
> The timelines would depend on whether we release whatever we merge to
> trunk in 2.9 or would we want to get in few other features which people
> would like to see in 2.9
> This would require some discussion with the stakeholders.
> We were thinking of having a short discussion with you guys as well to
> find out whether there are any further gaps in ATSv2 with respect to
> federation support and if they can be filled before 2.9 release.
>
> Assuming 2.9 is targetted for October end, we would have to start a merge
> vote in September end or October 1st week which leaves us with very little
> time to take up large changes anyways.
>
> We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though which
> we rebase with branch-2 regularly. So, if we decide to merge in branch-2
> without any additional changes, we would be able to go for branch-2 merge
> discussion and vote almost immediately.
>
> Regards,
> Varun Saxena.
>
>
>
>
> On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K <su...@gmail.com>
> wrote:
>
>> +1 (binding).
>>
>> I have been following the effort and had few design discussions around the
>> team especially about how it integrates with Federation. Overall I feel
>> it's a welcome improvement to YARN.
>>
>> What are the timelines you are looking for getting this into branch-2?
>>
>> Thanks,
>> Subru
>>
>> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
>>
>> > +1 (binding)
>> >
>> > I've built the current branch, and checked out a few basic areas
>> including
>> > documentation. Also perused the most recent changes that went in.
>> >
>> > Thanks much for the great team work! I look forward to seeing it in
>> action.
>> >
>> > Regards,
>> > Sangjin
>> >
>> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
>> > wrote:
>> >
>> > > +1 from my side.
>> > >
>> > > More from the perspective of ensuring there is no impact of ATSv2
>> when it
>> > > is off (by default), I deployed the latest YARN-5355 bits into a few
>> > > clusters and ran internal Smoke tests. The tests shows no impact when
>> > ATSv2
>> > > is off.
>> > >
>> > > Best,
>> > > Haibo
>> > >
>> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
>> > >
>> > > > Thank you very much Vrushali, Rohith, Varun and other folks who made
>> > this
>> > > > happen. Great work, really appreciate the same!!
>> > > >
>> > > > +1 (binding) from my side:
>> > > >
>> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
>> > > > # Accessed new YARN UI which shows various flows/flow activity etc.
>> > Seems
>> > > > fine.
>> > > > # Based on code, looks like all apis are compatible.
>> > > > # REST api docs looks fine as well, I guess we could improve that a
>> bit
>> > > > more post merge as well.
>> > > > # Adding to additional thoughts which are discussed here, native
>> > service
>> > > > also could publish events to atsv2. I think that work is also
>> happened
>> > in
>> > > > branch.
>> > > >
>> > > > Looking forward to a much wider adoption of ATSv2 with more
>> projects.
>> > > >
>> > > > Thanks
>> > > > Sunil
>> > > >
>> > > >
>> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
>> > > > vrushalic2016@gmail.com> wrote:
>> > > >
>> > > > > Hi folks,
>> > > > >
>> > > > > Per earlier discussion [1], I'd like to start a formal vote to
>> merge
>> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
>> > vote
>> > > > will
>> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
>> > > > >
>> > > > > We have previously completed one merge onto trunk [3] and Timeline
>> > > > Service
>> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
>> > > > >
>> > > > > Since then, we have been working on extending the capabilities of
>> > > > Timeline
>> > > > > Service v2 in a feature branch [2] for a while, and we are
>> reasonably
>> > > > > confident that the state of the feature meets the criteria to be
>> > merged
>> > > > > onto trunk and we'd love folks to get their hands on it in a test
>> > > > capacity
>> > > > > and provide valuable feedback so that we can make it
>> > production-ready.
>> > > > >
>> > > > > In a nutshell, Timeline Service v.2 delivers significant
>> scalability
>> > > and
>> > > > > usability improvements based on a new architecture. What we would
>> > like
>> > > to
>> > > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has
>> a
>> > > > > complete end-to-end read/write flow with security and read level
>> > > > > authorization via whitelists. You should be able to start setting
>> it
>> > up
>> > > > and
>> > > > > testing it.
>> > > > >
>> > > > > At a high level, the following are the key features that have been
>> > > > > implemented since alpha1:
>> > > > > - Security via Kerberos Authentication and delegation tokens
>> > > > > - Read side simple authorization via whitelist
>> > > > > - Client configurable entity sort ordering
>> > > > > - Richer REST APIs for apps, app attempts, containers, fetching
>> > metrics
>> > > > by
>> > > > > timerange, pagination, sub-app entities
>> > > > > - Support for storing sub-application entities (entities that
>> exist
>> > > > outside
>> > > > > the scope of an application)
>> > > > > - Configurable TTLs (time-to-live) for tables, configurable table
>> > > > prefixes,
>> > > > > configurable hbase cluster
>> > > > > - Flow level aggregations done as dynamic (table level)
>> coprocessors
>> > > > > - Uses latest stable HBase release 1.2.6
>> > > > >
>> > > > > There are a total of 82 subtasks that were completed as part of
>> this
>> > > > > effort.
>> > > > >
>> > > > > We paid close attention to ensure that once disabled Timeline
>> Service
>> > > v.2
>> > > > > does not impact existing functionality when disabled (by default).
>> > > > >
>> > > > > Special thanks to a team of folks who worked hard and contributed
>> > > towards
>> > > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
>> > > Varun
>> > > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
>> Joep
>> > > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> > > > >
>> > > > > Regards,
>> > > > > Vrushali
>> > > > >
>> > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
>> > > 383.html
>> > > > > [2] https://issues.apache.org/jira/browse/YARN-5355
>> > > > > [3] https://issues.apache.org/jira/browse/YARN-2928
>> > > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Thanks Subru for voting.
> What are the timelines you are looking for getting this into branch-2?
We haven't yet decided on it and were thinking of discussing this in detail
within the team after merge to trunk.
The timelines would depend on whether we release whatever we merge to trunk
in 2.9 or would we want to get in few other features which people would
like to see in 2.9
This would require some discussion with the stakeholders.
We were thinking of having a short discussion with you guys as well to find
out whether there are any further gaps in ATSv2 with respect to federation
support and if they can be filled before 2.9 release.
Assuming 2.9 is targetted for October end, we would have to start a merge
vote in September end or October 1st week which leaves us with very little
time to take up large changes anyways.
We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though which
we rebase with branch-2 regularly. So, if we decide to merge in branch-2
without any additional changes, we would be able to go for branch-2 merge
discussion and vote almost immediately.
Regards,
Varun Saxena.
On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K <su...@gmail.com> wrote:
> +1 (binding).
>
> I have been following the effort and had few design discussions around the
> team especially about how it integrates with Federation. Overall I feel
> it's a welcome improvement to YARN.
>
> What are the timelines you are looking for getting this into branch-2?
>
> Thanks,
> Subru
>
> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
>
> > +1 (binding)
> >
> > I've built the current branch, and checked out a few basic areas
> including
> > documentation. Also perused the most recent changes that went in.
> >
> > Thanks much for the great team work! I look forward to seeing it in
> action.
> >
> > Regards,
> > Sangjin
> >
> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
> > wrote:
> >
> > > +1 from my side.
> > >
> > > More from the perspective of ensuring there is no impact of ATSv2 when
> it
> > > is off (by default), I deployed the latest YARN-5355 bits into a few
> > > clusters and ran internal Smoke tests. The tests shows no impact when
> > ATSv2
> > > is off.
> > >
> > > Best,
> > > Haibo
> > >
> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> > >
> > > > Thank you very much Vrushali, Rohith, Varun and other folks who made
> > this
> > > > happen. Great work, really appreciate the same!!
> > > >
> > > > +1 (binding) from my side:
> > > >
> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > > > # Accessed new YARN UI which shows various flows/flow activity etc.
> > Seems
> > > > fine.
> > > > # Based on code, looks like all apis are compatible.
> > > > # REST api docs looks fine as well, I guess we could improve that a
> bit
> > > > more post merge as well.
> > > > # Adding to additional thoughts which are discussed here, native
> > service
> > > > also could publish events to atsv2. I think that work is also
> happened
> > in
> > > > branch.
> > > >
> > > > Looking forward to a much wider adoption of ATSv2 with more projects.
> > > >
> > > > Thanks
> > > > Sunil
> > > >
> > > >
> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > > > vrushalic2016@gmail.com> wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Per earlier discussion [1], I'd like to start a formal vote to
> merge
> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> > vote
> > > > will
> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
> > > > >
> > > > > We have previously completed one merge onto trunk [3] and Timeline
> > > > Service
> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > > >
> > > > > Since then, we have been working on extending the capabilities of
> > > > Timeline
> > > > > Service v2 in a feature branch [2] for a while, and we are
> reasonably
> > > > > confident that the state of the feature meets the criteria to be
> > merged
> > > > > onto trunk and we'd love folks to get their hands on it in a test
> > > > capacity
> > > > > and provide valuable feedback so that we can make it
> > production-ready.
> > > > >
> > > > > In a nutshell, Timeline Service v.2 delivers significant
> scalability
> > > and
> > > > > usability improvements based on a new architecture. What we would
> > like
> > > to
> > > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > > > complete end-to-end read/write flow with security and read level
> > > > > authorization via whitelists. You should be able to start setting
> it
> > up
> > > > and
> > > > > testing it.
> > > > >
> > > > > At a high level, the following are the key features that have been
> > > > > implemented since alpha1:
> > > > > - Security via Kerberos Authentication and delegation tokens
> > > > > - Read side simple authorization via whitelist
> > > > > - Client configurable entity sort ordering
> > > > > - Richer REST APIs for apps, app attempts, containers, fetching
> > metrics
> > > > by
> > > > > timerange, pagination, sub-app entities
> > > > > - Support for storing sub-application entities (entities that exist
> > > > outside
> > > > > the scope of an application)
> > > > > - Configurable TTLs (time-to-live) for tables, configurable table
> > > > prefixes,
> > > > > configurable hbase cluster
> > > > > - Flow level aggregations done as dynamic (table level)
> coprocessors
> > > > > - Uses latest stable HBase release 1.2.6
> > > > >
> > > > > There are a total of 82 subtasks that were completed as part of
> this
> > > > > effort.
> > > > >
> > > > > We paid close attention to ensure that once disabled Timeline
> Service
> > > v.2
> > > > > does not impact existing functionality when disabled (by default).
> > > > >
> > > > > Special thanks to a team of folks who worked hard and contributed
> > > towards
> > > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > > Varun
> > > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > > >
> > > > > Regards,
> > > > > Vrushali
> > > > >
> > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > > 383.html
> > > > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Thanks Subru for voting.
> What are the timelines you are looking for getting this into branch-2?
We haven't yet decided on it and were thinking of discussing this in detail
within the team after merge to trunk.
The timelines would depend on whether we release whatever we merge to trunk
in 2.9 or would we want to get in few other features which people would
like to see in 2.9
This would require some discussion with the stakeholders.
We were thinking of having a short discussion with you guys as well to find
out whether there are any further gaps in ATSv2 with respect to federation
support and if they can be filled before 2.9 release.
Assuming 2.9 is targetted for October end, we would have to start a merge
vote in September end or October 1st week which leaves us with very little
time to take up large changes anyways.
We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though which
we rebase with branch-2 regularly. So, if we decide to merge in branch-2
without any additional changes, we would be able to go for branch-2 merge
discussion and vote almost immediately.
Regards,
Varun Saxena.
On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K <su...@gmail.com> wrote:
> +1 (binding).
>
> I have been following the effort and had few design discussions around the
> team especially about how it integrates with Federation. Overall I feel
> it's a welcome improvement to YARN.
>
> What are the timelines you are looking for getting this into branch-2?
>
> Thanks,
> Subru
>
> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
>
> > +1 (binding)
> >
> > I've built the current branch, and checked out a few basic areas
> including
> > documentation. Also perused the most recent changes that went in.
> >
> > Thanks much for the great team work! I look forward to seeing it in
> action.
> >
> > Regards,
> > Sangjin
> >
> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
> > wrote:
> >
> > > +1 from my side.
> > >
> > > More from the perspective of ensuring there is no impact of ATSv2 when
> it
> > > is off (by default), I deployed the latest YARN-5355 bits into a few
> > > clusters and ran internal Smoke tests. The tests shows no impact when
> > ATSv2
> > > is off.
> > >
> > > Best,
> > > Haibo
> > >
> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> > >
> > > > Thank you very much Vrushali, Rohith, Varun and other folks who made
> > this
> > > > happen. Great work, really appreciate the same!!
> > > >
> > > > +1 (binding) from my side:
> > > >
> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > > > # Accessed new YARN UI which shows various flows/flow activity etc.
> > Seems
> > > > fine.
> > > > # Based on code, looks like all apis are compatible.
> > > > # REST api docs looks fine as well, I guess we could improve that a
> bit
> > > > more post merge as well.
> > > > # Adding to additional thoughts which are discussed here, native
> > service
> > > > also could publish events to atsv2. I think that work is also
> happened
> > in
> > > > branch.
> > > >
> > > > Looking forward to a much wider adoption of ATSv2 with more projects.
> > > >
> > > > Thanks
> > > > Sunil
> > > >
> > > >
> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > > > vrushalic2016@gmail.com> wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Per earlier discussion [1], I'd like to start a formal vote to
> merge
> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> > vote
> > > > will
> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
> > > > >
> > > > > We have previously completed one merge onto trunk [3] and Timeline
> > > > Service
> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > > >
> > > > > Since then, we have been working on extending the capabilities of
> > > > Timeline
> > > > > Service v2 in a feature branch [2] for a while, and we are
> reasonably
> > > > > confident that the state of the feature meets the criteria to be
> > merged
> > > > > onto trunk and we'd love folks to get their hands on it in a test
> > > > capacity
> > > > > and provide valuable feedback so that we can make it
> > production-ready.
> > > > >
> > > > > In a nutshell, Timeline Service v.2 delivers significant
> scalability
> > > and
> > > > > usability improvements based on a new architecture. What we would
> > like
> > > to
> > > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > > > complete end-to-end read/write flow with security and read level
> > > > > authorization via whitelists. You should be able to start setting
> it
> > up
> > > > and
> > > > > testing it.
> > > > >
> > > > > At a high level, the following are the key features that have been
> > > > > implemented since alpha1:
> > > > > - Security via Kerberos Authentication and delegation tokens
> > > > > - Read side simple authorization via whitelist
> > > > > - Client configurable entity sort ordering
> > > > > - Richer REST APIs for apps, app attempts, containers, fetching
> > metrics
> > > > by
> > > > > timerange, pagination, sub-app entities
> > > > > - Support for storing sub-application entities (entities that exist
> > > > outside
> > > > > the scope of an application)
> > > > > - Configurable TTLs (time-to-live) for tables, configurable table
> > > > prefixes,
> > > > > configurable hbase cluster
> > > > > - Flow level aggregations done as dynamic (table level)
> coprocessors
> > > > > - Uses latest stable HBase release 1.2.6
> > > > >
> > > > > There are a total of 82 subtasks that were completed as part of
> this
> > > > > effort.
> > > > >
> > > > > We paid close attention to ensure that once disabled Timeline
> Service
> > > v.2
> > > > > does not impact existing functionality when disabled (by default).
> > > > >
> > > > > Special thanks to a team of folks who worked hard and contributed
> > > towards
> > > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > > Varun
> > > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > > >
> > > > > Regards,
> > > > > Vrushali
> > > > >
> > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > > 383.html
> > > > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Thanks Subru for voting.
> What are the timelines you are looking for getting this into branch-2?
We haven't yet decided on it and were thinking of discussing this in detail
within the team after merge to trunk.
The timelines would depend on whether we release whatever we merge to trunk
in 2.9 or would we want to get in few other features which people would
like to see in 2.9
This would require some discussion with the stakeholders.
We were thinking of having a short discussion with you guys as well to find
out whether there are any further gaps in ATSv2 with respect to federation
support and if they can be filled before 2.9 release.
Assuming 2.9 is targetted for October end, we would have to start a merge
vote in September end or October 1st week which leaves us with very little
time to take up large changes anyways.
We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though which
we rebase with branch-2 regularly. So, if we decide to merge in branch-2
without any additional changes, we would be able to go for branch-2 merge
discussion and vote almost immediately.
Regards,
Varun Saxena.
On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K <su...@gmail.com> wrote:
> +1 (binding).
>
> I have been following the effort and had few design discussions around the
> team especially about how it integrates with Federation. Overall I feel
> it's a welcome improvement to YARN.
>
> What are the timelines you are looking for getting this into branch-2?
>
> Thanks,
> Subru
>
> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
>
> > +1 (binding)
> >
> > I've built the current branch, and checked out a few basic areas
> including
> > documentation. Also perused the most recent changes that went in.
> >
> > Thanks much for the great team work! I look forward to seeing it in
> action.
> >
> > Regards,
> > Sangjin
> >
> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
> > wrote:
> >
> > > +1 from my side.
> > >
> > > More from the perspective of ensuring there is no impact of ATSv2 when
> it
> > > is off (by default), I deployed the latest YARN-5355 bits into a few
> > > clusters and ran internal Smoke tests. The tests shows no impact when
> > ATSv2
> > > is off.
> > >
> > > Best,
> > > Haibo
> > >
> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> > >
> > > > Thank you very much Vrushali, Rohith, Varun and other folks who made
> > this
> > > > happen. Great work, really appreciate the same!!
> > > >
> > > > +1 (binding) from my side:
> > > >
> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > > > # Accessed new YARN UI which shows various flows/flow activity etc.
> > Seems
> > > > fine.
> > > > # Based on code, looks like all apis are compatible.
> > > > # REST api docs looks fine as well, I guess we could improve that a
> bit
> > > > more post merge as well.
> > > > # Adding to additional thoughts which are discussed here, native
> > service
> > > > also could publish events to atsv2. I think that work is also
> happened
> > in
> > > > branch.
> > > >
> > > > Looking forward to a much wider adoption of ATSv2 with more projects.
> > > >
> > > > Thanks
> > > > Sunil
> > > >
> > > >
> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > > > vrushalic2016@gmail.com> wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Per earlier discussion [1], I'd like to start a formal vote to
> merge
> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> > vote
> > > > will
> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
> > > > >
> > > > > We have previously completed one merge onto trunk [3] and Timeline
> > > > Service
> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > > >
> > > > > Since then, we have been working on extending the capabilities of
> > > > Timeline
> > > > > Service v2 in a feature branch [2] for a while, and we are
> reasonably
> > > > > confident that the state of the feature meets the criteria to be
> > merged
> > > > > onto trunk and we'd love folks to get their hands on it in a test
> > > > capacity
> > > > > and provide valuable feedback so that we can make it
> > production-ready.
> > > > >
> > > > > In a nutshell, Timeline Service v.2 delivers significant
> scalability
> > > and
> > > > > usability improvements based on a new architecture. What we would
> > like
> > > to
> > > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > > > complete end-to-end read/write flow with security and read level
> > > > > authorization via whitelists. You should be able to start setting
> it
> > up
> > > > and
> > > > > testing it.
> > > > >
> > > > > At a high level, the following are the key features that have been
> > > > > implemented since alpha1:
> > > > > - Security via Kerberos Authentication and delegation tokens
> > > > > - Read side simple authorization via whitelist
> > > > > - Client configurable entity sort ordering
> > > > > - Richer REST APIs for apps, app attempts, containers, fetching
> > metrics
> > > > by
> > > > > timerange, pagination, sub-app entities
> > > > > - Support for storing sub-application entities (entities that exist
> > > > outside
> > > > > the scope of an application)
> > > > > - Configurable TTLs (time-to-live) for tables, configurable table
> > > > prefixes,
> > > > > configurable hbase cluster
> > > > > - Flow level aggregations done as dynamic (table level)
> coprocessors
> > > > > - Uses latest stable HBase release 1.2.6
> > > > >
> > > > > There are a total of 82 subtasks that were completed as part of
> this
> > > > > effort.
> > > > >
> > > > > We paid close attention to ensure that once disabled Timeline
> Service
> > > v.2
> > > > > does not impact existing functionality when disabled (by default).
> > > > >
> > > > > Special thanks to a team of folks who worked hard and contributed
> > > towards
> > > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > > Varun
> > > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > > >
> > > > > Regards,
> > > > > Vrushali
> > > > >
> > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > > 383.html
> > > > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Thanks Subru for voting.
> What are the timelines you are looking for getting this into branch-2?
We haven't yet decided on it and were thinking of discussing this in detail
within the team after merge to trunk.
The timelines would depend on whether we release whatever we merge to trunk
in 2.9 or would we want to get in few other features which people would
like to see in 2.9
This would require some discussion with the stakeholders.
We were thinking of having a short discussion with you guys as well to find
out whether there are any further gaps in ATSv2 with respect to federation
support and if they can be filled before 2.9 release.
Assuming 2.9 is targetted for October end, we would have to start a merge
vote in September end or October 1st week which leaves us with very little
time to take up large changes anyways.
We do maintain a branch-2 version of ATSv2(YARN-5355_branch2) though which
we rebase with branch-2 regularly. So, if we decide to merge in branch-2
without any additional changes, we would be able to go for branch-2 merge
discussion and vote almost immediately.
Regards,
Varun Saxena.
On Sat, Aug 26, 2017 at 2:00 AM, Subramaniam V K <su...@gmail.com> wrote:
> +1 (binding).
>
> I have been following the effort and had few design discussions around the
> team especially about how it integrates with Federation. Overall I feel
> it's a welcome improvement to YARN.
>
> What are the timelines you are looking for getting this into branch-2?
>
> Thanks,
> Subru
>
> On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
>
> > +1 (binding)
> >
> > I've built the current branch, and checked out a few basic areas
> including
> > documentation. Also perused the most recent changes that went in.
> >
> > Thanks much for the great team work! I look forward to seeing it in
> action.
> >
> > Regards,
> > Sangjin
> >
> > On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
> > wrote:
> >
> > > +1 from my side.
> > >
> > > More from the perspective of ensuring there is no impact of ATSv2 when
> it
> > > is off (by default), I deployed the latest YARN-5355 bits into a few
> > > clusters and ran internal Smoke tests. The tests shows no impact when
> > ATSv2
> > > is off.
> > >
> > > Best,
> > > Haibo
> > >
> > > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> > >
> > > > Thank you very much Vrushali, Rohith, Varun and other folks who made
> > this
> > > > happen. Great work, really appreciate the same!!
> > > >
> > > > +1 (binding) from my side:
> > > >
> > > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > > > # Accessed new YARN UI which shows various flows/flow activity etc.
> > Seems
> > > > fine.
> > > > # Based on code, looks like all apis are compatible.
> > > > # REST api docs looks fine as well, I guess we could improve that a
> bit
> > > > more post merge as well.
> > > > # Adding to additional thoughts which are discussed here, native
> > service
> > > > also could publish events to atsv2. I think that work is also
> happened
> > in
> > > > branch.
> > > >
> > > > Looking forward to a much wider adoption of ATSv2 with more projects.
> > > >
> > > > Thanks
> > > > Sunil
> > > >
> > > >
> > > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > > > vrushalic2016@gmail.com> wrote:
> > > >
> > > > > Hi folks,
> > > > >
> > > > > Per earlier discussion [1], I'd like to start a formal vote to
> merge
> > > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> > vote
> > > > will
> > > > > run for 7 days, and will end August 29 11:00 PM PDT.
> > > > >
> > > > > We have previously completed one merge onto trunk [3] and Timeline
> > > > Service
> > > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > > >
> > > > > Since then, we have been working on extending the capabilities of
> > > > Timeline
> > > > > Service v2 in a feature branch [2] for a while, and we are
> reasonably
> > > > > confident that the state of the feature meets the criteria to be
> > merged
> > > > > onto trunk and we'd love folks to get their hands on it in a test
> > > > capacity
> > > > > and provide valuable feedback so that we can make it
> > production-ready.
> > > > >
> > > > > In a nutshell, Timeline Service v.2 delivers significant
> scalability
> > > and
> > > > > usability improvements based on a new architecture. What we would
> > like
> > > to
> > > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > > > complete end-to-end read/write flow with security and read level
> > > > > authorization via whitelists. You should be able to start setting
> it
> > up
> > > > and
> > > > > testing it.
> > > > >
> > > > > At a high level, the following are the key features that have been
> > > > > implemented since alpha1:
> > > > > - Security via Kerberos Authentication and delegation tokens
> > > > > - Read side simple authorization via whitelist
> > > > > - Client configurable entity sort ordering
> > > > > - Richer REST APIs for apps, app attempts, containers, fetching
> > metrics
> > > > by
> > > > > timerange, pagination, sub-app entities
> > > > > - Support for storing sub-application entities (entities that exist
> > > > outside
> > > > > the scope of an application)
> > > > > - Configurable TTLs (time-to-live) for tables, configurable table
> > > > prefixes,
> > > > > configurable hbase cluster
> > > > > - Flow level aggregations done as dynamic (table level)
> coprocessors
> > > > > - Uses latest stable HBase release 1.2.6
> > > > >
> > > > > There are a total of 82 subtasks that were completed as part of
> this
> > > > > effort.
> > > > >
> > > > > We paid close attention to ensure that once disabled Timeline
> Service
> > > v.2
> > > > > does not impact existing functionality when disabled (by default).
> > > > >
> > > > > Special thanks to a team of folks who worked hard and contributed
> > > towards
> > > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > > Varun
> > > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
> Joep
> > > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > > >
> > > > > Regards,
> > > > > Vrushali
> > > > >
> > > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > > 383.html
> > > > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > > > >
> > > >
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Subramaniam V K <su...@gmail.com>.
+1 (binding).
I have been following the effort and had few design discussions around the
team especially about how it integrates with Federation. Overall I feel
it's a welcome improvement to YARN.
What are the timelines you are looking for getting this into branch-2?
Thanks,
Subru
On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
> +1 (binding)
>
> I've built the current branch, and checked out a few basic areas including
> documentation. Also perused the most recent changes that went in.
>
> Thanks much for the great team work! I look forward to seeing it in action.
>
> Regards,
> Sangjin
>
> On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
> wrote:
>
> > +1 from my side.
> >
> > More from the perspective of ensuring there is no impact of ATSv2 when it
> > is off (by default), I deployed the latest YARN-5355 bits into a few
> > clusters and ran internal Smoke tests. The tests shows no impact when
> ATSv2
> > is off.
> >
> > Best,
> > Haibo
> >
> > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> >
> > > Thank you very much Vrushali, Rohith, Varun and other folks who made
> this
> > > happen. Great work, really appreciate the same!!
> > >
> > > +1 (binding) from my side:
> > >
> > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > > # Accessed new YARN UI which shows various flows/flow activity etc.
> Seems
> > > fine.
> > > # Based on code, looks like all apis are compatible.
> > > # REST api docs looks fine as well, I guess we could improve that a bit
> > > more post merge as well.
> > > # Adding to additional thoughts which are discussed here, native
> service
> > > also could publish events to atsv2. I think that work is also happened
> in
> > > branch.
> > >
> > > Looking forward to a much wider adoption of ATSv2 with more projects.
> > >
> > > Thanks
> > > Sunil
> > >
> > >
> > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > > vrushalic2016@gmail.com> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> vote
> > > will
> > > > run for 7 days, and will end August 29 11:00 PM PDT.
> > > >
> > > > We have previously completed one merge onto trunk [3] and Timeline
> > > Service
> > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >
> > > > Since then, we have been working on extending the capabilities of
> > > Timeline
> > > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > > confident that the state of the feature meets the criteria to be
> merged
> > > > onto trunk and we'd love folks to get their hands on it in a test
> > > capacity
> > > > and provide valuable feedback so that we can make it
> production-ready.
> > > >
> > > > In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > > > usability improvements based on a new architecture. What we would
> like
> > to
> > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > > complete end-to-end read/write flow with security and read level
> > > > authorization via whitelists. You should be able to start setting it
> up
> > > and
> > > > testing it.
> > > >
> > > > At a high level, the following are the key features that have been
> > > > implemented since alpha1:
> > > > - Security via Kerberos Authentication and delegation tokens
> > > > - Read side simple authorization via whitelist
> > > > - Client configurable entity sort ordering
> > > > - Richer REST APIs for apps, app attempts, containers, fetching
> metrics
> > > by
> > > > timerange, pagination, sub-app entities
> > > > - Support for storing sub-application entities (entities that exist
> > > outside
> > > > the scope of an application)
> > > > - Configurable TTLs (time-to-live) for tables, configurable table
> > > prefixes,
> > > > configurable hbase cluster
> > > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > > - Uses latest stable HBase release 1.2.6
> > > >
> > > > There are a total of 82 subtasks that were completed as part of this
> > > > effort.
> > > >
> > > > We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > > > does not impact existing functionality when disabled (by default).
> > > >
> > > > Special thanks to a team of folks who worked hard and contributed
> > towards
> > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >
> > > > Regards,
> > > > Vrushali
> > > >
> > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > 383.html
> > > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > > >
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Subramaniam V K <su...@gmail.com>.
+1 (binding).
I have been following the effort and had few design discussions around the
team especially about how it integrates with Federation. Overall I feel
it's a welcome improvement to YARN.
What are the timelines you are looking for getting this into branch-2?
Thanks,
Subru
On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
> +1 (binding)
>
> I've built the current branch, and checked out a few basic areas including
> documentation. Also perused the most recent changes that went in.
>
> Thanks much for the great team work! I look forward to seeing it in action.
>
> Regards,
> Sangjin
>
> On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
> wrote:
>
> > +1 from my side.
> >
> > More from the perspective of ensuring there is no impact of ATSv2 when it
> > is off (by default), I deployed the latest YARN-5355 bits into a few
> > clusters and ran internal Smoke tests. The tests shows no impact when
> ATSv2
> > is off.
> >
> > Best,
> > Haibo
> >
> > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> >
> > > Thank you very much Vrushali, Rohith, Varun and other folks who made
> this
> > > happen. Great work, really appreciate the same!!
> > >
> > > +1 (binding) from my side:
> > >
> > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > > # Accessed new YARN UI which shows various flows/flow activity etc.
> Seems
> > > fine.
> > > # Based on code, looks like all apis are compatible.
> > > # REST api docs looks fine as well, I guess we could improve that a bit
> > > more post merge as well.
> > > # Adding to additional thoughts which are discussed here, native
> service
> > > also could publish events to atsv2. I think that work is also happened
> in
> > > branch.
> > >
> > > Looking forward to a much wider adoption of ATSv2 with more projects.
> > >
> > > Thanks
> > > Sunil
> > >
> > >
> > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > > vrushalic2016@gmail.com> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> vote
> > > will
> > > > run for 7 days, and will end August 29 11:00 PM PDT.
> > > >
> > > > We have previously completed one merge onto trunk [3] and Timeline
> > > Service
> > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >
> > > > Since then, we have been working on extending the capabilities of
> > > Timeline
> > > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > > confident that the state of the feature meets the criteria to be
> merged
> > > > onto trunk and we'd love folks to get their hands on it in a test
> > > capacity
> > > > and provide valuable feedback so that we can make it
> production-ready.
> > > >
> > > > In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > > > usability improvements based on a new architecture. What we would
> like
> > to
> > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > > complete end-to-end read/write flow with security and read level
> > > > authorization via whitelists. You should be able to start setting it
> up
> > > and
> > > > testing it.
> > > >
> > > > At a high level, the following are the key features that have been
> > > > implemented since alpha1:
> > > > - Security via Kerberos Authentication and delegation tokens
> > > > - Read side simple authorization via whitelist
> > > > - Client configurable entity sort ordering
> > > > - Richer REST APIs for apps, app attempts, containers, fetching
> metrics
> > > by
> > > > timerange, pagination, sub-app entities
> > > > - Support for storing sub-application entities (entities that exist
> > > outside
> > > > the scope of an application)
> > > > - Configurable TTLs (time-to-live) for tables, configurable table
> > > prefixes,
> > > > configurable hbase cluster
> > > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > > - Uses latest stable HBase release 1.2.6
> > > >
> > > > There are a total of 82 subtasks that were completed as part of this
> > > > effort.
> > > >
> > > > We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > > > does not impact existing functionality when disabled (by default).
> > > >
> > > > Special thanks to a team of folks who worked hard and contributed
> > towards
> > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >
> > > > Regards,
> > > > Vrushali
> > > >
> > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > 383.html
> > > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > > >
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Subramaniam V K <su...@gmail.com>.
+1 (binding).
I have been following the effort and had few design discussions around the
team especially about how it integrates with Federation. Overall I feel
it's a welcome improvement to YARN.
What are the timelines you are looking for getting this into branch-2?
Thanks,
Subru
On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
> +1 (binding)
>
> I've built the current branch, and checked out a few basic areas including
> documentation. Also perused the most recent changes that went in.
>
> Thanks much for the great team work! I look forward to seeing it in action.
>
> Regards,
> Sangjin
>
> On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
> wrote:
>
> > +1 from my side.
> >
> > More from the perspective of ensuring there is no impact of ATSv2 when it
> > is off (by default), I deployed the latest YARN-5355 bits into a few
> > clusters and ran internal Smoke tests. The tests shows no impact when
> ATSv2
> > is off.
> >
> > Best,
> > Haibo
> >
> > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> >
> > > Thank you very much Vrushali, Rohith, Varun and other folks who made
> this
> > > happen. Great work, really appreciate the same!!
> > >
> > > +1 (binding) from my side:
> > >
> > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > > # Accessed new YARN UI which shows various flows/flow activity etc.
> Seems
> > > fine.
> > > # Based on code, looks like all apis are compatible.
> > > # REST api docs looks fine as well, I guess we could improve that a bit
> > > more post merge as well.
> > > # Adding to additional thoughts which are discussed here, native
> service
> > > also could publish events to atsv2. I think that work is also happened
> in
> > > branch.
> > >
> > > Looking forward to a much wider adoption of ATSv2 with more projects.
> > >
> > > Thanks
> > > Sunil
> > >
> > >
> > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > > vrushalic2016@gmail.com> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> vote
> > > will
> > > > run for 7 days, and will end August 29 11:00 PM PDT.
> > > >
> > > > We have previously completed one merge onto trunk [3] and Timeline
> > > Service
> > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >
> > > > Since then, we have been working on extending the capabilities of
> > > Timeline
> > > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > > confident that the state of the feature meets the criteria to be
> merged
> > > > onto trunk and we'd love folks to get their hands on it in a test
> > > capacity
> > > > and provide valuable feedback so that we can make it
> production-ready.
> > > >
> > > > In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > > > usability improvements based on a new architecture. What we would
> like
> > to
> > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > > complete end-to-end read/write flow with security and read level
> > > > authorization via whitelists. You should be able to start setting it
> up
> > > and
> > > > testing it.
> > > >
> > > > At a high level, the following are the key features that have been
> > > > implemented since alpha1:
> > > > - Security via Kerberos Authentication and delegation tokens
> > > > - Read side simple authorization via whitelist
> > > > - Client configurable entity sort ordering
> > > > - Richer REST APIs for apps, app attempts, containers, fetching
> metrics
> > > by
> > > > timerange, pagination, sub-app entities
> > > > - Support for storing sub-application entities (entities that exist
> > > outside
> > > > the scope of an application)
> > > > - Configurable TTLs (time-to-live) for tables, configurable table
> > > prefixes,
> > > > configurable hbase cluster
> > > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > > - Uses latest stable HBase release 1.2.6
> > > >
> > > > There are a total of 82 subtasks that were completed as part of this
> > > > effort.
> > > >
> > > > We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > > > does not impact existing functionality when disabled (by default).
> > > >
> > > > Special thanks to a team of folks who worked hard and contributed
> > towards
> > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >
> > > > Regards,
> > > > Vrushali
> > > >
> > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > 383.html
> > > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > > >
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Subramaniam V K <su...@gmail.com>.
+1 (binding).
I have been following the effort and had few design discussions around the
team especially about how it integrates with Federation. Overall I feel
it's a welcome improvement to YARN.
What are the timelines you are looking for getting this into branch-2?
Thanks,
Subru
On Fri, Aug 25, 2017 at 10:04 AM, Sangjin Lee <sj...@apache.org> wrote:
> +1 (binding)
>
> I've built the current branch, and checked out a few basic areas including
> documentation. Also perused the most recent changes that went in.
>
> Thanks much for the great team work! I look forward to seeing it in action.
>
> Regards,
> Sangjin
>
> On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com>
> wrote:
>
> > +1 from my side.
> >
> > More from the perspective of ensuring there is no impact of ATSv2 when it
> > is off (by default), I deployed the latest YARN-5355 bits into a few
> > clusters and ran internal Smoke tests. The tests shows no impact when
> ATSv2
> > is off.
> >
> > Best,
> > Haibo
> >
> > On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> >
> > > Thank you very much Vrushali, Rohith, Varun and other folks who made
> this
> > > happen. Great work, really appreciate the same!!
> > >
> > > +1 (binding) from my side:
> > >
> > > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > > # Accessed new YARN UI which shows various flows/flow activity etc.
> Seems
> > > fine.
> > > # Based on code, looks like all apis are compatible.
> > > # REST api docs looks fine as well, I guess we could improve that a bit
> > > more post merge as well.
> > > # Adding to additional thoughts which are discussed here, native
> service
> > > also could publish events to atsv2. I think that work is also happened
> in
> > > branch.
> > >
> > > Looking forward to a much wider adoption of ATSv2 with more projects.
> > >
> > > Thanks
> > > Sunil
> > >
> > >
> > > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > > vrushalic2016@gmail.com> wrote:
> > >
> > > > Hi folks,
> > > >
> > > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The
> vote
> > > will
> > > > run for 7 days, and will end August 29 11:00 PM PDT.
> > > >
> > > > We have previously completed one merge onto trunk [3] and Timeline
> > > Service
> > > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > > >
> > > > Since then, we have been working on extending the capabilities of
> > > Timeline
> > > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > > confident that the state of the feature meets the criteria to be
> merged
> > > > onto trunk and we'd love folks to get their hands on it in a test
> > > capacity
> > > > and provide valuable feedback so that we can make it
> production-ready.
> > > >
> > > > In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > > > usability improvements based on a new architecture. What we would
> like
> > to
> > > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > > complete end-to-end read/write flow with security and read level
> > > > authorization via whitelists. You should be able to start setting it
> up
> > > and
> > > > testing it.
> > > >
> > > > At a high level, the following are the key features that have been
> > > > implemented since alpha1:
> > > > - Security via Kerberos Authentication and delegation tokens
> > > > - Read side simple authorization via whitelist
> > > > - Client configurable entity sort ordering
> > > > - Richer REST APIs for apps, app attempts, containers, fetching
> metrics
> > > by
> > > > timerange, pagination, sub-app entities
> > > > - Support for storing sub-application entities (entities that exist
> > > outside
> > > > the scope of an application)
> > > > - Configurable TTLs (time-to-live) for tables, configurable table
> > > prefixes,
> > > > configurable hbase cluster
> > > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > > - Uses latest stable HBase release 1.2.6
> > > >
> > > > There are a total of 82 subtasks that were completed as part of this
> > > > effort.
> > > >
> > > > We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > > > does not impact existing functionality when disabled (by default).
> > > >
> > > > Special thanks to a team of folks who worked hard and contributed
> > towards
> > > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> > Varun
> > > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > > >
> > > > Regards,
> > > > Vrushali
> > > >
> > > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> > 383.html
> > > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > > >
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
+1 (binding)
I've built the current branch, and checked out a few basic areas including
documentation. Also perused the most recent changes that went in.
Thanks much for the great team work! I look forward to seeing it in action.
Regards,
Sangjin
On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com> wrote:
> +1 from my side.
>
> More from the perspective of ensuring there is no impact of ATSv2 when it
> is off (by default), I deployed the latest YARN-5355 bits into a few
> clusters and ran internal Smoke tests. The tests shows no impact when ATSv2
> is off.
>
> Best,
> Haibo
>
> On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
>
> > Thank you very much Vrushali, Rohith, Varun and other folks who made this
> > happen. Great work, really appreciate the same!!
> >
> > +1 (binding) from my side:
> >
> > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > # Accessed new YARN UI which shows various flows/flow activity etc. Seems
> > fine.
> > # Based on code, looks like all apis are compatible.
> > # REST api docs looks fine as well, I guess we could improve that a bit
> > more post merge as well.
> > # Adding to additional thoughts which are discussed here, native service
> > also could publish events to atsv2. I think that work is also happened in
> > branch.
> >
> > Looking forward to a much wider adoption of ATSv2 with more projects.
> >
> > Thanks
> > Sunil
> >
> >
> > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > vrushalic2016@gmail.com> wrote:
> >
> > > Hi folks,
> > >
> > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > will
> > > run for 7 days, and will end August 29 11:00 PM PDT.
> > >
> > > We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >
> > > Since then, we have been working on extending the capabilities of
> > Timeline
> > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > confident that the state of the feature meets the criteria to be merged
> > > onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > > and provide valuable feedback so that we can make it production-ready.
> > >
> > > In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> > > usability improvements based on a new architecture. What we would like
> to
> > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > complete end-to-end read/write flow with security and read level
> > > authorization via whitelists. You should be able to start setting it up
> > and
> > > testing it.
> > >
> > > At a high level, the following are the key features that have been
> > > implemented since alpha1:
> > > - Security via Kerberos Authentication and delegation tokens
> > > - Read side simple authorization via whitelist
> > > - Client configurable entity sort ordering
> > > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> > by
> > > timerange, pagination, sub-app entities
> > > - Support for storing sub-application entities (entities that exist
> > outside
> > > the scope of an application)
> > > - Configurable TTLs (time-to-live) for tables, configurable table
> > prefixes,
> > > configurable hbase cluster
> > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > - Uses latest stable HBase release 1.2.6
> > >
> > > There are a total of 82 subtasks that were completed as part of this
> > > effort.
> > >
> > > We paid close attention to ensure that once disabled Timeline Service
> v.2
> > > does not impact existing functionality when disabled (by default).
> > >
> > > Special thanks to a team of folks who worked hard and contributed
> towards
> > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >
> > > Regards,
> > > Vrushali
> > >
> > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> 383.html
> > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
+1 (binding)
I've built the current branch, and checked out a few basic areas including
documentation. Also perused the most recent changes that went in.
Thanks much for the great team work! I look forward to seeing it in action.
Regards,
Sangjin
On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com> wrote:
> +1 from my side.
>
> More from the perspective of ensuring there is no impact of ATSv2 when it
> is off (by default), I deployed the latest YARN-5355 bits into a few
> clusters and ran internal Smoke tests. The tests shows no impact when ATSv2
> is off.
>
> Best,
> Haibo
>
> On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
>
> > Thank you very much Vrushali, Rohith, Varun and other folks who made this
> > happen. Great work, really appreciate the same!!
> >
> > +1 (binding) from my side:
> >
> > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > # Accessed new YARN UI which shows various flows/flow activity etc. Seems
> > fine.
> > # Based on code, looks like all apis are compatible.
> > # REST api docs looks fine as well, I guess we could improve that a bit
> > more post merge as well.
> > # Adding to additional thoughts which are discussed here, native service
> > also could publish events to atsv2. I think that work is also happened in
> > branch.
> >
> > Looking forward to a much wider adoption of ATSv2 with more projects.
> >
> > Thanks
> > Sunil
> >
> >
> > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > vrushalic2016@gmail.com> wrote:
> >
> > > Hi folks,
> > >
> > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > will
> > > run for 7 days, and will end August 29 11:00 PM PDT.
> > >
> > > We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >
> > > Since then, we have been working on extending the capabilities of
> > Timeline
> > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > confident that the state of the feature meets the criteria to be merged
> > > onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > > and provide valuable feedback so that we can make it production-ready.
> > >
> > > In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> > > usability improvements based on a new architecture. What we would like
> to
> > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > complete end-to-end read/write flow with security and read level
> > > authorization via whitelists. You should be able to start setting it up
> > and
> > > testing it.
> > >
> > > At a high level, the following are the key features that have been
> > > implemented since alpha1:
> > > - Security via Kerberos Authentication and delegation tokens
> > > - Read side simple authorization via whitelist
> > > - Client configurable entity sort ordering
> > > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> > by
> > > timerange, pagination, sub-app entities
> > > - Support for storing sub-application entities (entities that exist
> > outside
> > > the scope of an application)
> > > - Configurable TTLs (time-to-live) for tables, configurable table
> > prefixes,
> > > configurable hbase cluster
> > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > - Uses latest stable HBase release 1.2.6
> > >
> > > There are a total of 82 subtasks that were completed as part of this
> > > effort.
> > >
> > > We paid close attention to ensure that once disabled Timeline Service
> v.2
> > > does not impact existing functionality when disabled (by default).
> > >
> > > Special thanks to a team of folks who worked hard and contributed
> towards
> > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >
> > > Regards,
> > > Vrushali
> > >
> > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> 383.html
> > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
+1 (binding)
I've built the current branch, and checked out a few basic areas including
documentation. Also perused the most recent changes that went in.
Thanks much for the great team work! I look forward to seeing it in action.
Regards,
Sangjin
On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com> wrote:
> +1 from my side.
>
> More from the perspective of ensuring there is no impact of ATSv2 when it
> is off (by default), I deployed the latest YARN-5355 bits into a few
> clusters and ran internal Smoke tests. The tests shows no impact when ATSv2
> is off.
>
> Best,
> Haibo
>
> On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
>
> > Thank you very much Vrushali, Rohith, Varun and other folks who made this
> > happen. Great work, really appreciate the same!!
> >
> > +1 (binding) from my side:
> >
> > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > # Accessed new YARN UI which shows various flows/flow activity etc. Seems
> > fine.
> > # Based on code, looks like all apis are compatible.
> > # REST api docs looks fine as well, I guess we could improve that a bit
> > more post merge as well.
> > # Adding to additional thoughts which are discussed here, native service
> > also could publish events to atsv2. I think that work is also happened in
> > branch.
> >
> > Looking forward to a much wider adoption of ATSv2 with more projects.
> >
> > Thanks
> > Sunil
> >
> >
> > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > vrushalic2016@gmail.com> wrote:
> >
> > > Hi folks,
> > >
> > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > will
> > > run for 7 days, and will end August 29 11:00 PM PDT.
> > >
> > > We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >
> > > Since then, we have been working on extending the capabilities of
> > Timeline
> > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > confident that the state of the feature meets the criteria to be merged
> > > onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > > and provide valuable feedback so that we can make it production-ready.
> > >
> > > In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> > > usability improvements based on a new architecture. What we would like
> to
> > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > complete end-to-end read/write flow with security and read level
> > > authorization via whitelists. You should be able to start setting it up
> > and
> > > testing it.
> > >
> > > At a high level, the following are the key features that have been
> > > implemented since alpha1:
> > > - Security via Kerberos Authentication and delegation tokens
> > > - Read side simple authorization via whitelist
> > > - Client configurable entity sort ordering
> > > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> > by
> > > timerange, pagination, sub-app entities
> > > - Support for storing sub-application entities (entities that exist
> > outside
> > > the scope of an application)
> > > - Configurable TTLs (time-to-live) for tables, configurable table
> > prefixes,
> > > configurable hbase cluster
> > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > - Uses latest stable HBase release 1.2.6
> > >
> > > There are a total of 82 subtasks that were completed as part of this
> > > effort.
> > >
> > > We paid close attention to ensure that once disabled Timeline Service
> v.2
> > > does not impact existing functionality when disabled (by default).
> > >
> > > Special thanks to a team of folks who worked hard and contributed
> towards
> > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >
> > > Regards,
> > > Vrushali
> > >
> > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> 383.html
> > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Sangjin Lee <sj...@apache.org>.
+1 (binding)
I've built the current branch, and checked out a few basic areas including
documentation. Also perused the most recent changes that went in.
Thanks much for the great team work! I look forward to seeing it in action.
Regards,
Sangjin
On Fri, Aug 25, 2017 at 9:27 AM, Haibo Chen <ha...@cloudera.com> wrote:
> +1 from my side.
>
> More from the perspective of ensuring there is no impact of ATSv2 when it
> is off (by default), I deployed the latest YARN-5355 bits into a few
> clusters and ran internal Smoke tests. The tests shows no impact when ATSv2
> is off.
>
> Best,
> Haibo
>
> On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
>
> > Thank you very much Vrushali, Rohith, Varun and other folks who made this
> > happen. Great work, really appreciate the same!!
> >
> > +1 (binding) from my side:
> >
> > # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> > # Accessed new YARN UI which shows various flows/flow activity etc. Seems
> > fine.
> > # Based on code, looks like all apis are compatible.
> > # REST api docs looks fine as well, I guess we could improve that a bit
> > more post merge as well.
> > # Adding to additional thoughts which are discussed here, native service
> > also could publish events to atsv2. I think that work is also happened in
> > branch.
> >
> > Looking forward to a much wider adoption of ATSv2 with more projects.
> >
> > Thanks
> > Sunil
> >
> >
> > On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> > vrushalic2016@gmail.com> wrote:
> >
> > > Hi folks,
> > >
> > > Per earlier discussion [1], I'd like to start a formal vote to merge
> > > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > will
> > > run for 7 days, and will end August 29 11:00 PM PDT.
> > >
> > > We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > > v2 has been part of Hadoop release 3.0.0-alpha1.
> > >
> > > Since then, we have been working on extending the capabilities of
> > Timeline
> > > Service v2 in a feature branch [2] for a while, and we are reasonably
> > > confident that the state of the feature meets the criteria to be merged
> > > onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > > and provide valuable feedback so that we can make it production-ready.
> > >
> > > In a nutshell, Timeline Service v.2 delivers significant scalability
> and
> > > usability improvements based on a new architecture. What we would like
> to
> > > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > > complete end-to-end read/write flow with security and read level
> > > authorization via whitelists. You should be able to start setting it up
> > and
> > > testing it.
> > >
> > > At a high level, the following are the key features that have been
> > > implemented since alpha1:
> > > - Security via Kerberos Authentication and delegation tokens
> > > - Read side simple authorization via whitelist
> > > - Client configurable entity sort ordering
> > > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> > by
> > > timerange, pagination, sub-app entities
> > > - Support for storing sub-application entities (entities that exist
> > outside
> > > the scope of an application)
> > > - Configurable TTLs (time-to-live) for tables, configurable table
> > prefixes,
> > > configurable hbase cluster
> > > - Flow level aggregations done as dynamic (table level) coprocessors
> > > - Uses latest stable HBase release 1.2.6
> > >
> > > There are a total of 82 subtasks that were completed as part of this
> > > effort.
> > >
> > > We paid close attention to ensure that once disabled Timeline Service
> v.2
> > > does not impact existing functionality when disabled (by default).
> > >
> > > Special thanks to a team of folks who worked hard and contributed
> towards
> > > this effort with patches, reviews and guidance: Rohith Sharma K S,
> Varun
> > > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> > >
> > > Regards,
> > > Vrushali
> > >
> > > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27
> 383.html
> > > [2] https://issues.apache.org/jira/browse/YARN-5355
> > > [3] https://issues.apache.org/jira/browse/YARN-2928
> > > [4] https://github.com/apache/hadoop/commits/YARN-5355
> > >
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Haibo Chen <ha...@cloudera.com>.
+1 from my side.
More from the perspective of ensuring there is no impact of ATSv2 when it
is off (by default), I deployed the latest YARN-5355 bits into a few
clusters and ran internal Smoke tests. The tests shows no impact when ATSv2
is off.
Best,
Haibo
On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> Thank you very much Vrushali, Rohith, Varun and other folks who made this
> happen. Great work, really appreciate the same!!
>
> +1 (binding) from my side:
>
> # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> # Accessed new YARN UI which shows various flows/flow activity etc. Seems
> fine.
> # Based on code, looks like all apis are compatible.
> # REST api docs looks fine as well, I guess we could improve that a bit
> more post merge as well.
> # Adding to additional thoughts which are discussed here, native service
> also could publish events to atsv2. I think that work is also happened in
> branch.
>
> Looking forward to a much wider adoption of ATSv2 with more projects.
>
> Thanks
> Sunil
>
>
> On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> > effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Haibo Chen <ha...@cloudera.com>.
+1 from my side.
More from the perspective of ensuring there is no impact of ATSv2 when it
is off (by default), I deployed the latest YARN-5355 bits into a few
clusters and ran internal Smoke tests. The tests shows no impact when ATSv2
is off.
Best,
Haibo
On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> Thank you very much Vrushali, Rohith, Varun and other folks who made this
> happen. Great work, really appreciate the same!!
>
> +1 (binding) from my side:
>
> # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> # Accessed new YARN UI which shows various flows/flow activity etc. Seems
> fine.
> # Based on code, looks like all apis are compatible.
> # REST api docs looks fine as well, I guess we could improve that a bit
> more post merge as well.
> # Adding to additional thoughts which are discussed here, native service
> also could publish events to atsv2. I think that work is also happened in
> branch.
>
> Looking forward to a much wider adoption of ATSv2 with more projects.
>
> Thanks
> Sunil
>
>
> On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> > effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Haibo Chen <ha...@cloudera.com>.
+1 from my side.
More from the perspective of ensuring there is no impact of ATSv2 when it
is off (by default), I deployed the latest YARN-5355 bits into a few
clusters and ran internal Smoke tests. The tests shows no impact when ATSv2
is off.
Best,
Haibo
On Thu, Aug 24, 2017 at 7:51 AM, Sunil G <su...@apache.org> wrote:
> Thank you very much Vrushali, Rohith, Varun and other folks who made this
> happen. Great work, really appreciate the same!!
>
> +1 (binding) from my side:
>
> # Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
> # Accessed new YARN UI which shows various flows/flow activity etc. Seems
> fine.
> # Based on code, looks like all apis are compatible.
> # REST api docs looks fine as well, I guess we could improve that a bit
> more post merge as well.
> # Adding to additional thoughts which are discussed here, native service
> also could publish events to atsv2. I think that work is also happened in
> branch.
>
> Looking forward to a much wider adoption of ATSv2 with more projects.
>
> Thanks
> Sunil
>
>
> On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> > effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Sunil G <su...@apache.org>.
Thank you very much Vrushali, Rohith, Varun and other folks who made this
happen. Great work, really appreciate the same!!
+1 (binding) from my side:
# Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
# Accessed new YARN UI which shows various flows/flow activity etc. Seems
fine.
# Based on code, looks like all apis are compatible.
# REST api docs looks fine as well, I guess we could improve that a bit
more post merge as well.
# Adding to additional thoughts which are discussed here, native service
also could publish events to atsv2. I think that work is also happened in
branch.
Looking forward to a much wider adoption of ATSv2 with more projects.
Thanks
Sunil
On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "J. Rottinghuis" <jr...@gmail.com>.
+1 (non-binding) for the merge
@Vinod I hope that means a +1 from you as well!
Cheers,
Joep
On Tue, Aug 22, 2017 at 11:15 AM, Vinod Kumar Vavilapalli <
vinodkv@apache.org> wrote:
> Such a great community effort - hats off, team!
>
> Thanks
> +Vinod
>
> > On Aug 21, 2017, at 11:32 PM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
> >
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "J. Rottinghuis" <jr...@gmail.com>.
+1 (non-binding) for the merge
@Vinod I hope that means a +1 from you as well!
Cheers,
Joep
On Tue, Aug 22, 2017 at 11:15 AM, Vinod Kumar Vavilapalli <
vinodkv@apache.org> wrote:
> Such a great community effort - hats off, team!
>
> Thanks
> +Vinod
>
> > On Aug 21, 2017, at 11:32 PM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
> >
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "J. Rottinghuis" <jr...@gmail.com>.
+1 (non-binding) for the merge
@Vinod I hope that means a +1 from you as well!
Cheers,
Joep
On Tue, Aug 22, 2017 at 11:15 AM, Vinod Kumar Vavilapalli <
vinodkv@apache.org> wrote:
> Such a great community effort - hats off, team!
>
> Thanks
> +Vinod
>
> > On Aug 21, 2017, at 11:32 PM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
> >
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
>
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to
trunk
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Such a great community effort - hats off, team!
Thanks
+Vinod
> On Aug 21, 2017, at 11:32 PM, Vrushali Channapattan <vr...@gmail.com> wrote:
>
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to
trunk
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Such a great community effort - hats off, team!
Thanks
+Vinod
> On Aug 21, 2017, at 11:32 PM, Vrushali Channapattan <vr...@gmail.com> wrote:
>
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
---------------------------------------------------------------------
To unsubscribe, e-mail: mapreduce-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Jason Lowe <jl...@oath.com.INVALID>.
+1 (binding)
I participated in the review for the reader authorization and verified that
ATSv2 has no significant impact when disabled. Looking forward to seeing
the next increment in functionality in a release. A big thank you to
everyone involved in this effort!
Jason
On Tue, Aug 22, 2017 at 1:32 AM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Sunil G <su...@apache.org>.
Thank you very much Vrushali, Rohith, Varun and other folks who made this
happen. Great work, really appreciate the same!!
+1 (binding) from my side:
# Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
# Accessed new YARN UI which shows various flows/flow activity etc. Seems
fine.
# Based on code, looks like all apis are compatible.
# REST api docs looks fine as well, I guess we could improve that a bit
more post merge as well.
# Adding to additional thoughts which are discussed here, native service
also could publish events to atsv2. I think that work is also happened in
branch.
Looking forward to a much wider adoption of ATSv2 with more projects.
Thanks
Sunil
On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Hi All,
Just to update.
Folks who may be interested in going through the documentation for the
feature can refer to current documentation attached in pdf format, on the
umbrella JIRA i.e. YARN-5355.
Furthermore, we have run the YARN-5355 patch(with all the changes made in
the branch) against trunk and the build is almost green barring a few
checkstyle issues. The test failures which come up in the build are
outstanding issues on trunk.
Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo
mmentId=16138266&page=com.atlassian.jira.plugin.system.
issuetabpanels:comment-tabpanel#comment-16138266
Thanks,
Varun Saxena.
On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Jason Lowe <jl...@oath.com.INVALID>.
+1 (binding)
I participated in the review for the reader authorization and verified that
ATSv2 has no significant impact when disabled. Looking forward to seeing
the next increment in functionality in a release. A big thank you to
everyone involved in this effort!
Jason
On Tue, Aug 22, 2017 at 1:32 AM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to
trunk
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Such a great community effort - hats off, team!
Thanks
+Vinod
> On Aug 21, 2017, at 11:32 PM, Vrushali Channapattan <vr...@gmail.com> wrote:
>
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
---------------------------------------------------------------------
To unsubscribe, e-mail: common-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
+1 (binding).
Kudos to all the team members for their great work!
Being part of the ATSv2 team, I have been involved with either development
or review of most of the JIRAs'.
Tested ATSv2 in both secure and non-secure mode. Also verified that there
is no impact when ATSv2 is turned off.
Regards,
Varun Saxena.
On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Chris Trezzo <ct...@gmail.com>.
+1
I helped deploy and run the ATSv2 aux service on a cluster with ~400 node
managers on it. I also verified that ATSv2 has no significant impact when
disabled.
Nice work everyone!
On Tue, Aug 29, 2017 at 6:17 AM, Aaron Gresch <ag...@gmail.com> wrote:
> +1 non-binding
>
> I did some testing with security off running both ATS v1 and v2.
>
> On Tue, Aug 22, 2017 at 1:32 AM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> > effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Chris Trezzo <ct...@gmail.com>.
+1
I helped deploy and run the ATSv2 aux service on a cluster with ~400 node
managers on it. I also verified that ATSv2 has no significant impact when
disabled.
Nice work everyone!
On Tue, Aug 29, 2017 at 6:17 AM, Aaron Gresch <ag...@gmail.com> wrote:
> +1 non-binding
>
> I did some testing with security off running both ATS v1 and v2.
>
> On Tue, Aug 22, 2017 at 1:32 AM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> > effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Chris Trezzo <ct...@gmail.com>.
+1
I helped deploy and run the ATSv2 aux service on a cluster with ~400 node
managers on it. I also verified that ATSv2 has no significant impact when
disabled.
Nice work everyone!
On Tue, Aug 29, 2017 at 6:17 AM, Aaron Gresch <ag...@gmail.com> wrote:
> +1 non-binding
>
> I did some testing with security off running both ATS v1 and v2.
>
> On Tue, Aug 22, 2017 at 1:32 AM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> > effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Chris Trezzo <ct...@gmail.com>.
+1
I helped deploy and run the ATSv2 aux service on a cluster with ~400 node
managers on it. I also verified that ATSv2 has no significant impact when
disabled.
Nice work everyone!
On Tue, Aug 29, 2017 at 6:17 AM, Aaron Gresch <ag...@gmail.com> wrote:
> +1 non-binding
>
> I did some testing with security off running both ATS v1 and v2.
>
> On Tue, Aug 22, 2017 at 1:32 AM, Vrushali Channapattan <
> vrushalic2016@gmail.com> wrote:
>
> > Hi folks,
> >
> > Per earlier discussion [1], I'd like to start a formal vote to merge
> > feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> will
> > run for 7 days, and will end August 29 11:00 PM PDT.
> >
> > We have previously completed one merge onto trunk [3] and Timeline
> Service
> > v2 has been part of Hadoop release 3.0.0-alpha1.
> >
> > Since then, we have been working on extending the capabilities of
> Timeline
> > Service v2 in a feature branch [2] for a while, and we are reasonably
> > confident that the state of the feature meets the criteria to be merged
> > onto trunk and we'd love folks to get their hands on it in a test
> capacity
> > and provide valuable feedback so that we can make it production-ready.
> >
> > In a nutshell, Timeline Service v.2 delivers significant scalability and
> > usability improvements based on a new architecture. What we would like to
> > merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > complete end-to-end read/write flow with security and read level
> > authorization via whitelists. You should be able to start setting it up
> and
> > testing it.
> >
> > At a high level, the following are the key features that have been
> > implemented since alpha1:
> > - Security via Kerberos Authentication and delegation tokens
> > - Read side simple authorization via whitelist
> > - Client configurable entity sort ordering
> > - Richer REST APIs for apps, app attempts, containers, fetching metrics
> by
> > timerange, pagination, sub-app entities
> > - Support for storing sub-application entities (entities that exist
> outside
> > the scope of an application)
> > - Configurable TTLs (time-to-live) for tables, configurable table
> prefixes,
> > configurable hbase cluster
> > - Flow level aggregations done as dynamic (table level) coprocessors
> > - Uses latest stable HBase release 1.2.6
> >
> > There are a total of 82 subtasks that were completed as part of this
> > effort.
> >
> > We paid close attention to ensure that once disabled Timeline Service v.2
> > does not impact existing functionality when disabled (by default).
> >
> > Special thanks to a team of folks who worked hard and contributed towards
> > this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> > Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> > Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
> >
> > Regards,
> > Vrushali
> >
> > [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> > [2] https://issues.apache.org/jira/browse/YARN-5355
> > [3] https://issues.apache.org/jira/browse/YARN-2928
> > [4] https://github.com/apache/hadoop/commits/YARN-5355
> >
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Aaron Gresch <ag...@gmail.com>.
+1 non-binding
I did some testing with security off running both ATS v1 and v2.
On Tue, Aug 22, 2017 at 1:32 AM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Sunil G <su...@apache.org>.
Thank you very much Vrushali, Rohith, Varun and other folks who made this
happen. Great work, really appreciate the same!!
+1 (binding) from my side:
# Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
# Accessed new YARN UI which shows various flows/flow activity etc. Seems
fine.
# Based on code, looks like all apis are compatible.
# REST api docs looks fine as well, I guess we could improve that a bit
more post merge as well.
# Adding to additional thoughts which are discussed here, native service
also could publish events to atsv2. I think that work is also happened in
branch.
Looking forward to a much wider adoption of ATSv2 with more projects.
Thanks
Sunil
On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Jason Lowe <jl...@oath.com.INVALID>.
+1 (binding)
I participated in the review for the reader authorization and verified that
ATSv2 has no significant impact when disabled. Looking forward to seeing
the next increment in functionality in a release. A big thank you to
everyone involved in this effort!
Jason
On Tue, Aug 22, 2017 at 1:32 AM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Aaron Gresch <ag...@gmail.com>.
+1 non-binding
I did some testing with security off running both ATS v1 and v2.
On Tue, Aug 22, 2017 at 1:32 AM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to
trunk
Posted by Vinod Kumar Vavilapalli <vi...@apache.org>.
Such a great community effort - hats off, team!
Thanks
+Vinod
> On Aug 21, 2017, at 11:32 PM, Vrushali Channapattan <vr...@gmail.com> wrote:
>
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-help@hadoop.apache.org
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Hi All,
Just to update.
Folks who may be interested in going through the documentation for the
feature can refer to current documentation attached in pdf format, on the
umbrella JIRA i.e. YARN-5355.
Furthermore, we have run the YARN-5355 patch(with all the changes made in
the branch) against trunk and the build is almost green barring a few
checkstyle issues. The test failures which come up in the build are
outstanding issues on trunk.
Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo
mmentId=16138266&page=com.atlassian.jira.plugin.system.
issuetabpanels:comment-tabpanel#comment-16138266
Thanks,
Varun Saxena.
On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Sunil G <su...@apache.org>.
Thank you very much Vrushali, Rohith, Varun and other folks who made this
happen. Great work, really appreciate the same!!
+1 (binding) from my side:
# Tested ATSv2 cluster in a secure cluster. Ran some basic jobs
# Accessed new YARN UI which shows various flows/flow activity etc. Seems
fine.
# Based on code, looks like all apis are compatible.
# REST api docs looks fine as well, I guess we could improve that a bit
more post merge as well.
# Adding to additional thoughts which are discussed here, native service
also could publish events to atsv2. I think that work is also happened in
branch.
Looking forward to a much wider adoption of ATSv2 with more projects.
Thanks
Sunil
On Tue, Aug 22, 2017 at 12:02 PM Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by Rohith Sharma K S <ro...@apache.org>.
+1 (binding)
Thank you very much for the great team work!
Built from source and deployed in secured cluster. The below are the test
result.
Deployment :
Standard hadoop security deployment authentication and authorization as
well.
Branch-2 Hadoop and Hbase security cluster.
Branch-3 Hadoop security cluster. HBase client is pointing to Branch-2
hbase cluster.
All security configurations are set in-place.
Each service is running with its own user. Say, HDFS is running with
hdfs, YARN user is running with yarn, Hbase is running with hbase
Smoke test user : test-user
Test Cases :
Authentication :
Verify for all daemons start up successful : OK
Run a MR job using test-user : OK
Verify for REST API’s with in the scope of application : OK
Verify for REST API’s newly added I.e outside scope of application : OK.
RM Restart/ NM restart / RM_work-preserving restart has executed and
verified for data : OK. (Entity validation is done, but not entity data
validation!
Token redistribution to AM, NM is verified.
Authorization :
1 . Basic whitelisting of users to read has been validated. Works as
expected!
Disabling TSv2 configuration is also being tested.
Thanks & Regards
Rohith Sharma K S
On 22 August 2017 at 12:02, Vrushali Channapattan <vr...@gmail.com>
wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
+1 (binding).
Kudos to all the team members for their great work!
Being part of the ATSv2 team, I have been involved with either development
or review of most of the JIRAs'.
Tested ATSv2 in both secure and non-secure mode. Also verified that there
is no impact when ATSv2 is turned off.
Regards,
Varun Saxena.
On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
Hi All,
Just to update.
Folks who may be interested in going through the documentation for the
feature can refer to current documentation attached in pdf format, on the
umbrella JIRA i.e. YARN-5355.
Furthermore, we have run the YARN-5355 patch(with all the changes made in
the branch) against trunk and the build is almost green barring a few
checkstyle issues. The test failures which come up in the build are
outstanding issues on trunk.
Refer to https://issues.apache.org/jira/browse/YARN-5355?focusedCo
mmentId=16138266&page=com.atlassian.jira.plugin.system.
issuetabpanels:comment-tabpanel#comment-16138266
Thanks,
Varun Saxena.
On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>
Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk
Posted by "varunsaxena@apache.org" <va...@gmail.com>.
+1 (binding).
Kudos to all the team members for their great work!
Being part of the ATSv2 team, I have been involved with either development
or review of most of the JIRAs'.
Tested ATSv2 in both secure and non-secure mode. Also verified that there
is no impact when ATSv2 is turned off.
Regards,
Varun Saxena.
On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
vrushalic2016@gmail.com> wrote:
> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>