You are viewing a plain text version of this content. The canonical link for it is here.

Posted to mapreduce-dev@hadoop.apache.org by Vrushali C <vr...@gmail.com> on 2017/09/07 23:57:36 UTC

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

Thanks everyone.

It has been over a week (~9 days) since TSv2 has been merged to trunk with
no problems thus far. We are now thinking about merging timeline service v2
to branch2 some time in the next few weeks.

So far, we have been maintaining a branch2 based YARN-5355_branch2 along
with our trunk based feature branch YARN-5355. Varun Saxena has been
diligently rebasing it to stay current with branch2.

Currently we are in the process of testing it just like we did our due
diligence with the trunk based YARN-5355 branch and will ensure the TSv2
branch2 code is a stable state to be merged.

We will send out another email when we are ready to merge to branch2.
thanks
Vrushali

On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
wrote:

> Good to see this merged. I have initiated a separate thread with a smaller
> set of stakeholders to discuss inclusion in 2.9. We'll report back to the
> 2.9 release thread as soon as we reach consensus.
>
> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
> wrote:
>
>> +1 to maintaining history.
>>
>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>> varun.saxena.apache@gmail.com> wrote:
>>
>> > Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
>> > Maintaining history I believe can be useful as it can make reverts
>> > easier if at all required.
>> > And can be an easy reference point to look at who had contributed what
>> > without having to go back to the branch.
>> >
>> > Regards,
>> > Varun Saxena.
>> >
>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>> > wrote:
>> >
>> > > Thanks Sangjin for the link to the previous discussions on this! I
>> think
>> > > that helps answer Steve's questions.
>> > >
>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>> merged
>> > to
>> > > trunk via "git merge --no-ff" .
>> > >
>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>> > > chose to develop on a feature branch YARN-5355 so that we could
>> control
>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>> > >
>> > > Is the latest merge causing any conflicts or issues for s3guard,
>> Steve?
>> > >
>> > > thanks
>> > > Vrushali
>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>> 6afd9ef
>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org
>> %3E
>> > >
>> > >
>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>> wrote:
>> > >
>> > >> I recall this discussion about a couple of years ago:
>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>> > >> dev.hadoop.apache.org%3E
>> > >>
>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>> stevel@hortonworks.com
>> > >
>> > >> wrote:
>> > >>
>> > >>> I'd have assumed it would have gone in as one single patch, rather
>> than
>> > >>> a full history. I don't see why the trunk needs all the evolutionary
>> > >>> history of a build.
>> > >>>
>> > >>> What should our policy/process be here?
>> > >>>
>> > >>> I do currently plan to merge the s3guard in as one single squashed
>> > >>> patch; just getting HADOOP-14809 sorted first.
>> > >>>
>> > >>>
>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>> > wrote:
>> > >>> >
>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>> > >>> >
>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get
>> on
>> > >>> with
>> > >>> > the merge to trunk shortly. Thanks everyone!
>> > >>> >
>> > >>> > Regards
>> > >>> > Vrushali
>> > >>> >
>> > >>> >
>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>> > >>> > varun.saxena.apache@gmail.com> wrote:
>> > >>> >
>> > >>> >> +1 (binding).
>> > >>> >>
>> > >>> >> Kudos to all the team members for their great work!
>> > >>> >>
>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>> > >>> development
>> > >>> >> or review of most of the JIRAs'.
>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>> that
>> > >>> there
>> > >>> >> is no impact when ATSv2 is turned off.
>> > >>> >>
>> > >>> >> Regards,
>> > >>> >> Varun Saxena.
>> > >>> >>
>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>> > >>> >> vrushalic2016@gmail.com> wrote:
>> > >>> >>
>> > >>> >>> Hi folks,
>> > >>> >>>
>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>> > merge
>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>> The
>> > >>> vote
>> > >>> >>> will
>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>> > >>> >>>
>> > >>> >>> We have previously completed one merge onto trunk [3] and
>> Timeline
>> > >>> Service
>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>> > >>> >>>
>> > >>> >>> Since then, we have been working on extending the capabilities
>> of
>> > >>> Timeline
>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>> > reasonably
>> > >>> >>> confident that the state of the feature meets the criteria to be
>> > >>> merged
>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>> test
>> > >>> capacity
>> > >>> >>> and provide valuable feedback so that we can make it
>> > >>> production-ready.
>> > >>> >>>
>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>> > scalability
>> > >>> and
>> > >>> >>> usability improvements based on a new architecture. What we
>> would
>> > >>> like to
>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>> has a
>> > >>> >>> complete end-to-end read/write flow with security and read level
>> > >>> >>> authorization via whitelists. You should be able to start
>> setting
>> > it
>> > >>> up
>> > >>> >>> and
>> > >>> >>> testing it.
>> > >>> >>>
>> > >>> >>> At a high level, the following are the key features that have
>> been
>> > >>> >>> implemented since alpha1:
>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>> > >>> >>> - Read side simple authorization via whitelist
>> > >>> >>> - Client configurable entity sort ordering
>> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>> > >>> metrics by
>> > >>> >>> timerange, pagination, sub-app entities
>> > >>> >>> - Support for storing sub-application entities (entities that
>> exist
>> > >>> >>> outside
>> > >>> >>> the scope of an application)
>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>> table
>> > >>> >>> prefixes,
>> > >>> >>> configurable hbase cluster
>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>> > coprocessors
>> > >>> >>> - Uses latest stable HBase release 1.2.6
>> > >>> >>>
>> > >>> >>> There are a total of 82 subtasks that were completed as part of
>> > this
>> > >>> >>> effort.
>> > >>> >>>
>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>> > >>> Service v.2
>> > >>> >>> does not impact existing functionality when disabled (by
>> default).
>> > >>> >>>
>> > >>> >>> Special thanks to a team of folks who worked hard and
>> contributed
>> > >>> towards
>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma K
>> S,
>> > >>> Varun
>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli,
>> > Joep
>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>> > >>> >>>
>> > >>> >>> Regards,
>> > >>> >>> Vrushali
>> > >>> >>>
>> > >>> >>> [1] http://www.mail-archive.com/ya
>> rn-dev@hadoop.apache.org/msg27
>> > >>> 383.html
>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>> > >>> >>>
>> > >>> >>
>> > >>> >>
>> > >>>
>> > >>>
>> > >>> ------------------------------------------------------------
>> ---------
>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>> > >>>
>> > >>>
>> > >>
>> > >
>> >
>>
>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

Posted by "varunsaxena@apache.org" <va...@gmail.com>.

Hi All,

Please note that ATSv2 has been merged to branch-2.
Refer to https://github.com/apache/hadoop/commit/
54a694172d7162b2ff45d6e1780c0523cfda66e7

We will continue monitoring if everything works well post-merge.

Regards,
Varun Saxena.

On Mon, Oct 16, 2017 at 9:54 PM, Vrushali C <vr...@gmail.com> wrote:

> Timeline Service v2 should be landing on branch2 shortly.
>
> thanks
> Vrushali
>
> On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com>
> wrote:
>
>> Thanks everyone.
>>
>> It has been over a week (~9 days) since TSv2 has been merged to trunk
>> with no problems thus far. We are now thinking about merging timeline
>> service v2 to branch2 some time in the next few weeks.
>>
>> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
>> with our trunk based feature branch YARN-5355. Varun Saxena has been
>> diligently rebasing it to stay current with branch2.
>>
>> Currently we are in the process of testing it just like we did our due
>> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
>> branch2 code is a stable state to be merged.
>>
>> We will send out another email when we are ready to merge to branch2.
>> thanks
>> Vrushali
>>
>> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
>> wrote:
>>
>>> Good to see this merged. I have initiated a separate thread with a
>>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>>> to the 2.9 release thread as soon as we reach consensus.
>>>
>>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>>> wrote:
>>>
>>>> +1 to maintaining history.
>>>>
>>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>>> varun.saxena.apache@gmail.com> wrote:
>>>>
>>>> > Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
>>>> > Maintaining history I believe can be useful as it can make reverts
>>>> > easier if at all required.
>>>> > And can be an easy reference point to look at who had contributed what
>>>> > without having to go back to the branch.
>>>> >
>>>> > Regards,
>>>> > Varun Saxena.
>>>> >
>>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>>> think
>>>> > > that helps answer Steve's questions.
>>>> > >
>>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>>> merged
>>>> > to
>>>> > > trunk via "git merge --no-ff" .
>>>> > >
>>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge,
>>>> we
>>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>>> control
>>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>>> > >
>>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>>> Steve?
>>>> > >
>>>> > > thanks
>>>> > > Vrushali
>>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>>> 6afd9ef
>>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>>> pache.org%3E
>>>> > >
>>>> > >
>>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>>> wrote:
>>>> > >
>>>> > >> I recall this discussion about a couple of years ago:
>>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>>> > >> dev.hadoop.apache.org%3E
>>>> > >>
>>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>>> stevel@hortonworks.com
>>>> > >
>>>> > >> wrote:
>>>> > >>
>>>> > >>> I'd have assumed it would have gone in as one single patch,
>>>> rather than
>>>> > >>> a full history. I don't see why the trunk needs all the
>>>> evolutionary
>>>> > >>> history of a build.
>>>> > >>>
>>>> > >>> What should our policy/process be here?
>>>> > >>>
>>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>>> > >>>
>>>> > >>>
>>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> > >>> >
>>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>>> > >>> >
>>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>>> get on
>>>> > >>> with
>>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>>> > >>> >
>>>> > >>> > Regards
>>>> > >>> > Vrushali
>>>> > >>> >
>>>> > >>> >
>>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>>> > >>> >
>>>> > >>> >> +1 (binding).
>>>> > >>> >>
>>>> > >>> >> Kudos to all the team members for their great work!
>>>> > >>> >>
>>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>>> > >>> development
>>>> > >>> >> or review of most of the JIRAs'.
>>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>>> that
>>>> > >>> there
>>>> > >>> >> is no impact when ATSv2 is turned off.
>>>> > >>> >>
>>>> > >>> >> Regards,
>>>> > >>> >> Varun Saxena.
>>>> > >>> >>
>>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>>> > >>> >>
>>>> > >>> >>> Hi folks,
>>>> > >>> >>>
>>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>>> > merge
>>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>>> The
>>>> > >>> vote
>>>> > >>> >>> will
>>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>>> > >>> >>>
>>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>>> Timeline
>>>> > >>> Service
>>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>> > >>> >>>
>>>> > >>> >>> Since then, we have been working on extending the
>>>> capabilities of
>>>> > >>> Timeline
>>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>>> > reasonably
>>>> > >>> >>> confident that the state of the feature meets the criteria to
>>>> be
>>>> > >>> merged
>>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>>> test
>>>> > >>> capacity
>>>> > >>> >>> and provide valuable feedback so that we can make it
>>>> > >>> production-ready.
>>>> > >>> >>>
>>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>>> > scalability
>>>> > >>> and
>>>> > >>> >>> usability improvements based on a new architecture. What we
>>>> would
>>>> > >>> like to
>>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>>> has a
>>>> > >>> >>> complete end-to-end read/write flow with security and read
>>>> level
>>>> > >>> >>> authorization via whitelists. You should be able to start
>>>> setting
>>>> > it
>>>> > >>> up
>>>> > >>> >>> and
>>>> > >>> >>> testing it.
>>>> > >>> >>>
>>>> > >>> >>> At a high level, the following are the key features that have
>>>> been
>>>> > >>> >>> implemented since alpha1:
>>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>>> > >>> >>> - Read side simple authorization via whitelist
>>>> > >>> >>> - Client configurable entity sort ordering
>>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers,
>>>> fetching
>>>> > >>> metrics by
>>>> > >>> >>> timerange, pagination, sub-app entities
>>>> > >>> >>> - Support for storing sub-application entities (entities that
>>>> exist
>>>> > >>> >>> outside
>>>> > >>> >>> the scope of an application)
>>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>>> table
>>>> > >>> >>> prefixes,
>>>> > >>> >>> configurable hbase cluster
>>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>>> > coprocessors
>>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>>> > >>> >>>
>>>> > >>> >>> There are a total of 82 subtasks that were completed as part
>>>> of
>>>> > this
>>>> > >>> >>> effort.
>>>> > >>> >>>
>>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>>> > >>> Service v.2
>>>> > >>> >>> does not impact existing functionality when disabled (by
>>>> default).
>>>> > >>> >>>
>>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>>> contributed
>>>> > >>> towards
>>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>>> K S,
>>>> > >>> Varun
>>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>>> Vavilapalli,
>>>> > Joep
>>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal
>>>> Stack.
>>>> > >>> >>>
>>>> > >>> >>> Regards,
>>>> > >>> >>> Vrushali
>>>> > >>> >>>
>>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>>> rn-dev@hadoop.apache.org/msg27
>>>> > >>> 383.html
>>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>>> > >>> >>>
>>>> > >>> >>
>>>> > >>> >>
>>>> > >>>
>>>> > >>>
>>>> > >>> ------------------------------------------------------------
>>>> ---------
>>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>> > >>>
>>>> > >>>
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

Posted by "varunsaxena@apache.org" <va...@gmail.com>.

Hi All,

Please note that ATSv2 has been merged to branch-2.
Refer to https://github.com/apache/hadoop/commit/
54a694172d7162b2ff45d6e1780c0523cfda66e7

We will continue monitoring if everything works well post-merge.

Regards,
Varun Saxena.

On Mon, Oct 16, 2017 at 9:54 PM, Vrushali C <vr...@gmail.com> wrote:

> Timeline Service v2 should be landing on branch2 shortly.
>
> thanks
> Vrushali
>
> On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com>
> wrote:
>
>> Thanks everyone.
>>
>> It has been over a week (~9 days) since TSv2 has been merged to trunk
>> with no problems thus far. We are now thinking about merging timeline
>> service v2 to branch2 some time in the next few weeks.
>>
>> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
>> with our trunk based feature branch YARN-5355. Varun Saxena has been
>> diligently rebasing it to stay current with branch2.
>>
>> Currently we are in the process of testing it just like we did our due
>> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
>> branch2 code is a stable state to be merged.
>>
>> We will send out another email when we are ready to merge to branch2.
>> thanks
>> Vrushali
>>
>> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
>> wrote:
>>
>>> Good to see this merged. I have initiated a separate thread with a
>>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>>> to the 2.9 release thread as soon as we reach consensus.
>>>
>>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>>> wrote:
>>>
>>>> +1 to maintaining history.
>>>>
>>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>>> varun.saxena.apache@gmail.com> wrote:
>>>>
>>>> > Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
>>>> > Maintaining history I believe can be useful as it can make reverts
>>>> > easier if at all required.
>>>> > And can be an easy reference point to look at who had contributed what
>>>> > without having to go back to the branch.
>>>> >
>>>> > Regards,
>>>> > Varun Saxena.
>>>> >
>>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>>> think
>>>> > > that helps answer Steve's questions.
>>>> > >
>>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>>> merged
>>>> > to
>>>> > > trunk via "git merge --no-ff" .
>>>> > >
>>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge,
>>>> we
>>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>>> control
>>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>>> > >
>>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>>> Steve?
>>>> > >
>>>> > > thanks
>>>> > > Vrushali
>>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>>> 6afd9ef
>>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>>> pache.org%3E
>>>> > >
>>>> > >
>>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>>> wrote:
>>>> > >
>>>> > >> I recall this discussion about a couple of years ago:
>>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>>> > >> dev.hadoop.apache.org%3E
>>>> > >>
>>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>>> stevel@hortonworks.com
>>>> > >
>>>> > >> wrote:
>>>> > >>
>>>> > >>> I'd have assumed it would have gone in as one single patch,
>>>> rather than
>>>> > >>> a full history. I don't see why the trunk needs all the
>>>> evolutionary
>>>> > >>> history of a build.
>>>> > >>>
>>>> > >>> What should our policy/process be here?
>>>> > >>>
>>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>>> > >>>
>>>> > >>>
>>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> > >>> >
>>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>>> > >>> >
>>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>>> get on
>>>> > >>> with
>>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>>> > >>> >
>>>> > >>> > Regards
>>>> > >>> > Vrushali
>>>> > >>> >
>>>> > >>> >
>>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>>> > >>> >
>>>> > >>> >> +1 (binding).
>>>> > >>> >>
>>>> > >>> >> Kudos to all the team members for their great work!
>>>> > >>> >>
>>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>>> > >>> development
>>>> > >>> >> or review of most of the JIRAs'.
>>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>>> that
>>>> > >>> there
>>>> > >>> >> is no impact when ATSv2 is turned off.
>>>> > >>> >>
>>>> > >>> >> Regards,
>>>> > >>> >> Varun Saxena.
>>>> > >>> >>
>>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>>> > >>> >>
>>>> > >>> >>> Hi folks,
>>>> > >>> >>>
>>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>>> > merge
>>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>>> The
>>>> > >>> vote
>>>> > >>> >>> will
>>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>>> > >>> >>>
>>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>>> Timeline
>>>> > >>> Service
>>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>> > >>> >>>
>>>> > >>> >>> Since then, we have been working on extending the
>>>> capabilities of
>>>> > >>> Timeline
>>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>>> > reasonably
>>>> > >>> >>> confident that the state of the feature meets the criteria to
>>>> be
>>>> > >>> merged
>>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>>> test
>>>> > >>> capacity
>>>> > >>> >>> and provide valuable feedback so that we can make it
>>>> > >>> production-ready.
>>>> > >>> >>>
>>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>>> > scalability
>>>> > >>> and
>>>> > >>> >>> usability improvements based on a new architecture. What we
>>>> would
>>>> > >>> like to
>>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>>> has a
>>>> > >>> >>> complete end-to-end read/write flow with security and read
>>>> level
>>>> > >>> >>> authorization via whitelists. You should be able to start
>>>> setting
>>>> > it
>>>> > >>> up
>>>> > >>> >>> and
>>>> > >>> >>> testing it.
>>>> > >>> >>>
>>>> > >>> >>> At a high level, the following are the key features that have
>>>> been
>>>> > >>> >>> implemented since alpha1:
>>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>>> > >>> >>> - Read side simple authorization via whitelist
>>>> > >>> >>> - Client configurable entity sort ordering
>>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers,
>>>> fetching
>>>> > >>> metrics by
>>>> > >>> >>> timerange, pagination, sub-app entities
>>>> > >>> >>> - Support for storing sub-application entities (entities that
>>>> exist
>>>> > >>> >>> outside
>>>> > >>> >>> the scope of an application)
>>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>>> table
>>>> > >>> >>> prefixes,
>>>> > >>> >>> configurable hbase cluster
>>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>>> > coprocessors
>>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>>> > >>> >>>
>>>> > >>> >>> There are a total of 82 subtasks that were completed as part
>>>> of
>>>> > this
>>>> > >>> >>> effort.
>>>> > >>> >>>
>>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>>> > >>> Service v.2
>>>> > >>> >>> does not impact existing functionality when disabled (by
>>>> default).
>>>> > >>> >>>
>>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>>> contributed
>>>> > >>> towards
>>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>>> K S,
>>>> > >>> Varun
>>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>>> Vavilapalli,
>>>> > Joep
>>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal
>>>> Stack.
>>>> > >>> >>>
>>>> > >>> >>> Regards,
>>>> > >>> >>> Vrushali
>>>> > >>> >>>
>>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>>> rn-dev@hadoop.apache.org/msg27
>>>> > >>> 383.html
>>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>>> > >>> >>>
>>>> > >>> >>
>>>> > >>> >>
>>>> > >>>
>>>> > >>>
>>>> > >>> ------------------------------------------------------------
>>>> ---------
>>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>> > >>>
>>>> > >>>
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

Posted by "varunsaxena@apache.org" <va...@gmail.com>.

Hi All,

Please note that ATSv2 has been merged to branch-2.
Refer to https://github.com/apache/hadoop/commit/
54a694172d7162b2ff45d6e1780c0523cfda66e7

We will continue monitoring if everything works well post-merge.

Regards,
Varun Saxena.

On Mon, Oct 16, 2017 at 9:54 PM, Vrushali C <vr...@gmail.com> wrote:

> Timeline Service v2 should be landing on branch2 shortly.
>
> thanks
> Vrushali
>
> On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com>
> wrote:
>
>> Thanks everyone.
>>
>> It has been over a week (~9 days) since TSv2 has been merged to trunk
>> with no problems thus far. We are now thinking about merging timeline
>> service v2 to branch2 some time in the next few weeks.
>>
>> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
>> with our trunk based feature branch YARN-5355. Varun Saxena has been
>> diligently rebasing it to stay current with branch2.
>>
>> Currently we are in the process of testing it just like we did our due
>> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
>> branch2 code is a stable state to be merged.
>>
>> We will send out another email when we are ready to merge to branch2.
>> thanks
>> Vrushali
>>
>> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
>> wrote:
>>
>>> Good to see this merged. I have initiated a separate thread with a
>>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>>> to the 2.9 release thread as soon as we reach consensus.
>>>
>>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>>> wrote:
>>>
>>>> +1 to maintaining history.
>>>>
>>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>>> varun.saxena.apache@gmail.com> wrote:
>>>>
>>>> > Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
>>>> > Maintaining history I believe can be useful as it can make reverts
>>>> > easier if at all required.
>>>> > And can be an easy reference point to look at who had contributed what
>>>> > without having to go back to the branch.
>>>> >
>>>> > Regards,
>>>> > Varun Saxena.
>>>> >
>>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>>> think
>>>> > > that helps answer Steve's questions.
>>>> > >
>>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>>> merged
>>>> > to
>>>> > > trunk via "git merge --no-ff" .
>>>> > >
>>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge,
>>>> we
>>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>>> control
>>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>>> > >
>>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>>> Steve?
>>>> > >
>>>> > > thanks
>>>> > > Vrushali
>>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>>> 6afd9ef
>>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>>> pache.org%3E
>>>> > >
>>>> > >
>>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>>> wrote:
>>>> > >
>>>> > >> I recall this discussion about a couple of years ago:
>>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>>> > >> dev.hadoop.apache.org%3E
>>>> > >>
>>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>>> stevel@hortonworks.com
>>>> > >
>>>> > >> wrote:
>>>> > >>
>>>> > >>> I'd have assumed it would have gone in as one single patch,
>>>> rather than
>>>> > >>> a full history. I don't see why the trunk needs all the
>>>> evolutionary
>>>> > >>> history of a build.
>>>> > >>>
>>>> > >>> What should our policy/process be here?
>>>> > >>>
>>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>>> > >>>
>>>> > >>>
>>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> > >>> >
>>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>>> > >>> >
>>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>>> get on
>>>> > >>> with
>>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>>> > >>> >
>>>> > >>> > Regards
>>>> > >>> > Vrushali
>>>> > >>> >
>>>> > >>> >
>>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>>> > >>> >
>>>> > >>> >> +1 (binding).
>>>> > >>> >>
>>>> > >>> >> Kudos to all the team members for their great work!
>>>> > >>> >>
>>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>>> > >>> development
>>>> > >>> >> or review of most of the JIRAs'.
>>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>>> that
>>>> > >>> there
>>>> > >>> >> is no impact when ATSv2 is turned off.
>>>> > >>> >>
>>>> > >>> >> Regards,
>>>> > >>> >> Varun Saxena.
>>>> > >>> >>
>>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>>> > >>> >>
>>>> > >>> >>> Hi folks,
>>>> > >>> >>>
>>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>>> > merge
>>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>>> The
>>>> > >>> vote
>>>> > >>> >>> will
>>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>>> > >>> >>>
>>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>>> Timeline
>>>> > >>> Service
>>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>> > >>> >>>
>>>> > >>> >>> Since then, we have been working on extending the
>>>> capabilities of
>>>> > >>> Timeline
>>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>>> > reasonably
>>>> > >>> >>> confident that the state of the feature meets the criteria to
>>>> be
>>>> > >>> merged
>>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>>> test
>>>> > >>> capacity
>>>> > >>> >>> and provide valuable feedback so that we can make it
>>>> > >>> production-ready.
>>>> > >>> >>>
>>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>>> > scalability
>>>> > >>> and
>>>> > >>> >>> usability improvements based on a new architecture. What we
>>>> would
>>>> > >>> like to
>>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>>> has a
>>>> > >>> >>> complete end-to-end read/write flow with security and read
>>>> level
>>>> > >>> >>> authorization via whitelists. You should be able to start
>>>> setting
>>>> > it
>>>> > >>> up
>>>> > >>> >>> and
>>>> > >>> >>> testing it.
>>>> > >>> >>>
>>>> > >>> >>> At a high level, the following are the key features that have
>>>> been
>>>> > >>> >>> implemented since alpha1:
>>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>>> > >>> >>> - Read side simple authorization via whitelist
>>>> > >>> >>> - Client configurable entity sort ordering
>>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers,
>>>> fetching
>>>> > >>> metrics by
>>>> > >>> >>> timerange, pagination, sub-app entities
>>>> > >>> >>> - Support for storing sub-application entities (entities that
>>>> exist
>>>> > >>> >>> outside
>>>> > >>> >>> the scope of an application)
>>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>>> table
>>>> > >>> >>> prefixes,
>>>> > >>> >>> configurable hbase cluster
>>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>>> > coprocessors
>>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>>> > >>> >>>
>>>> > >>> >>> There are a total of 82 subtasks that were completed as part
>>>> of
>>>> > this
>>>> > >>> >>> effort.
>>>> > >>> >>>
>>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>>> > >>> Service v.2
>>>> > >>> >>> does not impact existing functionality when disabled (by
>>>> default).
>>>> > >>> >>>
>>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>>> contributed
>>>> > >>> towards
>>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>>> K S,
>>>> > >>> Varun
>>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>>> Vavilapalli,
>>>> > Joep
>>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal
>>>> Stack.
>>>> > >>> >>>
>>>> > >>> >>> Regards,
>>>> > >>> >>> Vrushali
>>>> > >>> >>>
>>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>>> rn-dev@hadoop.apache.org/msg27
>>>> > >>> 383.html
>>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>>> > >>> >>>
>>>> > >>> >>
>>>> > >>> >>
>>>> > >>>
>>>> > >>>
>>>> > >>> ------------------------------------------------------------
>>>> ---------
>>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>> > >>>
>>>> > >>>
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

Posted by "varunsaxena@apache.org" <va...@gmail.com>.

Hi All,

Please note that ATSv2 has been merged to branch-2.
Refer to https://github.com/apache/hadoop/commit/
54a694172d7162b2ff45d6e1780c0523cfda66e7

We will continue monitoring if everything works well post-merge.

Regards,
Varun Saxena.

On Mon, Oct 16, 2017 at 9:54 PM, Vrushali C <vr...@gmail.com> wrote:

> Timeline Service v2 should be landing on branch2 shortly.
>
> thanks
> Vrushali
>
> On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com>
> wrote:
>
>> Thanks everyone.
>>
>> It has been over a week (~9 days) since TSv2 has been merged to trunk
>> with no problems thus far. We are now thinking about merging timeline
>> service v2 to branch2 some time in the next few weeks.
>>
>> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
>> with our trunk based feature branch YARN-5355. Varun Saxena has been
>> diligently rebasing it to stay current with branch2.
>>
>> Currently we are in the process of testing it just like we did our due
>> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
>> branch2 code is a stable state to be merged.
>>
>> We will send out another email when we are ready to merge to branch2.
>> thanks
>> Vrushali
>>
>> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
>> wrote:
>>
>>> Good to see this merged. I have initiated a separate thread with a
>>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>>> to the 2.9 release thread as soon as we reach consensus.
>>>
>>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>>> wrote:
>>>
>>>> +1 to maintaining history.
>>>>
>>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>>> varun.saxena.apache@gmail.com> wrote:
>>>>
>>>> > Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
>>>> > Maintaining history I believe can be useful as it can make reverts
>>>> > easier if at all required.
>>>> > And can be an easy reference point to look at who had contributed what
>>>> > without having to go back to the branch.
>>>> >
>>>> > Regards,
>>>> > Varun Saxena.
>>>> >
>>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> >
>>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>>> think
>>>> > > that helps answer Steve's questions.
>>>> > >
>>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>>> merged
>>>> > to
>>>> > > trunk via "git merge --no-ff" .
>>>> > >
>>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge,
>>>> we
>>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>>> control
>>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>>> > >
>>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>>> Steve?
>>>> > >
>>>> > > thanks
>>>> > > Vrushali
>>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>>> 6afd9ef
>>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>>> pache.org%3E
>>>> > >
>>>> > >
>>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>>> wrote:
>>>> > >
>>>> > >> I recall this discussion about a couple of years ago:
>>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>>> > >> dev.hadoop.apache.org%3E
>>>> > >>
>>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>>> stevel@hortonworks.com
>>>> > >
>>>> > >> wrote:
>>>> > >>
>>>> > >>> I'd have assumed it would have gone in as one single patch,
>>>> rather than
>>>> > >>> a full history. I don't see why the trunk needs all the
>>>> evolutionary
>>>> > >>> history of a build.
>>>> > >>>
>>>> > >>> What should our policy/process be here?
>>>> > >>>
>>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>>> > >>>
>>>> > >>>
>>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>>> > wrote:
>>>> > >>> >
>>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>>> > >>> >
>>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>>> get on
>>>> > >>> with
>>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>>> > >>> >
>>>> > >>> > Regards
>>>> > >>> > Vrushali
>>>> > >>> >
>>>> > >>> >
>>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>>> > >>> >
>>>> > >>> >> +1 (binding).
>>>> > >>> >>
>>>> > >>> >> Kudos to all the team members for their great work!
>>>> > >>> >>
>>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>>> > >>> development
>>>> > >>> >> or review of most of the JIRAs'.
>>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>>> that
>>>> > >>> there
>>>> > >>> >> is no impact when ATSv2 is turned off.
>>>> > >>> >>
>>>> > >>> >> Regards,
>>>> > >>> >> Varun Saxena.
>>>> > >>> >>
>>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>>> > >>> >>
>>>> > >>> >>> Hi folks,
>>>> > >>> >>>
>>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>>> > merge
>>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>>> The
>>>> > >>> vote
>>>> > >>> >>> will
>>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>>> > >>> >>>
>>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>>> Timeline
>>>> > >>> Service
>>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>>> > >>> >>>
>>>> > >>> >>> Since then, we have been working on extending the
>>>> capabilities of
>>>> > >>> Timeline
>>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>>> > reasonably
>>>> > >>> >>> confident that the state of the feature meets the criteria to
>>>> be
>>>> > >>> merged
>>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>>> test
>>>> > >>> capacity
>>>> > >>> >>> and provide valuable feedback so that we can make it
>>>> > >>> production-ready.
>>>> > >>> >>>
>>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>>> > scalability
>>>> > >>> and
>>>> > >>> >>> usability improvements based on a new architecture. What we
>>>> would
>>>> > >>> like to
>>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>>> has a
>>>> > >>> >>> complete end-to-end read/write flow with security and read
>>>> level
>>>> > >>> >>> authorization via whitelists. You should be able to start
>>>> setting
>>>> > it
>>>> > >>> up
>>>> > >>> >>> and
>>>> > >>> >>> testing it.
>>>> > >>> >>>
>>>> > >>> >>> At a high level, the following are the key features that have
>>>> been
>>>> > >>> >>> implemented since alpha1:
>>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>>> > >>> >>> - Read side simple authorization via whitelist
>>>> > >>> >>> - Client configurable entity sort ordering
>>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers,
>>>> fetching
>>>> > >>> metrics by
>>>> > >>> >>> timerange, pagination, sub-app entities
>>>> > >>> >>> - Support for storing sub-application entities (entities that
>>>> exist
>>>> > >>> >>> outside
>>>> > >>> >>> the scope of an application)
>>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>>> table
>>>> > >>> >>> prefixes,
>>>> > >>> >>> configurable hbase cluster
>>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>>> > coprocessors
>>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>>> > >>> >>>
>>>> > >>> >>> There are a total of 82 subtasks that were completed as part
>>>> of
>>>> > this
>>>> > >>> >>> effort.
>>>> > >>> >>>
>>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>>> > >>> Service v.2
>>>> > >>> >>> does not impact existing functionality when disabled (by
>>>> default).
>>>> > >>> >>>
>>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>>> contributed
>>>> > >>> towards
>>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>>> K S,
>>>> > >>> Varun
>>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>>> Vavilapalli,
>>>> > Joep
>>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal
>>>> Stack.
>>>> > >>> >>>
>>>> > >>> >>> Regards,
>>>> > >>> >>> Vrushali
>>>> > >>> >>>
>>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>>> rn-dev@hadoop.apache.org/msg27
>>>> > >>> 383.html
>>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>>> > >>> >>>
>>>> > >>> >>
>>>> > >>> >>
>>>> > >>>
>>>> > >>>
>>>> > >>> ------------------------------------------------------------
>>>> ---------
>>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>>> > >>>
>>>> > >>>
>>>> > >>
>>>> > >
>>>> >
>>>>
>>>
>>>
>>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

Posted by Vrushali C <vr...@gmail.com>.

Timeline Service v2 should be landing on branch2 shortly.

thanks
Vrushali

On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com> wrote:

> Thanks everyone.
>
> It has been over a week (~9 days) since TSv2 has been merged to trunk with
> no problems thus far. We are now thinking about merging timeline service v2
> to branch2 some time in the next few weeks.
>
> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
> with our trunk based feature branch YARN-5355. Varun Saxena has been
> diligently rebasing it to stay current with branch2.
>
> Currently we are in the process of testing it just like we did our due
> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
> branch2 code is a stable state to be merged.
>
> We will send out another email when we are ready to merge to branch2.
> thanks
> Vrushali
>
> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
> wrote:
>
>> Good to see this merged. I have initiated a separate thread with a
>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>> to the 2.9 release thread as soon as we reach consensus.
>>
>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>> wrote:
>>
>>> +1 to maintaining history.
>>>
>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>> varun.saxena.apache@gmail.com> wrote:
>>>
>>> > Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
>>> > Maintaining history I believe can be useful as it can make reverts
>>> > easier if at all required.
>>> > And can be an easy reference point to look at who had contributed what
>>> > without having to go back to the branch.
>>> >
>>> > Regards,
>>> > Varun Saxena.
>>> >
>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> >
>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>> think
>>> > > that helps answer Steve's questions.
>>> > >
>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>> merged
>>> > to
>>> > > trunk via "git merge --no-ff" .
>>> > >
>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>> control
>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>> > >
>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>> Steve?
>>> > >
>>> > > thanks
>>> > > Vrushali
>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>> 6afd9ef
>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>> pache.org%3E
>>> > >
>>> > >
>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>> wrote:
>>> > >
>>> > >> I recall this discussion about a couple of years ago:
>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>> > >> dev.hadoop.apache.org%3E
>>> > >>
>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>> stevel@hortonworks.com
>>> > >
>>> > >> wrote:
>>> > >>
>>> > >>> I'd have assumed it would have gone in as one single patch, rather
>>> than
>>> > >>> a full history. I don't see why the trunk needs all the
>>> evolutionary
>>> > >>> history of a build.
>>> > >>>
>>> > >>> What should our policy/process be here?
>>> > >>>
>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>> > >>>
>>> > >>>
>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> > >>> >
>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>> > >>> >
>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>> get on
>>> > >>> with
>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>> > >>> >
>>> > >>> > Regards
>>> > >>> > Vrushali
>>> > >>> >
>>> > >>> >
>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>> > >>> >
>>> > >>> >> +1 (binding).
>>> > >>> >>
>>> > >>> >> Kudos to all the team members for their great work!
>>> > >>> >>
>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>> > >>> development
>>> > >>> >> or review of most of the JIRAs'.
>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>> that
>>> > >>> there
>>> > >>> >> is no impact when ATSv2 is turned off.
>>> > >>> >>
>>> > >>> >> Regards,
>>> > >>> >> Varun Saxena.
>>> > >>> >>
>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>> > >>> >>
>>> > >>> >>> Hi folks,
>>> > >>> >>>
>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>> > merge
>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>> The
>>> > >>> vote
>>> > >>> >>> will
>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> > >>> >>>
>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>> Timeline
>>> > >>> Service
>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> > >>> >>>
>>> > >>> >>> Since then, we have been working on extending the capabilities
>>> of
>>> > >>> Timeline
>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>> > reasonably
>>> > >>> >>> confident that the state of the feature meets the criteria to
>>> be
>>> > >>> merged
>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>> test
>>> > >>> capacity
>>> > >>> >>> and provide valuable feedback so that we can make it
>>> > >>> production-ready.
>>> > >>> >>>
>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>> > scalability
>>> > >>> and
>>> > >>> >>> usability improvements based on a new architecture. What we
>>> would
>>> > >>> like to
>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>> has a
>>> > >>> >>> complete end-to-end read/write flow with security and read
>>> level
>>> > >>> >>> authorization via whitelists. You should be able to start
>>> setting
>>> > it
>>> > >>> up
>>> > >>> >>> and
>>> > >>> >>> testing it.
>>> > >>> >>>
>>> > >>> >>> At a high level, the following are the key features that have
>>> been
>>> > >>> >>> implemented since alpha1:
>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>> > >>> >>> - Read side simple authorization via whitelist
>>> > >>> >>> - Client configurable entity sort ordering
>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>>> > >>> metrics by
>>> > >>> >>> timerange, pagination, sub-app entities
>>> > >>> >>> - Support for storing sub-application entities (entities that
>>> exist
>>> > >>> >>> outside
>>> > >>> >>> the scope of an application)
>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>> table
>>> > >>> >>> prefixes,
>>> > >>> >>> configurable hbase cluster
>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>> > coprocessors
>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>> > >>> >>>
>>> > >>> >>> There are a total of 82 subtasks that were completed as part of
>>> > this
>>> > >>> >>> effort.
>>> > >>> >>>
>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>> > >>> Service v.2
>>> > >>> >>> does not impact existing functionality when disabled (by
>>> default).
>>> > >>> >>>
>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>> contributed
>>> > >>> towards
>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>> K S,
>>> > >>> Varun
>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>> Vavilapalli,
>>> > Joep
>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> > >>> >>>
>>> > >>> >>> Regards,
>>> > >>> >>> Vrushali
>>> > >>> >>>
>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>> rn-dev@hadoop.apache.org/msg27
>>> > >>> 383.html
>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> > >>> >>>
>>> > >>> >>
>>> > >>> >>
>>> > >>>
>>> > >>>
>>> > >>> ------------------------------------------------------------
>>> ---------
>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>> > >>>
>>> > >>>
>>> > >>
>>> > >
>>> >
>>>
>>
>>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

Posted by Vrushali C <vr...@gmail.com>.

Timeline Service v2 should be landing on branch2 shortly.

thanks
Vrushali

On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com> wrote:

> Thanks everyone.
>
> It has been over a week (~9 days) since TSv2 has been merged to trunk with
> no problems thus far. We are now thinking about merging timeline service v2
> to branch2 some time in the next few weeks.
>
> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
> with our trunk based feature branch YARN-5355. Varun Saxena has been
> diligently rebasing it to stay current with branch2.
>
> Currently we are in the process of testing it just like we did our due
> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
> branch2 code is a stable state to be merged.
>
> We will send out another email when we are ready to merge to branch2.
> thanks
> Vrushali
>
> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
> wrote:
>
>> Good to see this merged. I have initiated a separate thread with a
>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>> to the 2.9 release thread as soon as we reach consensus.
>>
>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>> wrote:
>>
>>> +1 to maintaining history.
>>>
>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>> varun.saxena.apache@gmail.com> wrote:
>>>
>>> > Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
>>> > Maintaining history I believe can be useful as it can make reverts
>>> > easier if at all required.
>>> > And can be an easy reference point to look at who had contributed what
>>> > without having to go back to the branch.
>>> >
>>> > Regards,
>>> > Varun Saxena.
>>> >
>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> >
>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>> think
>>> > > that helps answer Steve's questions.
>>> > >
>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>> merged
>>> > to
>>> > > trunk via "git merge --no-ff" .
>>> > >
>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>> control
>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>> > >
>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>> Steve?
>>> > >
>>> > > thanks
>>> > > Vrushali
>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>> 6afd9ef
>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>> pache.org%3E
>>> > >
>>> > >
>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>> wrote:
>>> > >
>>> > >> I recall this discussion about a couple of years ago:
>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>> > >> dev.hadoop.apache.org%3E
>>> > >>
>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>> stevel@hortonworks.com
>>> > >
>>> > >> wrote:
>>> > >>
>>> > >>> I'd have assumed it would have gone in as one single patch, rather
>>> than
>>> > >>> a full history. I don't see why the trunk needs all the
>>> evolutionary
>>> > >>> history of a build.
>>> > >>>
>>> > >>> What should our policy/process be here?
>>> > >>>
>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>> > >>>
>>> > >>>
>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> > >>> >
>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>> > >>> >
>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>> get on
>>> > >>> with
>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>> > >>> >
>>> > >>> > Regards
>>> > >>> > Vrushali
>>> > >>> >
>>> > >>> >
>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>> > >>> >
>>> > >>> >> +1 (binding).
>>> > >>> >>
>>> > >>> >> Kudos to all the team members for their great work!
>>> > >>> >>
>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>> > >>> development
>>> > >>> >> or review of most of the JIRAs'.
>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>> that
>>> > >>> there
>>> > >>> >> is no impact when ATSv2 is turned off.
>>> > >>> >>
>>> > >>> >> Regards,
>>> > >>> >> Varun Saxena.
>>> > >>> >>
>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>> > >>> >>
>>> > >>> >>> Hi folks,
>>> > >>> >>>
>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>> > merge
>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>> The
>>> > >>> vote
>>> > >>> >>> will
>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> > >>> >>>
>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>> Timeline
>>> > >>> Service
>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> > >>> >>>
>>> > >>> >>> Since then, we have been working on extending the capabilities
>>> of
>>> > >>> Timeline
>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>> > reasonably
>>> > >>> >>> confident that the state of the feature meets the criteria to
>>> be
>>> > >>> merged
>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>> test
>>> > >>> capacity
>>> > >>> >>> and provide valuable feedback so that we can make it
>>> > >>> production-ready.
>>> > >>> >>>
>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>> > scalability
>>> > >>> and
>>> > >>> >>> usability improvements based on a new architecture. What we
>>> would
>>> > >>> like to
>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>> has a
>>> > >>> >>> complete end-to-end read/write flow with security and read
>>> level
>>> > >>> >>> authorization via whitelists. You should be able to start
>>> setting
>>> > it
>>> > >>> up
>>> > >>> >>> and
>>> > >>> >>> testing it.
>>> > >>> >>>
>>> > >>> >>> At a high level, the following are the key features that have
>>> been
>>> > >>> >>> implemented since alpha1:
>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>> > >>> >>> - Read side simple authorization via whitelist
>>> > >>> >>> - Client configurable entity sort ordering
>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>>> > >>> metrics by
>>> > >>> >>> timerange, pagination, sub-app entities
>>> > >>> >>> - Support for storing sub-application entities (entities that
>>> exist
>>> > >>> >>> outside
>>> > >>> >>> the scope of an application)
>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>> table
>>> > >>> >>> prefixes,
>>> > >>> >>> configurable hbase cluster
>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>> > coprocessors
>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>> > >>> >>>
>>> > >>> >>> There are a total of 82 subtasks that were completed as part of
>>> > this
>>> > >>> >>> effort.
>>> > >>> >>>
>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>> > >>> Service v.2
>>> > >>> >>> does not impact existing functionality when disabled (by
>>> default).
>>> > >>> >>>
>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>> contributed
>>> > >>> towards
>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>> K S,
>>> > >>> Varun
>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>> Vavilapalli,
>>> > Joep
>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> > >>> >>>
>>> > >>> >>> Regards,
>>> > >>> >>> Vrushali
>>> > >>> >>>
>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>> rn-dev@hadoop.apache.org/msg27
>>> > >>> 383.html
>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> > >>> >>>
>>> > >>> >>
>>> > >>> >>
>>> > >>>
>>> > >>>
>>> > >>> ------------------------------------------------------------
>>> ---------
>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>> > >>>
>>> > >>>
>>> > >>
>>> > >
>>> >
>>>
>>
>>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

Posted by Vrushali C <vr...@gmail.com>.

Timeline Service v2 should be landing on branch2 shortly.

thanks
Vrushali

On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com> wrote:

> Thanks everyone.
>
> It has been over a week (~9 days) since TSv2 has been merged to trunk with
> no problems thus far. We are now thinking about merging timeline service v2
> to branch2 some time in the next few weeks.
>
> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
> with our trunk based feature branch YARN-5355. Varun Saxena has been
> diligently rebasing it to stay current with branch2.
>
> Currently we are in the process of testing it just like we did our due
> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
> branch2 code is a stable state to be merged.
>
> We will send out another email when we are ready to merge to branch2.
> thanks
> Vrushali
>
> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
> wrote:
>
>> Good to see this merged. I have initiated a separate thread with a
>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>> to the 2.9 release thread as soon as we reach consensus.
>>
>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>> wrote:
>>
>>> +1 to maintaining history.
>>>
>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>> varun.saxena.apache@gmail.com> wrote:
>>>
>>> > Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
>>> > Maintaining history I believe can be useful as it can make reverts
>>> > easier if at all required.
>>> > And can be an easy reference point to look at who had contributed what
>>> > without having to go back to the branch.
>>> >
>>> > Regards,
>>> > Varun Saxena.
>>> >
>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> >
>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>> think
>>> > > that helps answer Steve's questions.
>>> > >
>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>> merged
>>> > to
>>> > > trunk via "git merge --no-ff" .
>>> > >
>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>> control
>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>> > >
>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>> Steve?
>>> > >
>>> > > thanks
>>> > > Vrushali
>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>> 6afd9ef
>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>> pache.org%3E
>>> > >
>>> > >
>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>> wrote:
>>> > >
>>> > >> I recall this discussion about a couple of years ago:
>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>> > >> dev.hadoop.apache.org%3E
>>> > >>
>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>> stevel@hortonworks.com
>>> > >
>>> > >> wrote:
>>> > >>
>>> > >>> I'd have assumed it would have gone in as one single patch, rather
>>> than
>>> > >>> a full history. I don't see why the trunk needs all the
>>> evolutionary
>>> > >>> history of a build.
>>> > >>>
>>> > >>> What should our policy/process be here?
>>> > >>>
>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>> > >>>
>>> > >>>
>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> > >>> >
>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>> > >>> >
>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>> get on
>>> > >>> with
>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>> > >>> >
>>> > >>> > Regards
>>> > >>> > Vrushali
>>> > >>> >
>>> > >>> >
>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>> > >>> >
>>> > >>> >> +1 (binding).
>>> > >>> >>
>>> > >>> >> Kudos to all the team members for their great work!
>>> > >>> >>
>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>> > >>> development
>>> > >>> >> or review of most of the JIRAs'.
>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>> that
>>> > >>> there
>>> > >>> >> is no impact when ATSv2 is turned off.
>>> > >>> >>
>>> > >>> >> Regards,
>>> > >>> >> Varun Saxena.
>>> > >>> >>
>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>> > >>> >>
>>> > >>> >>> Hi folks,
>>> > >>> >>>
>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>> > merge
>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>> The
>>> > >>> vote
>>> > >>> >>> will
>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> > >>> >>>
>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>> Timeline
>>> > >>> Service
>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> > >>> >>>
>>> > >>> >>> Since then, we have been working on extending the capabilities
>>> of
>>> > >>> Timeline
>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>> > reasonably
>>> > >>> >>> confident that the state of the feature meets the criteria to
>>> be
>>> > >>> merged
>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>> test
>>> > >>> capacity
>>> > >>> >>> and provide valuable feedback so that we can make it
>>> > >>> production-ready.
>>> > >>> >>>
>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>> > scalability
>>> > >>> and
>>> > >>> >>> usability improvements based on a new architecture. What we
>>> would
>>> > >>> like to
>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>> has a
>>> > >>> >>> complete end-to-end read/write flow with security and read
>>> level
>>> > >>> >>> authorization via whitelists. You should be able to start
>>> setting
>>> > it
>>> > >>> up
>>> > >>> >>> and
>>> > >>> >>> testing it.
>>> > >>> >>>
>>> > >>> >>> At a high level, the following are the key features that have
>>> been
>>> > >>> >>> implemented since alpha1:
>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>> > >>> >>> - Read side simple authorization via whitelist
>>> > >>> >>> - Client configurable entity sort ordering
>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>>> > >>> metrics by
>>> > >>> >>> timerange, pagination, sub-app entities
>>> > >>> >>> - Support for storing sub-application entities (entities that
>>> exist
>>> > >>> >>> outside
>>> > >>> >>> the scope of an application)
>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>> table
>>> > >>> >>> prefixes,
>>> > >>> >>> configurable hbase cluster
>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>> > coprocessors
>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>> > >>> >>>
>>> > >>> >>> There are a total of 82 subtasks that were completed as part of
>>> > this
>>> > >>> >>> effort.
>>> > >>> >>>
>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>> > >>> Service v.2
>>> > >>> >>> does not impact existing functionality when disabled (by
>>> default).
>>> > >>> >>>
>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>> contributed
>>> > >>> towards
>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>> K S,
>>> > >>> Varun
>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>> Vavilapalli,
>>> > Joep
>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> > >>> >>>
>>> > >>> >>> Regards,
>>> > >>> >>> Vrushali
>>> > >>> >>>
>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>> rn-dev@hadoop.apache.org/msg27
>>> > >>> 383.html
>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> > >>> >>>
>>> > >>> >>
>>> > >>> >>
>>> > >>>
>>> > >>>
>>> > >>> ------------------------------------------------------------
>>> ---------
>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>> > >>>
>>> > >>>
>>> > >>
>>> > >
>>> >
>>>
>>
>>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

Posted by Vrushali C <vr...@gmail.com>.

Timeline Service v2 should be landing on branch2 shortly.

thanks
Vrushali

On Thu, Sep 7, 2017 at 4:57 PM, Vrushali C <vr...@gmail.com> wrote:

> Thanks everyone.
>
> It has been over a week (~9 days) since TSv2 has been merged to trunk with
> no problems thus far. We are now thinking about merging timeline service v2
> to branch2 some time in the next few weeks.
>
> So far, we have been maintaining a branch2 based YARN-5355_branch2 along
> with our trunk based feature branch YARN-5355. Varun Saxena has been
> diligently rebasing it to stay current with branch2.
>
> Currently we are in the process of testing it just like we did our due
> diligence with the trunk based YARN-5355 branch and will ensure the TSv2
> branch2 code is a stable state to be merged.
>
> We will send out another email when we are ready to merge to branch2.
> thanks
> Vrushali
>
> On Thu, Aug 31, 2017 at 12:33 PM, Subramaniam V K <su...@gmail.com>
> wrote:
>
>> Good to see this merged. I have initiated a separate thread with a
>> smaller set of stakeholders to discuss inclusion in 2.9. We'll report back
>> to the 2.9 release thread as soon as we reach consensus.
>>
>> On Thu, Aug 31, 2017 at 10:39 AM, Ravi Prakash <ra...@gmail.com>
>> wrote:
>>
>>> +1 to maintaining history.
>>>
>>> On Wed, Aug 30, 2017 at 11:38 PM, varunsaxena@apache.org <
>>> varun.saxena.apache@gmail.com> wrote:
>>>
>>> > Yes, I had used "git merge --no-ff"  while merging ATSv2 to trunk.
>>> > Maintaining history I believe can be useful as it can make reverts
>>> > easier if at all required.
>>> > And can be an easy reference point to look at who had contributed what
>>> > without having to go back to the branch.
>>> >
>>> > Regards,
>>> > Varun Saxena.
>>> >
>>> > On Thu, Aug 31, 2017 at 3:56 AM, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> >
>>> > > Thanks Sangjin for the link to the previous discussions on this! I
>>> think
>>> > > that helps answer Steve's questions.
>>> > >
>>> > > As decided on that thread [1], YARN-5355 as a feature branch was
>>> merged
>>> > to
>>> > > trunk via "git merge --no-ff" .
>>> > >
>>> > > Although trunk already had TSv2 code (alpha1) prior to this merge, we
>>> > > chose to develop on a feature branch YARN-5355 so that we could
>>> control
>>> > > when changes went into trunk and didn't inadvertently disrupt trunk.
>>> > >
>>> > > Is the latest merge causing any conflicts or issues for s3guard,
>>> Steve?
>>> > >
>>> > > thanks
>>> > > Vrushali
>>> > > [1] https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c7
>>> 6afd9ef
>>> > > f1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.a
>>> pache.org%3E
>>> > >
>>> > >
>>> > > On Wed, Aug 30, 2017 at 2:37 PM, Sangjin Lee <sj...@apache.org>
>>> wrote:
>>> > >
>>> > >> I recall this discussion about a couple of years ago:
>>> > >> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac
>>> > >> 2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-
>>> > >> dev.hadoop.apache.org%3E
>>> > >>
>>> > >> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran <
>>> stevel@hortonworks.com
>>> > >
>>> > >> wrote:
>>> > >>
>>> > >>> I'd have assumed it would have gone in as one single patch, rather
>>> than
>>> > >>> a full history. I don't see why the trunk needs all the
>>> evolutionary
>>> > >>> history of a build.
>>> > >>>
>>> > >>> What should our policy/process be here?
>>> > >>>
>>> > >>> I do currently plan to merge the s3guard in as one single squashed
>>> > >>> patch; just getting HADOOP-14809 sorted first.
>>> > >>>
>>> > >>>
>>> > >>> > On 30 Aug 2017, at 07:09, Vrushali C <vr...@gmail.com>
>>> > wrote:
>>> > >>> >
>>> > >>> > I'm adding my +1 (binding) to conclude the vote.
>>> > >>> >
>>> > >>> > With 13 +1's (11 binding) and no -1's, the vote passes. We'll
>>> get on
>>> > >>> with
>>> > >>> > the merge to trunk shortly. Thanks everyone!
>>> > >>> >
>>> > >>> > Regards
>>> > >>> > Vrushali
>>> > >>> >
>>> > >>> >
>>> > >>> > On Tue, Aug 29, 2017 at 10:54 AM, varunsaxena@apache.org <
>>> > >>> > varun.saxena.apache@gmail.com> wrote:
>>> > >>> >
>>> > >>> >> +1 (binding).
>>> > >>> >>
>>> > >>> >> Kudos to all the team members for their great work!
>>> > >>> >>
>>> > >>> >> Being part of the ATSv2 team, I have been involved with either
>>> > >>> development
>>> > >>> >> or review of most of the JIRAs'.
>>> > >>> >> Tested ATSv2 in both secure and non-secure mode. Also verified
>>> that
>>> > >>> there
>>> > >>> >> is no impact when ATSv2 is turned off.
>>> > >>> >>
>>> > >>> >> Regards,
>>> > >>> >> Varun Saxena.
>>> > >>> >>
>>> > >>> >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
>>> > >>> >> vrushalic2016@gmail.com> wrote:
>>> > >>> >>
>>> > >>> >>> Hi folks,
>>> > >>> >>>
>>> > >>> >>> Per earlier discussion [1], I'd like to start a formal vote to
>>> > merge
>>> > >>> >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk.
>>> The
>>> > >>> vote
>>> > >>> >>> will
>>> > >>> >>> run for 7 days, and will end August 29 11:00 PM PDT.
>>> > >>> >>>
>>> > >>> >>> We have previously completed one merge onto trunk [3] and
>>> Timeline
>>> > >>> Service
>>> > >>> >>> v2 has been part of Hadoop release 3.0.0-alpha1.
>>> > >>> >>>
>>> > >>> >>> Since then, we have been working on extending the capabilities
>>> of
>>> > >>> Timeline
>>> > >>> >>> Service v2 in a feature branch [2] for a while, and we are
>>> > reasonably
>>> > >>> >>> confident that the state of the feature meets the criteria to
>>> be
>>> > >>> merged
>>> > >>> >>> onto trunk and we'd love folks to get their hands on it in a
>>> test
>>> > >>> capacity
>>> > >>> >>> and provide valuable feedback so that we can make it
>>> > >>> production-ready.
>>> > >>> >>>
>>> > >>> >>> In a nutshell, Timeline Service v.2 delivers significant
>>> > scalability
>>> > >>> and
>>> > >>> >>> usability improvements based on a new architecture. What we
>>> would
>>> > >>> like to
>>> > >>> >>> merge to trunk is termed "alpha 2" (milestone 2). The feature
>>> has a
>>> > >>> >>> complete end-to-end read/write flow with security and read
>>> level
>>> > >>> >>> authorization via whitelists. You should be able to start
>>> setting
>>> > it
>>> > >>> up
>>> > >>> >>> and
>>> > >>> >>> testing it.
>>> > >>> >>>
>>> > >>> >>> At a high level, the following are the key features that have
>>> been
>>> > >>> >>> implemented since alpha1:
>>> > >>> >>> - Security via Kerberos Authentication and delegation tokens
>>> > >>> >>> - Read side simple authorization via whitelist
>>> > >>> >>> - Client configurable entity sort ordering
>>> > >>> >>> - Richer REST APIs for apps, app attempts, containers, fetching
>>> > >>> metrics by
>>> > >>> >>> timerange, pagination, sub-app entities
>>> > >>> >>> - Support for storing sub-application entities (entities that
>>> exist
>>> > >>> >>> outside
>>> > >>> >>> the scope of an application)
>>> > >>> >>> - Configurable TTLs (time-to-live) for tables, configurable
>>> table
>>> > >>> >>> prefixes,
>>> > >>> >>> configurable hbase cluster
>>> > >>> >>> - Flow level aggregations done as dynamic (table level)
>>> > coprocessors
>>> > >>> >>> - Uses latest stable HBase release 1.2.6
>>> > >>> >>>
>>> > >>> >>> There are a total of 82 subtasks that were completed as part of
>>> > this
>>> > >>> >>> effort.
>>> > >>> >>>
>>> > >>> >>> We paid close attention to ensure that once disabled Timeline
>>> > >>> Service v.2
>>> > >>> >>> does not impact existing functionality when disabled (by
>>> default).
>>> > >>> >>>
>>> > >>> >>> Special thanks to a team of folks who worked hard and
>>> contributed
>>> > >>> towards
>>> > >>> >>> this effort with patches, reviews and guidance: Rohith Sharma
>>> K S,
>>> > >>> Varun
>>> > >>> >>> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar
>>> Vavilapalli,
>>> > Joep
>>> > >>> >>> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>>> > >>> >>>
>>> > >>> >>> Regards,
>>> > >>> >>> Vrushali
>>> > >>> >>>
>>> > >>> >>> [1] http://www.mail-archive.com/ya
>>> rn-dev@hadoop.apache.org/msg27
>>> > >>> 383.html
>>> > >>> >>> [2] https://issues.apache.org/jira/browse/YARN-5355
>>> > >>> >>> [3] https://issues.apache.org/jira/browse/YARN-2928
>>> > >>> >>> [4] https://github.com/apache/hadoop/commits/YARN-5355
>>> > >>> >>>
>>> > >>> >>
>>> > >>> >>
>>> > >>>
>>> > >>>
>>> > >>> ------------------------------------------------------------
>>> ---------
>>> > >>> To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
>>> > >>> For additional commands, e-mail: yarn-dev-help@hadoop.apache.org
>>> > >>>
>>> > >>>
>>> > >>
>>> > >
>>> >
>>>
>>
>>
>