You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-dev@hadoop.apache.org by Sangjin Lee <sj...@apache.org> on 2015/10/28 01:00:17 UTC
[YARN-2928] first drop on trunk
In the past several months, the timeline service v.2 team has made a
tremendous progress. We have a working storage implementation based on
HBase, timeline collectors, timeline readers with some of the more
important queries and filters, integration with RM and NM, distributed
shell and mapreduce, and some basic UI to boot. We're getting real close to
a complete end-to-end flow (no pun intended). Kudos to the team (cc'ed
here)!
I think it is time to discuss defining a merge to trunk of an alpha-quality
release as our first milestone so that a wider audience has a chance to try
it out. This doesn't replace the timeline service (ATS) v.1 yet, but it
would be a great chance to get feedback.
I think the theme is essentially a basic but complete end-to-end flow that
includes the write path and the read path and some UI. These are the key
major things we may want to complete before we consider merging the first
milestone:
- application aggregation (YARN-3816
<https://issues.apache.org/jira/browse/YARN-3816>)
- flow run compaction work (YARN-4062
<https://issues.apache.org/jira/browse/YARN-4062>)
- finalize the metrics storage (YARN-4053
<https://issues.apache.org/jira/browse/YARN-4053>)
- improve queries and filters (YARN-3863
<https://issues.apache.org/jira/browse/YARN-3863>)
- UI POC based on the new YARN UI framework (YARN-4097
<https://issues.apache.org/jira/browse/YARN-4097>, YARN-4239
<https://issues.apache.org/jira/browse/YARN-4239>?)
In addition to these, we would like to close a few more JIRAs that we're
currently working on. Also, in terms of the app integration, we can debate
whether we stick to the distributed shell for now or spend some more effort
to round out the mapreduce support.
I also think the following major things are probably out of scope for this
first drop:
- time-based (offline) user/queue aggregation based on Phoenix (YARN-3817
<https://issues.apache.org/jira/browse/YARN-3817>)
- fault-tolerant storage (YARN-4061
<https://issues.apache.org/jira/browse/YARN-4061>)
- timeline collector as a separate daemon (YARN-3033
<https://issues.apache.org/jira/browse/YARN-3033>)
- timeline collector containerization
- compatibility with v.1 (YARN-3196
<https://issues.apache.org/jira/browse/YARN-3196>, YARN-3865
<https://issues.apache.org/jira/browse/YARN-3865>)
- support for off-cluster timeline clients (YARN-3981
<https://issues.apache.org/jira/browse/YARN-3981>)
We should discuss whether we agree on the theme of the first milestone
(mentioned above). Given that, then, we should discuss what makes it and
what doesn't (basically the above 2 lists).
We also should discuss the rough time frame to complete this. This email is
to open the discussion. Your thoughts are welcome. Thanks!
Sangjin
Re: [YARN-2928] first drop on trunk
Posted by Sangjin Lee <sj...@apache.org>.
Let's add the label and discuss the technical aspects on the JIRA. I just
added comments there too.
On Mon, Nov 23, 2015 at 9:25 PM, Varun Saxena <vs...@gmail.com>
wrote:
> I think this JIRA should be complete by the time we merge it into trunk.
>
> It would be nice to have the CLI work with ATSv2 if application is not
> found in RM.
>
> Regards,
> Varun Saxena.
>
> On Tue, 24 Nov 2015 at 10:48, Naganarasimha G R (Naga) <
> garlanaganarasimha@huawei.com> wrote:
>
>> Hi Sangjin,
>>
>> Seems like the list is fine, can we have YARN-4343 too in the list ?
>>
>> + Naga
>> ________________________________________
>> From: Varun Saxena [vsaxena.varun@gmail.com]
>> Sent: Saturday, November 14, 2015 01:46
>> To: Sangjin Lee
>> Cc: Joep Rottinghuis; Karthik Kambatla; yarn-dev@hadoop.apache.org; Li
>> Lu; Junping Du; zjshen@apache.org; Vinod Kumar Vavilapalli;
>> Naganarasimha Garla; Robert Kanter; Vrushali Channapattan
>> Subject: Re: [YARN-2928] first drop on trunk
>>
>> The list
>>
>> https://issues.apache.org/jira/issues/?jql=labels%20%3D%20yarn-2928-1st-milestone
>> looks good to me.
>>
>> Regards,
>> Varun Saxena.
>>
>>
>> On Sat, Nov 14, 2015 at 12:09 AM, Sangjin Lee <sj...@apache.org> wrote:
>>
>> > I took a stab at labeling JIRAs that I think we need for the first
>> > milestone drop, and came up with 17 JIRAs:
>> >
>> https://issues.apache.org/jira/browse/YARN-4239?jql=labels%20%3D%20yarn-2928-1st-milestone
>> >
>> > Please take a look at them, and comment on them.
>> >
>> > Thanks,
>> > Sangjin
>> >
>> > On Wed, Nov 4, 2015 at 5:10 PM, Joep Rottinghuis <jo...@twitter.com>
>> wrote:
>> >
>> >> Agreed with the theme and lists for the merge back of the first
>> >> alpha-quality release of ATS v2 to trunk. Thanks for putting forth the
>> >> suggestion with the two lists Sangjin.
>> >>
>> >> As we discussed in the weekly sync with the ATS v2 team, right after
>> >> we've merged back, the next thing to decide is what the story will be
>> to
>> >> run both ATS v1 and ATS v2 in parallel so that we can start discussing
>> when
>> >> we can do a merge to branch-2.
>> >> We'll want people (including ourselves) to be able to run ATS v2 on a
>> >> trial basis w/o impacting existing use-cases on a branch-2 based
>> release.
>> >> The merge-back to branch-2 will be a separate topic of discussion
>> though.
>> >>
>> >> Thanks,
>> >>
>> >> Joep
>> >>
>> >>
>> >> On Wed, Nov 4, 2015 at 12:18 PM, Karthik Kambatla <ka...@cloudera.com>
>> >> wrote:
>> >>
>> >>> +1
>> >>>
>> >>> In favor of merging a version working end-to-end at alpha quality to
>> >>> trunk.
>> >>>
>> >>> On Mon, Nov 2, 2015 at 10:51 AM, Sangjin Lee <sj...@apache.org>
>> wrote:
>> >>>
>> >>>> Just a reminder that I'd like input from folks that are working on
>> this
>> >>>> feature (as well as anyone who's interested or course!).
>> >>>>
>> >>>> I propose creating a label to start labeling things we'd like to see
>> >>>> done before the first milestone. How about "yarn-2928-1st-milestone"?
>> >>>>
>> >>>> Thanks,
>> >>>> Sangjin
>> >>>>
>> >>>> On Wed, Oct 28, 2015 at 10:30 AM, Varun Saxena <
>> vsaxena.varun@gmail.com
>> >>>> > wrote:
>> >>>>
>> >>>>> Hi Sangjin,
>> >>>>>
>> >>>>> Thanks for initiating the discussion.
>> >>>>> I think YARN-3862 needs to go in as well. YARN-3862 and YARN-3863
>> are
>> >>>>> somewhat related.
>> >>>>>
>> >>>>> Regards,
>> >>>>> Varun Saxena.
>> >>>>>
>> >>>>> On Wed, Oct 28, 2015 at 5:30 AM, Sangjin Lee <sj...@apache.org>
>> wrote:
>> >>>>>
>> >>>>>> In the past several months, the timeline service v.2 team has made
>> a
>> >>>>>> tremendous progress. We have a working storage implementation
>> based on
>> >>>>>> HBase, timeline collectors, timeline readers with some of the more
>> >>>>>> important queries and filters, integration with RM and NM,
>> distributed
>> >>>>>> shell and mapreduce, and some basic UI to boot. We're getting real
>> close to
>> >>>>>> a complete end-to-end flow (no pun intended). Kudos to the team
>> (cc'ed
>> >>>>>> here)!
>> >>>>>>
>> >>>>>> I think it is time to discuss defining a merge to trunk of an
>> >>>>>> alpha-quality release as our first milestone so that a wider
>> audience has a
>> >>>>>> chance to try it out. This doesn't replace the timeline service
>> (ATS) v.1
>> >>>>>> yet, but it would be a great chance to get feedback.
>> >>>>>>
>> >>>>>> I think the theme is essentially a basic but complete end-to-end
>> flow
>> >>>>>> that includes the write path and the read path and some UI. These
>> are the
>> >>>>>> key major things we may want to complete before we consider
>> merging the
>> >>>>>> first milestone:
>> >>>>>> - application aggregation (YARN-3816
>> >>>>>> <https://issues.apache.org/jira/browse/YARN-3816>)
>> >>>>>> - flow run compaction work (YARN-4062
>> >>>>>> <https://issues.apache.org/jira/browse/YARN-4062>)
>> >>>>>> - finalize the metrics storage (YARN-4053
>> >>>>>> <https://issues.apache.org/jira/browse/YARN-4053>)
>> >>>>>> - improve queries and filters (YARN-3863
>> >>>>>> <https://issues.apache.org/jira/browse/YARN-3863>)
>> >>>>>> - UI POC based on the new YARN UI framework (YARN-4097
>> >>>>>> <https://issues.apache.org/jira/browse/YARN-4097>, YARN-4239
>> >>>>>> <https://issues.apache.org/jira/browse/YARN-4239>?)
>> >>>>>>
>> >>>>>> In addition to these, we would like to close a few more JIRAs that
>> >>>>>> we're currently working on. Also, in terms of the app integration,
>> we can
>> >>>>>> debate whether we stick to the distributed shell for now or spend
>> some more
>> >>>>>> effort to round out the mapreduce support.
>> >>>>>>
>> >>>>>> I also think the following major things are probably out of scope
>> for
>> >>>>>> this first drop:
>> >>>>>> - time-based (offline) user/queue aggregation based on Phoenix (
>> >>>>>> YARN-3817 <https://issues.apache.org/jira/browse/YARN-3817>)
>> >>>>>> - fault-tolerant storage (YARN-4061
>> >>>>>> <https://issues.apache.org/jira/browse/YARN-4061>)
>> >>>>>> - timeline collector as a separate daemon (YARN-3033
>> >>>>>> <https://issues.apache.org/jira/browse/YARN-3033>)
>> >>>>>> - timeline collector containerization
>> >>>>>> - compatibility with v.1 (YARN-3196
>> >>>>>> <https://issues.apache.org/jira/browse/YARN-3196>, YARN-3865
>> >>>>>> <https://issues.apache.org/jira/browse/YARN-3865>)
>> >>>>>> - support for off-cluster timeline clients (YARN-3981
>> >>>>>> <https://issues.apache.org/jira/browse/YARN-3981>)
>> >>>>>>
>> >>>>>> We should discuss whether we agree on the theme of the first
>> >>>>>> milestone (mentioned above). Given that, then, we should discuss
>> what makes
>> >>>>>> it and what doesn't (basically the above 2 lists).
>> >>>>>>
>> >>>>>> We also should discuss the rough time frame to complete this. This
>> >>>>>> email is to open the discussion. Your thoughts are welcome. Thanks!
>> >>>>>>
>> >>>>>> Sangjin
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >
>>
>
Re: [YARN-2928] first drop on trunk
Posted by Varun Saxena <vs...@gmail.com>.
I think this JIRA should be complete by the time we merge it into trunk.
It would be nice to have the CLI work with ATSv2 if application is not
found in RM.
Regards,
Varun Saxena.
On Tue, 24 Nov 2015 at 10:48, Naganarasimha G R (Naga) <
garlanaganarasimha@huawei.com> wrote:
> Hi Sangjin,
>
> Seems like the list is fine, can we have YARN-4343 too in the list ?
>
> + Naga
> ________________________________________
> From: Varun Saxena [vsaxena.varun@gmail.com]
> Sent: Saturday, November 14, 2015 01:46
> To: Sangjin Lee
> Cc: Joep Rottinghuis; Karthik Kambatla; yarn-dev@hadoop.apache.org; Li
> Lu; Junping Du; zjshen@apache.org; Vinod Kumar Vavilapalli; Naganarasimha
> Garla; Robert Kanter; Vrushali Channapattan
> Subject: Re: [YARN-2928] first drop on trunk
>
> The list
>
> https://issues.apache.org/jira/issues/?jql=labels%20%3D%20yarn-2928-1st-milestone
> looks good to me.
>
> Regards,
> Varun Saxena.
>
>
> On Sat, Nov 14, 2015 at 12:09 AM, Sangjin Lee <sj...@apache.org> wrote:
>
> > I took a stab at labeling JIRAs that I think we need for the first
> > milestone drop, and came up with 17 JIRAs:
> >
> https://issues.apache.org/jira/browse/YARN-4239?jql=labels%20%3D%20yarn-2928-1st-milestone
> >
> > Please take a look at them, and comment on them.
> >
> > Thanks,
> > Sangjin
> >
> > On Wed, Nov 4, 2015 at 5:10 PM, Joep Rottinghuis <jo...@twitter.com>
> wrote:
> >
> >> Agreed with the theme and lists for the merge back of the first
> >> alpha-quality release of ATS v2 to trunk. Thanks for putting forth the
> >> suggestion with the two lists Sangjin.
> >>
> >> As we discussed in the weekly sync with the ATS v2 team, right after
> >> we've merged back, the next thing to decide is what the story will be to
> >> run both ATS v1 and ATS v2 in parallel so that we can start discussing
> when
> >> we can do a merge to branch-2.
> >> We'll want people (including ourselves) to be able to run ATS v2 on a
> >> trial basis w/o impacting existing use-cases on a branch-2 based
> release.
> >> The merge-back to branch-2 will be a separate topic of discussion
> though.
> >>
> >> Thanks,
> >>
> >> Joep
> >>
> >>
> >> On Wed, Nov 4, 2015 at 12:18 PM, Karthik Kambatla <ka...@cloudera.com>
> >> wrote:
> >>
> >>> +1
> >>>
> >>> In favor of merging a version working end-to-end at alpha quality to
> >>> trunk.
> >>>
> >>> On Mon, Nov 2, 2015 at 10:51 AM, Sangjin Lee <sj...@apache.org> wrote:
> >>>
> >>>> Just a reminder that I'd like input from folks that are working on
> this
> >>>> feature (as well as anyone who's interested or course!).
> >>>>
> >>>> I propose creating a label to start labeling things we'd like to see
> >>>> done before the first milestone. How about "yarn-2928-1st-milestone"?
> >>>>
> >>>> Thanks,
> >>>> Sangjin
> >>>>
> >>>> On Wed, Oct 28, 2015 at 10:30 AM, Varun Saxena <
> vsaxena.varun@gmail.com
> >>>> > wrote:
> >>>>
> >>>>> Hi Sangjin,
> >>>>>
> >>>>> Thanks for initiating the discussion.
> >>>>> I think YARN-3862 needs to go in as well. YARN-3862 and YARN-3863 are
> >>>>> somewhat related.
> >>>>>
> >>>>> Regards,
> >>>>> Varun Saxena.
> >>>>>
> >>>>> On Wed, Oct 28, 2015 at 5:30 AM, Sangjin Lee <sj...@apache.org>
> wrote:
> >>>>>
> >>>>>> In the past several months, the timeline service v.2 team has made a
> >>>>>> tremendous progress. We have a working storage implementation based
> on
> >>>>>> HBase, timeline collectors, timeline readers with some of the more
> >>>>>> important queries and filters, integration with RM and NM,
> distributed
> >>>>>> shell and mapreduce, and some basic UI to boot. We're getting real
> close to
> >>>>>> a complete end-to-end flow (no pun intended). Kudos to the team
> (cc'ed
> >>>>>> here)!
> >>>>>>
> >>>>>> I think it is time to discuss defining a merge to trunk of an
> >>>>>> alpha-quality release as our first milestone so that a wider
> audience has a
> >>>>>> chance to try it out. This doesn't replace the timeline service
> (ATS) v.1
> >>>>>> yet, but it would be a great chance to get feedback.
> >>>>>>
> >>>>>> I think the theme is essentially a basic but complete end-to-end
> flow
> >>>>>> that includes the write path and the read path and some UI. These
> are the
> >>>>>> key major things we may want to complete before we consider merging
> the
> >>>>>> first milestone:
> >>>>>> - application aggregation (YARN-3816
> >>>>>> <https://issues.apache.org/jira/browse/YARN-3816>)
> >>>>>> - flow run compaction work (YARN-4062
> >>>>>> <https://issues.apache.org/jira/browse/YARN-4062>)
> >>>>>> - finalize the metrics storage (YARN-4053
> >>>>>> <https://issues.apache.org/jira/browse/YARN-4053>)
> >>>>>> - improve queries and filters (YARN-3863
> >>>>>> <https://issues.apache.org/jira/browse/YARN-3863>)
> >>>>>> - UI POC based on the new YARN UI framework (YARN-4097
> >>>>>> <https://issues.apache.org/jira/browse/YARN-4097>, YARN-4239
> >>>>>> <https://issues.apache.org/jira/browse/YARN-4239>?)
> >>>>>>
> >>>>>> In addition to these, we would like to close a few more JIRAs that
> >>>>>> we're currently working on. Also, in terms of the app integration,
> we can
> >>>>>> debate whether we stick to the distributed shell for now or spend
> some more
> >>>>>> effort to round out the mapreduce support.
> >>>>>>
> >>>>>> I also think the following major things are probably out of scope
> for
> >>>>>> this first drop:
> >>>>>> - time-based (offline) user/queue aggregation based on Phoenix (
> >>>>>> YARN-3817 <https://issues.apache.org/jira/browse/YARN-3817>)
> >>>>>> - fault-tolerant storage (YARN-4061
> >>>>>> <https://issues.apache.org/jira/browse/YARN-4061>)
> >>>>>> - timeline collector as a separate daemon (YARN-3033
> >>>>>> <https://issues.apache.org/jira/browse/YARN-3033>)
> >>>>>> - timeline collector containerization
> >>>>>> - compatibility with v.1 (YARN-3196
> >>>>>> <https://issues.apache.org/jira/browse/YARN-3196>, YARN-3865
> >>>>>> <https://issues.apache.org/jira/browse/YARN-3865>)
> >>>>>> - support for off-cluster timeline clients (YARN-3981
> >>>>>> <https://issues.apache.org/jira/browse/YARN-3981>)
> >>>>>>
> >>>>>> We should discuss whether we agree on the theme of the first
> >>>>>> milestone (mentioned above). Given that, then, we should discuss
> what makes
> >>>>>> it and what doesn't (basically the above 2 lists).
> >>>>>>
> >>>>>> We also should discuss the rough time frame to complete this. This
> >>>>>> email is to open the discussion. Your thoughts are welcome. Thanks!
> >>>>>>
> >>>>>> Sangjin
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
> >
>
RE: [YARN-2928] first drop on trunk
Posted by "Naganarasimha G R (Naga)" <ga...@huawei.com>.
Hi Sangjin,
Seems like the list is fine, can we have YARN-4343 too in the list ?
+ Naga
________________________________________
From: Varun Saxena [vsaxena.varun@gmail.com]
Sent: Saturday, November 14, 2015 01:46
To: Sangjin Lee
Cc: Joep Rottinghuis; Karthik Kambatla; yarn-dev@hadoop.apache.org; Li Lu; Junping Du; zjshen@apache.org; Vinod Kumar Vavilapalli; Naganarasimha Garla; Robert Kanter; Vrushali Channapattan
Subject: Re: [YARN-2928] first drop on trunk
The list
https://issues.apache.org/jira/issues/?jql=labels%20%3D%20yarn-2928-1st-milestone
looks good to me.
Regards,
Varun Saxena.
On Sat, Nov 14, 2015 at 12:09 AM, Sangjin Lee <sj...@apache.org> wrote:
> I took a stab at labeling JIRAs that I think we need for the first
> milestone drop, and came up with 17 JIRAs:
> https://issues.apache.org/jira/browse/YARN-4239?jql=labels%20%3D%20yarn-2928-1st-milestone
>
> Please take a look at them, and comment on them.
>
> Thanks,
> Sangjin
>
> On Wed, Nov 4, 2015 at 5:10 PM, Joep Rottinghuis <jo...@twitter.com> wrote:
>
>> Agreed with the theme and lists for the merge back of the first
>> alpha-quality release of ATS v2 to trunk. Thanks for putting forth the
>> suggestion with the two lists Sangjin.
>>
>> As we discussed in the weekly sync with the ATS v2 team, right after
>> we've merged back, the next thing to decide is what the story will be to
>> run both ATS v1 and ATS v2 in parallel so that we can start discussing when
>> we can do a merge to branch-2.
>> We'll want people (including ourselves) to be able to run ATS v2 on a
>> trial basis w/o impacting existing use-cases on a branch-2 based release.
>> The merge-back to branch-2 will be a separate topic of discussion though.
>>
>> Thanks,
>>
>> Joep
>>
>>
>> On Wed, Nov 4, 2015 at 12:18 PM, Karthik Kambatla <ka...@cloudera.com>
>> wrote:
>>
>>> +1
>>>
>>> In favor of merging a version working end-to-end at alpha quality to
>>> trunk.
>>>
>>> On Mon, Nov 2, 2015 at 10:51 AM, Sangjin Lee <sj...@apache.org> wrote:
>>>
>>>> Just a reminder that I'd like input from folks that are working on this
>>>> feature (as well as anyone who's interested or course!).
>>>>
>>>> I propose creating a label to start labeling things we'd like to see
>>>> done before the first milestone. How about "yarn-2928-1st-milestone"?
>>>>
>>>> Thanks,
>>>> Sangjin
>>>>
>>>> On Wed, Oct 28, 2015 at 10:30 AM, Varun Saxena <vsaxena.varun@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Sangjin,
>>>>>
>>>>> Thanks for initiating the discussion.
>>>>> I think YARN-3862 needs to go in as well. YARN-3862 and YARN-3863 are
>>>>> somewhat related.
>>>>>
>>>>> Regards,
>>>>> Varun Saxena.
>>>>>
>>>>> On Wed, Oct 28, 2015 at 5:30 AM, Sangjin Lee <sj...@apache.org> wrote:
>>>>>
>>>>>> In the past several months, the timeline service v.2 team has made a
>>>>>> tremendous progress. We have a working storage implementation based on
>>>>>> HBase, timeline collectors, timeline readers with some of the more
>>>>>> important queries and filters, integration with RM and NM, distributed
>>>>>> shell and mapreduce, and some basic UI to boot. We're getting real close to
>>>>>> a complete end-to-end flow (no pun intended). Kudos to the team (cc'ed
>>>>>> here)!
>>>>>>
>>>>>> I think it is time to discuss defining a merge to trunk of an
>>>>>> alpha-quality release as our first milestone so that a wider audience has a
>>>>>> chance to try it out. This doesn't replace the timeline service (ATS) v.1
>>>>>> yet, but it would be a great chance to get feedback.
>>>>>>
>>>>>> I think the theme is essentially a basic but complete end-to-end flow
>>>>>> that includes the write path and the read path and some UI. These are the
>>>>>> key major things we may want to complete before we consider merging the
>>>>>> first milestone:
>>>>>> - application aggregation (YARN-3816
>>>>>> <https://issues.apache.org/jira/browse/YARN-3816>)
>>>>>> - flow run compaction work (YARN-4062
>>>>>> <https://issues.apache.org/jira/browse/YARN-4062>)
>>>>>> - finalize the metrics storage (YARN-4053
>>>>>> <https://issues.apache.org/jira/browse/YARN-4053>)
>>>>>> - improve queries and filters (YARN-3863
>>>>>> <https://issues.apache.org/jira/browse/YARN-3863>)
>>>>>> - UI POC based on the new YARN UI framework (YARN-4097
>>>>>> <https://issues.apache.org/jira/browse/YARN-4097>, YARN-4239
>>>>>> <https://issues.apache.org/jira/browse/YARN-4239>?)
>>>>>>
>>>>>> In addition to these, we would like to close a few more JIRAs that
>>>>>> we're currently working on. Also, in terms of the app integration, we can
>>>>>> debate whether we stick to the distributed shell for now or spend some more
>>>>>> effort to round out the mapreduce support.
>>>>>>
>>>>>> I also think the following major things are probably out of scope for
>>>>>> this first drop:
>>>>>> - time-based (offline) user/queue aggregation based on Phoenix (
>>>>>> YARN-3817 <https://issues.apache.org/jira/browse/YARN-3817>)
>>>>>> - fault-tolerant storage (YARN-4061
>>>>>> <https://issues.apache.org/jira/browse/YARN-4061>)
>>>>>> - timeline collector as a separate daemon (YARN-3033
>>>>>> <https://issues.apache.org/jira/browse/YARN-3033>)
>>>>>> - timeline collector containerization
>>>>>> - compatibility with v.1 (YARN-3196
>>>>>> <https://issues.apache.org/jira/browse/YARN-3196>, YARN-3865
>>>>>> <https://issues.apache.org/jira/browse/YARN-3865>)
>>>>>> - support for off-cluster timeline clients (YARN-3981
>>>>>> <https://issues.apache.org/jira/browse/YARN-3981>)
>>>>>>
>>>>>> We should discuss whether we agree on the theme of the first
>>>>>> milestone (mentioned above). Given that, then, we should discuss what makes
>>>>>> it and what doesn't (basically the above 2 lists).
>>>>>>
>>>>>> We also should discuss the rough time frame to complete this. This
>>>>>> email is to open the discussion. Your thoughts are welcome. Thanks!
>>>>>>
>>>>>> Sangjin
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [YARN-2928] first drop on trunk
Posted by Varun Saxena <vs...@gmail.com>.
The list
https://issues.apache.org/jira/issues/?jql=labels%20%3D%20yarn-2928-1st-milestone
looks good to me.
Regards,
Varun Saxena.
On Sat, Nov 14, 2015 at 12:09 AM, Sangjin Lee <sj...@apache.org> wrote:
> I took a stab at labeling JIRAs that I think we need for the first
> milestone drop, and came up with 17 JIRAs:
> https://issues.apache.org/jira/browse/YARN-4239?jql=labels%20%3D%20yarn-2928-1st-milestone
>
> Please take a look at them, and comment on them.
>
> Thanks,
> Sangjin
>
> On Wed, Nov 4, 2015 at 5:10 PM, Joep Rottinghuis <jo...@twitter.com> wrote:
>
>> Agreed with the theme and lists for the merge back of the first
>> alpha-quality release of ATS v2 to trunk. Thanks for putting forth the
>> suggestion with the two lists Sangjin.
>>
>> As we discussed in the weekly sync with the ATS v2 team, right after
>> we've merged back, the next thing to decide is what the story will be to
>> run both ATS v1 and ATS v2 in parallel so that we can start discussing when
>> we can do a merge to branch-2.
>> We'll want people (including ourselves) to be able to run ATS v2 on a
>> trial basis w/o impacting existing use-cases on a branch-2 based release.
>> The merge-back to branch-2 will be a separate topic of discussion though.
>>
>> Thanks,
>>
>> Joep
>>
>>
>> On Wed, Nov 4, 2015 at 12:18 PM, Karthik Kambatla <ka...@cloudera.com>
>> wrote:
>>
>>> +1
>>>
>>> In favor of merging a version working end-to-end at alpha quality to
>>> trunk.
>>>
>>> On Mon, Nov 2, 2015 at 10:51 AM, Sangjin Lee <sj...@apache.org> wrote:
>>>
>>>> Just a reminder that I'd like input from folks that are working on this
>>>> feature (as well as anyone who's interested or course!).
>>>>
>>>> I propose creating a label to start labeling things we'd like to see
>>>> done before the first milestone. How about "yarn-2928-1st-milestone"?
>>>>
>>>> Thanks,
>>>> Sangjin
>>>>
>>>> On Wed, Oct 28, 2015 at 10:30 AM, Varun Saxena <vsaxena.varun@gmail.com
>>>> > wrote:
>>>>
>>>>> Hi Sangjin,
>>>>>
>>>>> Thanks for initiating the discussion.
>>>>> I think YARN-3862 needs to go in as well. YARN-3862 and YARN-3863 are
>>>>> somewhat related.
>>>>>
>>>>> Regards,
>>>>> Varun Saxena.
>>>>>
>>>>> On Wed, Oct 28, 2015 at 5:30 AM, Sangjin Lee <sj...@apache.org> wrote:
>>>>>
>>>>>> In the past several months, the timeline service v.2 team has made a
>>>>>> tremendous progress. We have a working storage implementation based on
>>>>>> HBase, timeline collectors, timeline readers with some of the more
>>>>>> important queries and filters, integration with RM and NM, distributed
>>>>>> shell and mapreduce, and some basic UI to boot. We're getting real close to
>>>>>> a complete end-to-end flow (no pun intended). Kudos to the team (cc'ed
>>>>>> here)!
>>>>>>
>>>>>> I think it is time to discuss defining a merge to trunk of an
>>>>>> alpha-quality release as our first milestone so that a wider audience has a
>>>>>> chance to try it out. This doesn't replace the timeline service (ATS) v.1
>>>>>> yet, but it would be a great chance to get feedback.
>>>>>>
>>>>>> I think the theme is essentially a basic but complete end-to-end flow
>>>>>> that includes the write path and the read path and some UI. These are the
>>>>>> key major things we may want to complete before we consider merging the
>>>>>> first milestone:
>>>>>> - application aggregation (YARN-3816
>>>>>> <https://issues.apache.org/jira/browse/YARN-3816>)
>>>>>> - flow run compaction work (YARN-4062
>>>>>> <https://issues.apache.org/jira/browse/YARN-4062>)
>>>>>> - finalize the metrics storage (YARN-4053
>>>>>> <https://issues.apache.org/jira/browse/YARN-4053>)
>>>>>> - improve queries and filters (YARN-3863
>>>>>> <https://issues.apache.org/jira/browse/YARN-3863>)
>>>>>> - UI POC based on the new YARN UI framework (YARN-4097
>>>>>> <https://issues.apache.org/jira/browse/YARN-4097>, YARN-4239
>>>>>> <https://issues.apache.org/jira/browse/YARN-4239>?)
>>>>>>
>>>>>> In addition to these, we would like to close a few more JIRAs that
>>>>>> we're currently working on. Also, in terms of the app integration, we can
>>>>>> debate whether we stick to the distributed shell for now or spend some more
>>>>>> effort to round out the mapreduce support.
>>>>>>
>>>>>> I also think the following major things are probably out of scope for
>>>>>> this first drop:
>>>>>> - time-based (offline) user/queue aggregation based on Phoenix (
>>>>>> YARN-3817 <https://issues.apache.org/jira/browse/YARN-3817>)
>>>>>> - fault-tolerant storage (YARN-4061
>>>>>> <https://issues.apache.org/jira/browse/YARN-4061>)
>>>>>> - timeline collector as a separate daemon (YARN-3033
>>>>>> <https://issues.apache.org/jira/browse/YARN-3033>)
>>>>>> - timeline collector containerization
>>>>>> - compatibility with v.1 (YARN-3196
>>>>>> <https://issues.apache.org/jira/browse/YARN-3196>, YARN-3865
>>>>>> <https://issues.apache.org/jira/browse/YARN-3865>)
>>>>>> - support for off-cluster timeline clients (YARN-3981
>>>>>> <https://issues.apache.org/jira/browse/YARN-3981>)
>>>>>>
>>>>>> We should discuss whether we agree on the theme of the first
>>>>>> milestone (mentioned above). Given that, then, we should discuss what makes
>>>>>> it and what doesn't (basically the above 2 lists).
>>>>>>
>>>>>> We also should discuss the rough time frame to complete this. This
>>>>>> email is to open the discussion. Your thoughts are welcome. Thanks!
>>>>>>
>>>>>> Sangjin
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: [YARN-2928] first drop on trunk
Posted by Sangjin Lee <sj...@apache.org>.
I took a stab at labeling JIRAs that I think we need for the first
milestone drop, and came up with 17 JIRAs:
https://issues.apache.org/jira/browse/YARN-4239?jql=labels%20%3D%20yarn-2928-1st-milestone
Please take a look at them, and comment on them.
Thanks,
Sangjin
On Wed, Nov 4, 2015 at 5:10 PM, Joep Rottinghuis <jo...@twitter.com> wrote:
> Agreed with the theme and lists for the merge back of the first
> alpha-quality release of ATS v2 to trunk. Thanks for putting forth the
> suggestion with the two lists Sangjin.
>
> As we discussed in the weekly sync with the ATS v2 team, right after
> we've merged back, the next thing to decide is what the story will be to
> run both ATS v1 and ATS v2 in parallel so that we can start discussing when
> we can do a merge to branch-2.
> We'll want people (including ourselves) to be able to run ATS v2 on a
> trial basis w/o impacting existing use-cases on a branch-2 based release.
> The merge-back to branch-2 will be a separate topic of discussion though.
>
> Thanks,
>
> Joep
>
>
> On Wed, Nov 4, 2015 at 12:18 PM, Karthik Kambatla <ka...@cloudera.com>
> wrote:
>
>> +1
>>
>> In favor of merging a version working end-to-end at alpha quality to
>> trunk.
>>
>> On Mon, Nov 2, 2015 at 10:51 AM, Sangjin Lee <sj...@apache.org> wrote:
>>
>>> Just a reminder that I'd like input from folks that are working on this
>>> feature (as well as anyone who's interested or course!).
>>>
>>> I propose creating a label to start labeling things we'd like to see
>>> done before the first milestone. How about "yarn-2928-1st-milestone"?
>>>
>>> Thanks,
>>> Sangjin
>>>
>>> On Wed, Oct 28, 2015 at 10:30 AM, Varun Saxena <vs...@gmail.com>
>>> wrote:
>>>
>>>> Hi Sangjin,
>>>>
>>>> Thanks for initiating the discussion.
>>>> I think YARN-3862 needs to go in as well. YARN-3862 and YARN-3863 are
>>>> somewhat related.
>>>>
>>>> Regards,
>>>> Varun Saxena.
>>>>
>>>> On Wed, Oct 28, 2015 at 5:30 AM, Sangjin Lee <sj...@apache.org> wrote:
>>>>
>>>>> In the past several months, the timeline service v.2 team has made a
>>>>> tremendous progress. We have a working storage implementation based on
>>>>> HBase, timeline collectors, timeline readers with some of the more
>>>>> important queries and filters, integration with RM and NM, distributed
>>>>> shell and mapreduce, and some basic UI to boot. We're getting real close to
>>>>> a complete end-to-end flow (no pun intended). Kudos to the team (cc'ed
>>>>> here)!
>>>>>
>>>>> I think it is time to discuss defining a merge to trunk of an
>>>>> alpha-quality release as our first milestone so that a wider audience has a
>>>>> chance to try it out. This doesn't replace the timeline service (ATS) v.1
>>>>> yet, but it would be a great chance to get feedback.
>>>>>
>>>>> I think the theme is essentially a basic but complete end-to-end flow
>>>>> that includes the write path and the read path and some UI. These are the
>>>>> key major things we may want to complete before we consider merging the
>>>>> first milestone:
>>>>> - application aggregation (YARN-3816
>>>>> <https://issues.apache.org/jira/browse/YARN-3816>)
>>>>> - flow run compaction work (YARN-4062
>>>>> <https://issues.apache.org/jira/browse/YARN-4062>)
>>>>> - finalize the metrics storage (YARN-4053
>>>>> <https://issues.apache.org/jira/browse/YARN-4053>)
>>>>> - improve queries and filters (YARN-3863
>>>>> <https://issues.apache.org/jira/browse/YARN-3863>)
>>>>> - UI POC based on the new YARN UI framework (YARN-4097
>>>>> <https://issues.apache.org/jira/browse/YARN-4097>, YARN-4239
>>>>> <https://issues.apache.org/jira/browse/YARN-4239>?)
>>>>>
>>>>> In addition to these, we would like to close a few more JIRAs that
>>>>> we're currently working on. Also, in terms of the app integration, we can
>>>>> debate whether we stick to the distributed shell for now or spend some more
>>>>> effort to round out the mapreduce support.
>>>>>
>>>>> I also think the following major things are probably out of scope for
>>>>> this first drop:
>>>>> - time-based (offline) user/queue aggregation based on Phoenix (
>>>>> YARN-3817 <https://issues.apache.org/jira/browse/YARN-3817>)
>>>>> - fault-tolerant storage (YARN-4061
>>>>> <https://issues.apache.org/jira/browse/YARN-4061>)
>>>>> - timeline collector as a separate daemon (YARN-3033
>>>>> <https://issues.apache.org/jira/browse/YARN-3033>)
>>>>> - timeline collector containerization
>>>>> - compatibility with v.1 (YARN-3196
>>>>> <https://issues.apache.org/jira/browse/YARN-3196>, YARN-3865
>>>>> <https://issues.apache.org/jira/browse/YARN-3865>)
>>>>> - support for off-cluster timeline clients (YARN-3981
>>>>> <https://issues.apache.org/jira/browse/YARN-3981>)
>>>>>
>>>>> We should discuss whether we agree on the theme of the first milestone
>>>>> (mentioned above). Given that, then, we should discuss what makes it and
>>>>> what doesn't (basically the above 2 lists).
>>>>>
>>>>> We also should discuss the rough time frame to complete this. This
>>>>> email is to open the discussion. Your thoughts are welcome. Thanks!
>>>>>
>>>>> Sangjin
>>>>>
>>>>
>>>>
>>>
>>
>
Re: [YARN-2928] first drop on trunk
Posted by Karthik Kambatla <ka...@cloudera.com>.
+1
In favor of merging a version working end-to-end at alpha quality to trunk.
On Mon, Nov 2, 2015 at 10:51 AM, Sangjin Lee <sj...@apache.org> wrote:
> Just a reminder that I'd like input from folks that are working on this
> feature (as well as anyone who's interested or course!).
>
> I propose creating a label to start labeling things we'd like to see done
> before the first milestone. How about "yarn-2928-1st-milestone"?
>
> Thanks,
> Sangjin
>
> On Wed, Oct 28, 2015 at 10:30 AM, Varun Saxena <vs...@gmail.com>
> wrote:
>
>> Hi Sangjin,
>>
>> Thanks for initiating the discussion.
>> I think YARN-3862 needs to go in as well. YARN-3862 and YARN-3863 are
>> somewhat related.
>>
>> Regards,
>> Varun Saxena.
>>
>> On Wed, Oct 28, 2015 at 5:30 AM, Sangjin Lee <sj...@apache.org> wrote:
>>
>>> In the past several months, the timeline service v.2 team has made a
>>> tremendous progress. We have a working storage implementation based on
>>> HBase, timeline collectors, timeline readers with some of the more
>>> important queries and filters, integration with RM and NM, distributed
>>> shell and mapreduce, and some basic UI to boot. We're getting real close to
>>> a complete end-to-end flow (no pun intended). Kudos to the team (cc'ed
>>> here)!
>>>
>>> I think it is time to discuss defining a merge to trunk of an
>>> alpha-quality release as our first milestone so that a wider audience has a
>>> chance to try it out. This doesn't replace the timeline service (ATS) v.1
>>> yet, but it would be a great chance to get feedback.
>>>
>>> I think the theme is essentially a basic but complete end-to-end flow
>>> that includes the write path and the read path and some UI. These are the
>>> key major things we may want to complete before we consider merging the
>>> first milestone:
>>> - application aggregation (YARN-3816
>>> <https://issues.apache.org/jira/browse/YARN-3816>)
>>> - flow run compaction work (YARN-4062
>>> <https://issues.apache.org/jira/browse/YARN-4062>)
>>> - finalize the metrics storage (YARN-4053
>>> <https://issues.apache.org/jira/browse/YARN-4053>)
>>> - improve queries and filters (YARN-3863
>>> <https://issues.apache.org/jira/browse/YARN-3863>)
>>> - UI POC based on the new YARN UI framework (YARN-4097
>>> <https://issues.apache.org/jira/browse/YARN-4097>, YARN-4239
>>> <https://issues.apache.org/jira/browse/YARN-4239>?)
>>>
>>> In addition to these, we would like to close a few more JIRAs that we're
>>> currently working on. Also, in terms of the app integration, we can debate
>>> whether we stick to the distributed shell for now or spend some more effort
>>> to round out the mapreduce support.
>>>
>>> I also think the following major things are probably out of scope for
>>> this first drop:
>>> - time-based (offline) user/queue aggregation based on Phoenix (
>>> YARN-3817 <https://issues.apache.org/jira/browse/YARN-3817>)
>>> - fault-tolerant storage (YARN-4061
>>> <https://issues.apache.org/jira/browse/YARN-4061>)
>>> - timeline collector as a separate daemon (YARN-3033
>>> <https://issues.apache.org/jira/browse/YARN-3033>)
>>> - timeline collector containerization
>>> - compatibility with v.1 (YARN-3196
>>> <https://issues.apache.org/jira/browse/YARN-3196>, YARN-3865
>>> <https://issues.apache.org/jira/browse/YARN-3865>)
>>> - support for off-cluster timeline clients (YARN-3981
>>> <https://issues.apache.org/jira/browse/YARN-3981>)
>>>
>>> We should discuss whether we agree on the theme of the first milestone
>>> (mentioned above). Given that, then, we should discuss what makes it and
>>> what doesn't (basically the above 2 lists).
>>>
>>> We also should discuss the rough time frame to complete this. This email
>>> is to open the discussion. Your thoughts are welcome. Thanks!
>>>
>>> Sangjin
>>>
>>
>>
>
Re: [YARN-2928] first drop on trunk
Posted by Sangjin Lee <sj...@apache.org>.
Just a reminder that I'd like input from folks that are working on this
feature (as well as anyone who's interested or course!).
I propose creating a label to start labeling things we'd like to see done
before the first milestone. How about "yarn-2928-1st-milestone"?
Thanks,
Sangjin
On Wed, Oct 28, 2015 at 10:30 AM, Varun Saxena <vs...@gmail.com>
wrote:
> Hi Sangjin,
>
> Thanks for initiating the discussion.
> I think YARN-3862 needs to go in as well. YARN-3862 and YARN-3863 are
> somewhat related.
>
> Regards,
> Varun Saxena.
>
> On Wed, Oct 28, 2015 at 5:30 AM, Sangjin Lee <sj...@apache.org> wrote:
>
>> In the past several months, the timeline service v.2 team has made a
>> tremendous progress. We have a working storage implementation based on
>> HBase, timeline collectors, timeline readers with some of the more
>> important queries and filters, integration with RM and NM, distributed
>> shell and mapreduce, and some basic UI to boot. We're getting real close to
>> a complete end-to-end flow (no pun intended). Kudos to the team (cc'ed
>> here)!
>>
>> I think it is time to discuss defining a merge to trunk of an
>> alpha-quality release as our first milestone so that a wider audience has a
>> chance to try it out. This doesn't replace the timeline service (ATS) v.1
>> yet, but it would be a great chance to get feedback.
>>
>> I think the theme is essentially a basic but complete end-to-end flow
>> that includes the write path and the read path and some UI. These are the
>> key major things we may want to complete before we consider merging the
>> first milestone:
>> - application aggregation (YARN-3816
>> <https://issues.apache.org/jira/browse/YARN-3816>)
>> - flow run compaction work (YARN-4062
>> <https://issues.apache.org/jira/browse/YARN-4062>)
>> - finalize the metrics storage (YARN-4053
>> <https://issues.apache.org/jira/browse/YARN-4053>)
>> - improve queries and filters (YARN-3863
>> <https://issues.apache.org/jira/browse/YARN-3863>)
>> - UI POC based on the new YARN UI framework (YARN-4097
>> <https://issues.apache.org/jira/browse/YARN-4097>, YARN-4239
>> <https://issues.apache.org/jira/browse/YARN-4239>?)
>>
>> In addition to these, we would like to close a few more JIRAs that we're
>> currently working on. Also, in terms of the app integration, we can debate
>> whether we stick to the distributed shell for now or spend some more effort
>> to round out the mapreduce support.
>>
>> I also think the following major things are probably out of scope for
>> this first drop:
>> - time-based (offline) user/queue aggregation based on Phoenix (YARN-3817
>> <https://issues.apache.org/jira/browse/YARN-3817>)
>> - fault-tolerant storage (YARN-4061
>> <https://issues.apache.org/jira/browse/YARN-4061>)
>> - timeline collector as a separate daemon (YARN-3033
>> <https://issues.apache.org/jira/browse/YARN-3033>)
>> - timeline collector containerization
>> - compatibility with v.1 (YARN-3196
>> <https://issues.apache.org/jira/browse/YARN-3196>, YARN-3865
>> <https://issues.apache.org/jira/browse/YARN-3865>)
>> - support for off-cluster timeline clients (YARN-3981
>> <https://issues.apache.org/jira/browse/YARN-3981>)
>>
>> We should discuss whether we agree on the theme of the first milestone
>> (mentioned above). Given that, then, we should discuss what makes it and
>> what doesn't (basically the above 2 lists).
>>
>> We also should discuss the rough time frame to complete this. This email
>> is to open the discussion. Your thoughts are welcome. Thanks!
>>
>> Sangjin
>>
>
>
Re: [YARN-2928] first drop on trunk
Posted by Varun Saxena <vs...@gmail.com>.
Hi Sangjin,
Thanks for initiating the discussion.
I think YARN-3862 needs to go in as well. YARN-3862 and YARN-3863 are
somewhat related.
Regards,
Varun Saxena.
On Wed, Oct 28, 2015 at 5:30 AM, Sangjin Lee <sj...@apache.org> wrote:
> In the past several months, the timeline service v.2 team has made a
> tremendous progress. We have a working storage implementation based on
> HBase, timeline collectors, timeline readers with some of the more
> important queries and filters, integration with RM and NM, distributed
> shell and mapreduce, and some basic UI to boot. We're getting real close to
> a complete end-to-end flow (no pun intended). Kudos to the team (cc'ed
> here)!
>
> I think it is time to discuss defining a merge to trunk of an
> alpha-quality release as our first milestone so that a wider audience has a
> chance to try it out. This doesn't replace the timeline service (ATS) v.1
> yet, but it would be a great chance to get feedback.
>
> I think the theme is essentially a basic but complete end-to-end flow that
> includes the write path and the read path and some UI. These are the key
> major things we may want to complete before we consider merging the first
> milestone:
> - application aggregation (YARN-3816
> <https://issues.apache.org/jira/browse/YARN-3816>)
> - flow run compaction work (YARN-4062
> <https://issues.apache.org/jira/browse/YARN-4062>)
> - finalize the metrics storage (YARN-4053
> <https://issues.apache.org/jira/browse/YARN-4053>)
> - improve queries and filters (YARN-3863
> <https://issues.apache.org/jira/browse/YARN-3863>)
> - UI POC based on the new YARN UI framework (YARN-4097
> <https://issues.apache.org/jira/browse/YARN-4097>, YARN-4239
> <https://issues.apache.org/jira/browse/YARN-4239>?)
>
> In addition to these, we would like to close a few more JIRAs that we're
> currently working on. Also, in terms of the app integration, we can debate
> whether we stick to the distributed shell for now or spend some more effort
> to round out the mapreduce support.
>
> I also think the following major things are probably out of scope for this
> first drop:
> - time-based (offline) user/queue aggregation based on Phoenix (YARN-3817
> <https://issues.apache.org/jira/browse/YARN-3817>)
> - fault-tolerant storage (YARN-4061
> <https://issues.apache.org/jira/browse/YARN-4061>)
> - timeline collector as a separate daemon (YARN-3033
> <https://issues.apache.org/jira/browse/YARN-3033>)
> - timeline collector containerization
> - compatibility with v.1 (YARN-3196
> <https://issues.apache.org/jira/browse/YARN-3196>, YARN-3865
> <https://issues.apache.org/jira/browse/YARN-3865>)
> - support for off-cluster timeline clients (YARN-3981
> <https://issues.apache.org/jira/browse/YARN-3981>)
>
> We should discuss whether we agree on the theme of the first milestone
> (mentioned above). Given that, then, we should discuss what makes it and
> what doesn't (basically the above 2 lists).
>
> We also should discuss the rough time frame to complete this. This email
> is to open the discussion. Your thoughts are welcome. Thanks!
>
> Sangjin
>