You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by Naganarasimha Garla <na...@apache.org> on 2018/08/29 14:30:19 UTC
[Discuss] Merge discussion for Node attribute support feature YARN-3409
Hi All,
We would like to hear your thoughts on merging “Node Attributes Support in
YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
it in for HADOOP 3.2.
*Major work happened in this branch*
YARN-6858. Attribute Manager to store and provide node attributes in RM
YARN-7871. Support Node attributes reporting from NM to RM( distributed
node attributes)
YARN-7863. Modify placement constraints to support node attributes
YARN-7875. Node Attribute store for storing and recovering attributes
*Detailed Design:*
Please refer [1] for detailed design document.
*Testing Efforts:*
We did detailed tests for the feature in the last few weeks.
This feature will be enabled only when Node Attributes constraints are
specified through SchedulingRequest from AM.
Manager implementation will help to store and recover Node Attributes. This
works with existing placement constraints.
*Regarding to API stability:*
All newly added @Public APIs are @Unstable.
Documentation jira [3] could help to provide detailed configuration
details. This feature works from end-to-end and we tested this in our local
cluster. Branch code is run against trunk and tracked via [4].
We would love to get your thoughts before opening a voting thread.
Special thanks to a team of folks who worked hard and contributed towards
this efforts including design discussion / patch / reviews, etc.: Weiwei
Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
[1] :
https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
[2] : https://issues.apache.org/jira/browse/YARN-3409
[3] : https://issues.apache.org/jira/browse/YARN-7865
[4] : https://issues.apache.org/jira/browse/YARN-8718
Thanks,
+ Naga & Sunil Govindan
RE: [Discuss] Merge discussion for Node attribute support feature
YARN-3409
Posted by Bibinchundatt <bi...@huawei.com>.
+1 for merge. Fetaure would be a good addition to 3.2 release.
--------------------------------------------------
Bibin A Chundatt
M: +91-9742095715<tel:+91-9742095715>
E: bibin.chundatt@huawei.com<ma...@huawei.com>
2012<tel:2012>实验室-印研IT&Cloud BU分部
2012<tel:2012> Laboratories-IT&Cloud BU Branch Dept.
From:Naganarasimha Garla
To:common-dev@hadoop.apache.org,Hdfs-dev,yarn-dev@hadoop.apache.org,mapreduce-dev@hadoop.apache.org,
Date:2018-08-29 20:00:44
Subject:[Discuss] Merge discussion for Node attribute support feature YARN-3409
Hi All,
We would like to hear your thoughts on merging “Node Attributes Support in
YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
it in for HADOOP 3.2.
*Major work happened in this branch*
YARN-6858. Attribute Manager to store and provide node attributes in RM
YARN-7871. Support Node attributes reporting from NM to RM( distributed
node attributes)
YARN-7863. Modify placement constraints to support node attributes
YARN-7875. Node Attribute store for storing and recovering attributes
*Detailed Design:*
Please refer [1] for detailed design document.
*Testing Efforts:*
We did detailed tests for the feature in the last few weeks.
This feature will be enabled only when Node Attributes constraints are
specified through SchedulingRequest from AM.
Manager implementation will help to store and recover Node Attributes. This
works with existing placement constraints.
*Regarding to API stability:*
All newly added @Public APIs are @Unstable.
Documentation jira [3] could help to provide detailed configuration
details. This feature works from end-to-end and we tested this in our local
cluster. Branch code is run against trunk and tracked via [4].
We would love to get your thoughts before opening a voting thread.
Special thanks to a team of folks who worked hard and contributed towards
this efforts including design discussion / patch / reviews, etc.: Weiwei
Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
[1] :
https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
[2] : https://issues.apache.org/jira/browse/YARN-3409
[3] : https://issues.apache.org/jira/browse/YARN-7865
[4] : https://issues.apache.org/jira/browse/YARN-8718
Thanks,
+ Naga & Sunil Govindan
Re: [Discuss] Merge discussion for Node attribute support feature YARN-3409
Posted by Weiwei Yang <ab...@gmail.com>.
Hi Naga
+1 for the merge. We need it for 3.2.0.
This brings a lot of value when it works with placement constraints APIs,
it will be very important for long-running jobs and service type tasks.
Thanks!
--
Weiwei
On Wed, Aug 29, 2018 at 10:30 PM Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> Hi All,
>
> We would like to hear your thoughts on merging “Node Attributes Support in
> YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
> it in for HADOOP 3.2.
>
> *Major work happened in this branch*
>
> YARN-6858. Attribute Manager to store and provide node attributes in RM
> YARN-7871. Support Node attributes reporting from NM to RM( distributed
> node attributes)
> YARN-7863. Modify placement constraints to support node attributes
> YARN-7875. Node Attribute store for storing and recovering attributes
>
> *Detailed Design:*
>
> Please refer [1] for detailed design document.
>
> *Testing Efforts:*
>
> We did detailed tests for the feature in the last few weeks.
> This feature will be enabled only when Node Attributes constraints are
> specified through SchedulingRequest from AM.
> Manager implementation will help to store and recover Node Attributes. This
> works with existing placement constraints.
>
> *Regarding to API stability:*
>
> All newly added @Public APIs are @Unstable.
>
> Documentation jira [3] could help to provide detailed configuration
> details. This feature works from end-to-end and we tested this in our local
> cluster. Branch code is run against trunk and tracked via [4].
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Weiwei
> Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
> Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
>
> [1] :
>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> [2] : https://issues.apache.org/jira/browse/YARN-3409
> [3] : https://issues.apache.org/jira/browse/YARN-7865
> [4] : https://issues.apache.org/jira/browse/YARN-8718
>
> Thanks,
> + Naga & Sunil Govindan
>
--
Weiwei Yang
Re: [Discuss] Merge discussion for Node attribute support feature YARN-3409
Posted by Sunil G <su...@apache.org>.
+1 for merge.
Quickly checked all basic test runs in the branch
- Add/replace/remove of attributes work good.
- scheduler can now handle attribute based placement constraints.
- tested DS shell with various constructs like java=1.8, python!=3, AND/OR
constraints etc.
- Documentation on attributes is also looks good
Thanks Naga, Weiwei, Bibin and all the folks who supported in designing and
reviewing the branch.
- Sunil
On Wed, Aug 29, 2018 at 8:00 PM Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> Hi All,
>
> We would like to hear your thoughts on merging “Node Attributes Support in
> YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
> it in for HADOOP 3.2.
>
> *Major work happened in this branch*
>
> YARN-6858. Attribute Manager to store and provide node attributes in RM
> YARN-7871. Support Node attributes reporting from NM to RM( distributed
> node attributes)
> YARN-7863. Modify placement constraints to support node attributes
> YARN-7875. Node Attribute store for storing and recovering attributes
>
> *Detailed Design:*
>
> Please refer [1] for detailed design document.
>
> *Testing Efforts:*
>
> We did detailed tests for the feature in the last few weeks.
> This feature will be enabled only when Node Attributes constraints are
> specified through SchedulingRequest from AM.
> Manager implementation will help to store and recover Node Attributes. This
> works with existing placement constraints.
>
> *Regarding to API stability:*
>
> All newly added @Public APIs are @Unstable.
>
> Documentation jira [3] could help to provide detailed configuration
> details. This feature works from end-to-end and we tested this in our local
> cluster. Branch code is run against trunk and tracked via [4].
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Weiwei
> Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
> Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
>
> [1] :
>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> [2] : https://issues.apache.org/jira/browse/YARN-3409
> [3] : https://issues.apache.org/jira/browse/YARN-7865
> [4] : https://issues.apache.org/jira/browse/YARN-8718
>
> Thanks,
> + Naga & Sunil Govindan
>
Re: [Discuss] Merge discussion for Node attribute support feature YARN-3409
Posted by Sunil G <su...@apache.org>.
+1 for merge.
Quickly checked all basic test runs in the branch
- Add/replace/remove of attributes work good.
- scheduler can now handle attribute based placement constraints.
- tested DS shell with various constructs like java=1.8, python!=3, AND/OR
constraints etc.
- Documentation on attributes is also looks good
Thanks Naga, Weiwei, Bibin and all the folks who supported in designing and
reviewing the branch.
- Sunil
On Wed, Aug 29, 2018 at 8:00 PM Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> Hi All,
>
> We would like to hear your thoughts on merging “Node Attributes Support in
> YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
> it in for HADOOP 3.2.
>
> *Major work happened in this branch*
>
> YARN-6858. Attribute Manager to store and provide node attributes in RM
> YARN-7871. Support Node attributes reporting from NM to RM( distributed
> node attributes)
> YARN-7863. Modify placement constraints to support node attributes
> YARN-7875. Node Attribute store for storing and recovering attributes
>
> *Detailed Design:*
>
> Please refer [1] for detailed design document.
>
> *Testing Efforts:*
>
> We did detailed tests for the feature in the last few weeks.
> This feature will be enabled only when Node Attributes constraints are
> specified through SchedulingRequest from AM.
> Manager implementation will help to store and recover Node Attributes. This
> works with existing placement constraints.
>
> *Regarding to API stability:*
>
> All newly added @Public APIs are @Unstable.
>
> Documentation jira [3] could help to provide detailed configuration
> details. This feature works from end-to-end and we tested this in our local
> cluster. Branch code is run against trunk and tracked via [4].
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Weiwei
> Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
> Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
>
> [1] :
>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> [2] : https://issues.apache.org/jira/browse/YARN-3409
> [3] : https://issues.apache.org/jira/browse/YARN-7865
> [4] : https://issues.apache.org/jira/browse/YARN-8718
>
> Thanks,
> + Naga & Sunil Govindan
>
RE: [Discuss] Merge discussion for Node attribute support feature
YARN-3409
Posted by Bibinchundatt <bi...@huawei.com>.
+1 for merge. Fetaure would be a good addition to 3.2 release.
--------------------------------------------------
Bibin A Chundatt
M: +91-9742095715<tel:+91-9742095715>
E: bibin.chundatt@huawei.com<ma...@huawei.com>
2012<tel:2012>实验室-印研IT&Cloud BU分部
2012<tel:2012> Laboratories-IT&Cloud BU Branch Dept.
From:Naganarasimha Garla
To:common-dev@hadoop.apache.org,Hdfs-dev,yarn-dev@hadoop.apache.org,mapreduce-dev@hadoop.apache.org,
Date:2018-08-29 20:00:44
Subject:[Discuss] Merge discussion for Node attribute support feature YARN-3409
Hi All,
We would like to hear your thoughts on merging “Node Attributes Support in
YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
it in for HADOOP 3.2.
*Major work happened in this branch*
YARN-6858. Attribute Manager to store and provide node attributes in RM
YARN-7871. Support Node attributes reporting from NM to RM( distributed
node attributes)
YARN-7863. Modify placement constraints to support node attributes
YARN-7875. Node Attribute store for storing and recovering attributes
*Detailed Design:*
Please refer [1] for detailed design document.
*Testing Efforts:*
We did detailed tests for the feature in the last few weeks.
This feature will be enabled only when Node Attributes constraints are
specified through SchedulingRequest from AM.
Manager implementation will help to store and recover Node Attributes. This
works with existing placement constraints.
*Regarding to API stability:*
All newly added @Public APIs are @Unstable.
Documentation jira [3] could help to provide detailed configuration
details. This feature works from end-to-end and we tested this in our local
cluster. Branch code is run against trunk and tracked via [4].
We would love to get your thoughts before opening a voting thread.
Special thanks to a team of folks who worked hard and contributed towards
this efforts including design discussion / patch / reviews, etc.: Weiwei
Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
[1] :
https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
[2] : https://issues.apache.org/jira/browse/YARN-3409
[3] : https://issues.apache.org/jira/browse/YARN-7865
[4] : https://issues.apache.org/jira/browse/YARN-8718
Thanks,
+ Naga & Sunil Govindan
RE: [Discuss] Merge discussion for Node attribute support feature
YARN-3409
Posted by Bibinchundatt <bi...@huawei.com>.
+1 for merge. Fetaure would be a good addition to 3.2 release.
--------------------------------------------------
Bibin A Chundatt
M: +91-9742095715<tel:+91-9742095715>
E: bibin.chundatt@huawei.com<ma...@huawei.com>
2012<tel:2012>实验室-印研IT&Cloud BU分部
2012<tel:2012> Laboratories-IT&Cloud BU Branch Dept.
From:Naganarasimha Garla
To:common-dev@hadoop.apache.org,Hdfs-dev,yarn-dev@hadoop.apache.org,mapreduce-dev@hadoop.apache.org,
Date:2018-08-29 20:00:44
Subject:[Discuss] Merge discussion for Node attribute support feature YARN-3409
Hi All,
We would like to hear your thoughts on merging “Node Attributes Support in
YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
it in for HADOOP 3.2.
*Major work happened in this branch*
YARN-6858. Attribute Manager to store and provide node attributes in RM
YARN-7871. Support Node attributes reporting from NM to RM( distributed
node attributes)
YARN-7863. Modify placement constraints to support node attributes
YARN-7875. Node Attribute store for storing and recovering attributes
*Detailed Design:*
Please refer [1] for detailed design document.
*Testing Efforts:*
We did detailed tests for the feature in the last few weeks.
This feature will be enabled only when Node Attributes constraints are
specified through SchedulingRequest from AM.
Manager implementation will help to store and recover Node Attributes. This
works with existing placement constraints.
*Regarding to API stability:*
All newly added @Public APIs are @Unstable.
Documentation jira [3] could help to provide detailed configuration
details. This feature works from end-to-end and we tested this in our local
cluster. Branch code is run against trunk and tracked via [4].
We would love to get your thoughts before opening a voting thread.
Special thanks to a team of folks who worked hard and contributed towards
this efforts including design discussion / patch / reviews, etc.: Weiwei
Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
[1] :
https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
[2] : https://issues.apache.org/jira/browse/YARN-3409
[3] : https://issues.apache.org/jira/browse/YARN-7865
[4] : https://issues.apache.org/jira/browse/YARN-8718
Thanks,
+ Naga & Sunil Govindan
Re: [Discuss] Merge discussion for Node attribute support feature YARN-3409
Posted by Weiwei Yang <ab...@gmail.com>.
Hi Naga
+1 for the merge. We need it for 3.2.0.
This brings a lot of value when it works with placement constraints APIs,
it will be very important for long-running jobs and service type tasks.
Thanks!
--
Weiwei
On Wed, Aug 29, 2018 at 10:30 PM Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> Hi All,
>
> We would like to hear your thoughts on merging “Node Attributes Support in
> YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
> it in for HADOOP 3.2.
>
> *Major work happened in this branch*
>
> YARN-6858. Attribute Manager to store and provide node attributes in RM
> YARN-7871. Support Node attributes reporting from NM to RM( distributed
> node attributes)
> YARN-7863. Modify placement constraints to support node attributes
> YARN-7875. Node Attribute store for storing and recovering attributes
>
> *Detailed Design:*
>
> Please refer [1] for detailed design document.
>
> *Testing Efforts:*
>
> We did detailed tests for the feature in the last few weeks.
> This feature will be enabled only when Node Attributes constraints are
> specified through SchedulingRequest from AM.
> Manager implementation will help to store and recover Node Attributes. This
> works with existing placement constraints.
>
> *Regarding to API stability:*
>
> All newly added @Public APIs are @Unstable.
>
> Documentation jira [3] could help to provide detailed configuration
> details. This feature works from end-to-end and we tested this in our local
> cluster. Branch code is run against trunk and tracked via [4].
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Weiwei
> Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
> Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
>
> [1] :
>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> [2] : https://issues.apache.org/jira/browse/YARN-3409
> [3] : https://issues.apache.org/jira/browse/YARN-7865
> [4] : https://issues.apache.org/jira/browse/YARN-8718
>
> Thanks,
> + Naga & Sunil Govindan
>
--
Weiwei Yang
Re: [Discuss] Merge discussion for Node attribute support feature YARN-3409
Posted by Sunil G <su...@apache.org>.
+1 for merge.
Quickly checked all basic test runs in the branch
- Add/replace/remove of attributes work good.
- scheduler can now handle attribute based placement constraints.
- tested DS shell with various constructs like java=1.8, python!=3, AND/OR
constraints etc.
- Documentation on attributes is also looks good
Thanks Naga, Weiwei, Bibin and all the folks who supported in designing and
reviewing the branch.
- Sunil
On Wed, Aug 29, 2018 at 8:00 PM Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> Hi All,
>
> We would like to hear your thoughts on merging “Node Attributes Support in
> YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
> it in for HADOOP 3.2.
>
> *Major work happened in this branch*
>
> YARN-6858. Attribute Manager to store and provide node attributes in RM
> YARN-7871. Support Node attributes reporting from NM to RM( distributed
> node attributes)
> YARN-7863. Modify placement constraints to support node attributes
> YARN-7875. Node Attribute store for storing and recovering attributes
>
> *Detailed Design:*
>
> Please refer [1] for detailed design document.
>
> *Testing Efforts:*
>
> We did detailed tests for the feature in the last few weeks.
> This feature will be enabled only when Node Attributes constraints are
> specified through SchedulingRequest from AM.
> Manager implementation will help to store and recover Node Attributes. This
> works with existing placement constraints.
>
> *Regarding to API stability:*
>
> All newly added @Public APIs are @Unstable.
>
> Documentation jira [3] could help to provide detailed configuration
> details. This feature works from end-to-end and we tested this in our local
> cluster. Branch code is run against trunk and tracked via [4].
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Weiwei
> Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
> Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
>
> [1] :
>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> [2] : https://issues.apache.org/jira/browse/YARN-3409
> [3] : https://issues.apache.org/jira/browse/YARN-7865
> [4] : https://issues.apache.org/jira/browse/YARN-8718
>
> Thanks,
> + Naga & Sunil Govindan
>
Re: [Discuss] Merge discussion for Node attribute support feature YARN-3409
Posted by Sunil G <su...@apache.org>.
+1 for merge.
Quickly checked all basic test runs in the branch
- Add/replace/remove of attributes work good.
- scheduler can now handle attribute based placement constraints.
- tested DS shell with various constructs like java=1.8, python!=3, AND/OR
constraints etc.
- Documentation on attributes is also looks good
Thanks Naga, Weiwei, Bibin and all the folks who supported in designing and
reviewing the branch.
- Sunil
On Wed, Aug 29, 2018 at 8:00 PM Naganarasimha Garla <
naganarasimha_gr@apache.org> wrote:
> Hi All,
>
> We would like to hear your thoughts on merging “Node Attributes Support in
> YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
> it in for HADOOP 3.2.
>
> *Major work happened in this branch*
>
> YARN-6858. Attribute Manager to store and provide node attributes in RM
> YARN-7871. Support Node attributes reporting from NM to RM( distributed
> node attributes)
> YARN-7863. Modify placement constraints to support node attributes
> YARN-7875. Node Attribute store for storing and recovering attributes
>
> *Detailed Design:*
>
> Please refer [1] for detailed design document.
>
> *Testing Efforts:*
>
> We did detailed tests for the feature in the last few weeks.
> This feature will be enabled only when Node Attributes constraints are
> specified through SchedulingRequest from AM.
> Manager implementation will help to store and recover Node Attributes. This
> works with existing placement constraints.
>
> *Regarding to API stability:*
>
> All newly added @Public APIs are @Unstable.
>
> Documentation jira [3] could help to provide detailed configuration
> details. This feature works from end-to-end and we tested this in our local
> cluster. Branch code is run against trunk and tracked via [4].
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Weiwei
> Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
> Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
>
> [1] :
>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> [2] : https://issues.apache.org/jira/browse/YARN-3409
> [3] : https://issues.apache.org/jira/browse/YARN-7865
> [4] : https://issues.apache.org/jira/browse/YARN-8718
>
> Thanks,
> + Naga & Sunil Govindan
>
Re: [Discuss] Merge discussion for Node attribute support feature YARN-3409
Posted by Wangda Tan <wh...@gmail.com>.
+1 for the merge, it gonna be a great addition to 3.2.0 release. Thanks to
everybody for pushing this feature to complete.
Best,
Wangda
On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <bi...@huawei.com>
wrote:
> +1 for merge. Fetaure would be a good addition to 3.2 release.
>
> --------------------------------------------------
> Bibin A Chundatt
> M: +91-9742095715<tel:+91-9742095715>
> E: bibin.chundatt@huawei.com<ma...@huawei.com>
> 2012<tel:2012>实验室-印研IT&Cloud BU分部
> 2012<tel:2012> Laboratories-IT&Cloud BU Branch Dept.
> From:Naganarasimha Garla
> To:common-dev@hadoop.apache.org,Hdfs-dev,yarn-dev@hadoop.apache.org,
> mapreduce-dev@hadoop.apache.org,
> Date:2018-08-29 20:00:44
> Subject:[Discuss] Merge discussion for Node attribute support feature
> YARN-3409
>
> Hi All,
>
> We would like to hear your thoughts on merging “Node Attributes Support in
> YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
> it in for HADOOP 3.2.
>
> *Major work happened in this branch*
>
> YARN-6858. Attribute Manager to store and provide node attributes in RM
> YARN-7871. Support Node attributes reporting from NM to RM( distributed
> node attributes)
> YARN-7863. Modify placement constraints to support node attributes
> YARN-7875. Node Attribute store for storing and recovering attributes
>
> *Detailed Design:*
>
> Please refer [1] for detailed design document.
>
> *Testing Efforts:*
>
> We did detailed tests for the feature in the last few weeks.
> This feature will be enabled only when Node Attributes constraints are
> specified through SchedulingRequest from AM.
> Manager implementation will help to store and recover Node Attributes. This
> works with existing placement constraints.
>
> *Regarding to API stability:*
>
> All newly added @Public APIs are @Unstable.
>
> Documentation jira [3] could help to provide detailed configuration
> details. This feature works from end-to-end and we tested this in our local
> cluster. Branch code is run against trunk and tracked via [4].
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Weiwei
> Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
> Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
>
> [1] :
>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> [2] : https://issues.apache.org/jira/browse/YARN-3409
> [3] : https://issues.apache.org/jira/browse/YARN-7865
> [4] : https://issues.apache.org/jira/browse/YARN-8718
>
> Thanks,
> + Naga & Sunil Govindan
>
Re: [Discuss] Merge discussion for Node attribute support feature YARN-3409
Posted by Wangda Tan <wh...@gmail.com>.
+1 for the merge, it gonna be a great addition to 3.2.0 release. Thanks to
everybody for pushing this feature to complete.
Best,
Wangda
On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <bi...@huawei.com>
wrote:
> +1 for merge. Fetaure would be a good addition to 3.2 release.
>
> --------------------------------------------------
> Bibin A Chundatt
> M: +91-9742095715<tel:+91-9742095715>
> E: bibin.chundatt@huawei.com<ma...@huawei.com>
> 2012<tel:2012>实验室-印研IT&Cloud BU分部
> 2012<tel:2012> Laboratories-IT&Cloud BU Branch Dept.
> From:Naganarasimha Garla
> To:common-dev@hadoop.apache.org,Hdfs-dev,yarn-dev@hadoop.apache.org,
> mapreduce-dev@hadoop.apache.org,
> Date:2018-08-29 20:00:44
> Subject:[Discuss] Merge discussion for Node attribute support feature
> YARN-3409
>
> Hi All,
>
> We would like to hear your thoughts on merging “Node Attributes Support in
> YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
> it in for HADOOP 3.2.
>
> *Major work happened in this branch*
>
> YARN-6858. Attribute Manager to store and provide node attributes in RM
> YARN-7871. Support Node attributes reporting from NM to RM( distributed
> node attributes)
> YARN-7863. Modify placement constraints to support node attributes
> YARN-7875. Node Attribute store for storing and recovering attributes
>
> *Detailed Design:*
>
> Please refer [1] for detailed design document.
>
> *Testing Efforts:*
>
> We did detailed tests for the feature in the last few weeks.
> This feature will be enabled only when Node Attributes constraints are
> specified through SchedulingRequest from AM.
> Manager implementation will help to store and recover Node Attributes. This
> works with existing placement constraints.
>
> *Regarding to API stability:*
>
> All newly added @Public APIs are @Unstable.
>
> Documentation jira [3] could help to provide detailed configuration
> details. This feature works from end-to-end and we tested this in our local
> cluster. Branch code is run against trunk and tracked via [4].
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Weiwei
> Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
> Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
>
> [1] :
>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> [2] : https://issues.apache.org/jira/browse/YARN-3409
> [3] : https://issues.apache.org/jira/browse/YARN-7865
> [4] : https://issues.apache.org/jira/browse/YARN-8718
>
> Thanks,
> + Naga & Sunil Govindan
>
Re: [Discuss] Merge discussion for Node attribute support feature YARN-3409
Posted by Wangda Tan <wh...@gmail.com>.
+1 for the merge, it gonna be a great addition to 3.2.0 release. Thanks to
everybody for pushing this feature to complete.
Best,
Wangda
On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <bi...@huawei.com>
wrote:
> +1 for merge. Fetaure would be a good addition to 3.2 release.
>
> --------------------------------------------------
> Bibin A Chundatt
> M: +91-9742095715<tel:+91-9742095715>
> E: bibin.chundatt@huawei.com<ma...@huawei.com>
> 2012<tel:2012>实验室-印研IT&Cloud BU分部
> 2012<tel:2012> Laboratories-IT&Cloud BU Branch Dept.
> From:Naganarasimha Garla
> To:common-dev@hadoop.apache.org,Hdfs-dev,yarn-dev@hadoop.apache.org,
> mapreduce-dev@hadoop.apache.org,
> Date:2018-08-29 20:00:44
> Subject:[Discuss] Merge discussion for Node attribute support feature
> YARN-3409
>
> Hi All,
>
> We would like to hear your thoughts on merging “Node Attributes Support in
> YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
> it in for HADOOP 3.2.
>
> *Major work happened in this branch*
>
> YARN-6858. Attribute Manager to store and provide node attributes in RM
> YARN-7871. Support Node attributes reporting from NM to RM( distributed
> node attributes)
> YARN-7863. Modify placement constraints to support node attributes
> YARN-7875. Node Attribute store for storing and recovering attributes
>
> *Detailed Design:*
>
> Please refer [1] for detailed design document.
>
> *Testing Efforts:*
>
> We did detailed tests for the feature in the last few weeks.
> This feature will be enabled only when Node Attributes constraints are
> specified through SchedulingRequest from AM.
> Manager implementation will help to store and recover Node Attributes. This
> works with existing placement constraints.
>
> *Regarding to API stability:*
>
> All newly added @Public APIs are @Unstable.
>
> Documentation jira [3] could help to provide detailed configuration
> details. This feature works from end-to-end and we tested this in our local
> cluster. Branch code is run against trunk and tracked via [4].
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Weiwei
> Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
> Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
>
> [1] :
>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> [2] : https://issues.apache.org/jira/browse/YARN-3409
> [3] : https://issues.apache.org/jira/browse/YARN-7865
> [4] : https://issues.apache.org/jira/browse/YARN-8718
>
> Thanks,
> + Naga & Sunil Govindan
>
Re: [Discuss] Merge discussion for Node attribute support feature YARN-3409
Posted by Wangda Tan <wh...@gmail.com>.
+1 for the merge, it gonna be a great addition to 3.2.0 release. Thanks to
everybody for pushing this feature to complete.
Best,
Wangda
On Tue, Sep 4, 2018 at 8:25 AM Bibinchundatt <bi...@huawei.com>
wrote:
> +1 for merge. Fetaure would be a good addition to 3.2 release.
>
> --------------------------------------------------
> Bibin A Chundatt
> M: +91-9742095715<tel:+91-9742095715>
> E: bibin.chundatt@huawei.com<ma...@huawei.com>
> 2012<tel:2012>实验室-印研IT&Cloud BU分部
> 2012<tel:2012> Laboratories-IT&Cloud BU Branch Dept.
> From:Naganarasimha Garla
> To:common-dev@hadoop.apache.org,Hdfs-dev,yarn-dev@hadoop.apache.org,
> mapreduce-dev@hadoop.apache.org,
> Date:2018-08-29 20:00:44
> Subject:[Discuss] Merge discussion for Node attribute support feature
> YARN-3409
>
> Hi All,
>
> We would like to hear your thoughts on merging “Node Attributes Support in
> YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
> it in for HADOOP 3.2.
>
> *Major work happened in this branch*
>
> YARN-6858. Attribute Manager to store and provide node attributes in RM
> YARN-7871. Support Node attributes reporting from NM to RM( distributed
> node attributes)
> YARN-7863. Modify placement constraints to support node attributes
> YARN-7875. Node Attribute store for storing and recovering attributes
>
> *Detailed Design:*
>
> Please refer [1] for detailed design document.
>
> *Testing Efforts:*
>
> We did detailed tests for the feature in the last few weeks.
> This feature will be enabled only when Node Attributes constraints are
> specified through SchedulingRequest from AM.
> Manager implementation will help to store and recover Node Attributes. This
> works with existing placement constraints.
>
> *Regarding to API stability:*
>
> All newly added @Public APIs are @Unstable.
>
> Documentation jira [3] could help to provide detailed configuration
> details. This feature works from end-to-end and we tested this in our local
> cluster. Branch code is run against trunk and tracked via [4].
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Weiwei
> Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
> Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
>
> [1] :
>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> [2] : https://issues.apache.org/jira/browse/YARN-3409
> [3] : https://issues.apache.org/jira/browse/YARN-7865
> [4] : https://issues.apache.org/jira/browse/YARN-8718
>
> Thanks,
> + Naga & Sunil Govindan
>