You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Dmytro Sen <ds...@hortonworks.com> on 2015/11/18 09:18:49 UTC

Review Request 40433: Add support for ATS v1.5 for HDP versions 2.3 and higher

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40433/
-----------------------------------------------------------

Review request for Ambari, Andrew Onischuk and Vitalyi Brodetskyi.


Bugs: AMBARI-13936
    https://issues.apache.org/jira/browse/AMBARI-13936


Repository: ambari


Description
-------

Starting HDP-2.3, Ambari should support ATSv1.5. This involves:
Adding new configs as specified in the doc below. Lets add only the non-advanced configs to the HDP-2.3 yarn-site (skip all configs after the section ADVANCED CONFIGS)
Adding one new config to tez-site
Ensure that the DFS paths are created and have right permissions
property: yarn.timeline-service.entity-file-cache-store.active-dir (FSPermission 01777)
property: yarn.timeline-service.entity-file-cache-store.done-dir (FSPermission 0700)
Thing(s) to make sure:
yarn.timeline-service.entity-file-cache-store.active-dir and done-dir exist on the cluster on hdfs. Active dir should have permission 01777, owned by yarn-user, group hadoop-group. Done dir should have permission 0700, owned by yarn-user, group hadoop-group.
The Tez cache plugin class is in timeline server’s classpath - it should be the case even now, but lets check
yarn.timeline-service.leveldb-timeline-store.path is a path on local filesystem and needs to be created with permissions 755 ( owner being the same user as the one which timeline server is running as ) - this should be happening as well


Diffs
-----

  ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py cc42481 
  ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py aa79fe4 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/configuration/tez-site.xml 37db69f 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/yarn-site.xml c9e617b 

Diff: https://reviews.apache.org/r/40433/diff/


Testing
-------

Unit tests passed


Thanks,

Dmytro Sen

Re: Review Request 40433: Add support for ATS v1.5 for HDP versions 2.3 and higher

Posted by Vitalyi Brodetskyi <vb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40433/#review107015
-----------------------------------------------------------

Ship it!


Ship It!

- Vitalyi Brodetskyi


On Лис. 18, 2015, 12:07 після полудня, Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40433/
> -----------------------------------------------------------
> 
> (Updated Лис. 18, 2015, 12:07 після полудня)
> 
> 
> Review request for Ambari, Andrew Onischuk and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-13936
>     https://issues.apache.org/jira/browse/AMBARI-13936
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Starting HDP-2.3, Ambari should support ATSv1.5. This involves:
> Adding new configs as specified in the doc below. Lets add only the non-advanced configs to the HDP-2.3 yarn-site (skip all configs after the section ADVANCED CONFIGS)
> Adding one new config to tez-site
> Ensure that the DFS paths are created and have right permissions
> property: yarn.timeline-service.entity-file-cache-store.active-dir (FSPermission 01777)
> property: yarn.timeline-service.entity-file-cache-store.done-dir (FSPermission 0700)
> Thing(s) to make sure:
> yarn.timeline-service.entity-file-cache-store.active-dir and done-dir exist on the cluster on hdfs. Active dir should have permission 01777, owned by yarn-user, group hadoop-group. Done dir should have permission 0700, owned by yarn-user, group hadoop-group.
> The Tez cache plugin class is in timeline server’s classpath - it should be the case even now, but lets check
> yarn.timeline-service.leveldb-timeline-store.path is a path on local filesystem and needs to be created with permissions 755 ( owner being the same user as the one which timeline server is running as ) - this should be happening as well
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py cc42481 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py aa79fe4 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/configuration/tez-site.xml 37db69f 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/yarn-site.xml c9e617b 
>   ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py e949bb0 
>   ambari-server/src/test/python/stacks/2.3/YARN/test_ats_1_5.py PRE-CREATION 
>   ambari-server/src/test/python/stacks/2.3/configs/ats_1_5.json PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40433/diff/
> 
> 
> Testing
> -------
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 40433: Add support for ATS v1.5 for HDP versions 2.3 and higher

Posted by Andrew Onischuk <ao...@hortonworks.com>.

> On Nov. 18, 2015, 4:17 p.m., Andrew Onischuk wrote:
> > ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py, line 259
> > <https://reviews.apache.org/r/40433/diff/2/?file=1129446#file1129446line259>
> >
> >     This makes every yarn component not to create HDfs dirs not ATS, which slows down deploy. Can we fix this with aditional commit?

nm actually, seems like there is a check for that above what reviewboard showed me.


- Andrew


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40433/#review107039
-----------------------------------------------------------


On Nov. 18, 2015, 12:07 p.m., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40433/
> -----------------------------------------------------------
> 
> (Updated Nov. 18, 2015, 12:07 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-13936
>     https://issues.apache.org/jira/browse/AMBARI-13936
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Starting HDP-2.3, Ambari should support ATSv1.5. This involves:
> Adding new configs as specified in the doc below. Lets add only the non-advanced configs to the HDP-2.3 yarn-site (skip all configs after the section ADVANCED CONFIGS)
> Adding one new config to tez-site
> Ensure that the DFS paths are created and have right permissions
> property: yarn.timeline-service.entity-file-cache-store.active-dir (FSPermission 01777)
> property: yarn.timeline-service.entity-file-cache-store.done-dir (FSPermission 0700)
> Thing(s) to make sure:
> yarn.timeline-service.entity-file-cache-store.active-dir and done-dir exist on the cluster on hdfs. Active dir should have permission 01777, owned by yarn-user, group hadoop-group. Done dir should have permission 0700, owned by yarn-user, group hadoop-group.
> The Tez cache plugin class is in timeline server’s classpath - it should be the case even now, but lets check
> yarn.timeline-service.leveldb-timeline-store.path is a path on local filesystem and needs to be created with permissions 755 ( owner being the same user as the one which timeline server is running as ) - this should be happening as well
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py cc42481 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py aa79fe4 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/configuration/tez-site.xml 37db69f 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/yarn-site.xml c9e617b 
>   ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py e949bb0 
>   ambari-server/src/test/python/stacks/2.3/YARN/test_ats_1_5.py PRE-CREATION 
>   ambari-server/src/test/python/stacks/2.3/configs/ats_1_5.json PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40433/diff/
> 
> 
> Testing
> -------
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 40433: Add support for ATS v1.5 for HDP versions 2.3 and higher

Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40433/#review107039
-----------------------------------------------------------



ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py (line 259)
<https://reviews.apache.org/r/40433/#comment165932>

    This makes every yarn component not to create HDfs dirs not ATS, which slows down deploy. Can we fix this with aditional commit?


- Andrew Onischuk


On Nov. 18, 2015, 12:07 p.m., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40433/
> -----------------------------------------------------------
> 
> (Updated Nov. 18, 2015, 12:07 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-13936
>     https://issues.apache.org/jira/browse/AMBARI-13936
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Starting HDP-2.3, Ambari should support ATSv1.5. This involves:
> Adding new configs as specified in the doc below. Lets add only the non-advanced configs to the HDP-2.3 yarn-site (skip all configs after the section ADVANCED CONFIGS)
> Adding one new config to tez-site
> Ensure that the DFS paths are created and have right permissions
> property: yarn.timeline-service.entity-file-cache-store.active-dir (FSPermission 01777)
> property: yarn.timeline-service.entity-file-cache-store.done-dir (FSPermission 0700)
> Thing(s) to make sure:
> yarn.timeline-service.entity-file-cache-store.active-dir and done-dir exist on the cluster on hdfs. Active dir should have permission 01777, owned by yarn-user, group hadoop-group. Done dir should have permission 0700, owned by yarn-user, group hadoop-group.
> The Tez cache plugin class is in timeline server’s classpath - it should be the case even now, but lets check
> yarn.timeline-service.leveldb-timeline-store.path is a path on local filesystem and needs to be created with permissions 755 ( owner being the same user as the one which timeline server is running as ) - this should be happening as well
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py cc42481 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py aa79fe4 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/configuration/tez-site.xml 37db69f 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/yarn-site.xml c9e617b 
>   ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py e949bb0 
>   ambari-server/src/test/python/stacks/2.3/YARN/test_ats_1_5.py PRE-CREATION 
>   ambari-server/src/test/python/stacks/2.3/configs/ats_1_5.json PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40433/diff/
> 
> 
> Testing
> -------
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 40433: Add support for ATS v1.5 for HDP versions 2.3 and higher

Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40433/#review107049
-----------------------------------------------------------



ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py (line 259)
<https://reviews.apache.org/r/40433/#comment165952>

    +1, should only be needed on ATS, not the clients


- Alejandro Fernandez


On Nov. 18, 2015, 12:07 p.m., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40433/
> -----------------------------------------------------------
> 
> (Updated Nov. 18, 2015, 12:07 p.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-13936
>     https://issues.apache.org/jira/browse/AMBARI-13936
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Starting HDP-2.3, Ambari should support ATSv1.5. This involves:
> Adding new configs as specified in the doc below. Lets add only the non-advanced configs to the HDP-2.3 yarn-site (skip all configs after the section ADVANCED CONFIGS)
> Adding one new config to tez-site
> Ensure that the DFS paths are created and have right permissions
> property: yarn.timeline-service.entity-file-cache-store.active-dir (FSPermission 01777)
> property: yarn.timeline-service.entity-file-cache-store.done-dir (FSPermission 0700)
> Thing(s) to make sure:
> yarn.timeline-service.entity-file-cache-store.active-dir and done-dir exist on the cluster on hdfs. Active dir should have permission 01777, owned by yarn-user, group hadoop-group. Done dir should have permission 0700, owned by yarn-user, group hadoop-group.
> The Tez cache plugin class is in timeline server’s classpath - it should be the case even now, but lets check
> yarn.timeline-service.leveldb-timeline-store.path is a path on local filesystem and needs to be created with permissions 755 ( owner being the same user as the one which timeline server is running as ) - this should be happening as well
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py cc42481 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py aa79fe4 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/configuration/tez-site.xml 37db69f 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/yarn-site.xml c9e617b 
>   ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py e949bb0 
>   ambari-server/src/test/python/stacks/2.3/YARN/test_ats_1_5.py PRE-CREATION 
>   ambari-server/src/test/python/stacks/2.3/configs/ats_1_5.json PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/40433/diff/
> 
> 
> Testing
> -------
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 40433: Add support for ATS v1.5 for HDP versions 2.3 and higher

Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40433/
-----------------------------------------------------------

(Updated Ноя. 18, 2015, 12:07 п.п.)


Review request for Ambari, Andrew Onischuk and Vitalyi Brodetskyi.


Bugs: AMBARI-13936
    https://issues.apache.org/jira/browse/AMBARI-13936


Repository: ambari


Description
-------

Starting HDP-2.3, Ambari should support ATSv1.5. This involves:
Adding new configs as specified in the doc below. Lets add only the non-advanced configs to the HDP-2.3 yarn-site (skip all configs after the section ADVANCED CONFIGS)
Adding one new config to tez-site
Ensure that the DFS paths are created and have right permissions
property: yarn.timeline-service.entity-file-cache-store.active-dir (FSPermission 01777)
property: yarn.timeline-service.entity-file-cache-store.done-dir (FSPermission 0700)
Thing(s) to make sure:
yarn.timeline-service.entity-file-cache-store.active-dir and done-dir exist on the cluster on hdfs. Active dir should have permission 01777, owned by yarn-user, group hadoop-group. Done dir should have permission 0700, owned by yarn-user, group hadoop-group.
The Tez cache plugin class is in timeline server’s classpath - it should be the case even now, but lets check
yarn.timeline-service.leveldb-timeline-store.path is a path on local filesystem and needs to be created with permissions 755 ( owner being the same user as the one which timeline server is running as ) - this should be happening as well


Diffs (updated)
-----

  ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py cc42481 
  ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py aa79fe4 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/configuration/tez-site.xml 37db69f 
  ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/yarn-site.xml c9e617b 
  ambari-server/src/test/python/stacks/2.1/YARN/test_apptimelineserver.py e949bb0 
  ambari-server/src/test/python/stacks/2.3/YARN/test_ats_1_5.py PRE-CREATION 
  ambari-server/src/test/python/stacks/2.3/configs/ats_1_5.json PRE-CREATION 

Diff: https://reviews.apache.org/r/40433/diff/


Testing
-------

Unit tests passed


Thanks,

Dmytro Sen

Re: Review Request 40433: Add support for ATS v1.5 for HDP versions 2.3 and higher

Posted by Andrew Onischuk <ao...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40433/#review106997
-----------------------------------------------------------

Ship it!



ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py (line 103)
<https://reviews.apache.org/r/40433/#comment165877>

    We can move out this check as well, properties which are not found will not lead to failure. As we have lazy failing implemented for such a cases.



ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py (line 259)
<https://reviews.apache.org/r/40433/#comment165876>

    Is that possible to remove this check and simply check for existance of the properties as you do below, this way it should work for all the stacks.
    
    And no need to add this hacky stack check.


- Andrew Onischuk


On Nov. 18, 2015, 8:18 a.m., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40433/
> -----------------------------------------------------------
> 
> (Updated Nov. 18, 2015, 8:18 a.m.)
> 
> 
> Review request for Ambari, Andrew Onischuk and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-13936
>     https://issues.apache.org/jira/browse/AMBARI-13936
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Starting HDP-2.3, Ambari should support ATSv1.5. This involves:
> Adding new configs as specified in the doc below. Lets add only the non-advanced configs to the HDP-2.3 yarn-site (skip all configs after the section ADVANCED CONFIGS)
> Adding one new config to tez-site
> Ensure that the DFS paths are created and have right permissions
> property: yarn.timeline-service.entity-file-cache-store.active-dir (FSPermission 01777)
> property: yarn.timeline-service.entity-file-cache-store.done-dir (FSPermission 0700)
> Thing(s) to make sure:
> yarn.timeline-service.entity-file-cache-store.active-dir and done-dir exist on the cluster on hdfs. Active dir should have permission 01777, owned by yarn-user, group hadoop-group. Done dir should have permission 0700, owned by yarn-user, group hadoop-group.
> The Tez cache plugin class is in timeline server’s classpath - it should be the case even now, but lets check
> yarn.timeline-service.leveldb-timeline-store.path is a path on local filesystem and needs to be created with permissions 755 ( owner being the same user as the one which timeline server is running as ) - this should be happening as well
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py cc42481 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py aa79fe4 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/configuration/tez-site.xml 37db69f 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/yarn-site.xml c9e617b 
> 
> Diff: https://reviews.apache.org/r/40433/diff/
> 
> 
> Testing
> -------
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>

Re: Review Request 40433: Add support for ATS v1.5 for HDP versions 2.3 and higher

Posted by Vitalyi Brodetskyi <vb...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40433/#review106996
-----------------------------------------------------------


I think we should add some tests

- Vitalyi Brodetskyi


On Лис. 18, 2015, 8:18 до полудня, Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40433/
> -----------------------------------------------------------
> 
> (Updated Лис. 18, 2015, 8:18 до полудня)
> 
> 
> Review request for Ambari, Andrew Onischuk and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-13936
>     https://issues.apache.org/jira/browse/AMBARI-13936
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Starting HDP-2.3, Ambari should support ATSv1.5. This involves:
> Adding new configs as specified in the doc below. Lets add only the non-advanced configs to the HDP-2.3 yarn-site (skip all configs after the section ADVANCED CONFIGS)
> Adding one new config to tez-site
> Ensure that the DFS paths are created and have right permissions
> property: yarn.timeline-service.entity-file-cache-store.active-dir (FSPermission 01777)
> property: yarn.timeline-service.entity-file-cache-store.done-dir (FSPermission 0700)
> Thing(s) to make sure:
> yarn.timeline-service.entity-file-cache-store.active-dir and done-dir exist on the cluster on hdfs. Active dir should have permission 01777, owned by yarn-user, group hadoop-group. Done dir should have permission 0700, owned by yarn-user, group hadoop-group.
> The Tez cache plugin class is in timeline server’s classpath - it should be the case even now, but lets check
> yarn.timeline-service.leveldb-timeline-store.path is a path on local filesystem and needs to be created with permissions 755 ( owner being the same user as the one which timeline server is running as ) - this should be happening as well
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/params_linux.py cc42481 
>   ambari-server/src/main/resources/common-services/YARN/2.1.0.2.0/package/scripts/yarn.py aa79fe4 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/TEZ/configuration/tez-site.xml 37db69f 
>   ambari-server/src/main/resources/stacks/HDP/2.3/services/YARN/configuration/yarn-site.xml c9e617b 
> 
> Diff: https://reviews.apache.org/r/40433/diff/
> 
> 
> Testing
> -------
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>