You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@ambari.apache.org by Oliver Szabo <os...@hortonworks.com> on 2017/06/12 12:44:23 UTC

Review Request 59954: Drop JDK 7 support for Ambari Server and Ambari managed services (AMS, LogSearch, Infra)

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59954/
-----------------------------------------------------------

Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Levas, Robert Nettleton, Sumit Mohanty, Sebastian Toader, and Sid Wagle.


Bugs: AMBARI-21099
    https://issues.apache.org/jira/browse/AMBARI-21099


Repository: ambari


Description
-------

Requirements:
- ambari should drop JDK 7 support for ambari-server and ambari-managed services (infra, ams, logsearch), so force to use jdk8
- HDP 2.x should be still able to use JDK 1.7
Changes in ambari setup:
1. First the usual ambari setup

Choose JDK for Ambari and Stack services:
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
[3] Custom JDK

2. If JDK 7 is picked, that cannot be used for ambari (only for the stack services)

JDK 7 is detected (JDK 8 required for ambari-server), install JDK 8 as well:
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[3] Custom JDK


(this commands can be used is silent mode: ambari-server setup --java-home=... --stack-java-home=...)

After the commands are executed, the following properties would appear in ambari.properties

stack.java.home=/usr/jdk64/jdk1.7.0_67
stack.jdk.name=jdk-7u67linux-x64.tar.gz
stack.jce.name=UnlimitedJCEPolicyJDK7.zip

During command.json generating, commandParameters would be filled with proper ambari jdk,jce info (different for infra/logsearch/ams from other services) - not hostLevelParams, as multiple command could use the same hostLevelParams (like start all services: that can start zookeeper and ams as well with the same hostLevelParams)

On ambari-upgrade, ambari process wont be able to start with jdk7 so it will be required to run ambari-server setup first (you can skip the first change jdk part, but then the user will see its needed to setup a different jdk for ambari)

In the future, it will be recommended to do the JDK upgrade before upgrading to HDP 3.0 (to avoid some issues during HDP upgrade)

next step: 
- ams, logsearch, infra should use ambari_java_home (if it exists) 
- add function to ambari common to validate jdk version (inorder to fail starting any service which require jdk 8)
- make sure during HDP upgrade, minJdk and maxJdk is using the stack java versions, not ambari ones. (also add 1.8 as minJdk to HDP 3.0)
- there is one java version check in hive llap stack code, that should use stack.java.version (which i added), if that is included. (not ambari one - that will get the java.version from the ambari-server process, which can be invalid in that case)

open question:
- there can be a more flexible solution to include different jdks in ambari (based on jdk releases list) and send all of them in the command.json, and chose the proper one for different services. also that can cause some issues with custom jdks (adding a name to custom ones?), and also because the jdk names can be anything, we would need to validate/get the jdk version during sending a command to the agent. all of this work has done about to support older version of HDP, so im not sure about to create a flexible solution at all, as managing multiple JDKs can be really hard for a customer, so i would not recommend to use multiple ones.


Diffs
-----

  ambari-server/docs/configuration/index.md 2394264 
  ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java 17b89b7 
  ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java fb06e6d 
  ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java 2fa965e 
  ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 554ae1d 
  ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 2a9d6c9 
  ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java 181d323 
  ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 1d21ccd 
  ambari-server/src/main/python/ambari-server.py d84e833 
  ambari-server/src/main/python/ambari_server/serverConfiguration.py 4780338 
  ambari-server/src/main/python/ambari_server/serverSetup.py c6de088 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 4052d1d 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py 1f17cd1 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py 3488e75 
  ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/shared_initialization.py 148d235 
  ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-ANY/scripts/params.py 9be9101 
  ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-ANY/scripts/shared_initialization.py 5d79084 
  ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-START/scripts/params.py a3830f7 
  ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-START/scripts/shared_initialization.py aed1124 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-ANY/scripts/params.py 2c2c901 
  ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-ANY/scripts/shared_initialization.py 7dc1a48 
  ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProviderTest.java 29b0476 
  ambari-server/src/test/java/org/apache/ambari/server/utils/StageUtilsTest.java b1cce55 
  ambari-server/src/test/python/TestAmbariServer.py 7f0cb93 


Diff: https://reviews.apache.org/r/59954/diff/1/


Testing
-------

Tests done.
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 34:05 min
[INFO] Finished at: 2017-06-09T21:24:55+02:00
[INFO] Final Memory: 79M/910M
[INFO] ------------------------------------------------------------------------

FT: ambari-server setup tests done, jdk/jce downloaded properly in case I changed to jdk name / jce name. still some use cases is in progress...


Thanks,

Oliver Szabo

Re: Review Request 59954: Drop JDK 7 support for Ambari Server and Ambari managed services (AMS, LogSearch, Infra)

Posted by Sebastian Toader <st...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59954/#review177628
-----------------------------------------------------------


Ship it!




Ship It!

- Sebastian Toader


On June 12, 2017, 2:44 p.m., Oliver Szabo wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59954/
> -----------------------------------------------------------
> 
> (Updated June 12, 2017, 2:44 p.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Levas, Robert Nettleton, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21099
>     https://issues.apache.org/jira/browse/AMBARI-21099
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Requirements:
> - ambari should drop JDK 7 support for ambari-server and ambari-managed services (infra, ams, logsearch), so force to use jdk8
> - HDP 2.x should be still able to use JDK 1.7
> Changes in ambari setup:
> 1. First the usual ambari setup
> 
> Choose JDK for Ambari and Stack services:
> [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
> [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
> [3] Custom JDK
> 
> 2. If JDK 7 is picked, that cannot be used for ambari (only for the stack services)
> 
> JDK 7 is detected (JDK 8 required for ambari-server), install JDK 8 as well:
> [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
> [3] Custom JDK
> 
> 
> (this commands can be used is silent mode: ambari-server setup --java-home=... --stack-java-home=...)
> 
> After the commands are executed, the following properties would appear in ambari.properties
> 
> stack.java.home=/usr/jdk64/jdk1.7.0_67
> stack.jdk.name=jdk-7u67linux-x64.tar.gz
> stack.jce.name=UnlimitedJCEPolicyJDK7.zip
> 
> During command.json generating, commandParameters would be filled with proper ambari jdk,jce info (different for infra/logsearch/ams from other services) - not hostLevelParams, as multiple command could use the same hostLevelParams (like start all services: that can start zookeeper and ams as well with the same hostLevelParams)
> 
> On ambari-upgrade, ambari process wont be able to start with jdk7 so it will be required to run ambari-server setup first (you can skip the first change jdk part, but then the user will see its needed to setup a different jdk for ambari)
> 
> In the future, it will be recommended to do the JDK upgrade before upgrading to HDP 3.0 (to avoid some issues during HDP upgrade)
> 
> next step: 
> - ams, logsearch, infra should use ambari_java_home (if it exists) 
> - add function to ambari common to validate jdk version (inorder to fail starting any service which require jdk 8)
> - make sure during HDP upgrade, minJdk and maxJdk is using the stack java versions, not ambari ones. (also add 1.8 as minJdk to HDP 3.0)
> - there is one java version check in hive llap stack code, that should use stack.java.version (which i added), if that is included. (not ambari one - that will get the java.version from the ambari-server process, which can be invalid in that case)
> 
> open question:
> - there can be a more flexible solution to include different jdks in ambari (based on jdk releases list) and send all of them in the command.json, and chose the proper one for different services. also that can cause some issues with custom jdks (adding a name to custom ones?), and also because the jdk names can be anything, we would need to validate/get the jdk version during sending a command to the agent. all of this work has done about to support older version of HDP, so im not sure about to create a flexible solution at all, as managing multiple JDKs can be really hard for a customer, so i would not recommend to use multiple ones.
> 
> 
> Diffs
> -----
> 
>   ambari-server/docs/configuration/index.md 2394264 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java 17b89b7 
>   ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java fb06e6d 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java 2fa965e 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 554ae1d 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 2a9d6c9 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java 181d323 
>   ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 1d21ccd 
>   ambari-server/src/main/python/ambari-server.py d84e833 
>   ambari-server/src/main/python/ambari_server/serverConfiguration.py 4780338 
>   ambari-server/src/main/python/ambari_server/serverSetup.py c6de088 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 4052d1d 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py 1f17cd1 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py 3488e75 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/shared_initialization.py 148d235 
>   ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-ANY/scripts/params.py 9be9101 
>   ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-ANY/scripts/shared_initialization.py 5d79084 
>   ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-START/scripts/params.py a3830f7 
>   ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-START/scripts/shared_initialization.py aed1124 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-ANY/scripts/params.py 2c2c901 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-ANY/scripts/shared_initialization.py 7dc1a48 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProviderTest.java 29b0476 
>   ambari-server/src/test/java/org/apache/ambari/server/utils/StageUtilsTest.java b1cce55 
>   ambari-server/src/test/python/TestAmbariServer.py 7f0cb93 
> 
> 
> Diff: https://reviews.apache.org/r/59954/diff/1/
> 
> 
> Testing
> -------
> 
> Tests done.
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 34:05 min
> [INFO] Finished at: 2017-06-09T21:24:55+02:00
> [INFO] Final Memory: 79M/910M
> [INFO] ------------------------------------------------------------------------
> 
> FT: ambari-server setup tests done, jdk/jce downloaded properly in case I changed to jdk name / jce name. still some use cases is in progress...
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>

Re: Review Request 59954: Drop JDK 7 support for Ambari Server and Ambari managed services (AMS, LogSearch, Infra)

Posted by Nate Cole <nc...@hortonworks.com>.

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/59954/#review177880
-----------------------------------------------------------


Ship it!




Ship It!

- Nate Cole


On June 13, 2017, 10:44 a.m., Oliver Szabo wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/59954/
> -----------------------------------------------------------
> 
> (Updated June 13, 2017, 10:44 a.m.)
> 
> 
> Review request for Ambari, Jonathan Hurley, Nate Cole, Robert Levas, Robert Nettleton, Sumit Mohanty, Sebastian Toader, and Sid Wagle.
> 
> 
> Bugs: AMBARI-21099
>     https://issues.apache.org/jira/browse/AMBARI-21099
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> Requirements:
> - ambari should drop JDK 7 support for ambari-server and ambari-managed services (infra, ams, logsearch), so force to use jdk8
> - HDP 2.x should be still able to use JDK 1.7
> Changes in ambari setup:
> 1. First the usual ambari setup
> 
> Choose JDK for Ambari and Stack services:
> [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
> [2] Oracle JDK 1.7 + Java Cryptography Extension (JCE) Policy Files 7
> [3] Custom JDK
> 
> 2. If JDK 7 is picked, that cannot be used for ambari (only for the stack services)
> 
> JDK 7 is detected (JDK 8 required for ambari-server), install JDK 8 as well:
> [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
> [3] Custom JDK
> 
> 
> (this commands can be used is silent mode: ambari-server setup --java-home=... --stack-java-home=...)
> 
> After the commands are executed, the following properties would appear in ambari.properties
> 
> stack.java.home=/usr/jdk64/jdk1.7.0_67
> stack.jdk.name=jdk-7u67linux-x64.tar.gz
> stack.jce.name=UnlimitedJCEPolicyJDK7.zip
> 
> During command.json generating, commandParameters would be filled with proper ambari jdk,jce info (different for infra/logsearch/ams from other services) - not hostLevelParams, as multiple command could use the same hostLevelParams (like start all services: that can start zookeeper and ams as well with the same hostLevelParams)
> 
> On ambari-upgrade, ambari process wont be able to start with jdk7 so it will be required to run ambari-server setup first (you can skip the first change jdk part, but then the user will see its needed to setup a different jdk for ambari)
> 
> In the future, it will be recommended to do the JDK upgrade before upgrading to HDP 3.0 (to avoid some issues during HDP upgrade)
> 
> next step: 
> - add function to ambari common to validate jdk version (inorder to fail starting any service which require jdk 8)
> 
> open question:
> - there can be a more flexible solution to include different jdks in ambari (based on jdk releases list) and send all of them in the command.json, and chose the proper one for different services. also that can cause some issues with custom jdks (adding a name to custom ones?), and also because the jdk names can be anything, we would need to validate/get the jdk version during sending a command to the agent. all of this work has done about to support older version of HDP, so im not sure about to create a flexible solution at all, as managing multiple JDKs can be really hard for a customer, so i would not recommend to use multiple ones.
> 
> 
> Diffs
> -----
> 
>   ambari-server/docs/configuration/index.md 2394264 
>   ambari-server/src/main/java/org/apache/ambari/server/agent/ExecutionCommand.java 60df8cf 
>   ambari-server/src/main/java/org/apache/ambari/server/configuration/Configuration.java fb06e6d 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariActionExecutionHelper.java 1b0e0e0 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariCustomCommandExecutionHelper.java 07f6e30 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/AmbariManagementControllerImpl.java 8d262e2 
>   ambari-server/src/main/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProvider.java 21cf16c 
>   ambari-server/src/main/java/org/apache/ambari/server/utils/StageUtils.java 9930148 
>   ambari-server/src/main/python/ambari-server.py d84e833 
>   ambari-server/src/main/python/ambari_server/serverConfiguration.py 4780338 
>   ambari-server/src/main/python/ambari_server/serverSetup.py c6de088 
>   ambari-server/src/main/resources/common-services/AMBARI_INFRA/0.1.0/package/scripts/params.py 6eb3ba8 
>   ambari-server/src/main/resources/common-services/AMBARI_METRICS/0.1.0/package/scripts/params.py b8c14f4 
>   ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/metadata.py 36c4598 
>   ambari-server/src/main/resources/common-services/ATLAS/0.1.0.2.3/package/scripts/params.py d26df33 
>   ambari-server/src/main/resources/common-services/LOGSEARCH/0.5.0/package/scripts/params.py 6738c5c 
>   ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/params.py 094d239 
>   ambari-server/src/main/resources/common-services/RANGER/0.4.0/package/scripts/setup_ranger_xml.py 85eb796 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/params.py 4052d1d 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-ANY/scripts/shared_initialization.py 1f17cd1 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/params.py 3488e75 
>   ambari-server/src/main/resources/stacks/HDP/2.0.6/hooks/before-START/scripts/shared_initialization.py 148d235 
>   ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-ANY/scripts/params.py 9be9101 
>   ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-ANY/scripts/shared_initialization.py 5d79084 
>   ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-START/scripts/params.py a3830f7 
>   ambari-server/src/main/resources/stacks/HDP/3.0/hooks/before-START/scripts/shared_initialization.py aed1124 
>   ambari-server/src/main/resources/stacks/HDP/3.0/metainfo.xml 0364d41 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-ANY/scripts/params.py 2c2c901 
>   ambari-server/src/main/resources/stacks/PERF/1.0/hooks/before-ANY/scripts/shared_initialization.py 7dc1a48 
>   ambari-server/src/test/java/org/apache/ambari/server/controller/internal/ClientConfigResourceProviderTest.java 29b0476 
>   ambari-server/src/test/java/org/apache/ambari/server/utils/StageUtilsTest.java b1cce55 
>   ambari-server/src/test/python/TestAmbariServer.py 7f0cb93 
> 
> 
> Diff: https://reviews.apache.org/r/59954/diff/3/
> 
> 
> Testing
> -------
> 
> Tests done.
> [INFO] ------------------------------------------------------------------------
> [INFO] BUILD SUCCESS
> [INFO] ------------------------------------------------------------------------
> [INFO] Total time: 34:05 min
> [INFO] Finished at: 2017-06-09T21:24:55+02:00
> [INFO] Final Memory: 79M/910M
> [INFO] ------------------------------------------------------------------------
> 
> FT: ambari-server setup tests done, jdk/jce downloaded properly in case I changed to jdk name / jce name. deploying cluster with ams/infra has done with using different jdk for those services.
> 
> 
> Thanks,
> 
> Oliver Szabo
> 
>