You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Dmytro Sen <ds...@hortonworks.com> on 2017/03/16 16:01:04 UTC

Review Request 57692: HDP 3.0 support for Spark with configs, kerberos, widgets, metrics, quicklinks, and themes

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57692/
-----------------------------------------------------------

Review request for Ambari, Alejandro Fernandez, Sid Wagle, and Vitalyi Brodetskyi.


Bugs: AMBARI-19834
    https://issues.apache.org/jira/browse/AMBARI-19834


Repository: ambari


Description
-------

HDP 3.0 support for Spark with configs, kerberos, widgets, metrics, quicklinks, and themes
Flatten from HDP 2.0.6 - 2.6 into common-services, and reference in HDP 3.0
IMPORTANT: In HDP 3.0, Spark1 is being removed, and Spark2 is being rebranded as "Spark"
In HDP 3.0, we have created a new stack definition that does not inherit from other stacks, in order to reduce the complexity of having to analyze older stacks.
This means that we need to create a service definition (metainfo.xml, configs, kerberos, widgets, metrics, quicklinks, and themes) that is equivalent to what is inherit and deleted from all of the previous stacks.
A merge needs to account for additions, overrides, and deletions.
metainfo.xml and configs perform a merge of older versions
kerberos.json always seems to override the previous file
Because the bits for this service may not yet be available in the HDP 3.0 repo, 
the task is to ensure that /api/v1/stacks/HDP/versions/2.6/services/SPARK2 (which uses inheritance) is equivalent to the flattening of /api/v1/stacks/HDP/versions/3.0/services/SPARK .
Please take a look at how this was done for ZK, HDFS, and YARN/MR.
This means that you will not be able to actually install the service for now, but can still perform validation during the Install Wizard that the correct components and configs show up.


Diffs
-----

  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/alerts.json PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/livy-conf.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/livy-env.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/livy-log4j-properties.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/livy-spark-blacklist.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-defaults.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-env.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-hive-site-override.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-log4j-properties.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-logsearch-conf.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-metrics-properties.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-thrift-fairscheduler.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-thrift-sparkconf.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/kerberos.json PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/metainfo.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/quicklinks/quicklinks.json PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/job_history_server.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/livy_server.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/livy_service.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/params.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/service_check.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/setup_livy.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/setup_spark.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/spark_client.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/spark_service.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/spark_thrift_server.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/status_params.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/3.0/services/SPARK/metainfo.xml PRE-CREATION 


Diff: https://reviews.apache.org/r/57692/diff/1/


Testing
-------

Unit tests passed


Thanks,

Dmytro Sen


Re: Review Request 57692: HDP 3.0 support for Spark with configs, kerberos, widgets, metrics, quicklinks, and themes

Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57692/#review169226
-----------------------------------------------------------


Fix it, then Ship it!





ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/alerts.json
Lines 1 (patched)
<https://reviews.apache.org/r/57692/#comment241565>

    Why are we calling this SPARK_REBRANDED?
    Let's just go with SPARK.



ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-defaults.xml
Lines 42 (patched)
<https://reviews.apache.org/r/57692/#comment241566>

    Let's set all of these to <on-ambari-upgrade add="false" />


- Alejandro Fernandez


On March 16, 2017, 4:01 p.m., Dmytro Sen wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57692/
> -----------------------------------------------------------
> 
> (Updated March 16, 2017, 4:01 p.m.)
> 
> 
> Review request for Ambari, Alejandro Fernandez, Sid Wagle, and Vitalyi Brodetskyi.
> 
> 
> Bugs: AMBARI-19834
>     https://issues.apache.org/jira/browse/AMBARI-19834
> 
> 
> Repository: ambari
> 
> 
> Description
> -------
> 
> HDP 3.0 support for Spark with configs, kerberos, widgets, metrics, quicklinks, and themes
> Flatten from HDP 2.0.6 - 2.6 into common-services, and reference in HDP 3.0
> IMPORTANT: In HDP 3.0, Spark1 is being removed, and Spark2 is being rebranded as "Spark"
> In HDP 3.0, we have created a new stack definition that does not inherit from other stacks, in order to reduce the complexity of having to analyze older stacks.
> This means that we need to create a service definition (metainfo.xml, configs, kerberos, widgets, metrics, quicklinks, and themes) that is equivalent to what is inherit and deleted from all of the previous stacks.
> A merge needs to account for additions, overrides, and deletions.
> metainfo.xml and configs perform a merge of older versions
> kerberos.json always seems to override the previous file
> Because the bits for this service may not yet be available in the HDP 3.0 repo, 
> the task is to ensure that /api/v1/stacks/HDP/versions/2.6/services/SPARK2 (which uses inheritance) is equivalent to the flattening of /api/v1/stacks/HDP/versions/3.0/services/SPARK .
> Please take a look at how this was done for ZK, HDFS, and YARN/MR.
> This means that you will not be able to actually install the service for now, but can still perform validation during the Install Wizard that the correct components and configs show up.
> 
> 
> Diffs
> -----
> 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/alerts.json PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/livy-conf.xml PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/livy-env.xml PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/livy-log4j-properties.xml PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/livy-spark-blacklist.xml PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-defaults.xml PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-env.xml PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-hive-site-override.xml PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-log4j-properties.xml PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-logsearch-conf.xml PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-metrics-properties.xml PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-thrift-fairscheduler.xml PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/configuration/spark-thrift-sparkconf.xml PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/kerberos.json PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/metainfo.xml PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/quicklinks/quicklinks.json PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/job_history_server.py PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/livy_server.py PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/livy_service.py PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/params.py PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/service_check.py PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/setup_livy.py PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/setup_spark.py PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/spark_client.py PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/spark_service.py PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/spark_thrift_server.py PRE-CREATION 
>   ambari-server/src/main/resources/common-services/SPARK_REBRANDED/2.2.0.3.0/scripts/status_params.py PRE-CREATION 
>   ambari-server/src/main/resources/stacks/HDP/3.0/services/SPARK/metainfo.xml PRE-CREATION 
> 
> 
> Diff: https://reviews.apache.org/r/57692/diff/1/
> 
> 
> Testing
> -------
> 
> Unit tests passed
> 
> 
> Thanks,
> 
> Dmytro Sen
> 
>


Re: Review Request 57692: HDP 3.0 support for Spark with configs, kerberos, widgets, metrics, quicklinks, and themes

Posted by Dmytro Sen <ds...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57692/
-----------------------------------------------------------

(Updated \u041c\u0430\u0440\u0442 17, 2017, 11:25 \u0434.\u043f.)


Review request for Ambari, Alejandro Fernandez, Sid Wagle, and Vitalyi Brodetskyi.


Bugs: AMBARI-19834
    https://issues.apache.org/jira/browse/AMBARI-19834


Repository: ambari


Description
-------

HDP 3.0 support for Spark with configs, kerberos, widgets, metrics, quicklinks, and themes
Flatten from HDP 2.0.6 - 2.6 into common-services, and reference in HDP 3.0
IMPORTANT: In HDP 3.0, Spark1 is being removed, and Spark2 is being rebranded as "Spark"
In HDP 3.0, we have created a new stack definition that does not inherit from other stacks, in order to reduce the complexity of having to analyze older stacks.
This means that we need to create a service definition (metainfo.xml, configs, kerberos, widgets, metrics, quicklinks, and themes) that is equivalent to what is inherit and deleted from all of the previous stacks.
A merge needs to account for additions, overrides, and deletions.
metainfo.xml and configs perform a merge of older versions
kerberos.json always seems to override the previous file
Because the bits for this service may not yet be available in the HDP 3.0 repo, 
the task is to ensure that /api/v1/stacks/HDP/versions/2.6/services/SPARK2 (which uses inheritance) is equivalent to the flattening of /api/v1/stacks/HDP/versions/3.0/services/SPARK .
Please take a look at how this was done for ZK, HDFS, and YARN/MR.
This means that you will not be able to actually install the service for now, but can still perform validation during the Install Wizard that the correct components and configs show up.


Diffs (updated)
-----

  ambari-server/src/main/resources/common-services/SPARK/2.2.0/alerts.json PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/configuration/livy-conf.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/configuration/livy-env.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/configuration/livy-log4j-properties.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/configuration/livy-spark-blacklist.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/configuration/spark-defaults.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/configuration/spark-env.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/configuration/spark-hive-site-override.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/configuration/spark-log4j-properties.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/configuration/spark-logsearch-conf.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/configuration/spark-metrics-properties.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/configuration/spark-thrift-fairscheduler.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/configuration/spark-thrift-sparkconf.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/kerberos.json PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/metainfo.xml PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/quicklinks/quicklinks.json PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/scripts/job_history_server.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/scripts/livy_server.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/scripts/livy_service.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/scripts/params.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/scripts/service_check.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/scripts/setup_livy.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/scripts/setup_spark.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/scripts/spark_client.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/scripts/spark_service.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/scripts/spark_thrift_server.py PRE-CREATION 
  ambari-server/src/main/resources/common-services/SPARK/2.2.0/scripts/status_params.py PRE-CREATION 
  ambari-server/src/main/resources/stacks/HDP/3.0/services/SPARK/metainfo.xml PRE-CREATION 


Diff: https://reviews.apache.org/r/57692/diff/2/

Changes: https://reviews.apache.org/r/57692/diff/1-2/


Testing
-------

Unit tests passed


Thanks,

Dmytro Sen