You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@ambari.apache.org by Saisai Shao <sa...@gmail.com> on 2015/11/20 08:34:36 UTC
Review Request 40530: [AMBARI-13983] Change Spark TS definition to
make it work under secure and non-secure environment
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40530/
-----------------------------------------------------------
Review request for Ambari and Sumit Mohanty.
Repository: ambari
Description
-------
[AMBARI-13983] Change Spark TS definition to make it work under secure and non-secure environment
Diffs
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 985c2e3
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py debed40
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py b274c61
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py bb0d35f
ambari-server/src/main/resources/common-services/SPARK/1.4.1.2.3/kerberos.json 3868a62
ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/configuration/spark-hive-site-override.xml 2de64c5
ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 90d3f0a
ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 798feb8
ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 05745ae
Diff: https://reviews.apache.org/r/40530/diff/
Testing
-------
Local testing is done.
Thanks,
Saisai Shao
Re: Review Request 40530: [AMBARI-13983] Change Spark TS definition
to make it work under secure and non-secure environment
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40530/#review107506
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/configuration/spark-hive-site-override.xml (line 23)
<https://reviews.apache.org/r/40530/#comment166650>
I think this is also used by HS2 under some condition. You can search all .xml files under ambari-server/src/main/resources to see if a port number is used by any services - as known to Ambari
- Sumit Mohanty
On Nov. 22, 2015, 10:04 a.m., Saisai Shao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40530/
> -----------------------------------------------------------
>
> (Updated Nov. 22, 2015, 10:04 a.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> [AMBARI-13983] Change Spark TS definition to make it work under secure and non-secure environment
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 985c2e3
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py debed40
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py b274c61
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py bb0d35f
> ambari-server/src/main/resources/common-services/SPARK/1.4.1.2.3/kerberos.json 3868a62
> ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/configuration/spark-hive-site-override.xml 2de64c5
> ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 90d3f0a
> ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 798feb8
> ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 05745ae
>
> Diff: https://reviews.apache.org/r/40530/diff/
>
>
> Testing
> -------
>
> Local testing is done.
>
>
> Thanks,
>
> Saisai Shao
>
>
Re: Review Request 40530: [AMBARI-13983] Change Spark TS definition
to make it work under secure and non-secure environment
Posted by Robert Levas <rl...@hortonworks.com>.
> On Nov. 22, 2015, 6:15 a.m., Robert Levas wrote:
> > I am not totoally convinced that this is the correct solution - making Spark runs a Hive - but I am no Hive expert, so I will need to take your word on that this is ok.
Correction: "Spark runs a Hive" --> "Spark runs as Hive"
- Robert
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40530/#review107501
-----------------------------------------------------------
On Nov. 22, 2015, 5:04 a.m., Saisai Shao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40530/
> -----------------------------------------------------------
>
> (Updated Nov. 22, 2015, 5:04 a.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> [AMBARI-13983] Change Spark TS definition to make it work under secure and non-secure environment
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 985c2e3
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py debed40
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py b274c61
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py bb0d35f
> ambari-server/src/main/resources/common-services/SPARK/1.4.1.2.3/kerberos.json 3868a62
> ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/configuration/spark-hive-site-override.xml 2de64c5
> ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 90d3f0a
> ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 798feb8
> ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 05745ae
>
> Diff: https://reviews.apache.org/r/40530/diff/
>
>
> Testing
> -------
>
> Local testing is done.
>
>
> Thanks,
>
> Saisai Shao
>
>
Re: Review Request 40530: [AMBARI-13983] Change Spark TS definition
to make it work under secure and non-secure environment
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40530/#review107501
-----------------------------------------------------------
Ship it!
I am not totoally convinced that this is the correct solution - making Spark runs a Hive - but I am no Hive expert, so I will need to take your word on that this is ok.
- Robert Levas
On Nov. 22, 2015, 5:04 a.m., Saisai Shao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40530/
> -----------------------------------------------------------
>
> (Updated Nov. 22, 2015, 5:04 a.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> [AMBARI-13983] Change Spark TS definition to make it work under secure and non-secure environment
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 985c2e3
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py debed40
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py b274c61
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py bb0d35f
> ambari-server/src/main/resources/common-services/SPARK/1.4.1.2.3/kerberos.json 3868a62
> ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/configuration/spark-hive-site-override.xml 2de64c5
> ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 90d3f0a
> ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 798feb8
> ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 05745ae
>
> Diff: https://reviews.apache.org/r/40530/diff/
>
>
> Testing
> -------
>
> Local testing is done.
>
>
> Thanks,
>
> Saisai Shao
>
>
Re: Review Request 40530: [AMBARI-13983] Change Spark TS definition
to make it work under secure and non-secure environment
Posted by Sumit Mohanty <sm...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40530/#review107534
-----------------------------------------------------------
Ship it!
Ship It!
- Sumit Mohanty
On Nov. 23, 2015, 12:59 a.m., Saisai Shao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40530/
> -----------------------------------------------------------
>
> (Updated Nov. 23, 2015, 12:59 a.m.)
>
>
> Review request for Ambari, Robert Levas and Sumit Mohanty.
>
>
> Bugs: AMBARI-13983
> https://issues.apache.org/jira/browse/AMBARI-13983
>
>
> Repository: ambari
>
>
> Description
> -------
>
> [AMBARI-13983] Change Spark TS definition to make it work under secure and non-secure environment
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 985c2e3
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py debed40
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py b274c61
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py bb0d35f
> ambari-server/src/main/resources/common-services/SPARK/1.4.1.2.3/kerberos.json 3868a62
> ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/configuration/spark-hive-site-override.xml 2de64c5
> ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 90d3f0a
> ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 798feb8
> ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 05745ae
>
> Diff: https://reviews.apache.org/r/40530/diff/
>
>
> Testing
> -------
>
> Local testing is done.
>
>
> Thanks,
>
> Saisai Shao
>
>
Re: Review Request 40530: [AMBARI-13983] Change Spark TS definition
to make it work under secure and non-secure environment
Posted by Saisai Shao <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40530/
-----------------------------------------------------------
(Updated Nov. 23, 2015, 12:59 a.m.)
Review request for Ambari, Robert Levas and Sumit Mohanty.
Changes
-------
Change the SparkTS default port number
Bugs: AMBARI-13983
https://issues.apache.org/jira/browse/AMBARI-13983
Repository: ambari
Description
-------
[AMBARI-13983] Change Spark TS definition to make it work under secure and non-secure environment
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 985c2e3
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py debed40
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py b274c61
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py bb0d35f
ambari-server/src/main/resources/common-services/SPARK/1.4.1.2.3/kerberos.json 3868a62
ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/configuration/spark-hive-site-override.xml 2de64c5
ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 90d3f0a
ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 798feb8
ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 05745ae
Diff: https://reviews.apache.org/r/40530/diff/
Testing
-------
Local testing is done.
Thanks,
Saisai Shao
Re: Review Request 40530: [AMBARI-13983] Change Spark TS definition
to make it work under secure and non-secure environment
Posted by Saisai Shao <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40530/
-----------------------------------------------------------
(Updated Nov. 23, 2015, 12:18 a.m.)
Review request for Ambari, Robert Levas and Sumit Mohanty.
Bugs: AMBARI-13983
https://issues.apache.org/jira/browse/AMBARI-13983
Repository: ambari
Description
-------
[AMBARI-13983] Change Spark TS definition to make it work under secure and non-secure environment
Diffs
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 985c2e3
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py debed40
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py b274c61
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py bb0d35f
ambari-server/src/main/resources/common-services/SPARK/1.4.1.2.3/kerberos.json 3868a62
ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/configuration/spark-hive-site-override.xml 2de64c5
ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 90d3f0a
ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 798feb8
ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 05745ae
Diff: https://reviews.apache.org/r/40530/diff/
Testing
-------
Local testing is done.
Thanks,
Saisai Shao
Re: Review Request 40530: [AMBARI-13983] Change Spark TS definition
to make it work under secure and non-secure environment
Posted by Saisai Shao <sa...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40530/
-----------------------------------------------------------
(Updated Nov. 22, 2015, 10:04 a.m.)
Review request for Ambari and Sumit Mohanty.
Changes
-------
make fqdn lower case
Repository: ambari
Description
-------
[AMBARI-13983] Change Spark TS definition to make it work under secure and non-secure environment
Diffs (updated)
-----
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 985c2e3
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py debed40
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py b274c61
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py bb0d35f
ambari-server/src/main/resources/common-services/SPARK/1.4.1.2.3/kerberos.json 3868a62
ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/configuration/spark-hive-site-override.xml 2de64c5
ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 90d3f0a
ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 798feb8
ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 05745ae
Diff: https://reviews.apache.org/r/40530/diff/
Testing
-------
Local testing is done.
Thanks,
Saisai Shao
Re: Review Request 40530: [AMBARI-13983] Change Spark TS definition
to make it work under secure and non-secure environment
Posted by Saisai Shao <sa...@gmail.com>.
> On Nov. 20, 2015, 10:54 a.m., Robert Levas wrote:
> > ambari-server/src/main/resources/common-services/SPARK/1.4.1.2.3/kerberos.json, lines 56-58
> > <https://reviews.apache.org/r/40530/diff/1/?file=1133969#file1133969line56>
> >
> > Spark should be using it own Kerberos identity not Hive's, unless there is a really good reason for it.
Because we are using hive account to start Spark TS, so from my understanding using hive identity is the only choice.
- Saisai
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40530/#review107346
-----------------------------------------------------------
On Nov. 20, 2015, 7:34 a.m., Saisai Shao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40530/
> -----------------------------------------------------------
>
> (Updated Nov. 20, 2015, 7:34 a.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> [AMBARI-13983] Change Spark TS definition to make it work under secure and non-secure environment
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 985c2e3
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py debed40
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py b274c61
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py bb0d35f
> ambari-server/src/main/resources/common-services/SPARK/1.4.1.2.3/kerberos.json 3868a62
> ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/configuration/spark-hive-site-override.xml 2de64c5
> ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 90d3f0a
> ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 798feb8
> ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 05745ae
>
> Diff: https://reviews.apache.org/r/40530/diff/
>
>
> Testing
> -------
>
> Local testing is done.
>
>
> Thanks,
>
> Saisai Shao
>
>
Re: Review Request 40530: [AMBARI-13983] Change Spark TS definition
to make it work under secure and non-secure environment
Posted by Robert Levas <rl...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/40530/#review107346
-----------------------------------------------------------
ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py (line 54)
<https://reviews.apache.org/r/40530/#comment166439>
Typically we use the hostname set in the configuration:
```
hostname = config["hostname"].lower()
```
But maybe socket.getfqdn is better?
To be consistent, you should probably force this value to be lowercase.
ambari-server/src/main/resources/common-services/SPARK/1.4.1.2.3/kerberos.json (lines 56 - 58)
<https://reviews.apache.org/r/40530/#comment166437>
Spark should be using it own Kerberos identity not Hive's, unless there is a really good reason for it.
- Robert Levas
On Nov. 20, 2015, 2:34 a.m., Saisai Shao wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/40530/
> -----------------------------------------------------------
>
> (Updated Nov. 20, 2015, 2:34 a.m.)
>
>
> Review request for Ambari and Sumit Mohanty.
>
>
> Repository: ambari
>
>
> Description
> -------
>
> [AMBARI-13983] Change Spark TS definition to make it work under secure and non-secure environment
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/params.py 985c2e3
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/setup_spark.py debed40
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/spark_service.py b274c61
> ambari-server/src/main/resources/common-services/SPARK/1.2.0.2.2/package/scripts/status_params.py bb0d35f
> ambari-server/src/main/resources/common-services/SPARK/1.4.1.2.3/kerberos.json 3868a62
> ambari-server/src/main/resources/stacks/HDP/2.3/services/SPARK/configuration/spark-hive-site-override.xml 2de64c5
> ambari-server/src/test/python/stacks/2.2/SPARK/test_job_history_server.py 90d3f0a
> ambari-server/src/test/python/stacks/2.2/SPARK/test_spark_client.py 798feb8
> ambari-server/src/test/python/stacks/2.3/SPARK/test_spark_thrift_server.py 05745ae
>
> Diff: https://reviews.apache.org/r/40530/diff/
>
>
> Testing
> -------
>
> Local testing is done.
>
>
> Thanks,
>
> Saisai Shao
>
>