You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@ambari.apache.org by Di Li <di...@ca.ibm.com> on 2017/07/12 21:00:35 UTC
Review Request 60814: AMBARI-21461 Spark thrift server restart fails
during migration EU
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60814/
-----------------------------------------------------------
Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
Bugs: AMBARI-21461
https://issues.apache.org/jira/browse/AMBARI-21461
Repository: ambari
Description
-------
Spark in IOP 4.2 BigInsights stack uses script name "thrift_server.py" instead of "spark_thrift_server.py" used by the common service Spark and the IOP 4.2.5 BigInsights stack.
This seemed to have caused the following error during migration EU where Spark thrift server failed to restart.
"""
Caught an exception while executing custom service command: <class 'ambari_agent.AgentException.AgentException'>: 'Script /var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/thrift_server.py does not exist'; 'Script /var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/thrift_server.py does not exist'
"""
We should be able to fix it by renaming "thrift_server.py" used in BI 4.2 stack to "spark_thrift_server.py" and updating metainfo.xml files accordingly.
Diffs
-----
ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/metainfo.xml 02abc62
ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/package/scripts/spark_thrift_server.py PRE-CREATION
ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/package/scripts/thrift_server.py 39e15d3
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/metainfo.xml 7b6bae1
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/package/scripts/spark_thrift_server.py PRE-CREATION
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/package/scripts/thrift_server.py d1a8b67
Diff: https://reviews.apache.org/r/60814/diff/1/
Testing
-------
to be tested in integration tests
Thanks,
Di Li
Re: Review Request 60814: AMBARI-21461 Spark thrift server restart
fails during migration EU
Posted by Tim Thorpe <tt...@ca.ibm.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60814/#review180426
-----------------------------------------------------------
Ship it!
Ship It!
- Tim Thorpe
On July 12, 2017, 9 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60814/
> -----------------------------------------------------------
>
> (Updated July 12, 2017, 9 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21461
> https://issues.apache.org/jira/browse/AMBARI-21461
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Spark in IOP 4.2 BigInsights stack uses script name "thrift_server.py" instead of "spark_thrift_server.py" used by the common service Spark and the IOP 4.2.5 BigInsights stack.
> This seemed to have caused the following error during migration EU where Spark thrift server failed to restart.
> """
> Caught an exception while executing custom service command: <class 'ambari_agent.AgentException.AgentException'>: 'Script /var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/thrift_server.py does not exist'; 'Script /var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/thrift_server.py does not exist'
> """
> We should be able to fix it by renaming "thrift_server.py" used in BI 4.2 stack to "spark_thrift_server.py" and updating metainfo.xml files accordingly.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/metainfo.xml 02abc62
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/package/scripts/spark_thrift_server.py PRE-CREATION
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/package/scripts/thrift_server.py 39e15d3
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/metainfo.xml 7b6bae1
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/package/scripts/spark_thrift_server.py PRE-CREATION
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/package/scripts/thrift_server.py d1a8b67
>
>
> Diff: https://reviews.apache.org/r/60814/diff/1/
>
>
> Testing
> -------
>
> to be tested in integration tests
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 60814: AMBARI-21461 Spark thrift server restart
fails during migration EU
Posted by Di Li <di...@ca.ibm.com>.
> On July 12, 2017, 9:59 p.m., Alejandro Fernandez wrote:
> >
All I am doing is to rename the file in BI 4.0 and 4.2 stack to make it the same name as common services and BI 4.2.5. This is a quick fix for EU error. I am not reimplementing the script as that's not what this JIRA is about. The script already exists on an IOP 4.2 GA cluster. The cluster would have been bad if the script did not function. The piont here is the file names need to be consistent >> the real issue, thus the patch I proposed.
"""
Caught an exception while executing custom service command: <class 'ambari_agent.AgentException.AgentException'>: 'Script /var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/thrift_server.py does not exist'; 'Script /var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/thrift_server.py does not exist'
"""
- Di
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60814/#review180361
-----------------------------------------------------------
On July 12, 2017, 9 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60814/
> -----------------------------------------------------------
>
> (Updated July 12, 2017, 9 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21461
> https://issues.apache.org/jira/browse/AMBARI-21461
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Spark in IOP 4.2 BigInsights stack uses script name "thrift_server.py" instead of "spark_thrift_server.py" used by the common service Spark and the IOP 4.2.5 BigInsights stack.
> This seemed to have caused the following error during migration EU where Spark thrift server failed to restart.
> """
> Caught an exception while executing custom service command: <class 'ambari_agent.AgentException.AgentException'>: 'Script /var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/thrift_server.py does not exist'; 'Script /var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/thrift_server.py does not exist'
> """
> We should be able to fix it by renaming "thrift_server.py" used in BI 4.2 stack to "spark_thrift_server.py" and updating metainfo.xml files accordingly.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/metainfo.xml 02abc62
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/package/scripts/spark_thrift_server.py PRE-CREATION
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/package/scripts/thrift_server.py 39e15d3
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/metainfo.xml 7b6bae1
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/package/scripts/spark_thrift_server.py PRE-CREATION
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/package/scripts/thrift_server.py d1a8b67
>
>
> Diff: https://reviews.apache.org/r/60814/diff/1/
>
>
> Testing
> -------
>
> to be tested in integration tests
>
>
> Thanks,
>
> Di Li
>
>
Re: Review Request 60814: AMBARI-21461 Spark thrift server restart
fails during migration EU
Posted by Alejandro Fernandez <af...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60814/#review180361
-----------------------------------------------------------
ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/package/scripts/spark_thrift_server.py
Lines 24 (patched)
<https://reviews.apache.org/r/60814/#comment255509>
Import * has been known to cause a lot of problems on certain environments
ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/package/scripts/spark_thrift_server.py
Lines 44 (patched)
<https://reviews.apache.org/r/60814/#comment255510>
Use stack features.json
ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/package/scripts/spark_thrift_server.py
Lines 82 (patched)
<https://reviews.apache.org/r/60814/#comment255511>
If this config is not defined, it needs a default value.
ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/package/scripts/spark_thrift_server.py
Lines 83 (patched)
<https://reviews.apache.org/r/60814/#comment255512>
I believe the hostname case may matter in the kinit command.
ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/package/scripts/spark_thrift_server.py
Lines 24 (patched)
<https://reviews.apache.org/r/60814/#comment255513>
Don't use imort *
- Alejandro Fernandez
On July 12, 2017, 9 p.m., Di Li wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60814/
> -----------------------------------------------------------
>
> (Updated July 12, 2017, 9 p.m.)
>
>
> Review request for Ambari, Dmytro Grinenko, Di Li, Dmitro Lisnichenko, Jonathan Hurley, Sumit Mohanty, Sid Wagle, and Tim Thorpe.
>
>
> Bugs: AMBARI-21461
> https://issues.apache.org/jira/browse/AMBARI-21461
>
>
> Repository: ambari
>
>
> Description
> -------
>
> Spark in IOP 4.2 BigInsights stack uses script name "thrift_server.py" instead of "spark_thrift_server.py" used by the common service Spark and the IOP 4.2.5 BigInsights stack.
> This seemed to have caused the following error during migration EU where Spark thrift server failed to restart.
> """
> Caught an exception while executing custom service command: <class 'ambari_agent.AgentException.AgentException'>: 'Script /var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/thrift_server.py does not exist'; 'Script /var/lib/ambari-agent/cache/common-services/SPARK/1.2.1/package/scripts/thrift_server.py does not exist'
> """
> We should be able to fix it by renaming "thrift_server.py" used in BI 4.2 stack to "spark_thrift_server.py" and updating metainfo.xml files accordingly.
>
>
> Diffs
> -----
>
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/metainfo.xml 02abc62
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/package/scripts/spark_thrift_server.py PRE-CREATION
> ambari-server/src/main/resources/stacks/BigInsights/4.0/services/SPARK/package/scripts/thrift_server.py 39e15d3
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/metainfo.xml 7b6bae1
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/package/scripts/spark_thrift_server.py PRE-CREATION
> ambari-server/src/main/resources/stacks/BigInsights/4.2/services/SPARK/package/scripts/thrift_server.py d1a8b67
>
>
> Diff: https://reviews.apache.org/r/60814/diff/1/
>
>
> Testing
> -------
>
> to be tested in integration tests
>
>
> Thanks,
>
> Di Li
>
>