You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Jeff Zhang (JIRA)" <ji...@apache.org> on 2016/11/26 01:02:39 UTC
[jira] [Comment Edited] (SPARK-18405) Add yarn-cluster mode support
to Spark Thrift Server
[ https://issues.apache.org/jira/browse/SPARK-18405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15696978#comment-15696978 ]
Jeff Zhang edited comment on SPARK-18405 at 11/26/16 1:01 AM:
--------------------------------------------------------------
I think he mean to launch multiple spark thrift server in yarn-cluster mode so each we can evenly distribute workload across them(e.g. one spark thrift server per department). And it is much easier to request large container for AM in yarn-cluster while the the memory of the host in yarn-client mode may be limited and without capacity control under yarn.
was (Author: zjffdu):
I think he mean to launch multiple spark thrift server in yarn-cluster mode so each we can evenly distribute workload across them. And it is much easier to request large container for AM in yarn-cluster while the the memory of the host in yarn-client mode may be limited and without capacity control under yarn.
> Add yarn-cluster mode support to Spark Thrift Server
> ----------------------------------------------------
>
> Key: SPARK-18405
> URL: https://issues.apache.org/jira/browse/SPARK-18405
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.6.0, 1.6.2, 2.0.0, 2.0.1
> Reporter: Prabhu Kasinathan
> Labels: Spark, ThriftServer2
>
> Currently, spark thrift server can run only on yarn-client mode.
> Can we add Yarn-Cluster mode support to spark thrift server?
> This will help us to launch multiple spark thrift server with different spark configurations and it really help in large distributed clusters where there is requirement to run complex sqls through STS. With client mode, there is a chance to overload local host with too much driver memory.
> Please let me know your thoughts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org