You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@hive.apache.org by Tao Li <tl...@hortonworks.com> on 2016/08/23 00:53:16 UTC

Question about Hive on Spark

Hi,

Looks like the Spark client (SparkClientImpl class) submits Sparks jobs to the YARN cluster by forking a process and kicking off spark-submit script. Are we provisioning new containers every time we submit a job? There could be a perf hit by doing that.

Thanks.

Re: Question about Hive on Spark

Posted by Xuefu Zhang <xu...@uber.com>.

That happens per session or if certain configs are changed in the session.

On Mon, Aug 22, 2016 at 5:53 PM, Tao Li <tl...@hortonworks.com> wrote:

> Hi,
>
> Looks like the Spark client (SparkClientImpl class) submits Sparks jobs to
> the YARN cluster by forking a process and kicking off spark-submit script.
> Are we provisioning new containers every time we submit a job? There could
> be a perf hit by doing that.
>
> Thanks.
>