You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by emiretsk <eu...@gmail.com> on 2016/07/01 15:42:32 UTC
How are threads created in SQL Executor?
It seems like threads are created by SQLExecution.withExecutionId, which is
called inside BroadcastExchangeExec.scala.
When does the plan executor execute a BroadcaseExchange, and is there a way
to control the number of threads? We have a job that writes DataFrames to an
external DB, and it seems like each task creates as many threads as there
are cores.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-are-threads-created-in-SQL-Executor-tp27260.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org
Re: How are threads created in SQL Executor?
Posted by Takeshi Yamamuro <li...@gmail.com>.
You mean `spark.sql.shuffle.partitions`?
http://spark.apache.org/docs/latest/sql-programming-guide.html#other-configuration-options
// maropu
On Fri, Jul 1, 2016 at 8:42 AM, emiretsk <eu...@gmail.com> wrote:
> It seems like threads are created by SQLExecution.withExecutionId, which is
> called inside BroadcastExchangeExec.scala.
> When does the plan executor execute a BroadcaseExchange, and is there a way
> to control the number of threads? We have a job that writes DataFrames to
> an
> external DB, and it seems like each task creates as many threads as there
> are cores.
>
>
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/How-are-threads-created-in-SQL-Executor-tp27260.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: user-unsubscribe@spark.apache.org
>
>
--
---
Takeshi Yamamuro