You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by radoburansky <ra...@gmail.com> on 2016/01/18 21:13:06 UTC
Number of CPU cores for a Spark Streaming app in Standalone mode
I somehow don't want to believe this waste of resources. Is it really true
that if I have 20 input streams I must have at least 21 CPU cores? Even if I
read only once per minute and only a few messages? I still hope that I miss
an important information. Thanks a lot
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Number-of-CPU-cores-for-a-Spark-Streaming-app-in-Standalone-mode-tp25997.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Number of CPU cores for a Spark Streaming app in Standalone
mode
Posted by radoburansky <ra...@gmail.com>.
I am adding an answer from SO:
http://stackoverflow.com/questions/34861947/read-more-kafka-topics-than-number-of-cpu-cores
<http://stackoverflow.com/questions/34861947/read-more-kafka-topics-than-number-of-cpu-cores>
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Number-of-CPU-cores-for-a-Spark-Streaming-app-in-Standalone-mode-tp25997p26001.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Number of CPU cores for a Spark Streaming app in Standalone mode
Posted by Tathagata Das <td...@databricks.com>.
If you are using receiver-based input streams, then you have to dedicate 1
core to each receiver. If you read only once per minute on each receiver,
than consider consolidating the data reading pipeline such that you can use
fewer receivers.
On Mon, Jan 18, 2016 at 12:13 PM, radoburansky <ra...@gmail.com>
wrote:
> I somehow don't want to believe this waste of resources. Is it really true
> that if I have 20 input streams I must have at least 21 CPU cores? Even if
> I
> read only once per minute and only a few messages? I still hope that I miss
> an important information. Thanks a lot
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Number-of-CPU-cores-for-a-Spark-Streaming-app-in-Standalone-mode-tp25997.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>