You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Rahul Tanwani <ta...@gmail.com> on 2016/04/17 19:27:49 UTC
Impact of STW GC events for the driver JVM on overall cluster
Hi Devs,
In case of stop the world GC events on the driver JVM, since all the
application threads will be stopped, there won't be any new task scheduled /
launched on the executors. In cases where the full collection is happening,
the applications threads may be stopped for a long time, and if the running
tasks are small, entire cluster will be idle till the GC is over and
application threads are resumed.
Is my understanding in this regard correct?
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Impact-of-STW-GC-events-for-the-driver-JVM-on-overall-cluster-tp17183.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Impact of STW GC events for the driver JVM on overall cluster
Posted by Rahul Tanwani <ta...@gmail.com>.
Does that not mean, GC settings with concurrent collectors should be
preferred over parallel collectors atleast on the driver side? If so, why
not have concurrent collectors specified by default when the driver JVM is
launched without any overriding on this part?
--
View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Impact-of-STW-GC-events-for-the-driver-JVM-on-overall-cluster-tp17183p17186.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
For additional commands, e-mail: dev-help@spark.apache.org
Re: Impact of STW GC events for the driver JVM on overall cluster
Posted by Reynold Xin <rx...@databricks.com>.
Your understanding is correct. If the driver is stuck in GC, then during
that period it cannot schedule any tasks.
On Sun, Apr 17, 2016 at 10:27 AM, Rahul Tanwani <ta...@gmail.com>
wrote:
> Hi Devs,
>
> In case of stop the world GC events on the driver JVM, since all the
> application threads will be stopped, there won't be any new task scheduled
> /
> launched on the executors. In cases where the full collection is happening,
> the applications threads may be stopped for a long time, and if the running
> tasks are small, entire cluster will be idle till the GC is over and
> application threads are resumed.
>
> Is my understanding in this regard correct?
>
>
>
> --
> View this message in context:
> http://apache-spark-developers-list.1001551.n3.nabble.com/Impact-of-STW-GC-events-for-the-driver-JVM-on-overall-cluster-tp17183.html
> Sent from the Apache Spark Developers List mailing list archive at
> Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@spark.apache.org
> For additional commands, e-mail: dev-help@spark.apache.org
>
>