You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Egor Pahomov <pa...@gmail.com> on 2014/11/14 20:32:51 UTC
Elastic allocation(spark.dynamicAllocation.enabled) results in task
never being executed.
Hi.
I execute ipython notebook + pyspark with spark.dynamicAllocation.enabled =
true. Task never ends.
Code:
import sys
from random import random
from operator import add
partitions = 10
n = 100000 * partitions
def f(_):
x = random() * 2 - 1
y = random() * 2 - 1
return 1 if x ** 2 + y ** 2 < 1 else 0
count = sc.parallelize(xrange(1, n + 1), partitions).map(f).reduce(add)
print "Pi is roughly %f" % (4.0 * count / n)
Run notebook:
IPYTHON_ARGS="notebook --profile=ydf --port $IPYTHON_PORT
--port-retries=0 --ip='*' --no-browser"
pyspark \
--verbose \
--master yarn-client \
--conf spark.driver.port=$((RANDOM_PORT + 2)) \
--conf spark.broadcast.port=$((RANDOM_PORT + 3)) \
--conf spark.replClassServer.port=$((RANDOM_PORT + 4)) \
--conf spark.blockManager.port=$((RANDOM_PORT + 5)) \
--conf spark.executor.port=$((RANDOM_PORT + 6)) \
--conf spark.fileserver.port=$((RANDOM_PORT + 7)) \
--conf spark.shuffle.service.enabled=true \
--conf spark.dynamicAllocation.enabled=true \
--conf spark.dynamicAllocation.minExecutors=1 \
--conf spark.dynamicAllocation.maxExecutors=10 \
--conf spark.ui.port=$SPARK_UI_PORT
Spark/Ipython log is in attachment.
--
*Sincerely yoursEgor PakhomovScala Developer, Yandex*
Re: Elastic allocation(spark.dynamicAllocation.enabled) results in
task never being executed.
Posted by Dhiraj Peechara <dh...@gmail.com>.
Yes Andrew, I saw this page and got it working. I do have a question
though. I am manually adding spark-<version>-yarn-shuffle.jar to one of the
existing folders(yarn/lib/) which is already in the classpath on each
nodemanager. I am wondering what is the best way to add
spark-<version>-yarn-shuffle.jar
to the classpath on all the nodemanagers.
Thank you
On Wed, Jan 7, 2015 at 2:56 PM, Andrew Or <an...@databricks.com> wrote:
> Did you end up getting it working? By the way this might be a nicer view
> of the docs:
>
> https://github.com/apache/spark/blob/60e2d9e2902b132b14191c9791c71e8f0d42ce9d/docs/job-scheduling.md
>
> We will update the latest Spark docs to include this shortly.
> -Andrew
>
> 2015-01-04 4:44 GMT-08:00 Tsuyoshi Ozawa <oz...@apache.org>:
>
> Please check the document added by Andrew. I could run tasks with Spark
>> 1.2.0.
>>
>> *
>> https://github.com/apache/spark/pull/3731/files#diff-c3cbe4cabe90562520f22d2306aa9116R86
>> *
>> https://github.com/apache/spark/pull/3757/files#diff-c3cbe4cabe90562520f22d2306aa9116R101
>>
>> Thanks,
>> - Tsuyoshi
>>
>> On Sun, Jan 4, 2015 at 11:54 AM, firemonk9 <dh...@gmail.com>
>> wrote:
>> > I am running into similar problem. Have you found any resolution to this
>> > issue ?
>> >
>> >
>> >
>> > --
>> > View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Elastic-allocation-spark-dynamicAllocation-enabled-results-in-task-never-being-executed-tp18969p20957.html
>> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> > For additional commands, e-mail: user-help@spark.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Elastic allocation(spark.dynamicAllocation.enabled) results in
task never being executed.
Posted by Andrew Or <an...@databricks.com>.
Did you end up getting it working? By the way this might be a nicer view of
the docs:
https://github.com/apache/spark/blob/60e2d9e2902b132b14191c9791c71e8f0d42ce9d/docs/job-scheduling.md
We will update the latest Spark docs to include this shortly.
-Andrew
2015-01-04 4:44 GMT-08:00 Tsuyoshi Ozawa <oz...@apache.org>:
> Please check the document added by Andrew. I could run tasks with Spark
> 1.2.0.
>
> *
> https://github.com/apache/spark/pull/3731/files#diff-c3cbe4cabe90562520f22d2306aa9116R86
> *
> https://github.com/apache/spark/pull/3757/files#diff-c3cbe4cabe90562520f22d2306aa9116R101
>
> Thanks,
> - Tsuyoshi
>
> On Sun, Jan 4, 2015 at 11:54 AM, firemonk9 <dh...@gmail.com>
> wrote:
> > I am running into similar problem. Have you found any resolution to this
> > issue ?
> >
> >
> >
> > --
> > View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Elastic-allocation-spark-dynamicAllocation-enabled-results-in-task-never-being-executed-tp18969p20957.html
> > Sent from the Apache Spark User List mailing list archive at Nabble.com.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> > For additional commands, e-mail: user-help@spark.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: Elastic allocation(spark.dynamicAllocation.enabled) results in
task never being executed.
Posted by Tsuyoshi Ozawa <oz...@apache.org>.
Please check the document added by Andrew. I could run tasks with Spark 1.2.0.
* https://github.com/apache/spark/pull/3731/files#diff-c3cbe4cabe90562520f22d2306aa9116R86
* https://github.com/apache/spark/pull/3757/files#diff-c3cbe4cabe90562520f22d2306aa9116R101
Thanks,
- Tsuyoshi
On Sun, Jan 4, 2015 at 11:54 AM, firemonk9 <dh...@gmail.com> wrote:
> I am running into similar problem. Have you found any resolution to this
> issue ?
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Elastic-allocation-spark-dynamicAllocation-enabled-results-in-task-never-being-executed-tp18969p20957.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Elastic allocation(spark.dynamicAllocation.enabled) results in
task never being executed.
Posted by firemonk9 <dh...@gmail.com>.
I am running into similar problem. Have you found any resolution to this
issue ?
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Elastic-allocation-spark-dynamicAllocation-enabled-results-in-task-never-being-executed-tp18969p20957.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Elastic allocation(spark.dynamicAllocation.enabled) results in
task never being executed.
Posted by Andrew Or <an...@databricks.com>.
Hey Egor,
Have you checked the AM logs? My guess is that it threw an exception or
something such that no executors (not even the initial set) have registered
with your driver. You may already know this, but you can go to the http://<RM
address>:8088 page and click into the application to access this.
Alternatively you could run "yarn logs -applicationId <appId>" after
quitting the application.
-Andrew
2014-11-14 14:23 GMT-08:00 Sandy Ryza <sa...@cloudera.com>:
> That would be helpful as well. Can you confirm that when you try it with
> dynamic allocation the cluster has free resources?
>
> On Fri, Nov 14, 2014 at 12:17 PM, Egor Pahomov <pa...@gmail.com>
> wrote:
>
>> It's successful without dynamic allocation. I can provide spark log for
>> that scenario if it can help.
>>
>> 2014-11-14 21:36 GMT+02:00 Sandy Ryza <sa...@cloudera.com>:
>>
>>> Hi Egor,
>>>
>>> Is it successful without dynamic allocation? From your log, it looks
>>> like the job is unable to acquire resources from YARN, which could be
>>> because other jobs are using up all the resources.
>>>
>>> -Sandy
>>>
>>> On Fri, Nov 14, 2014 at 11:32 AM, Egor Pahomov <pa...@gmail.com>
>>> wrote:
>>>
>>>> Hi.
>>>> I execute ipython notebook + pyspark with
>>>> spark.dynamicAllocation.enabled = true. Task never ends.
>>>> Code:
>>>>
>>>> import sys
>>>> from random import random
>>>> from operator import add
>>>> partitions = 10
>>>> n = 100000 * partitions
>>>>
>>>> def f(_):
>>>> x = random() * 2 - 1
>>>> y = random() * 2 - 1
>>>> return 1 if x ** 2 + y ** 2 < 1 else 0
>>>>
>>>> count = sc.parallelize(xrange(1, n + 1), partitions).map(f).reduce(add)
>>>> print "Pi is roughly %f" % (4.0 * count / n)
>>>>
>>>>
>>>>
>>>> Run notebook:
>>>>
>>>> IPYTHON_ARGS="notebook --profile=ydf --port $IPYTHON_PORT --port-retries=0 --ip='*' --no-browser"
>>>> pyspark \
>>>> --verbose \
>>>> --master yarn-client \
>>>> --conf spark.driver.port=$((RANDOM_PORT + 2)) \
>>>> --conf spark.broadcast.port=$((RANDOM_PORT + 3)) \
>>>> --conf spark.replClassServer.port=$((RANDOM_PORT + 4)) \
>>>> --conf spark.blockManager.port=$((RANDOM_PORT + 5)) \
>>>> --conf spark.executor.port=$((RANDOM_PORT + 6)) \
>>>> --conf spark.fileserver.port=$((RANDOM_PORT + 7)) \
>>>> --conf spark.shuffle.service.enabled=true \
>>>> --conf spark.dynamicAllocation.enabled=true \
>>>> --conf spark.dynamicAllocation.minExecutors=1 \
>>>> --conf spark.dynamicAllocation.maxExecutors=10 \
>>>> --conf spark.ui.port=$SPARK_UI_PORT
>>>>
>>>>
>>>> Spark/Ipython log is in attachment.
>>>>
>>>> --
>>>>
>>>>
>>>>
>>>> *Sincerely yoursEgor PakhomovScala Developer, Yandex*
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>
>>>
>>
>>
>> --
>>
>>
>>
>> *Sincerely yoursEgor PakhomovScala Developer, Yandex*
>>
>
>
Re: Elastic allocation(spark.dynamicAllocation.enabled) results in
task never being executed.
Posted by Sandy Ryza <sa...@cloudera.com>.
That would be helpful as well. Can you confirm that when you try it with
dynamic allocation the cluster has free resources?
On Fri, Nov 14, 2014 at 12:17 PM, Egor Pahomov <pa...@gmail.com>
wrote:
> It's successful without dynamic allocation. I can provide spark log for
> that scenario if it can help.
>
> 2014-11-14 21:36 GMT+02:00 Sandy Ryza <sa...@cloudera.com>:
>
>> Hi Egor,
>>
>> Is it successful without dynamic allocation? From your log, it looks like
>> the job is unable to acquire resources from YARN, which could be because
>> other jobs are using up all the resources.
>>
>> -Sandy
>>
>> On Fri, Nov 14, 2014 at 11:32 AM, Egor Pahomov <pa...@gmail.com>
>> wrote:
>>
>>> Hi.
>>> I execute ipython notebook + pyspark with
>>> spark.dynamicAllocation.enabled = true. Task never ends.
>>> Code:
>>>
>>> import sys
>>> from random import random
>>> from operator import add
>>> partitions = 10
>>> n = 100000 * partitions
>>>
>>> def f(_):
>>> x = random() * 2 - 1
>>> y = random() * 2 - 1
>>> return 1 if x ** 2 + y ** 2 < 1 else 0
>>>
>>> count = sc.parallelize(xrange(1, n + 1), partitions).map(f).reduce(add)
>>> print "Pi is roughly %f" % (4.0 * count / n)
>>>
>>>
>>>
>>> Run notebook:
>>>
>>> IPYTHON_ARGS="notebook --profile=ydf --port $IPYTHON_PORT --port-retries=0 --ip='*' --no-browser"
>>> pyspark \
>>> --verbose \
>>> --master yarn-client \
>>> --conf spark.driver.port=$((RANDOM_PORT + 2)) \
>>> --conf spark.broadcast.port=$((RANDOM_PORT + 3)) \
>>> --conf spark.replClassServer.port=$((RANDOM_PORT + 4)) \
>>> --conf spark.blockManager.port=$((RANDOM_PORT + 5)) \
>>> --conf spark.executor.port=$((RANDOM_PORT + 6)) \
>>> --conf spark.fileserver.port=$((RANDOM_PORT + 7)) \
>>> --conf spark.shuffle.service.enabled=true \
>>> --conf spark.dynamicAllocation.enabled=true \
>>> --conf spark.dynamicAllocation.minExecutors=1 \
>>> --conf spark.dynamicAllocation.maxExecutors=10 \
>>> --conf spark.ui.port=$SPARK_UI_PORT
>>>
>>>
>>> Spark/Ipython log is in attachment.
>>>
>>> --
>>>
>>>
>>>
>>> *Sincerely yoursEgor PakhomovScala Developer, Yandex*
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>
>>
>
>
> --
>
>
>
> *Sincerely yoursEgor PakhomovScala Developer, Yandex*
>
Re: Elastic allocation(spark.dynamicAllocation.enabled) results in
task never being executed.
Posted by Egor Pahomov <pa...@gmail.com>.
It's successful without dynamic allocation. I can provide spark log for
that scenario if it can help.
2014-11-14 21:36 GMT+02:00 Sandy Ryza <sa...@cloudera.com>:
> Hi Egor,
>
> Is it successful without dynamic allocation? From your log, it looks like
> the job is unable to acquire resources from YARN, which could be because
> other jobs are using up all the resources.
>
> -Sandy
>
> On Fri, Nov 14, 2014 at 11:32 AM, Egor Pahomov <pa...@gmail.com>
> wrote:
>
>> Hi.
>> I execute ipython notebook + pyspark with spark.dynamicAllocation.enabled
>> = true. Task never ends.
>> Code:
>>
>> import sys
>> from random import random
>> from operator import add
>> partitions = 10
>> n = 100000 * partitions
>>
>> def f(_):
>> x = random() * 2 - 1
>> y = random() * 2 - 1
>> return 1 if x ** 2 + y ** 2 < 1 else 0
>>
>> count = sc.parallelize(xrange(1, n + 1), partitions).map(f).reduce(add)
>> print "Pi is roughly %f" % (4.0 * count / n)
>>
>>
>>
>> Run notebook:
>>
>> IPYTHON_ARGS="notebook --profile=ydf --port $IPYTHON_PORT --port-retries=0 --ip='*' --no-browser"
>> pyspark \
>> --verbose \
>> --master yarn-client \
>> --conf spark.driver.port=$((RANDOM_PORT + 2)) \
>> --conf spark.broadcast.port=$((RANDOM_PORT + 3)) \
>> --conf spark.replClassServer.port=$((RANDOM_PORT + 4)) \
>> --conf spark.blockManager.port=$((RANDOM_PORT + 5)) \
>> --conf spark.executor.port=$((RANDOM_PORT + 6)) \
>> --conf spark.fileserver.port=$((RANDOM_PORT + 7)) \
>> --conf spark.shuffle.service.enabled=true \
>> --conf spark.dynamicAllocation.enabled=true \
>> --conf spark.dynamicAllocation.minExecutors=1 \
>> --conf spark.dynamicAllocation.maxExecutors=10 \
>> --conf spark.ui.port=$SPARK_UI_PORT
>>
>>
>> Spark/Ipython log is in attachment.
>>
>> --
>>
>>
>>
>> *Sincerely yoursEgor PakhomovScala Developer, Yandex*
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>
>
--
*Sincerely yoursEgor PakhomovScala Developer, Yandex*
Re: Elastic allocation(spark.dynamicAllocation.enabled) results in
task never being executed.
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Egor,
Is it successful without dynamic allocation? From your log, it looks like
the job is unable to acquire resources from YARN, which could be because
other jobs are using up all the resources.
-Sandy
On Fri, Nov 14, 2014 at 11:32 AM, Egor Pahomov <pa...@gmail.com>
wrote:
> Hi.
> I execute ipython notebook + pyspark with spark.dynamicAllocation.enabled
> = true. Task never ends.
> Code:
>
> import sys
> from random import random
> from operator import add
> partitions = 10
> n = 100000 * partitions
>
> def f(_):
> x = random() * 2 - 1
> y = random() * 2 - 1
> return 1 if x ** 2 + y ** 2 < 1 else 0
>
> count = sc.parallelize(xrange(1, n + 1), partitions).map(f).reduce(add)
> print "Pi is roughly %f" % (4.0 * count / n)
>
>
>
> Run notebook:
>
> IPYTHON_ARGS="notebook --profile=ydf --port $IPYTHON_PORT --port-retries=0 --ip='*' --no-browser"
> pyspark \
> --verbose \
> --master yarn-client \
> --conf spark.driver.port=$((RANDOM_PORT + 2)) \
> --conf spark.broadcast.port=$((RANDOM_PORT + 3)) \
> --conf spark.replClassServer.port=$((RANDOM_PORT + 4)) \
> --conf spark.blockManager.port=$((RANDOM_PORT + 5)) \
> --conf spark.executor.port=$((RANDOM_PORT + 6)) \
> --conf spark.fileserver.port=$((RANDOM_PORT + 7)) \
> --conf spark.shuffle.service.enabled=true \
> --conf spark.dynamicAllocation.enabled=true \
> --conf spark.dynamicAllocation.minExecutors=1 \
> --conf spark.dynamicAllocation.maxExecutors=10 \
> --conf spark.ui.port=$SPARK_UI_PORT
>
>
> Spark/Ipython log is in attachment.
>
> --
>
>
>
> *Sincerely yoursEgor PakhomovScala Developer, Yandex*
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>