You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Aaron <aa...@target.com> on 2014/08/19 19:47:09 UTC

Python script runs fine in local mode, errors in other modes

Hello,I have a relatively simple python program that works just find in local
most (--master local) but produces a strange error when I try to run it via
Yarn ( --deploy-mode client --master yarn) or just execute the code through
pyspark.Here's the code:sc = SparkContext(appName="foo")input =
sc.textFile("hdfs://[valid hdfs path]")mappedToLines = input.map(lambda
myline: myline.split(","))The third line yields this error:TypeError: 'bool'
object is not callableBut "myline" seems to be a valid string if I look at
it this way:>>> mappedToLines = input.map(lambda myline: len(myline))>>>
mappedToLines.collect()[84, 104, 109, 89, 108, 92, 89, 90, 93, 102]I just
now have access to a Hadoop cluster with Spark installed, so hopefully I'm
running into some simple issues that I never had to deal with when testing
in my own sandbox in purely local mode before. Any help would be
appreciated, thanks!  -Aaron



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Python-script-runs-fine-in-local-mode-errors-in-other-modes-tp12390.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

Re: Python script runs fine in local mode, errors in other modes

Posted by Davies Liu <da...@databricks.com>.
I got another report about this recently, and figured out that it's
caused by having different versions of python in driver and YARN:

http://stackoverflow.com/questions/28879803/spark-runs-in-local-but-not-in-yarn/28931934#28931934

Created JIRA: https://issues.apache.org/jira/browse/SPARK-6216?filter=-1

On Tue, Aug 19, 2014 at 12:12 PM, Davies Liu <da...@databricks.com> wrote:
> This script run very well without your CSV file. Could download you
> CSV file into local disks, and narrow down to the lines which triggle
> this issue?
>
> On Tue, Aug 19, 2014 at 12:02 PM, Aaron <aa...@target.com> wrote:
>> These three lines of python code cause the error for me:
>>
>> sc = SparkContext(appName="foo")
>> input = sc.textFile("hdfs://[valid hdfs path]")
>> mappedToLines = input.map(lambda myline: myline.split(","))
>>
>> The file I'm loading is a simple CSV.
>>
>>
>>
>> --
>> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Python-script-runs-fine-in-local-mode-errors-in-other-modes-tp12390p12398.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Python script runs fine in local mode, errors in other modes

Posted by Davies Liu <da...@databricks.com>.
This script run very well without your CSV file. Could download you
CSV file into local disks, and narrow down to the lines which triggle
this issue?

On Tue, Aug 19, 2014 at 12:02 PM, Aaron <aa...@target.com> wrote:
> These three lines of python code cause the error for me:
>
> sc = SparkContext(appName="foo")
> input = sc.textFile("hdfs://[valid hdfs path]")
> mappedToLines = input.map(lambda myline: myline.split(","))
>
> The file I'm loading is a simple CSV.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Python-script-runs-fine-in-local-mode-errors-in-other-modes-tp12390p12398.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Python script runs fine in local mode, errors in other modes

Posted by Aaron <aa...@target.com>.
These three lines of python code cause the error for me:  

sc = SparkContext(appName="foo")   
input = sc.textFile("hdfs://[valid hdfs path]")   
mappedToLines = input.map(lambda myline: myline.split(","))  

The file I'm loading is a simple CSV.  



--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Python-script-runs-fine-in-local-mode-errors-in-other-modes-tp12390p12398.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Python script runs fine in local mode, errors in other modes

Posted by Davies Liu <da...@databricks.com>.
On Tue, Aug 19, 2014 at 11:20 AM, Aaron <aa...@target.com> wrote:
> Sure thing, this is the stacktrace from pyspark.  It's repeated a few times,
> but I think this is the unique stuff.
>
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File
> "/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/pyspark/rdd.py",
> line 583, in collect
>     bytesInJava = self._jrdd.collect().iterator()
>   File
> "/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py",
> line 537, in __call__
>   File
> "/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py",
> line 300, in get_return_value
> py4j.protocol.Py4JJavaError: An error occurred while calling o24.collect.
> : org.apache.spark.SparkException: Job aborted due to stage failure: Task
> 0.0:0 failed 4 times, most recent failure: Exception failure in TID 5 on
> host [hostname redacted]: org.apache.spark.api.python.PythonException:
> Traceback (most recent call last):
>   File
> "/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/pyspark/worker.py",
> line 77, in main
>     serializer.dump_stream(func(split_index, iterator), outfile)
>   File
> "/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/pyspark/serializers.py",
> line 191, in dump_stream
>     self.serializer.dump_stream(self._batched(iterator), stream)
>   File
> "/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/pyspark/serializers.py",
> line 123, in dump_stream
>     for obj in iterator:
>   File
> "/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/pyspark/serializers.py",
> line 180, in _batched
>     for item in iterator:
>   File "<stdin>", line 1, in <lambda>
> TypeError: 'bool' object is not callable

This seems very strange, "myline.split" became a bool?

Could you show me how to reproduce this issue? thanks.

Davies

> org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:118)
>
> org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:148)
>         org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:81)
>         org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>         org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
>         org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
>         org.apache.spark.scheduler.Task.run(Task.scala:51)
>
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         java.lang.Thread.run(Thread.java:745)
> Driver stacktrace:
>         at
> org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1033)
>         at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017)
>         at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015)
>         at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>         at
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
>         at
> org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015)
>         at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
>         at
> org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
>         at scala.Option.foreach(Option.scala:236)
>         at
> org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:633)
>         at
> org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1207)
>         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
>         at akka.actor.ActorCell.invoke(ActorCell.scala:456)
>         at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
>         at akka.dispatch.Mailbox.run(Mailbox.scala:219)
>         at
> akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
>         at
> scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
>         at
> scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
>         at
> scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
>         at
> scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
>
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Python-script-runs-fine-in-local-mode-errors-in-other-modes-tp12390p12392.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Python script runs fine in local mode, errors in other modes

Posted by Aaron <aa...@target.com>.
Sure thing, this is the stacktrace from pyspark.  It's repeated a few times,
but I think this is the unique stuff.

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File
"/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/pyspark/rdd.py",
line 583, in collect
    bytesInJava = self._jrdd.collect().iterator()
  File
"/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py",
line 537, in __call__
  File
"/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py",
line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o24.collect.
: org.apache.spark.SparkException: Job aborted due to stage failure: Task
0.0:0 failed 4 times, most recent failure: Exception failure in TID 5 on
host [hostname redacted]: org.apache.spark.api.python.PythonException:
Traceback (most recent call last):
  File
"/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/pyspark/worker.py",
line 77, in main
    serializer.dump_stream(func(split_index, iterator), outfile)
  File
"/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/pyspark/serializers.py",
line 191, in dump_stream
    self.serializer.dump_stream(self._batched(iterator), stream)
  File
"/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/pyspark/serializers.py",
line 123, in dump_stream
    for obj in iterator:
  File
"/opt/cloudera/parcels/CDH-5.1.0-1.cdh5.1.0.p0.53/lib/spark/python/pyspark/serializers.py",
line 180, in _batched
    for item in iterator:
  File "<stdin>", line 1, in <lambda>
TypeError: 'bool' object is not callable

       
org.apache.spark.api.python.PythonRDD$$anon$1.read(PythonRDD.scala:118)
       
org.apache.spark.api.python.PythonRDD$$anon$1.<init>(PythonRDD.scala:148)
        org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:81)
        org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
        org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
        org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:111)
        org.apache.spark.scheduler.Task.run(Task.scala:51)
       
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
       
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
       
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
        at
org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1033)
        at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1017)
        at
org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1015)
        at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
        at
org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1015)
        at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
        at
org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:633)
        at scala.Option.foreach(Option.scala:236)
        at
org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:633)
        at
org.apache.spark.scheduler.DAGSchedulerEventProcessActor$$anonfun$receive$2.applyOrElse(DAGScheduler.scala:1207)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
        at akka.actor.ActorCell.invoke(ActorCell.scala:456)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:237)
        at akka.dispatch.Mailbox.run(Mailbox.scala:219)
        at
akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:386)
        at
scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at
scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at
scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at
scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Python-script-runs-fine-in-local-mode-errors-in-other-modes-tp12390p12392.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Python script runs fine in local mode, errors in other modes

Posted by Davies Liu <da...@databricks.com>.
Could you post the completed stacktrace?

On Tue, Aug 19, 2014 at 10:47 AM, Aaron <aa...@target.com> wrote:
> Hello, I have a relatively simple python program that works just find in
> local most (--master local) but produces a strange error when I try to run
> it via Yarn ( --deploy-mode client --master yarn) or just execute the code
> through pyspark. Here's the code: sc = SparkContext(appName="foo") input =
> sc.textFile("hdfs://[valid hdfs path]") mappedToLines = input.map(lambda
> myline: myline.split(",")) The third line yields this error: TypeError:
> 'bool' object is not callable But "myline" seems to be a valid string if I
> look at it this way: >>> mappedToLines = input.map(lambda myline:
> len(myline)) >>> mappedToLines.collect() [84, 104, 109, 89, 108, 92, 89, 90,
> 93, 102] I just now have access to a Hadoop cluster with Spark installed, so
> hopefully I'm running into some simple issues that I never had to deal with
> when testing in my own sandbox in purely local mode before. Any help would
> be appreciated, thanks! -Aaron
> ________________________________
> View this message in context: Python script runs fine in local mode, errors
> in other modes
> Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org


Re: Python script runs fine in local mode, errors in other modes

Posted by Aaron <aa...@target.com>.
Hi! Yeah, the problem was inconsistent versions of python across the cluster. I was launching from a node with Python 2.7.9, but the job ran on nodes with 2.6.6.

From: "devmacrile [via Apache Spark User List]" <ml...@n3.nabble.com>>
Date: Monday, September 28, 2015 at 1:35 PM
To: Aaron Dossett <Aa...@target.com>>
Subject: Re: Python script runs fine in local mode, errors in other modes

Was there any eventual solution to this that you discovered?

________________________________
If you reply to this email, your message will be added to the discussion below:
http://apache-spark-user-list.1001560.n3.nabble.com/Python-script-runs-fine-in-local-mode-errors-in-other-modes-tp12390p24847.html
To unsubscribe from Python script runs fine in local mode, errors in other modes, click here<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=12390&code=YWFyb24uZG9zc2V0dEB0YXJnZXQuY29tfDEyMzkwfDEzNzY3MTk4OQ==>.
NAML<http://apache-spark-user-list.1001560.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Python-script-runs-fine-in-local-mode-errors-in-other-modes-tp12390p24848.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.