You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Nicholas Chammas <ni...@gmail.com> on 2014/05/06 02:52:09 UTC
How can adding a random count() change the behavior of my program?
I’m running into something very strange today. I’m getting an error on the
follow innocuous operations.
a = sc.textFile('s3n://...')
a = a.repartition(8)
a = a.map(...)
c = a.countByKey() # ERRORs out on this action. See below for traceback. [1]
If I add a count() right after the repartition(), this error magically goes
away.
a = sc.textFile('s3n://...')
a = a.repartition(8)
print a.count()
a = a.map(...)
c = a.countByKey() # A-OK! WTF?
To top it off, this “fix” is inconsistent. Sometimes, I still get this
error.
This is strange. How do I get to the bottom of this?
Nick
[1] Here’s the traceback:
Traceback (most recent call last):
File "<stdin>", line 7, in <module>
File "file.py", line 187, in function_blah
c = a.countByKey()
File "/root/spark/python/pyspark/rdd.py", line 778, in countByKey
return self.map(lambda x: x[0]).countByValue()
File "/root/spark/python/pyspark/rdd.py", line 624, in countByValue
return self.mapPartitions(countPartition).reduce(mergeMaps)
File "/root/spark/python/pyspark/rdd.py", line 505, in reduce
vals = self.mapPartitions(func).collect()
File "/root/spark/python/pyspark/rdd.py", line 469, in collect
bytesInJava = self._jrdd.collect().iterator()
File "/root/spark/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py",
line 537, in __call__
File "/root/spark/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py",
line 300, in get_return_value
py4j.protocol.Py4JJavaError: An error occurred while calling o46.collect.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/How-can-adding-a-random-count-change-the-behavior-of-my-program-tp5406.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
Re: How can adding a random count() change the behavior of my program?
Posted by Nicholas Chammas <ni...@gmail.com>.
Yeah, I believe repartition() is a lazy operation, but it’s strange that
adding the count() can affect anything. I wonder if it has anything to do
with defining an RDD as a transformation on itself in PySpark. Maybe
combining lazy transformations with Python’s
labels-not-variables<http://foobarnbaz.com/2012/07/08/understanding-python-variables/>design
is somehow problematic? I have no idea.
As in:
# bad?
a = textFile(...)
a = a.repartition(8)
# better?
a = textFile(...)
b = a.repartition(8)
Nick
On Fri, May 9, 2014 at 6:29 PM, Walrus theCat <wa...@gmail.com>wrote:
> Nick,
>
> I have encountered strange things like this before (usually when
> programming with mutable structures and side-effects), and for me, the
> answer was that, until .count (or .first, or similar), is called, your
> variable 'a' refers to a set of instructions that only get executed to form
> the object you expect when you're asking something of it. Back before I
> was using side-effect-free techniques on immutable data structures, I had
> to call .first or .count or similar to get the behavior I wanted. There
> are still special cases where I have to purposefully "collapse" the RDD for
> some reason or another. This may not be new information to you, but I've
> encountered similar behavior before and highly suspect this is playing a
> role here.
>
>
> On Mon, May 5, 2014 at 5:52 PM, Nicholas Chammas <
> nicholas.chammas@gmail.com> wrote:
>
>> I’m running into something very strange today. I’m getting an error on
>> the follow innocuous operations.
>>
>> a = sc.textFile('s3n://...')
>> a = a.repartition(8)
>> a = a.map(...)
>> c = a.countByKey() # ERRORs out on this action. See below for traceback. [1]
>>
>> If I add a count() right after the repartition(), this error magically
>> goes away.
>>
>> a = sc.textFile('s3n://...')
>> a = a.repartition(8)
>> print a.count()
>> a = a.map(...)
>> c = a.countByKey() # A-OK! WTF?
>>
>> To top it off, this “fix” is inconsistent. Sometimes, I still get this
>> error.
>>
>> This is strange. How do I get to the bottom of this?
>>
>> Nick
>>
>> [1] Here’s the traceback:
>>
>> Traceback (most recent call last):
>> File "<stdin>", line 7, in <module>
>> File "file.py", line 187, in function_blah
>> c = a.countByKey()
>> File "/root/spark/python/pyspark/rdd.py", line 778, in countByKey
>> return self.map(lambda x: x[0]).countByValue()
>> File "/root/spark/python/pyspark/rdd.py", line 624, in countByValue
>> return self.mapPartitions(countPartition).reduce(mergeMaps)
>> File "/root/spark/python/pyspark/rdd.py", line 505, in reduce
>> vals = self.mapPartitions(func).collect()
>> File "/root/spark/python/pyspark/rdd.py", line 469, in collect
>> bytesInJava = self._jrdd.collect().iterator()
>> File "/root/spark/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", line 537, in __call__
>> File "/root/spark/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py", line 300, in get_return_value
>> py4j.protocol.Py4JJavaError: An error occurred while calling o46.collect.
>>
>>
>> ------------------------------
>> View this message in context: How can adding a random count() change the
>> behavior of my program?<http://apache-spark-user-list.1001560.n3.nabble.com/How-can-adding-a-random-count-change-the-behavior-of-my-program-tp5406.html>
>> Sent from the Apache Spark User List mailing list archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com.
>>
>
>
Re: How can adding a random count() change the behavior of my program?
Posted by Walrus theCat <wa...@gmail.com>.
Nick,
I have encountered strange things like this before (usually when
programming with mutable structures and side-effects), and for me, the
answer was that, until .count (or .first, or similar), is called, your
variable 'a' refers to a set of instructions that only get executed to form
the object you expect when you're asking something of it. Back before I
was using side-effect-free techniques on immutable data structures, I had
to call .first or .count or similar to get the behavior I wanted. There
are still special cases where I have to purposefully "collapse" the RDD for
some reason or another. This may not be new information to you, but I've
encountered similar behavior before and highly suspect this is playing a
role here.
On Mon, May 5, 2014 at 5:52 PM, Nicholas Chammas <nicholas.chammas@gmail.com
> wrote:
> I’m running into something very strange today. I’m getting an error on the
> follow innocuous operations.
>
> a = sc.textFile('s3n://...')
> a = a.repartition(8)
> a = a.map(...)
> c = a.countByKey() # ERRORs out on this action. See below for traceback. [1]
>
> If I add a count() right after the repartition(), this error magically
> goes away.
>
> a = sc.textFile('s3n://...')
> a = a.repartition(8)
> print a.count()
> a = a.map(...)
> c = a.countByKey() # A-OK! WTF?
>
> To top it off, this “fix” is inconsistent. Sometimes, I still get this
> error.
>
> This is strange. How do I get to the bottom of this?
>
> Nick
>
> [1] Here’s the traceback:
>
> Traceback (most recent call last):
> File "<stdin>", line 7, in <module>
> File "file.py", line 187, in function_blah
> c = a.countByKey()
> File "/root/spark/python/pyspark/rdd.py", line 778, in countByKey
> return self.map(lambda x: x[0]).countByValue()
> File "/root/spark/python/pyspark/rdd.py", line 624, in countByValue
> return self.mapPartitions(countPartition).reduce(mergeMaps)
> File "/root/spark/python/pyspark/rdd.py", line 505, in reduce
> vals = self.mapPartitions(func).collect()
> File "/root/spark/python/pyspark/rdd.py", line 469, in collect
> bytesInJava = self._jrdd.collect().iterator()
> File "/root/spark/python/lib/py4j-0.8.1-src.zip/py4j/java_gateway.py", line 537, in __call__
> File "/root/spark/python/lib/py4j-0.8.1-src.zip/py4j/protocol.py", line 300, in get_return_value
> py4j.protocol.Py4JJavaError: An error occurred while calling o46.collect.
>
>
> ------------------------------
> View this message in context: How can adding a random count() change the
> behavior of my program?<http://apache-spark-user-list.1001560.n3.nabble.com/How-can-adding-a-random-count-change-the-behavior-of-my-program-tp5406.html>
> Sent from the Apache Spark User List mailing list archive<http://apache-spark-user-list.1001560.n3.nabble.com/>at Nabble.com.
>