You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by edmond_huo <hu...@gmail.com> on 2014/09/17 07:03:14 UTC
The difference between pyspark.rdd.PipelinedRDD and pyspark.rdd.RDD
Hi,
I am a freshman about spark. I tried to run a job like wordcount example in
python. But when I tried to get the top 10 popular words in the file, I got
the message:AttributeError: 'PipelinedRDD' object has no attribute
'sortByKey'.
So my question is what is the difference between PipelinedRDD and RDD? and
if I want to sort the data in PipelinedRDD, how can I do it?
Thanks
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/The-difference-between-pyspark-rdd-PipelinedRDD-and-pyspark-rdd-RDD-tp14421.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: The difference between pyspark.rdd.PipelinedRDD and
pyspark.rdd.RDD
Posted by edmond_huo <hu...@gmail.com>.
Hi Davis,
Thank you for you answer. This is my code. I think it is very similar with
word count example in spark
lines = sc.textFile(sys.argv[2])
sie = lines.map(lambda l: (l.strip().split(',')[4],1)).reduceByKey(lambda
a, b: a + b)
sort_sie = sie.sortByKey(False)
Thanks again.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/The-difference-between-pyspark-rdd-PipelinedRDD-and-pyspark-rdd-RDD-tp14421p14448.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: The difference between pyspark.rdd.PipelinedRDD and pyspark.rdd.RDD
Posted by Davies Liu <da...@databricks.com>.
On Wed, Sep 17, 2014 at 7:37 AM, edmond_huo <hu...@gmail.com> wrote:
> Hi Davis,
>
> When I run your code in pyspark, I still get the same error:
>
>>>> sc.parallelize(range(10)).map(lambda x: (x, str(x))).sortByKey().count()
> Traceback (most recent call last):
> File "<stdin>", line 1, in <module>
> AttributeError: 'PipelinedRDD' object has no attribute 'sortByKey'
>
> Is it the matter with spark version? I am using spark-0.7.3.
This is the problem, this version is so old, could you upgrade to 1.1?
There are bunch of improvements out of the box, it will save your life :-)
Davies
> Thanks.
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/The-difference-between-pyspark-rdd-PipelinedRDD-and-pyspark-rdd-RDD-tp14421p14449.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: The difference between pyspark.rdd.PipelinedRDD and
pyspark.rdd.RDD
Posted by edmond_huo <hu...@gmail.com>.
Hi Davis,
When I run your code in pyspark, I still get the same error:
>>> sc.parallelize(range(10)).map(lambda x: (x, str(x))).sortByKey().count()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: 'PipelinedRDD' object has no attribute 'sortByKey'
Is it the matter with spark version? I am using spark-0.7.3.
Thanks.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/The-difference-between-pyspark-rdd-PipelinedRDD-and-pyspark-rdd-RDD-tp14421p14449.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: The difference between pyspark.rdd.PipelinedRDD and pyspark.rdd.RDD
Posted by Davies Liu <da...@databricks.com>.
PipelinedRDD is an RDD generated by Python mapper/reducer, such as
rdd.map(func) will be PipelinedRDD.
PipelinedRDD is an subclass of RDD, so it should have all the APIs which
RDD has.
>>> sc.parallelize(range(10)).map(lambda x: (x, str(x))).sortByKey().count()
10
I'm wondering that how can you trigger this error?
Davies
On Tue, Sep 16, 2014 at 10:03 PM, edmond_huo <hu...@gmail.com> wrote:
> Hi,
>
> I am a freshman about spark. I tried to run a job like wordcount example in
> python. But when I tried to get the top 10 popular words in the file, I got
> the message:AttributeError: 'PipelinedRDD' object has no attribute
> 'sortByKey'.
>
> So my question is what is the difference between PipelinedRDD and RDD? and
> if I want to sort the data in PipelinedRDD, how can I do it?
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/The-difference-between-pyspark-rdd-PipelinedRDD-and-pyspark-rdd-RDD-tp14421.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org