You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by DEEPAK SHARMA <de...@outlook.com> on 2016/08/16 02:58:14 UTC
Re: Apache Spark toDebugString producing different output for python
and scala repl
Hi All,
Below is the small piece of code in scala and python REPL in Apache Spark.However I am getting different output in both the language when I execute toDebugString.I am using cloudera quick start VM.
PYTHON
rdd2 = sc.textFile('file:/home/training/training_materials/data/frostroad.txt').map(lambda x:x.upper()).filter(lambda x : 'THE' in x)
print rdd2.toDebugString()
(1) PythonRDD[56] at RDD at PythonRDD.scala:42 []
| file:/home/training/training_materials/data/frostroad.txt MapPartitionsRDD[55] at textFile at NativeMethodAccessorImpl.java:-2 []
| file:/home/training/training_materials/data/frostroad.txt HadoopRDD[54] at textFile at ......
SCALA
val rdd2 = sc.textFile("file:/home/training/training_materials/data/frostroad.txt").map(x => x.toUpperCase()).filter(x => x.contains("THE"))
rdd2.toDebugString
res1: String =
(1) MapPartitionsRDD[3] at filter at <console>:21 []
| MapPartitionsRDD[2] at map at <console>:21 []
| file:/home/training/training_materials/data/frostroad.txt MapPartitionsRDD[1] at textFile at <console>:21 []
| file:/home/training/training_materials/data/frostroad.txt HadoopRDD[0] at textFile at <
Also one of cloudera slides say that the default partitions is 2 however its 1 (looking at output of toDebugString).
Appreciate any help.
Thanks
Deepak Sharma
Re: Apache Spark toDebugString producing different output for python
and scala repl
Posted by Saisai Shao <sa...@gmail.com>.
The implementation inside the Python API and Scala API for RDD is slightly
different, so the difference of RDD lineage you printed is expected.
On Tue, Aug 16, 2016 at 10:58 AM, DEEPAK SHARMA <deepak_dehradun@outlook.com
> wrote:
> Hi All,
>
>
> Below is the small piece of code in scala and python REPL in Apache
> Spark.However I am getting different output in both the language when I
> execute toDebugString.I am using cloudera quick start VM.
>
> PYTHON
>
> rdd2 = sc.textFile('file:/home/training/training_materials/
> data/frostroad.txt').map(lambda x:x.upper()).filter(lambda x : 'THE' in x)
>
> print rdd2.toDebugString()(1) PythonRDD[56] at RDD at PythonRDD.scala:42 []
> | file:/home/training/training_materials/data/frostroad.txt MapPartitionsRDD[55] at textFile at NativeMethodAccessorImpl.java:-2 []
> | file:/home/training/training_materials/data/frostroad.txt HadoopRDD[54] at textFile at ......
>
> SCALA
>
> val rdd2 = sc.textFile("file:/home/training/training_materials/data/frostroad.txt").map(x => x.toUpperCase()).filter(x => x.contains("THE"))
>
>
>
> rdd2.toDebugString
> res1: String = (1) MapPartitionsRDD[3] at filter at <console>:21 []
> | MapPartitionsRDD[2] at map at <console>:21 []
> | file:/home/training/training_materials/data/frostroad.txt MapPartitionsRDD[1] at textFile at <console>:21 []
> | file:/home/training/training_materials/data/frostroad.txt HadoopRDD[0] at textFile at <
>
>
> Also one of cloudera slides say that the default partitions is 2 however
> its 1 (looking at output of toDebugString).
>
>
> Appreciate any help.
>
>
> Thanks
>
> Deepak Sharma
>