You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@spark.apache.org by Gokula Krishnan D <em...@gmail.com> on 2015/12/04 14:01:58 UTC

How to get the list of available Transformations and actions for a RDD in Spark-Shell

Hello All -

In spark-shell when we press tab after <RDD_Name>. ; we could see the
possible list of transformations and actions.

But unable to see all the list. is there any other way to get the rest of
the list. I'm mainly looking for sortByKey()

val sales_RDD = sc.textFile("Data/Scala/phone_sales.txt")
val sales_map = sales_RDD.map(sales=>{val x=sales.split(","); (x(0),x(1))})

Layout of phone_sales.txt is (Brand, #.of Phones sold)

I am mainly looking for SortByKey() but when I do Sales_map or sales_RDD, I
could see only sortBy() but not SortByKey().

By the way, I am using spark 1.3.0 with CDH 5.4

[image: Inline image 1]



Thanks
Gokul

Re: How to get the list of available Transformations and actions for a RDD in Spark-Shell

Posted by Ted Yu <yu...@gmail.com>.

Did a quick test:

rdd2: org.apache.spark.rdd.RDD[(Int, Int)] = MapPartitionsRDD[1] at map at
<console>:29

I think sales_map is MapPartitionsRDD

FYI

On Fri, Dec 4, 2015 at 6:18 AM, Gokula Krishnan D <em...@gmail.com>
wrote:

> Thanks Ayan for the updates.
>
> But in my example, I hope "sales_map" is a pair_RDD , isn't it?.
>
> Thanks & Regards,
> Gokula Krishnan* (Gokul)*
>
>
> On Fri, Dec 4, 2015 at 8:16 AM, ayan guha <gu...@gmail.com> wrote:
>
>> sortByKey() is a property of pairRDD as it requires key value pair to
>> work. I think in scala their are transformation such as .toPairRDD().
>>
>> On Sat, Dec 5, 2015 at 12:01 AM, Gokula Krishnan D <em...@gmail.com>
>> wrote:
>>
>>> Hello All -
>>>
>>> In spark-shell when we press tab after <RDD_Name>. ; we could see the
>>> possible list of transformations and actions.
>>>
>>> But unable to see all the list. is there any other way to get the rest
>>> of the list. I'm mainly looking for sortByKey()
>>>
>>> val sales_RDD = sc.textFile("Data/Scala/phone_sales.txt")
>>> val sales_map = sales_RDD.map(sales=>{val x=sales.split(",");
>>> (x(0),x(1))})
>>>
>>> Layout of phone_sales.txt is (Brand, #.of Phones sold)
>>>
>>> I am mainly looking for SortByKey() but when I do Sales_map or
>>> sales_RDD, I could see only sortBy() but not SortByKey().
>>>
>>> By the way, I am using spark 1.3.0 with CDH 5.4
>>>
>>> [image: Inline image 1]
>>>
>>>
>>>
>>> Thanks
>>> Gokul
>>>
>>>
>>
>>
>> --
>> Best Regards,
>> Ayan Guha
>>
>
>

Re: How to get the list of available Transformations and actions for a RDD in Spark-Shell

Posted by Gokula Krishnan D <em...@gmail.com>.

Thanks Ayan for the updates.

But in my example, I hope "sales_map" is a pair_RDD , isn't it?.

Thanks & Regards,
Gokula Krishnan* (Gokul)*


On Fri, Dec 4, 2015 at 8:16 AM, ayan guha <gu...@gmail.com> wrote:

> sortByKey() is a property of pairRDD as it requires key value pair to
> work. I think in scala their are transformation such as .toPairRDD().
>
> On Sat, Dec 5, 2015 at 12:01 AM, Gokula Krishnan D <em...@gmail.com>
> wrote:
>
>> Hello All -
>>
>> In spark-shell when we press tab after <RDD_Name>. ; we could see the
>> possible list of transformations and actions.
>>
>> But unable to see all the list. is there any other way to get the rest of
>> the list. I'm mainly looking for sortByKey()
>>
>> val sales_RDD = sc.textFile("Data/Scala/phone_sales.txt")
>> val sales_map = sales_RDD.map(sales=>{val x=sales.split(",");
>> (x(0),x(1))})
>>
>> Layout of phone_sales.txt is (Brand, #.of Phones sold)
>>
>> I am mainly looking for SortByKey() but when I do Sales_map or sales_RDD,
>> I could see only sortBy() but not SortByKey().
>>
>> By the way, I am using spark 1.3.0 with CDH 5.4
>>
>> [image: Inline image 1]
>>
>>
>>
>> Thanks
>> Gokul
>>
>>
>
>
> --
> Best Regards,
> Ayan Guha
>

Re: How to get the list of available Transformations and actions for a RDD in Spark-Shell

Posted by ayan guha <gu...@gmail.com>.

sortByKey() is a property of pairRDD as it requires key value pair to work.
I think in scala their are transformation such as .toPairRDD().

On Sat, Dec 5, 2015 at 12:01 AM, Gokula Krishnan D <em...@gmail.com>
wrote:

> Hello All -
>
> In spark-shell when we press tab after <RDD_Name>. ; we could see the
> possible list of transformations and actions.
>
> But unable to see all the list. is there any other way to get the rest of
> the list. I'm mainly looking for sortByKey()
>
> val sales_RDD = sc.textFile("Data/Scala/phone_sales.txt")
> val sales_map = sales_RDD.map(sales=>{val x=sales.split(","); (x(0),x(1))})
>
> Layout of phone_sales.txt is (Brand, #.of Phones sold)
>
> I am mainly looking for SortByKey() but when I do Sales_map or sales_RDD,
> I could see only sortBy() but not SortByKey().
>
> By the way, I am using spark 1.3.0 with CDH 5.4
>
> [image: Inline image 1]
>
>
>
> Thanks
> Gokul
>
>


-- 
Best Regards,
Ayan Guha