You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Arun Luthra <ar...@gmail.com> on 2015/08/27 23:08:05 UTC
types allowed for saveasobjectfile?
What types of RDD can saveAsObjectFile(path) handle? I tried a naive test
with an RDD[Array[String]], but when I tried to read back the result with
sc.objectFile(path).take(5).foreach(println), I got a non-promising output
looking like:
[Ljava.lang.String;@46123a
[Ljava.lang.String;@76123b
[Ljava.lang.String;@13144c
[Ljava.lang.String;@75146d
[Ljava.lang.String;@79118f
Arun
Re: types allowed for saveasobjectfile?
Posted by Holden Karau <ho...@pigscanfly.ca>.
Yes, any java serializable object. Its important to note that since its
saving serialized objects it is as brittle as java serialization when it
comes to version changes, so if you can make your data fit in something
like sequence files, parquet, avro, or similar it can be not only more
space efficient but also more future proof.
On Thu, Aug 27, 2015 at 2:24 PM, Arun Luthra <ar...@gmail.com> wrote:
> Ah, yes, that did the trick.
>
> So more generally, can this handle any serializable object?
>
> On Thu, Aug 27, 2015 at 2:11 PM, Jonathan Coveney <jc...@gmail.com>
> wrote:
>
>> array[String] doesn't pretty print by default. Use .mkString(",") for
>> example
>>
>>
>> El jueves, 27 de agosto de 2015, Arun Luthra <ar...@gmail.com>
>> escribió:
>>
>>> What types of RDD can saveAsObjectFile(path) handle? I tried a naive
>>> test with an RDD[Array[String]], but when I tried to read back the result
>>> with sc.objectFile(path).take(5).foreach(println), I got a non-promising
>>> output looking like:
>>>
>>> [Ljava.lang.String;@46123a
>>> [Ljava.lang.String;@76123b
>>> [Ljava.lang.String;@13144c
>>> [Ljava.lang.String;@75146d
>>> [Ljava.lang.String;@79118f
>>>
>>>
>>> Arun
>>>
>>
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Linked In: https://www.linkedin.com/in/holdenkarau
Re: types allowed for saveasobjectfile?
Posted by Arun Luthra <ar...@gmail.com>.
Ah, yes, that did the trick.
So more generally, can this handle any serializable object?
On Thu, Aug 27, 2015 at 2:11 PM, Jonathan Coveney <jc...@gmail.com>
wrote:
> array[String] doesn't pretty print by default. Use .mkString(",") for
> example
>
>
> El jueves, 27 de agosto de 2015, Arun Luthra <ar...@gmail.com>
> escribió:
>
>> What types of RDD can saveAsObjectFile(path) handle? I tried a naive test
>> with an RDD[Array[String]], but when I tried to read back the result with
>> sc.objectFile(path).take(5).foreach(println), I got a non-promising output
>> looking like:
>>
>> [Ljava.lang.String;@46123a
>> [Ljava.lang.String;@76123b
>> [Ljava.lang.String;@13144c
>> [Ljava.lang.String;@75146d
>> [Ljava.lang.String;@79118f
>>
>>
>> Arun
>>
>
Re: types allowed for saveasobjectfile?
Posted by Jonathan Coveney <jc...@gmail.com>.
array[String] doesn't pretty print by default. Use .mkString(",") for
example
El jueves, 27 de agosto de 2015, Arun Luthra <ar...@gmail.com>
escribió:
> What types of RDD can saveAsObjectFile(path) handle? I tried a naive test
> with an RDD[Array[String]], but when I tried to read back the result with
> sc.objectFile(path).take(5).foreach(println), I got a non-promising output
> looking like:
>
> [Ljava.lang.String;@46123a
> [Ljava.lang.String;@76123b
> [Ljava.lang.String;@13144c
> [Ljava.lang.String;@75146d
> [Ljava.lang.String;@79118f
>
>
> Arun
>
Re: types allowed for saveasobjectfile?
Posted by Holden Karau <ho...@pigscanfly.ca>.
So println of any array of strings will look like that. The
java.util.Arrays class has some options to print arrays nicely.
On Thu, Aug 27, 2015 at 2:08 PM, Arun Luthra <ar...@gmail.com> wrote:
> What types of RDD can saveAsObjectFile(path) handle? I tried a naive test
> with an RDD[Array[String]], but when I tried to read back the result with
> sc.objectFile(path).take(5).foreach(println), I got a non-promising output
> looking like:
>
> [Ljava.lang.String;@46123a
> [Ljava.lang.String;@76123b
> [Ljava.lang.String;@13144c
> [Ljava.lang.String;@75146d
> [Ljava.lang.String;@79118f
>
>
> Arun
>
--
Cell : 425-233-8271
Twitter: https://twitter.com/holdenkarau
Linked In: https://www.linkedin.com/in/holdenkarau