You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by jeff saremi <je...@hotmail.com> on 2015/11/13 19:29:58 UTC
SequenceFile and object reuse
So we tried reading a sequencefile in Spark and realized that all our records have ended up becoming the same.
THen one of us found this:
Note: Because Hadoop's RecordReader class re-uses the same Writable object for each record, directly caching the returned RDD or directly passing it to an aggregation or shuffle operation will create many references to the same object. If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first copy them using a map function.
Is there anyone that can shed some light on this bizzare behavior and the decisions behind it?
And I also would like to know if anyone's able to read a binary file and not to incur the additional map() as suggested by the above? What format did you use?
thanksJeff
RE: SequenceFile and object reuse
Posted by an...@thomsonreuters.com.
As I understand it, it's down to how Hadoop FileInputFormats work, and
questions of mutability. If you were to read a file from Hadoop via an
InputFormat with a simple Java program, the InputFormat's RecordReader
creates a single, mutable instance of the Writable key class and a single,
mutable instance of the Writable value. When you loop through the records,
the RecordReader reuses those Writable instances by deserializing the
underlying bytes from the file into the instances 1 record at a time. It's
up to the application to then copy whatever's needed out of those Writable
instances into something else if it wants to do something with them.
It's exactly the same when using Spark as the application. When you create
an RDD of Writable objects by calling .sequenceFile, the RDD contains many
identical references to the exact same object instance. Therefore, when
Spark does a sort, cache or shuffle, (I believe) it optimizes because it
assumes that objects are immutable. Therefore, the map step is necessary,
because it creates a distinct, immutable copy of each record.
This is just an issue with the Hadoop InputFormat class. If you can write a
way of reading files from HDFS that don't use Hadoop's classes (though I'm
not sure why you would, a simple map is far easier), then the map would
potentially be unnecessary.
Andrew
From: jeff saremi [mailto:jeffsaremi@hotmail.com]
Sent: 19 November 2015 05:35
To: Jeff Zhang <zj...@gmail.com>
Cc: dev@spark.apache.org
Subject: RE: SequenceFile and object reuse
You're not seeing the issue because you perform one additional "map".
map{case (k,v) => (k.get(), v.toString)}
Instead of being able to use the read Text you had to create a tuple
(single) out of the string of the text.
That is exactly why I asked this question.
Why do we have t do this additional processing? What is the rationale behind
it?
Is there other ways of reading a hadoop file (or any other file) that would
not incur this additional step?
thanks
_____
Date: Thu, 19 Nov 2015 13:26:31 +0800
Subject: Re: FW: SequenceFile and object reuse
From: zjffdu@gmail.com <ma...@gmail.com>
To: jeffsaremi@hotmail.com <ma...@hotmail.com>
CC: dev@spark.apache.org <ma...@spark.apache.org>
Would this be an issue on the raw data ? I use the following simple code,
and don't hit the issue you mentioned. Or it would be better to share your
code.
val rdd =sc.sequenceFile("/Users/hadoop/Temp/Seq", classOf[IntWritable],
classOf[Text])
rdd.map{case (k,v) => (k.get(), v.toString)}.collect() foreach println
On Thu, Nov 19, 2015 at 12:04 PM, jeff saremi <jeffsaremi@hotmail.com
<ma...@hotmail.com> > wrote:
I sent this to the user forum. I got no responses. Could someone here please
help? thanks
jeff
_____
From: jeffsaremi@hotmail.com <ma...@hotmail.com>
To: user@spark.apache.org <ma...@spark.apache.org>
Subject: SequenceFile and object reuse
Date: Fri, 13 Nov 2015 13:29:58 -0500
So we tried reading a sequencefile in Spark and realized that all our
records have ended up becoming the same.
THen one of us found this:
Note: Because Hadoop's RecordReader class re-uses the same Writable object
for each record, directly caching the returned RDD or directly passing it to
an aggregation or shuffle operation will create many references to the same
object. If you plan to directly cache, sort, or aggregate Hadoop writable
objects, you should first copy them using a map function.
Is there anyone that can shed some light on this bizzare behavior and the
decisions behind it?
And I also would like to know if anyone's able to read a binary file and not
to incur the additional map() as suggested by the above? What format did you
use?
thanks
Jeff
--
Best Regards
Jeff Zhang
RE: SequenceFile and object reuse
Posted by jeff saremi <je...@hotmail.com>.
You're not seeing the issue because you perform one additional "map".
map{case (k,v) => (k.get(), v.toString)}Instead of being able to use the read Text you had to create a tuple (single) out of the string of the text.
That is exactly why I asked this question.Why do we have t do this additional processing? What is the rationale behind it?
Is there other ways of reading a hadoop file (or any other file) that would not incur this additional step?thanks
Date: Thu, 19 Nov 2015 13:26:31 +0800
Subject: Re: FW: SequenceFile and object reuse
From: zjffdu@gmail.com
To: jeffsaremi@hotmail.com
CC: dev@spark.apache.org
Would this be an issue on the raw data ? I use the following simple code, and don't hit the issue you mentioned. Or it would be better to share your code.
val rdd =sc.sequenceFile("/Users/hadoop/Temp/Seq", classOf[IntWritable], classOf[Text])
rdd.map{case (k,v) => (k.get(), v.toString)}.collect() foreach println
On Thu, Nov 19, 2015 at 12:04 PM, jeff saremi <je...@hotmail.com> wrote:
I sent this to the user forum. I got no responses. Could someone here please help? thanks
jeff
From: jeffsaremi@hotmail.com
To: user@spark.apache.org
Subject: SequenceFile and object reuse
Date: Fri, 13 Nov 2015 13:29:58 -0500
So we tried reading a sequencefile in Spark and realized that all our records have ended up becoming the same.
THen one of us found this:
Note: Because Hadoop's RecordReader class re-uses the same Writable object for each record, directly caching the returned RDD or directly passing it to an aggregation or shuffle operation will create many references to the same object. If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first copy them using a map function.
Is there anyone that can shed some light on this bizzare behavior and the decisions behind it?
And I also would like to know if anyone's able to read a binary file and not to incur the additional map() as suggested by the above? What format did you use?
thanksJeff
--
Best Regards
Jeff Zhang
Re: FW: SequenceFile and object reuse
Posted by Jeff Zhang <zj...@gmail.com>.
Would this be an issue on the raw data ? I use the following simple code,
and don't hit the issue you mentioned. Or it would be better to share your
code.
val rdd =sc.sequenceFile("/Users/hadoop/Temp/Seq",
classOf[IntWritable], classOf[Text])
rdd.map{case (k,v) => (k.get(), v.toString)}.collect() foreach println
On Thu, Nov 19, 2015 at 12:04 PM, jeff saremi <je...@hotmail.com>
wrote:
> I sent this to the user forum. I got no responses. Could someone here
> please help? thanks
> jeff
>
> ------------------------------
> From: jeffsaremi@hotmail.com
> To: user@spark.apache.org
> Subject: SequenceFile and object reuse
> Date: Fri, 13 Nov 2015 13:29:58 -0500
>
>
> So we tried reading a sequencefile in Spark and realized that all our
> records have ended up becoming the same.
> THen one of us found this:
>
> Note: Because Hadoop's RecordReader class re-uses the same Writable object
> for each record, directly caching the returned RDD or directly passing it
> to an aggregation or shuffle operation will create many references to the
> same object. If you plan to directly cache, sort, or aggregate Hadoop
> writable objects, you should first copy them using a map function.
>
> Is there anyone that can shed some light on this bizzare behavior and the
> decisions behind it?
> And I also would like to know if anyone's able to read a binary file and
> not to incur the additional map() as suggested by the above? What format
> did you use?
>
> thanks
> Jeff
>
--
Best Regards
Jeff Zhang
FW: SequenceFile and object reuse
Posted by jeff saremi <je...@hotmail.com>.
I sent this to the user forum. I got no responses. Could someone here please help? thanks
jeff
From: jeffsaremi@hotmail.com
To: user@spark.apache.org
Subject: SequenceFile and object reuse
Date: Fri, 13 Nov 2015 13:29:58 -0500
So we tried reading a sequencefile in Spark and realized that all our records have ended up becoming the same.
THen one of us found this:
Note: Because Hadoop's RecordReader class re-uses the same Writable object for each record, directly caching the returned RDD or directly passing it to an aggregation or shuffle operation will create many references to the same object. If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first copy them using a map function.
Is there anyone that can shed some light on this bizzare behavior and the decisions behind it?
And I also would like to know if anyone's able to read a binary file and not to incur the additional map() as suggested by the above? What format did you use?
thanksJeff
RE: SequenceFile and object reuse
Posted by jeff saremi <je...@hotmail.com>.
Sandy, Ryan, Andrew
Thanks very much. I think i now understand it better.
Jeff
From: ryan.blake.williams@gmail.com
Date: Thu, 19 Nov 2015 06:00:30 +0000
Subject: Re: SequenceFile and object reuse
To: sandy.ryza@cloudera.com; jeffsaremi@hotmail.com
CC: user@spark.apache.org
Hey Jeff, in addition to what Sandy said, there are two more reasons that this might not be as bad as it seems; I may be incorrect in my understanding though.
First, the "additional step" you're referring to is not likely to be adding any overhead; the "extra map" is really just materializing the data once (as opposed to zero times), which is what you want (assuming your access pattern couldn't be reformulated in the way Sandy described, i.e. where all the objects in a partition don't need to be in memory at the same time).
Secondly, even if this was an "extra" map step, it wouldn't add any extra stages to a given pipeline, being a "narrow" dependency, so it would likely be low-cost anyway.
Let me know if any of the above seems incorrect, thanks!
On Thu, Nov 19, 2015 at 12:41 AM Sandy Ryza <sa...@cloudera.com> wrote:
Hi Jeff,
Many access patterns simply take the result of hadoopFile and use it to create some other object, and thus have no need for each input record to refer to a different object. In those cases, the current API is more performant than an alternative that would create an object for each record, because it avoids the unnecessary overhead of creating Java objects. As you've pointed out, this is at the expense of making the code more verbose when caching.
-Sandy
On Fri, Nov 13, 2015 at 10:29 AM, jeff saremi <je...@hotmail.com> wrote:
So we tried reading a sequencefile in Spark and realized that all our records have ended up becoming the same.
THen one of us found this:
Note: Because Hadoop's RecordReader class re-uses the same Writable object for each record, directly caching the returned RDD or directly passing it to an aggregation or shuffle operation will create many references to the same object. If you plan to directly cache, sort, or aggregate Hadoop writable objects, you should first copy them using a map function.
Is there anyone that can shed some light on this bizzare behavior and the decisions behind it?
And I also would like to know if anyone's able to read a binary file and not to incur the additional map() as suggested by the above? What format did you use?
thanksJeff
Re: SequenceFile and object reuse
Posted by Ryan Williams <ry...@gmail.com>.
Hey Jeff, in addition to what Sandy said, there are two more reasons that
this might not be as bad as it seems; I may be incorrect in my
understanding though.
First, the "additional step" you're referring to is not likely to be adding
any overhead; the "extra map" is really just materializing the data once
(as opposed to zero times), which is what you want (assuming your access
pattern couldn't be reformulated in the way Sandy described, i.e. where all
the objects in a partition don't need to be in memory at the same time).
Secondly, even if this was an "extra" map step, it wouldn't add any extra
stages to a given pipeline, being a "narrow" dependency, so it would likely
be low-cost anyway.
Let me know if any of the above seems incorrect, thanks!
On Thu, Nov 19, 2015 at 12:41 AM Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Jeff,
>
> Many access patterns simply take the result of hadoopFile and use it to
> create some other object, and thus have no need for each input record to
> refer to a different object. In those cases, the current API is more
> performant than an alternative that would create an object for each record,
> because it avoids the unnecessary overhead of creating Java objects. As
> you've pointed out, this is at the expense of making the code more verbose
> when caching.
>
> -Sandy
>
> On Fri, Nov 13, 2015 at 10:29 AM, jeff saremi <je...@hotmail.com>
> wrote:
>
>> So we tried reading a sequencefile in Spark and realized that all our
>> records have ended up becoming the same.
>> THen one of us found this:
>>
>> Note: Because Hadoop's RecordReader class re-uses the same Writable
>> object for each record, directly caching the returned RDD or directly
>> passing it to an aggregation or shuffle operation will create many
>> references to the same object. If you plan to directly cache, sort, or
>> aggregate Hadoop writable objects, you should first copy them using a map
>> function.
>>
>> Is there anyone that can shed some light on this bizzare behavior and the
>> decisions behind it?
>> And I also would like to know if anyone's able to read a binary file and
>> not to incur the additional map() as suggested by the above? What format
>> did you use?
>>
>> thanks
>> Jeff
>>
>
>
Re: SequenceFile and object reuse
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Jeff,
Many access patterns simply take the result of hadoopFile and use it to
create some other object, and thus have no need for each input record to
refer to a different object. In those cases, the current API is more
performant than an alternative that would create an object for each record,
because it avoids the unnecessary overhead of creating Java objects. As
you've pointed out, this is at the expense of making the code more verbose
when caching.
-Sandy
On Fri, Nov 13, 2015 at 10:29 AM, jeff saremi <je...@hotmail.com>
wrote:
> So we tried reading a sequencefile in Spark and realized that all our
> records have ended up becoming the same.
> THen one of us found this:
>
> Note: Because Hadoop's RecordReader class re-uses the same Writable object
> for each record, directly caching the returned RDD or directly passing it
> to an aggregation or shuffle operation will create many references to the
> same object. If you plan to directly cache, sort, or aggregate Hadoop
> writable objects, you should first copy them using a map function.
>
> Is there anyone that can shed some light on this bizzare behavior and the
> decisions behind it?
> And I also would like to know if anyone's able to read a binary file and
> not to incur the additional map() as suggested by the above? What format
> did you use?
>
> thanks
> Jeff
>