You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Owen O'Malley (JIRA)" <ji...@apache.org> on 2007/02/13 06:04:05 UTC
[jira] Created: (HADOOP-1014) map/reduce is corrupting data between
map and reduce
map/reduce is corrupting data between map and reduce
----------------------------------------------------
Key: HADOOP-1014
URL: https://issues.apache.org/jira/browse/HADOOP-1014
Project: Hadoop
Issue Type: Bug
Components: mapred
Affects Versions: 0.11.1
Reporter: Owen O'Malley
Assigned To: Devaraj Das
Priority: Blocker
It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
from Mike Smith:
The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
when you rerun the same job. I have observed this inconsistency of the map
output in different jobs. A simple test to double check is to use hadoop
0.11 with nutch trunk.
from Albert Chern:
I am having the same problem with my own map reduce jobs. I have a job
which requires two pieces of data per key, and just as a sanity check I make
sure that it gets both in the reducer, but sometimes it doesn't. What's
even stranger is, the same tasks that complain about missing key/value pairs
will maybe fail two or three times, but then succeed on a subsequent try,
which leads me to believe that the bug has to do with randomization (I'm not
sure, but I think the map outputs are shuffled?).
All of my code works perfectly with 0.9, so I went back and just compared
the sizes of the outputs. For some jobs, the outputs from 0.11 were
consistently 4 bytes larger, probably due to changes in SequenceFile. But
for others, the output sizes were all over the place. Some partitions were
empty, some were correct, and some were missing data. There seems to be
something seriously wrong with 0.11, so I suggest you use 0.9. I've been
trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (HADOOP-1014) map/reduce is corrupting data between map and reduce
Posted by Nt Never <nt...@gmail.com>.
No, the assertion exception is the only exception, everything else runs
smoothly. I will upload the patch to TestMapRed() in a few minutes (it will
apply to the 0.11.1 release)
On 2/13/07, Devaraj Das (JIRA) <ji...@apache.org> wrote:
>
>
> [
> https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472998]
>
> Devaraj Das commented on HADOOP-1014:
> -------------------------------------
>
> Do you see any other error/exception for the failing job (other than the
> assertion exception)?
>
> > map/reduce is corrupting data between map and reduce
> > ----------------------------------------------------
> >
> > Key: HADOOP-1014
> > URL: https://issues.apache.org/jira/browse/HADOOP-1014
> > Project: Hadoop
> > Issue Type: Bug
> > Components: mapred
> > Affects Versions: 0.11.1
> > Reporter: Owen O'Malley
> > Assigned To: Devaraj Das
> > Priority: Blocker
> > Fix For: 0.11.2
> >
> >
> > It appears that a random data corruption is happening between the map
> and the reduce. This looks to be a blocker until it is resolved. There were
> two relevant messages on hadoop-dev:
> > from Mike Smith:
> > The map/reduce jobs are not consistent in hadoop 0.11 release and trunk
> both
> > when you rerun the same job. I have observed this inconsistency of the
> map
> > output in different jobs. A simple test to double check is to use hadoop
> > 0.11 with nutch trunk.
> > from Albert Chern:
> > I am having the same problem with my own map reduce jobs. I have a job
> > which requires two pieces of data per key, and just as a sanity check I
> make
> > sure that it gets both in the reducer, but sometimes it doesn't. What's
> > even stranger is, the same tasks that complain about missing key/value
> pairs
> > will maybe fail two or three times, but then succeed on a subsequent
> try,
> > which leads me to believe that the bug has to do with randomization (I'm
> not
> > sure, but I think the map outputs are shuffled?).
> > All of my code works perfectly with 0.9, so I went back and just
> compared
> > the sizes of the outputs. For some jobs, the outputs from 0.11 were
> > consistently 4 bytes larger, probably due to changes in
> SequenceFile. But
> > for others, the output sizes were all over the place. Some partitions
> were
> > empty, some were correct, and some were missing data. There seems to be
> > something seriously wrong with 0.11, so I suggest you use 0.9. I've
> been
> > trying to pinpoint the bug but its random nature is really annoying.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
Re: [jira] Updated: (HADOOP-1014) map/reduce is corrupting data between map and reduce
Posted by Nt Never <nt...@gmail.com>.
You are totally right, my bad. Your patched version passes all the JUnit
tests now. I will now test it on my largest jobs and compare with 0.9.2.
Should take about 7-8 hours. Thanks.
On 2/15/07, Devaraj Das (JIRA) <ji...@apache.org> wrote:
>
>
> [
> https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel]
>
> Devaraj Das updated HADOOP-1014:
> --------------------------------
>
> Attachment: zero-size-inmem-fs.patch
> TestMapRed.java
>
> Riccardo, the problem with your testcase was that in the "readFields"
> method of the WritableWrapper class you were not setting the "type" field so
> that "write" would always write the value '0' for 'type' in the map output
> file. Hence the reduces won't get the intended map outputs. I have attached
> the fixed TestMapRed.java.
> In your map method, you instantiate a WritableWrapper object which has the
> type field correctly set. After that you do output.collect which would
> have different behaviors in the versions 0.9 and 0.10+. In the version 0.9,
> the output.collect will write the data directly to the final map output
> file. However, with the 0.10+ versions, the output gets buffered and
> written later and there is deserialization/serialization happening when the
> output is finally written to disk from the buffer. In your code, the
> deserialization code (readFields) was not setting the type field and hence
> the serialization (write) would always write 0 (type - INT_WRITABLE) for the
> type field. The reducer, thus, would never see UTF8.
>
> Also attached a patch that would disable inmem merge (basically sets the
> buffer size for the ramfs to 0, and does some checks for that). This should
> remove the blocker.
>
> Mike, Albert and Riccardo - please comment whether this solves the issues
> you reported for the time being. I will continue to debug the inmem merge.
> Must be some race condition somewhere since the failures are not consistent.
> Thanks.
>
> > map/reduce is corrupting data between map and reduce
> > ----------------------------------------------------
> >
> > Key: HADOOP-1014
> > URL: https://issues.apache.org/jira/browse/HADOOP-1014
> > Project: Hadoop
> > Issue Type: Bug
> > Components: mapred
> > Affects Versions: 0.11.1
> > Reporter: Owen O'Malley
> > Assigned To: Devaraj Das
> > Priority: Blocker
> > Fix For: 0.11.2
> >
> > Attachments: TestMapRed.java, TestMapRed.patch,
> TestMapRed2.patch, zero-size-inmem-fs.patch
> >
> >
> > It appears that a random data corruption is happening between the map
> and the reduce. This looks to be a blocker until it is resolved. There were
> two relevant messages on hadoop-dev:
> > from Mike Smith:
> > The map/reduce jobs are not consistent in hadoop 0.11 release and trunk
> both
> > when you rerun the same job. I have observed this inconsistency of the
> map
> > output in different jobs. A simple test to double check is to use hadoop
> > 0.11 with nutch trunk.
> > from Albert Chern:
> > I am having the same problem with my own map reduce jobs. I have a job
> > which requires two pieces of data per key, and just as a sanity check I
> make
> > sure that it gets both in the reducer, but sometimes it doesn't. What's
> > even stranger is, the same tasks that complain about missing key/value
> pairs
> > will maybe fail two or three times, but then succeed on a subsequent
> try,
> > which leads me to believe that the bug has to do with randomization (I'm
> not
> > sure, but I think the map outputs are shuffled?).
> > All of my code works perfectly with 0.9, so I went back and just
> compared
> > the sizes of the outputs. For some jobs, the outputs from 0.11 were
> > consistently 4 bytes larger, probably due to changes in
> SequenceFile. But
> > for others, the output sizes were all over the place. Some partitions
> were
> > empty, some were correct, and some were missing data. There seems to be
> > something seriously wrong with 0.11, so I suggest you use 0.9. I've
> been
> > trying to pinpoint the bug but its random nature is really annoying.
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>
[jira] Commented: (HADOOP-1014) map/reduce is corrupting data
between map and reduce
Posted by "Mike Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473441 ]
Mike Smith commented on HADOOP-1014:
------------------------------------
Devaraj,
As I email you yesterday, that patch solved my problem. But, I will do some more tests and will let you know the results.
Mike
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
> Attachments: TestMapRed.java, TestMapRed.patch, TestMapRed2.patch, zero-size-inmem-fs.patch
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1014) map/reduce is corrupting data between
map and reduce
Posted by "Riccardo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Riccardo updated HADOOP-1014:
-----------------------------
Attachment: TestMapRed.patch
Here it is. It was created against the Hadoop 0.11.1 release.
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
> Attachments: TestMapRed.patch
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1014) map/reduce is corrupting data
between map and reduce
Posted by "Albert Chern (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473442 ]
Albert Chern commented on HADOOP-1014:
--------------------------------------
Disabling the in-mem merge has fixed this for me. I went back and checked my outputs from 0.9 against the patched 0.11 and they match perfectly. Thanks Devaraj!
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
> Attachments: TestMapRed.java, TestMapRed.patch, TestMapRed2.patch, zero-size-inmem-fs.patch
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1014) map/reduce is corrupting data between
map and reduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-1014:
---------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks, Devaraj.
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
> Attachments: TestMapRed.java, TestMapRed.patch, TestMapRed2.patch, zero-size-inmem-fs.patch
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1014) map/reduce is corrupting data
between map and reduce
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473462 ]
Hadoop QA commented on HADOOP-1014:
-----------------------------------
+1, because http://issues.apache.org/jira/secure/attachment/12351227/zero-size-inmem-fs.patch applied and successfully tested against trunk revision r507276.
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
> Attachments: TestMapRed.java, TestMapRed.patch, TestMapRed2.patch, zero-size-inmem-fs.patch
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1014) map/reduce is corrupting data
between map and reduce
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473302 ]
Andrzej Bialecki commented on HADOOP-1014:
-------------------------------------------
I didn't debug this, it's just an idea ... Could this be related to the fix to HADOOP-917?
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
> Attachments: TestMapRed.patch, TestMapRed2.patch
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1014) map/reduce is corrupting data
between map and reduce
Posted by "Riccardo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473170 ]
Riccardo commented on HADOOP-1014:
----------------------------------
No exceptions besides the assertion exceptions. I am uploading the patch to TestMapRed(). As I mentioned before it doesn't fail consistently, but when it does fail it is simply because it loses (key,value) pairs, no other exception is reported. At least on our cluster, the failure rate is 0% for less than 10M keys, and close to 40-50% for 100M+ keys. Hope this helps, but if I can do anything else to help out please let me know.
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1014) map/reduce is corrupting data
between map and reduce
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473453 ]
Owen O'Malley commented on HADOOP-1014:
---------------------------------------
This looks like a reasonable workaround for right now. Although it seems like a better answer for getPercentUsed would be 1.0f instead of 0.1f.
+1
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
> Attachments: TestMapRed.java, TestMapRed.patch, TestMapRed2.patch, zero-size-inmem-fs.patch
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1014) map/reduce is corrupting data
between map and reduce
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473570 ]
Owen O'Malley commented on HADOOP-1014:
---------------------------------------
We got a ConcurrentModificationException in the ramfs, which is likely relevant:
2007-02-15 02:58:08,997 INFO org.apache.hadoop.mapred.TaskRunner: task_0002_r_002131_0 Copying task_0002_m_001581_0 output from X.
2007-02-15 02:58:09,000 WARN org.apache.hadoop.mapred.TaskRunner: Merge of the inmemory files threw an exception: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:787)
at java.util.HashMap$KeyIterator.next(HashMap.java:823)
at org.apache.hadoop.fs.InMemoryFileSystem.getFiles(InMemoryFileSystem.java:370)
at org.apache.hadoop.mapred.ReduceTaskRunner$InMemFSMergeThread.run(ReduceTaskRunner.java:731)
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
> Attachments: TestMapRed.java, TestMapRed.patch, TestMapRed2.patch, zero-size-inmem-fs.patch
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1014) map/reduce is corrupting data between
map and reduce
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated HADOOP-1014:
----------------------------------
Status: Patch Available (was: Open)
Submitting patch for review on behalf of Devaraj (whose ISP is wonky right now) since we need to fix this 'blocker'...
Thanks to Albert and Mike for helping us test this.
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
> Attachments: TestMapRed.java, TestMapRed.patch, TestMapRed2.patch, zero-size-inmem-fs.patch
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1014) map/reduce is corrupting data
between map and reduce
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473775 ]
Devaraj Das commented on HADOOP-1014:
-------------------------------------
Regarding the comment on 0.1f vs 1.0f, I think it is better to leave it as is. I would like to avoid modifying this patch (which seems to remove the instability) & instead work towards making the ramfs stable. Makes sense?
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
> Attachments: TestMapRed.java, TestMapRed.patch, TestMapRed2.patch, zero-size-inmem-fs.patch
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1014) map/reduce is corrupting data
between map and reduce
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472990 ]
Devaraj Das commented on HADOOP-1014:
-------------------------------------
Riccardo, could you please upload your test as a patch on this issue? That will be helpful. Thanks.
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1014) map/reduce is corrupting data between
map and reduce
Posted by "Riccardo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Riccardo updated HADOOP-1014:
-----------------------------
Attachment: TestMapRed2.patch
Here is a much more compelling case. Same M/R job but instead of using GenericWritable() I use my own WritableWrapper() code (extremely simple). This JUnit fails for any number of keys (try just 50,000) with Hadoop 0.10.0 or newer, while it runs with no problems whatsoever on 0.9.2.
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
> Attachments: TestMapRed.patch, TestMapRed2.patch
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1014) map/reduce is corrupting data
between map and reduce
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12473897 ]
Owen O'Malley commented on HADOOP-1014:
---------------------------------------
I see the cause of the ConcurrentModificationException and likely the corruption problem. From the JavaDoc on the Collections.synchronizedMap:
| It is imperative that the user manually synchronize on the returned map when iterating over any of its
| collection views:
|
| Map m = Collections.synchronizedMap(new HashMap());
| ...
| Set s = m.keySet(); // Needn't be in synchronized block
| ...
| synchronized(m) { // Synchronizing on m, not s!
| Iterator i = s.iterator(); // Must be in synchronized block
| while (i.hasNext())
| foo(i.next());
| }
|
|
| Failure to follow this advice may result in non-deterministic behavior.
The InMemoryFileSystem locks the InMemoryFileSystem rather than the synchronizedMap. This leads to the non-determinism since the other operations (add, remove, etc.) are locking the synchronizedMap.
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
> Attachments: TestMapRed.java, TestMapRed.patch, TestMapRed2.patch, zero-size-inmem-fs.patch
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1014) map/reduce is corrupting data between
map and reduce
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-1014:
--------------------------------
Attachment: zero-size-inmem-fs.patch
TestMapRed.java
Riccardo, the problem with your testcase was that in the "readFields" method of the WritableWrapper class you were not setting the "type" field so that "write" would always write the value '0' for 'type' in the map output file. Hence the reduces won't get the intended map outputs. I have attached the fixed TestMapRed.java.
In your map method, you instantiate a WritableWrapper object which has the type field correctly set. After that you do output.collect which would have different behaviors in the versions 0.9 and 0.10+. In the version 0.9, the output.collect will write the data directly to the final map output file. However, with the 0.10+ versions, the output gets buffered and written later and there is deserialization/serialization happening when the output is finally written to disk from the buffer. In your code, the deserialization code (readFields) was not setting the type field and hence the serialization (write) would always write 0 (type - INT_WRITABLE) for the type field. The reducer, thus, would never see UTF8.
Also attached a patch that would disable inmem merge (basically sets the buffer size for the ramfs to 0, and does some checks for that). This should remove the blocker.
Mike, Albert and Riccardo - please comment whether this solves the issues you reported for the time being. I will continue to debug the inmem merge. Must be some race condition somewhere since the failures are not consistent. Thanks.
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
> Attachments: TestMapRed.java, TestMapRed.patch, TestMapRed2.patch, zero-size-inmem-fs.patch
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1014) map/reduce is corrupting data
between map and reduce
Posted by "Riccardo (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472972 ]
Riccardo commented on HADOOP-1014:
----------------------------------
Same problem here. I have a bunch of M/R jobs that process some 100Gb of text data and collect different types of stats. Since Hadoop 0.10.0 I started noticing that some of the reduce outputs were missing data (some of them were empty). I modified the TestMapRed() JUnit to run the following test. I create two separate sequence files by generating N random integers and storing them in one file as IntWritable() values and as Text() writables in the other one. Matching integers have the same key. Then I ran the M/R job with an identity mapper (I use GenericWritable() to wrap the different value types) and in the reducer I check if both the integer and the string representation are received for a given key. The job runs just fine until up about 50M keys, then it starts failing for 100M keys or more. The assertion exception is caused by the reducer receiving only one value not both. One problem is that the job does not fail consistently and it always throws the assertion exceptions for different keys and within different reduce tasks. Our M/R reduce cluster is composed of 10 high-end dual-core linux boxes with about 8Tb of aggregate HD capacity. I cannot get this JUnit (or any of my jobs) to fail with Hadoop 0.9.2. They actually run on it like a breeze.
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-1014) map/reduce is corrupting data between
map and reduce
Posted by "Doug Cutting (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Doug Cutting updated HADOOP-1014:
---------------------------------
Fix Version/s: 0.11.2
This warrants a point release. I've scheduled one for this Friday, hoping we find a fix by then.
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-1014) map/reduce is corrupting data
between map and reduce
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12472998 ]
Devaraj Das commented on HADOOP-1014:
-------------------------------------
Do you see any other error/exception for the failing job (other than the assertion exception)?
> map/reduce is corrupting data between map and reduce
> ----------------------------------------------------
>
> Key: HADOOP-1014
> URL: https://issues.apache.org/jira/browse/HADOOP-1014
> Project: Hadoop
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.11.1
> Reporter: Owen O'Malley
> Assigned To: Devaraj Das
> Priority: Blocker
> Fix For: 0.11.2
>
>
> It appears that a random data corruption is happening between the map and the reduce. This looks to be a blocker until it is resolved. There were two relevant messages on hadoop-dev:
> from Mike Smith:
> The map/reduce jobs are not consistent in hadoop 0.11 release and trunk both
> when you rerun the same job. I have observed this inconsistency of the map
> output in different jobs. A simple test to double check is to use hadoop
> 0.11 with nutch trunk.
> from Albert Chern:
> I am having the same problem with my own map reduce jobs. I have a job
> which requires two pieces of data per key, and just as a sanity check I make
> sure that it gets both in the reducer, but sometimes it doesn't. What's
> even stranger is, the same tasks that complain about missing key/value pairs
> will maybe fail two or three times, but then succeed on a subsequent try,
> which leads me to believe that the bug has to do with randomization (I'm not
> sure, but I think the map outputs are shuffled?).
> All of my code works perfectly with 0.9, so I went back and just compared
> the sizes of the outputs. For some jobs, the outputs from 0.11 were
> consistently 4 bytes larger, probably due to changes in SequenceFile. But
> for others, the output sizes were all over the place. Some partitions were
> empty, some were correct, and some were missing data. There seems to be
> something seriously wrong with 0.11, so I suggest you use 0.9. I've been
> trying to pinpoint the bug but its random nature is really annoying.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.