You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org> on 2008/06/30 03:33:44 UTC
[jira] Created: (HADOOP-3666) SequenceFile RecordReader should skip
bad records
SequenceFile RecordReader should skip bad records
-------------------------------------------------
Key: HADOOP-3666
URL: https://issues.apache.org/jira/browse/HADOOP-3666
Project: Hadoop Core
Issue Type: Bug
Components: mapred
Affects Versions: 0.17.0
Reporter: Joydeep Sen Sarma
Currently a bad record in a sequencefile leads to entire job being failed. the best workaround is to skip an errant file manually (by looking at what map task failed). This is a sucky option because it's manual and because one should be able to skip a sequencefile block (instead of entire file).
While we don't see this often (and i don't know why this corruption happened) - here's an example stack:
Status : FAILED java.lang.NegativeArraySizeException
at org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:96)
at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:75)
at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:130)
at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1640)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712)
at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79)
at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:176)
Ideally the recordreader should just skip the entire chunk if it gets an unrecoverable error while reading.
This was the consensus in hadoop-153 as well (that data corruptions should be handled by recordreaders) and hadoop-3144 did something similar for textinputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3666) SequenceFile RecordReader should
skip bad records
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12610582#action_12610582 ]
Joydeep Sen Sarma commented on HADOOP-3666:
-------------------------------------------
cool - that sounds perfect.
> SequenceFile RecordReader should skip bad records
> -------------------------------------------------
>
> Key: HADOOP-3666
> URL: https://issues.apache.org/jira/browse/HADOOP-3666
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Joydeep Sen Sarma
>
> Currently a bad record in a sequencefile leads to entire job being failed. the best workaround is to skip an errant file manually (by looking at what map task failed). This is a sucky option because it's manual and because one should be able to skip a sequencefile block (instead of entire file).
> While we don't see this often (and i don't know why this corruption happened) - here's an example stack:
> Status : FAILED java.lang.NegativeArraySizeException
> at org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:96)
> at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:75)
> at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:130)
> at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1640)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712)
> at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79)
> at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:176)
> Ideally the recordreader should just skip the entire chunk if it gets an unrecoverable error while reading.
> This was the consensus in hadoop-153 as well (that data corruptions should be handled by recordreaders) and hadoop-3144 did something similar for textinputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3666) SequenceFile RecordReader should
skip bad records
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12610271#action_12610271 ]
Joydeep Sen Sarma commented on HADOOP-3666:
-------------------------------------------
sorry - i didn't see Owen's comment earlier: - yes #2 was the intent.
I agree that this causes the absence of any policy hooks (max number of errors tolerable, reporting errors out to JobTracker.) - and i think that's leading to Sharad's comment. One problem with his proposal is how will the RecordReader differentiate between the first and second next() call?
One simple method to integrate this with the policy framework would be for the recordreader to export an error counter (as an additional interface). The TT/JT can make go/no-go decision based on the number of errors they observe. By default a recordreader would attempt to skip bad records, while raising the error count - but based on job spec - this may lead to job being aborted by Hadoop.
One wrinkle is that the number of errors is not the same as the amount of data skipped. That may be a design point (report bytes skipped versus # of bad records).
> SequenceFile RecordReader should skip bad records
> -------------------------------------------------
>
> Key: HADOOP-3666
> URL: https://issues.apache.org/jira/browse/HADOOP-3666
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Joydeep Sen Sarma
>
> Currently a bad record in a sequencefile leads to entire job being failed. the best workaround is to skip an errant file manually (by looking at what map task failed). This is a sucky option because it's manual and because one should be able to skip a sequencefile block (instead of entire file).
> While we don't see this often (and i don't know why this corruption happened) - here's an example stack:
> Status : FAILED java.lang.NegativeArraySizeException
> at org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:96)
> at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:75)
> at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:130)
> at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1640)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712)
> at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79)
> at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:176)
> Ideally the recordreader should just skip the entire chunk if it gets an unrecoverable error while reading.
> This was the consensus in hadoop-153 as well (that data corruptions should be handled by recordreaders) and hadoop-3144 did something similar for textinputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HADOOP-3666) SequenceFile RecordReader should
skip bad records
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Owen O'Malley resolved HADOOP-3666.
-----------------------------------
Resolution: Duplicate
This is a duplicate of HADOOP-153.
> SequenceFile RecordReader should skip bad records
> -------------------------------------------------
>
> Key: HADOOP-3666
> URL: https://issues.apache.org/jira/browse/HADOOP-3666
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Joydeep Sen Sarma
>
> Currently a bad record in a sequencefile leads to entire job being failed. the best workaround is to skip an errant file manually (by looking at what map task failed). This is a sucky option because it's manual and because one should be able to skip a sequencefile block (instead of entire file).
> While we don't see this often (and i don't know why this corruption happened) - here's an example stack:
> Status : FAILED java.lang.NegativeArraySizeException
> at org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:96)
> at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:75)
> at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:130)
> at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1640)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712)
> at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79)
> at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:176)
> Ideally the recordreader should just skip the entire chunk if it gets an unrecoverable error while reading.
> This was the consensus in hadoop-153 as well (that data corruptions should be handled by recordreaders) and hadoop-3144 did something similar for textinputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Reopened: (HADOOP-3666) SequenceFile RecordReader should
skip bad records
Posted by "Joydeep Sen Sarma (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Joydeep Sen Sarma reopened HADOOP-3666:
---------------------------------------
hadoop-153 is addressing application errors only at this point. i thought there was agreement that recordreaders should implement any logic for skipping corrupt records internally (and outside the scope of 153). please see:
http://issues.apache.org/jira/browse/HADOOP-153?focusedCommentId=12593425#action_12593425
> SequenceFile RecordReader should skip bad records
> -------------------------------------------------
>
> Key: HADOOP-3666
> URL: https://issues.apache.org/jira/browse/HADOOP-3666
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Joydeep Sen Sarma
>
> Currently a bad record in a sequencefile leads to entire job being failed. the best workaround is to skip an errant file manually (by looking at what map task failed). This is a sucky option because it's manual and because one should be able to skip a sequencefile block (instead of entire file).
> While we don't see this often (and i don't know why this corruption happened) - here's an example stack:
> Status : FAILED java.lang.NegativeArraySizeException
> at org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:96)
> at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:75)
> at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:130)
> at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1640)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712)
> at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79)
> at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:176)
> Ideally the recordreader should just skip the entire chunk if it gets an unrecoverable error while reading.
> This was the consensus in hadoop-153 as well (that data corruptions should be handled by recordreaders) and hadoop-3144 did something similar for textinputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3666) SequenceFile RecordReader should
skip bad records
Posted by "Owen O'Malley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609288#action_12609288 ]
Owen O'Malley commented on HADOOP-3666:
---------------------------------------
So what is the approach here?
1. Internal errors (negative lengths, any others?)
throw exception and skip to next sync block
2. reader errors (ie. errors in the deserializer)
skip to next record
Is that your intent?
> SequenceFile RecordReader should skip bad records
> -------------------------------------------------
>
> Key: HADOOP-3666
> URL: https://issues.apache.org/jira/browse/HADOOP-3666
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Joydeep Sen Sarma
>
> Currently a bad record in a sequencefile leads to entire job being failed. the best workaround is to skip an errant file manually (by looking at what map task failed). This is a sucky option because it's manual and because one should be able to skip a sequencefile block (instead of entire file).
> While we don't see this often (and i don't know why this corruption happened) - here's an example stack:
> Status : FAILED java.lang.NegativeArraySizeException
> at org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:96)
> at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:75)
> at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:130)
> at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1640)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712)
> at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79)
> at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:176)
> Ideally the recordreader should just skip the entire chunk if it gets an unrecoverable error while reading.
> This was the consensus in hadoop-153 as well (that data corruptions should be handled by recordreaders) and hadoop-3144 did something similar for textinputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3666) SequenceFile RecordReader should
skip bad records
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12610458#action_12610458 ]
Sharad Agarwal commented on HADOOP-3666:
----------------------------------------
{quote}One problem with his proposal is how will the RecordReader differentiate between the first and second next() call? {quote}
Does RecordReader really need to differentiate? When next() is called the first time, the RecordReader would skip to the sane record boundary, BEFORE throwing an Exception (a subclass of IOException, something like SkippedRecordException), so that framework knows that the record has been skipped. Calling next() again would read the next() record.
This way we are also not forcing all RecordReaders to implement this feature.
{quote} One simple method to integrate this with the policy framework would be for the Recordreader to export an error counter (as an additional interface). {quote}
That would be the better way. But given that lot of user code may get impacted, we can try to avoid interface change as far as possible.
makes sense?
> SequenceFile RecordReader should skip bad records
> -------------------------------------------------
>
> Key: HADOOP-3666
> URL: https://issues.apache.org/jira/browse/HADOOP-3666
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Joydeep Sen Sarma
>
> Currently a bad record in a sequencefile leads to entire job being failed. the best workaround is to skip an errant file manually (by looking at what map task failed). This is a sucky option because it's manual and because one should be able to skip a sequencefile block (instead of entire file).
> While we don't see this often (and i don't know why this corruption happened) - here's an example stack:
> Status : FAILED java.lang.NegativeArraySizeException
> at org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:96)
> at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:75)
> at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:130)
> at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1640)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712)
> at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79)
> at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:176)
> Ideally the recordreader should just skip the entire chunk if it gets an unrecoverable error while reading.
> This was the consensus in hadoop-153 as well (that data corruptions should be handled by recordreaders) and hadoop-3144 did something similar for textinputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3666) SequenceFile RecordReader should
skip bad records
Posted by "Sharad Agarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12610193#action_12610193 ]
Sharad Agarwal commented on HADOOP-3666:
----------------------------------------
The recordReader should somehow inform the framework about skipping any of the records. I think it wont be acceptable for the user job to skip a record and not notify about it.
Doing it may require RecordReader interface to change. One way of doing that without changing the interface would be throw a specific exception from next(). The framework can catch it and based on the exception type can decide to again call next() or just fail the job. Also it would allow framework to keep something like skipped records counter.
> SequenceFile RecordReader should skip bad records
> -------------------------------------------------
>
> Key: HADOOP-3666
> URL: https://issues.apache.org/jira/browse/HADOOP-3666
> Project: Hadoop Core
> Issue Type: Bug
> Components: mapred
> Affects Versions: 0.17.0
> Reporter: Joydeep Sen Sarma
>
> Currently a bad record in a sequencefile leads to entire job being failed. the best workaround is to skip an errant file manually (by looking at what map task failed). This is a sucky option because it's manual and because one should be able to skip a sequencefile block (instead of entire file).
> While we don't see this often (and i don't know why this corruption happened) - here's an example stack:
> Status : FAILED java.lang.NegativeArraySizeException
> at org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:96)
> at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:75)
> at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:130)
> at org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:1640)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1712)
> at org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:79)
> at org.apache.hadoop.mapred.MapTask$1.next(MapTask.java:176)
> Ideally the recordreader should just skip the entire chunk if it gets an unrecoverable error while reading.
> This was the consensus in hadoop-153 as well (that data corruptions should be handled by recordreaders) and hadoop-3144 did something similar for textinputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.