You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Koji Noguchi (JIRA)" <ji...@apache.org> on 2008/05/28 22:19:46 UTC
[jira] Created: (HADOOP-3460) SequenceFileAsBinaryOutputFormat
SequenceFileAsBinaryOutputFormat
--------------------------------
Key: HADOOP-3460
URL: https://issues.apache.org/jira/browse/HADOOP-3460
Project: Hadoop Core
Issue Type: New Feature
Components: mapred
Reporter: Koji Noguchi
Priority: Minor
Add an OutputFormat to write raw bytes as keys and values to a SequenceFile.
In C++-Pipes, we're using SequenceFileAsBinaryInputFormat to read Sequencefiles.
However, we current don't have a way to *write* a sequencefile efficiently without going through extra (de)serializations.
I'd like to store the correct classnames for key/values but use BytesWritable to write
(in order for the next java or pig code to be able to read this sequencefile).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3460) SequenceFileAsBinaryOutputFormat
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12603147#action_12603147 ]
Hadoop QA commented on HADOOP-3460:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12383575/HADOOP-3460-part3.patch
against trunk revision 664041.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2608/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2608/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2608/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/2608/console
This message is automatically generated.
> SequenceFileAsBinaryOutputFormat
> --------------------------------
>
> Key: HADOOP-3460
> URL: https://issues.apache.org/jira/browse/HADOOP-3460
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.18.0
>
> Attachments: HADOOP-3460-part1.patch, HADOOP-3460-part2.patch, HADOOP-3460-part3.patch
>
>
> Add an OutputFormat to write raw bytes as keys and values to a SequenceFile.
> In C++-Pipes, we're using SequenceFileAsBinaryInputFormat to read Sequencefiles.
> However, we current don't have a way to *write* a sequencefile efficiently without going through extra (de)serializations.
> I'd like to store the correct classnames for key/values but use BytesWritable to write
> (in order for the next java or pig code to be able to read this sequencefile).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3460) SequenceFileAsBinaryOutputFormat
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3460:
------------------------------------
Release Note: Created SequenceFileAsBinaryOutputFormat to write raw bytes as keys and values to a SequenceFile. (was: Adds an OutputFormat, SequenceFileAsBinaryOutputFormat, to write raw bytes as keys and values to a SequenceFile.)
> SequenceFileAsBinaryOutputFormat
> --------------------------------
>
> Key: HADOOP-3460
> URL: https://issues.apache.org/jira/browse/HADOOP-3460
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.18.0
>
> Attachments: HADOOP-3460-part1.patch, HADOOP-3460-part2.patch, HADOOP-3460-part3.patch
>
>
> Add an OutputFormat to write raw bytes as keys and values to a SequenceFile.
> In C++-Pipes, we're using SequenceFileAsBinaryInputFormat to read Sequencefiles.
> However, we current don't have a way to *write* a sequencefile efficiently without going through extra (de)serializations.
> I'd like to store the correct classnames for key/values but use BytesWritable to write
> (in order for the next java or pig code to be able to read this sequencefile).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3460) SequenceFileAsBinaryOutputFormat
Posted by "Devaraj Das (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Devaraj Das updated HADOOP-3460:
--------------------------------
Release Note: Adds an OutputFormat, SequenceFileAsBinaryOutputFormat, to write raw bytes as keys and values to a SequenceFile.
> SequenceFileAsBinaryOutputFormat
> --------------------------------
>
> Key: HADOOP-3460
> URL: https://issues.apache.org/jira/browse/HADOOP-3460
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.18.0
>
> Attachments: HADOOP-3460-part1.patch, HADOOP-3460-part2.patch, HADOOP-3460-part3.patch
>
>
> Add an OutputFormat to write raw bytes as keys and values to a SequenceFile.
> In C++-Pipes, we're using SequenceFileAsBinaryInputFormat to read Sequencefiles.
> However, we current don't have a way to *write* a sequencefile efficiently without going through extra (de)serializations.
> I'd like to store the correct classnames for key/values but use BytesWritable to write
> (in order for the next java or pig code to be able to read this sequencefile).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3460) SequenceFileAsBinaryOutputFormat
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated HADOOP-3460:
---------------------------------
Attachment: HADOOP-3460-part1.patch
First try.
> SequenceFileAsBinaryOutputFormat
> --------------------------------
>
> Key: HADOOP-3460
> URL: https://issues.apache.org/jira/browse/HADOOP-3460
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Koji Noguchi
> Priority: Minor
> Attachments: HADOOP-3460-part1.patch
>
>
> Add an OutputFormat to write raw bytes as keys and values to a SequenceFile.
> In C++-Pipes, we're using SequenceFileAsBinaryInputFormat to read Sequencefiles.
> However, we current don't have a way to *write* a sequencefile efficiently without going through extra (de)serializations.
> I'd like to store the correct classnames for key/values but use BytesWritable to write
> (in order for the next java or pig code to be able to read this sequencefile).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3460) SequenceFileAsBinaryOutputFormat
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated HADOOP-3460:
---------------------------------
Attachment: HADOOP-3460-part3.patch
bq. 1. The testcase: doesn't need a main method, you might want to break up the check for forbidding record compression into a separate test,
Separted the test into three. testbinary, testSequenceOutputClassDefaultsToMapRedOutputClass, and testcheckOutputSpecsForbidRecordCompression.
Also, I had a bug in the testing such that checkOutputSpecs was throwing an exception because output path was not set and not because RECORD compression was being set.
Fixed it.
bq. and the call to JobConf::setInputPath is generating a warning (replace with FileInputFormat::addInputPath)
Ah. I should have compiled with "-Djavac.args="-Xlint -Xmaxwarns 1000".
Done.
bq. 2. WritableValueBytes::writeCompressedBytes no longer throws IllegalArgumentException, so that can be removed from its signature
I left it in since the original SequenceFile.ValueBytes has a signature
{noformat}
public void writeCompressedBytes(DataOutputStream outStream)
throws IllegalArgumentException, IOException;
{noformat}
Should I still take it out?
bq. 3. SeqFABOF::checkOutputSpecs doesn't need to list InvalidJobConfException
Done.
> SequenceFileAsBinaryOutputFormat
> --------------------------------
>
> Key: HADOOP-3460
> URL: https://issues.apache.org/jira/browse/HADOOP-3460
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Attachments: HADOOP-3460-part1.patch, HADOOP-3460-part2.patch, HADOOP-3460-part3.patch
>
>
> Add an OutputFormat to write raw bytes as keys and values to a SequenceFile.
> In C++-Pipes, we're using SequenceFileAsBinaryInputFormat to read Sequencefiles.
> However, we current don't have a way to *write* a sequencefile efficiently without going through extra (de)serializations.
> I'd like to store the correct classnames for key/values but use BytesWritable to write
> (in order for the next java or pig code to be able to read this sequencefile).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3460) SequenceFileAsBinaryOutputFormat
Posted by "Koji Noguchi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Koji Noguchi updated HADOOP-3460:
---------------------------------
Attachment: HADOOP-3460-part2.patch
Chris, thanks for the review.
bq. * Different properties for the output key/value classes aren't necessary; you can use the existing methods, like JobConf::getOutputKeyClass.
Reason I did it this way is, I want to use this ouputformat with C++Pipes.
{noformat}
(1) c++-reducer ---> (2) Java-PipesReducer ---> (3) collector ---> (4) SequenceFile(AsBinary)...
{noformat}
And (2) mapred/pipes/PipesReducer calls job.getOutputKeyClass() and job.getOutputValueClass(),
but I want those outputs to be BytesWritable and not the key/value classes of the SequenceFile.
How about this. Just like mapoutputkeyclass uses outputkeyclass as the default class, we'll use
outputkeyclass if SequenceFileOutputKeyClass is not being defined in the config.
bq. * The generic signature on the RecordWriter can be <BytesWritable,BytesWritable> if the signature on SeqFileOF were correct:
Done. Modified SequenceFile.java. Added @SuppressWarnings("unchecked") for MultipleSequenceFileOutputFormat.getBaseRecordWriter.
bq. * Since record compression is not supported, it might be worthwhile to override OutputFormat::checkOutputSpecs and throw if it's attempted
Done. Test added.
bq. * This should be in o.a.h.mapred.lib rather than o.a.h.mapred
Yes. Except that SequenceFileAsBinaryInputFormat is in o.a.h.mapred.
For now, I'll leave this to o.a.h.mapred and we can create a new Jira to move both of them to o.a.h.mapred.lib.
bq. * Keeping a WritableValueBytes instance around (and adding a reset method) might be useful, so a new one isn't created for each write.
Done. (Not sure if I did it correctly.)
bq. * The IllegalArgumentException in WritableValueBytes should probably be an UnsupportedOperationException
Done.
bq. * WritableValueBytes should be a _static_ inner class
Done.
bq. * The indentation on the anonymous RecordWriter::close should be consistent with the standards
Done
> SequenceFileAsBinaryOutputFormat
> --------------------------------
>
> Key: HADOOP-3460
> URL: https://issues.apache.org/jira/browse/HADOOP-3460
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Attachments: HADOOP-3460-part1.patch, HADOOP-3460-part2.patch
>
>
> Add an OutputFormat to write raw bytes as keys and values to a SequenceFile.
> In C++-Pipes, we're using SequenceFileAsBinaryInputFormat to read Sequencefiles.
> However, we current don't have a way to *write* a sequencefile efficiently without going through extra (de)serializations.
> I'd like to store the correct classnames for key/values but use BytesWritable to write
> (in order for the next java or pig code to be able to read this sequencefile).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3460) SequenceFileAsBinaryOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602924#action_12602924 ]
Chris Douglas commented on HADOOP-3460:
---------------------------------------
Just a few quick changes and I think this is ready to commit:
# The testcase: doesn't need a main method, you might want to break up the check for forbidding record compression into a separate test, and the call to JobConf::setInputPath is generating a warning (replace with FileInputFormat::addInputPath)
# WritableValueBytes::writeCompressedBytes no longer throws IllegalArgumentException, so that can be removed from its signature
# SeqFABOF::checkOutputSpecs doesn't need to list InvalidJobConfException
> SequenceFileAsBinaryOutputFormat
> --------------------------------
>
> Key: HADOOP-3460
> URL: https://issues.apache.org/jira/browse/HADOOP-3460
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Attachments: HADOOP-3460-part1.patch, HADOOP-3460-part2.patch
>
>
> Add an OutputFormat to write raw bytes as keys and values to a SequenceFile.
> In C++-Pipes, we're using SequenceFileAsBinaryInputFormat to read Sequencefiles.
> However, we current don't have a way to *write* a sequencefile efficiently without going through extra (de)serializations.
> I'd like to store the correct classnames for key/values but use BytesWritable to write
> (in order for the next java or pig code to be able to read this sequencefile).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3460) SequenceFileAsBinaryOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12602446#action_12602446 ]
Chris Douglas commented on HADOOP-3460:
---------------------------------------
This looks great; just a few points:
* Different properties for the output key/value classes aren't necessary; you can use the existing methods, like JobConf::getOutputKeyClass.
* The generic signature on the RecordWriter can be <BytesWritable,BytesWritable> if the signature on SeqFileOF were correct:
{noformat}
-public class SequenceFileOutputFormat
-extends FileOutputFormat<WritableComparable, Writable> {
+public class SequenceFileOutputFormat<K extends WritableComparable,
+ V extends Writable>
+ extends FileOutputFormat<K,V> {
{noformat}
Permitting SeqFABOF:
{noformat}
public class SequenceFileAsBinaryOutputFormat
extends SequenceFileOutputFormat<BytesWritable,BytesWritable> {
{noformat}
This generates a warning in MultipleSequenceFileOutputFormat, but it's spurious and can be suppressed.
* Since record compression is not supported, it might be worthwhile to override OutputFormat::checkOutputSpecs and throw if it's attempted
* This should be in o.a.h.mapred.lib rather than o.a.h.mapred
* Keeping a WritableValueBytes instance around (and adding a reset method) might be useful, so a new one isn't created for each write.
* The IllegalArgumentException in WritableValueBytes should probably be an UnsupportedOperationException
* WritableValueBytes should be a _static_ inner class
* The indentation on the anonymous RecordWriter::close should be consistent with the standards
> SequenceFileAsBinaryOutputFormat
> --------------------------------
>
> Key: HADOOP-3460
> URL: https://issues.apache.org/jira/browse/HADOOP-3460
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Koji Noguchi
> Priority: Minor
> Attachments: HADOOP-3460-part1.patch
>
>
> Add an OutputFormat to write raw bytes as keys and values to a SequenceFile.
> In C++-Pipes, we're using SequenceFileAsBinaryInputFormat to read Sequencefiles.
> However, we current don't have a way to *write* a sequencefile efficiently without going through extra (de)serializations.
> I'd like to store the correct classnames for key/values but use BytesWritable to write
> (in order for the next java or pig code to be able to read this sequencefile).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3460) SequenceFileAsBinaryOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3460:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
{noformat}
[exec] +1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] +1 tests included. The patch appears to include 3 new or modified tests.
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
{noformat}
I just committed this. Thanks, Koji
> SequenceFileAsBinaryOutputFormat
> --------------------------------
>
> Key: HADOOP-3460
> URL: https://issues.apache.org/jira/browse/HADOOP-3460
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.18.0
>
> Attachments: HADOOP-3460-part1.patch, HADOOP-3460-part2.patch, HADOOP-3460-part3.patch
>
>
> Add an OutputFormat to write raw bytes as keys and values to a SequenceFile.
> In C++-Pipes, we're using SequenceFileAsBinaryInputFormat to read Sequencefiles.
> However, we current don't have a way to *write* a sequencefile efficiently without going through extra (de)serializations.
> I'd like to store the correct classnames for key/values but use BytesWritable to write
> (in order for the next java or pig code to be able to read this sequencefile).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3460) SequenceFileAsBinaryOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas updated HADOOP-3460:
----------------------------------
Fix Version/s: 0.18.0
Hadoop Flags: [Reviewed]
Status: Patch Available (was: Open)
> SequenceFileAsBinaryOutputFormat
> --------------------------------
>
> Key: HADOOP-3460
> URL: https://issues.apache.org/jira/browse/HADOOP-3460
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Fix For: 0.18.0
>
> Attachments: HADOOP-3460-part1.patch, HADOOP-3460-part2.patch, HADOOP-3460-part3.patch
>
>
> Add an OutputFormat to write raw bytes as keys and values to a SequenceFile.
> In C++-Pipes, we're using SequenceFileAsBinaryInputFormat to read Sequencefiles.
> However, we current don't have a way to *write* a sequencefile efficiently without going through extra (de)serializations.
> I'd like to store the correct classnames for key/values but use BytesWritable to write
> (in order for the next java or pig code to be able to read this sequencefile).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-3460) SequenceFileAsBinaryOutputFormat
Posted by "Chris Douglas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Douglas reassigned HADOOP-3460:
-------------------------------------
Assignee: Koji Noguchi
> SequenceFileAsBinaryOutputFormat
> --------------------------------
>
> Key: HADOOP-3460
> URL: https://issues.apache.org/jira/browse/HADOOP-3460
> Project: Hadoop Core
> Issue Type: New Feature
> Components: mapred
> Reporter: Koji Noguchi
> Assignee: Koji Noguchi
> Priority: Minor
> Attachments: HADOOP-3460-part1.patch
>
>
> Add an OutputFormat to write raw bytes as keys and values to a SequenceFile.
> In C++-Pipes, we're using SequenceFileAsBinaryInputFormat to read Sequencefiles.
> However, we current don't have a way to *write* a sequencefile efficiently without going through extra (de)serializations.
> I'd like to store the correct classnames for key/values but use BytesWritable to write
> (in order for the next java or pig code to be able to read this sequencefile).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.