You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Greg Roelofs (JIRA)" <ji...@apache.org> on 2010/07/08 23:59:05 UTC
[jira] Created: (MAPREDUCE-1927) unit test for HADOOP-6835
(concatenated gzip support)
unit test for HADOOP-6835 (concatenated gzip support)
-----------------------------------------------------
Key: MAPREDUCE-1927
URL: https://issues.apache.org/jira/browse/MAPREDUCE-1927
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 0.22.0
Reporter: Greg Roelofs
Assignee: Greg Roelofs
Priority: Minor
Fix For: 0.22.0
More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1927) unit test for HADOOP-6835
(concatenated gzip support)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Roelofs updated MAPREDUCE-1927:
------------------------------------
Attachment: MR-1927.v13.trunk-hadoop-mapreduce.patch.txt
Fixed the internal structure of the extra field in one of the test files (not relevant to the test, but a matter of spec correctness), as well as the corresponding file length in the unit test (3409 -> 3413 bytes). No substantive changes since v12.
> unit test for HADOOP-6835 (concatenated gzip support)
> -----------------------------------------------------
>
> Key: MAPREDUCE-1927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1927
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Greg Roelofs
> Assignee: Greg Roelofs
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: construct-all-gz-header-types-20100812.zip, MR-1927.v12.trunk-hadoop-mapreduce.patch, MR-1927.v13.trunk-hadoop-mapreduce.patch.txt, MR-1927.v9.trunk-hadoop-mapreduce.patch
>
>
> More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1927) unit test for HADOOP-6835
(concatenated gzip support)
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12886590#action_12886590 ]
Hadoop QA commented on MAPREDUCE-1927:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12449019/MR-1927.v9.trunk-hadoop-mapreduce.patch
against trunk revision 961578.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 21 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
-1 contrib tests. The patch failed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/294/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/294/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/294/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Mapreduce-Patch-h4.grid.sp2.yahoo.net/294/console
This message is automatically generated.
> unit test for HADOOP-6835 (concatenated gzip support)
> -----------------------------------------------------
>
> Key: MAPREDUCE-1927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1927
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Greg Roelofs
> Assignee: Greg Roelofs
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MR-1927.v9.trunk-hadoop-mapreduce.patch
>
>
> More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1927) unit test for HADOOP-6835
(concatenated gzip support)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Roelofs updated MAPREDUCE-1927:
------------------------------------
Attachment: construct-all-gz-header-types-20100812.zip
Script and gzip component subfiles to generate all 32 unencrypted flagbits variants (that is, text/binary, comment/no comment, header CRC/no header CRC, etc.).
These are not currently used by the unit test, but they could be. Is it worth it?
(Note that gzip versions up through 1.4 cannot be used to test the 16 header-CRC variants. A fix was checked into gzip's trunk on 20100703, however: http://git.savannah.gnu.org/cgit/gzip.git/commit/?id=97e225c0e8ee24f3fc9ff7c5cec63e74aefbb6a0 . zlib-based gzip decoders [zlib >= 1.2.1.2] can read all 32 variants successfully.)
> unit test for HADOOP-6835 (concatenated gzip support)
> -----------------------------------------------------
>
> Key: MAPREDUCE-1927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1927
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Greg Roelofs
> Assignee: Greg Roelofs
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: construct-all-gz-header-types-20100812.zip, MR-1927.v9.trunk-hadoop-mapreduce.patch
>
>
> More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1927) unit test for HADOOP-6835
(concatenated gzip support)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898497#action_12898497 ]
Greg Roelofs commented on MAPREDUCE-1927:
-----------------------------------------
Looks like Hudson's gone off the deep end again. Here's my "ant test-patch" results:
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 21 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
BUILD SUCCESSFUL
Total time: 39 minutes 15 seconds
> unit test for HADOOP-6835 (concatenated gzip support)
> -----------------------------------------------------
>
> Key: MAPREDUCE-1927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1927
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Greg Roelofs
> Assignee: Greg Roelofs
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: construct-all-gz-header-types-20100812.zip, MR-1927.v12.trunk-hadoop-mapreduce.patch, MR-1927.v9.trunk-hadoop-mapreduce.patch
>
>
> More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1927) unit test for HADOOP-6835
(concatenated gzip support)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Roelofs updated MAPREDUCE-1927:
------------------------------------
Attachment: MR-1927.v12.trunk-hadoop-mapreduce.patch
Updated (final?) patch. This corrects the internals of the gzip extra field in one of the test files (not relevant to the test, but matter of spec-compliance), changes a native-libs-only assertion to a warning to work around Apache Hudson's lack of support, and removes all but three commented-out code blocks.
The three remaining blocks are all a result of other bugs: the aforementioned Hudson native-library one (lack of coverage of a key infrastructure component is a problem, and it's getting worse with every new native codec); HADOOP-6799 (broken reset() on gzip compression); and HADOOP-6852 (broken bzip2 decompression, apparently). I realize commented-out code is frowned upon as a style issue, but I would rather not remove useful tests that are currently blocked by things outside my (immediate) control. I'll defer to committer's opinion, however.
This passed "ant test-patch" on a 10-node cluster, so I'm going to "submit patch" to see if it gets past Apache's Hudson, too.
> unit test for HADOOP-6835 (concatenated gzip support)
> -----------------------------------------------------
>
> Key: MAPREDUCE-1927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1927
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Greg Roelofs
> Assignee: Greg Roelofs
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: construct-all-gz-header-types-20100812.zip, MR-1927.v12.trunk-hadoop-mapreduce.patch, MR-1927.v9.trunk-hadoop-mapreduce.patch
>
>
> More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1927) unit test for HADOOP-6835
(concatenated gzip support)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Roelofs updated MAPREDUCE-1927:
------------------------------------
Status: Patch Available (was: Open)
> unit test for HADOOP-6835 (concatenated gzip support)
> -----------------------------------------------------
>
> Key: MAPREDUCE-1927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1927
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Greg Roelofs
> Assignee: Greg Roelofs
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MR-1927.v9.trunk-hadoop-mapreduce.patch
>
>
> More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1927) unit test for HADOOP-6835
(concatenated gzip support)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Roelofs updated MAPREDUCE-1927:
------------------------------------
Attachment: MR-1927.v9.trunk-hadoop-mapreduce.patch
Test code, six associated test files, and build.xml changes to copy them to the appropriate location.
> unit test for HADOOP-6835 (concatenated gzip support)
> -----------------------------------------------------
>
> Key: MAPREDUCE-1927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1927
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Greg Roelofs
> Assignee: Greg Roelofs
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MR-1927.v9.trunk-hadoop-mapreduce.patch
>
>
> More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1927) unit test for HADOOP-6835
(concatenated gzip support)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Roelofs updated MAPREDUCE-1927:
------------------------------------
Status: Patch Available (was: Open)
Extended hadoop-mapreduce unit test for hadoop-common's gzip and bzip2 codecs. (Supplements TestCodec in hadoop-common.)
> unit test for HADOOP-6835 (concatenated gzip support)
> -----------------------------------------------------
>
> Key: MAPREDUCE-1927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1927
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Greg Roelofs
> Assignee: Greg Roelofs
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: construct-all-gz-header-types-20100812.zip, MR-1927.v12.trunk-hadoop-mapreduce.patch, MR-1927.v9.trunk-hadoop-mapreduce.patch
>
>
> More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1927) unit test for HADOOP-6835
(concatenated gzip support)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Roelofs updated MAPREDUCE-1927:
------------------------------------
Status: Patch Available (was: Open)
> unit test for HADOOP-6835 (concatenated gzip support)
> -----------------------------------------------------
>
> Key: MAPREDUCE-1927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1927
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Greg Roelofs
> Assignee: Greg Roelofs
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: construct-all-gz-header-types-20100812.zip, MR-1927.v12.trunk-hadoop-mapreduce.patch, MR-1927.v13.trunk-hadoop-mapreduce.patch.txt, MR-1927.v9.trunk-hadoop-mapreduce.patch
>
>
> More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1927) unit test for HADOOP-6835
(concatenated gzip support)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Roelofs updated MAPREDUCE-1927:
------------------------------------
Status: Open (was: Patch Available)
> unit test for HADOOP-6835 (concatenated gzip support)
> -----------------------------------------------------
>
> Key: MAPREDUCE-1927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1927
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Greg Roelofs
> Assignee: Greg Roelofs
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: construct-all-gz-header-types-20100812.zip, MR-1927.v12.trunk-hadoop-mapreduce.patch, MR-1927.v9.trunk-hadoop-mapreduce.patch
>
>
> More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-1927) unit test for HADOOP-6835
(concatenated gzip support)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12898079#action_12898079 ]
Greg Roelofs commented on MAPREDUCE-1927:
-----------------------------------------
Oops, small correction: zlib must be newer than 1.2.1.2 to handle header CRCs correctly. RHEL4 ships with 1.2.1.2, and it's buggy.
> unit test for HADOOP-6835 (concatenated gzip support)
> -----------------------------------------------------
>
> Key: MAPREDUCE-1927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1927
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Greg Roelofs
> Assignee: Greg Roelofs
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: construct-all-gz-header-types-20100812.zip, MR-1927.v9.trunk-hadoop-mapreduce.patch
>
>
> More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-1927) unit test for HADOOP-6835
(concatenated gzip support)
Posted by "Greg Roelofs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Greg Roelofs updated MAPREDUCE-1927:
------------------------------------
Status: Open (was: Patch Available)
Updated, test-patch-passing patch coming shortly.
> unit test for HADOOP-6835 (concatenated gzip support)
> -----------------------------------------------------
>
> Key: MAPREDUCE-1927
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-1927
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Affects Versions: 0.22.0
> Reporter: Greg Roelofs
> Assignee: Greg Roelofs
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: MR-1927.v9.trunk-hadoop-mapreduce.patch
>
>
> More extensive test of concatenated gzip (and bzip2) decoding support for HADOOP-6835 (and HADOOP-4012 and HADOOP-6852).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.