You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Daehyun Kim (JIRA)" <ji...@apache.org> on 2009/07/16 04:23:14 UTC
[jira] Created: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
RAgzip: multiple map tasks for a large gzipped file
---------------------------------------------------
Key: HADOOP-6153
URL: https://issues.apache.org/jira/browse/HADOOP-6153
Project: Hadoop Common
Issue Type: Bug
Components: io, native
Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
Reporter: Daehyun Kim
Assignee: Daehyun Kim
Priority: Minor
It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-6153:
--------------------------------
Status: Open (was: Patch Available)
This patch fell out of date. I'm regenerating a new patch against trunk, will resubmit to Hudson soon.
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Affects Versions: 0.21.0
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Work started: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Daehyun Kim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HADOOP-6153 started by Daehyun Kim.
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Daehyun Kim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daehyun Kim updated HADOOP-6153:
--------------------------------
Attachment: HADOOP-6153.patch
My patch brings the hunk error.
I think the reason I copied the text of the patch in my linux terminal and pasted the text to notepad.
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Daehyun Kim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daehyun Kim updated HADOOP-6153:
--------------------------------
Status: In Progress (was: Patch Available)
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12791736#action_12791736 ]
Hadoop QA commented on HADOOP-6153:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12428245/hadoop-6153.txt
against trunk revision 891511.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/214/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/214/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/214/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-h4.grid.sp2.yahoo.net/214/console
This message is automatically generated.
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Affects Versions: 0.21.0
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-6153.patch, hadoop-6153.txt
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Daehyun Kim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daehyun Kim updated HADOOP-6153:
--------------------------------
Issue Type: Improvement (was: Bug)
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-6153:
--------------------------------
Attachment: hadoop-6153.txt
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Affects Versions: 0.21.0
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-6153.patch, hadoop-6153.txt
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Daehyun Kim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daehyun Kim updated HADOOP-6153:
--------------------------------
Attachment: HADOOP-6153.patch
1. To merge RAGzip with GzipCodec class, I add a method 'getDecompressorForRandomAccess(...)' to GzipCodec class. This method return the Decompressor for random access.
2. I add the '-makeap' option that is FS Shell Command. This option supports to make the access point of the gzip.
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Daehyun Kim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daehyun Kim updated HADOOP-6153:
--------------------------------
Affects Version/s: 0.21.0
Fix Version/s: 0.21.0
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Affects Versions: 0.21.0
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735962#action_12735962 ]
Hadoop QA commented on HADOOP-6153:
-----------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12414706/HADOOP-6153.patch
against trunk revision 798247.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/590/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/590/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/590/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/590/console
This message is automatically generated.
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Daehyun Kim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daehyun Kim updated HADOOP-6153:
--------------------------------
Status: Patch Available (was: In Progress)
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Daehyun Kim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daehyun Kim updated HADOOP-6153:
--------------------------------
Attachment: (was: HADOOP-6153.patch)
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Daehyun Kim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daehyun Kim updated HADOOP-6153:
--------------------------------
Hadoop Flags: [Reviewed]
Status: Patch Available (was: In Progress)
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12928648#action_12928648 ]
Hadoop QA commented on HADOOP-6153:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12428245/hadoop-6153.txt
against trunk revision 1031422.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://hudson.apache.org/hudson/job/PreCommit-HADOOP-Build/72//console
This message is automatically generated.
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Affects Versions: 0.21.0
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Attachments: HADOOP-6153.patch, hadoop-6153.txt
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12735632#action_12735632 ]
Hadoop QA commented on HADOOP-6153:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12414595/HADOOP-6153.patch
against trunk revision 798093.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 6 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/589/console
This message is automatically generated.
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Todd Lipcon updated HADOOP-6153:
--------------------------------
Status: Patch Available (was: Open)
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Affects Versions: 0.21.0
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-6153.patch, hadoop-6153.txt
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-6153) RAgzip: multiple map tasks for a
large gzipped file
Posted by "Daehyun Kim (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-6153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daehyun Kim updated HADOOP-6153:
--------------------------------
Hadoop Flags: (was: [Reviewed])
> RAgzip: multiple map tasks for a large gzipped file
> ---------------------------------------------------
>
> Key: HADOOP-6153
> URL: https://issues.apache.org/jira/browse/HADOOP-6153
> Project: Hadoop Common
> Issue Type: Improvement
> Components: io, native
> Environment: It requires zlib 1.2.2.4 or higher. (We tested on zlib 1.2.3)
> Reporter: Daehyun Kim
> Assignee: Daehyun Kim
> Priority: Minor
> Attachments: HADOOP-6153.patch
>
>
> It support to enable multiple map tasks for one large gzipped file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.