You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Pradeep Kamath (JIRA)" <ji...@apache.org> on 2010/03/22 20:18:27 UTC
[jira] Created: (PIG-1316) TextLoader should use
Bzip2TextInputFormat for bzip files so that bzip files can be efficiently
processed by splitting the files
TextLoader should use Bzip2TextInputFormat for bzip files so that bzip files can be efficiently processed by splitting the files
--------------------------------------------------------------------------------------------------------------------------------
Key: PIG-1316
URL: https://issues.apache.org/jira/browse/PIG-1316
Project: Pig
Issue Type: Improvement
Affects Versions: 0.7.0
Reporter: Pradeep Kamath
Assignee: Pradeep Kamath
Fix For: 0.7.0
Currently TextLoader uses TextInputFormat which does not split bzip files - this can be fixed by using Bzip2TextInputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1316) TextLoader should use
Bzip2TextInputFormat for bzip files so that bzip files can be efficiently
processed by splitting the files
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850386#action_12850386 ]
Pradeep Kamath commented on PIG-1316:
-------------------------------------
All unit tests passed - will commit shorty:
...
[junit] Running org.apache.pig.test.TestUnion
[junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 49.523 sec
test-contrib:
BUILD SUCCESSFUL
Total time: 278 minutes 11 seconds
> TextLoader should use Bzip2TextInputFormat for bzip files so that bzip files can be efficiently processed by splitting the files
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1316
> URL: https://issues.apache.org/jira/browse/PIG-1316
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1316.patch
>
>
> Currently TextLoader uses TextInputFormat which does not split bzip files - this can be fixed by using Bzip2TextInputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1316) TextLoader should use
Bzip2TextInputFormat for bzip files so that bzip files can be efficiently
processed by splitting the files
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1316:
--------------------------------
Attachment: PIG-1316.patch
Attached patch makes the required changes in TextLoader to use BZip2TextInputFormat if the load location ends with extension ".bz" or ".bz2" like PigStorage. Also for non bzip data, TextLoader will now use PigTextInputFormat rather than TextInputFormat so that input directories can be recursively traversed. I have also changed BZip2TextInputFormat to extend PigFileInputFormat instead of FileInputFormat for the same reason.
> TextLoader should use Bzip2TextInputFormat for bzip files so that bzip files can be efficiently processed by splitting the files
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1316
> URL: https://issues.apache.org/jira/browse/PIG-1316
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1316.patch
>
>
> Currently TextLoader uses TextInputFormat which does not split bzip files - this can be fixed by using Bzip2TextInputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1316) TextLoader should use
Bzip2TextInputFormat for bzip files so that bzip files can be efficiently
processed by splitting the files
Posted by "Richard Ding (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850246#action_12850246 ]
Richard Ding commented on PIG-1316:
-----------------------------------
+1
> TextLoader should use Bzip2TextInputFormat for bzip files so that bzip files can be efficiently processed by splitting the files
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1316
> URL: https://issues.apache.org/jira/browse/PIG-1316
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1316.patch
>
>
> Currently TextLoader uses TextInputFormat which does not split bzip files - this can be fixed by using Bzip2TextInputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1316) TextLoader should use
Bzip2TextInputFormat for bzip files so that bzip files can be efficiently
processed by splitting the files
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1316:
--------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Patch committed to trunk and branch-0.7
> TextLoader should use Bzip2TextInputFormat for bzip files so that bzip files can be efficiently processed by splitting the files
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1316
> URL: https://issues.apache.org/jira/browse/PIG-1316
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1316.patch
>
>
> Currently TextLoader uses TextInputFormat which does not split bzip files - this can be fixed by using Bzip2TextInputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (PIG-1316) TextLoader should use
Bzip2TextInputFormat for bzip files so that bzip files can be efficiently
processed by splitting the files
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12850266#action_12850266 ]
Pradeep Kamath commented on PIG-1316:
-------------------------------------
"test-patch" ant target results frm running locally:
[exec]
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] +1 tests included. The patch appears to include 13 new or modified tests.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the total number of release audit warnings.
[exec]
[exec]
[exec]
[exec]
[exec] ======================================================================
[exec] ======================================================================
[exec] Finished build.
[exec] ======================================================================
[exec] ======================================================================
Am also running the unit tests locally - the tests require some manual data setup (the data file for the test needs to be created before test run - the patch cannot handle these actions) - will update with results.
> TextLoader should use Bzip2TextInputFormat for bzip files so that bzip files can be efficiently processed by splitting the files
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1316
> URL: https://issues.apache.org/jira/browse/PIG-1316
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1316.patch
>
>
> Currently TextLoader uses TextInputFormat which does not split bzip files - this can be fixed by using Bzip2TextInputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (PIG-1316) TextLoader should use
Bzip2TextInputFormat for bzip files so that bzip files can be efficiently
processed by splitting the files
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai closed PIG-1316.
---------------------------
> TextLoader should use Bzip2TextInputFormat for bzip files so that bzip files can be efficiently processed by splitting the files
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1316
> URL: https://issues.apache.org/jira/browse/PIG-1316
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1316.patch
>
>
> Currently TextLoader uses TextInputFormat which does not split bzip files - this can be fixed by using Bzip2TextInputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (PIG-1316) TextLoader should use
Bzip2TextInputFormat for bzip files so that bzip files can be efficiently
processed by splitting the files
Posted by "Pradeep Kamath (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Pradeep Kamath updated PIG-1316:
--------------------------------
Status: Patch Available (was: Open)
> TextLoader should use Bzip2TextInputFormat for bzip files so that bzip files can be efficiently processed by splitting the files
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: PIG-1316
> URL: https://issues.apache.org/jira/browse/PIG-1316
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.7.0
> Reporter: Pradeep Kamath
> Assignee: Pradeep Kamath
> Fix For: 0.7.0
>
> Attachments: PIG-1316.patch
>
>
> Currently TextLoader uses TextInputFormat which does not split bzip files - this can be fixed by using Bzip2TextInputformat.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.