You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Vivek Padmanabhan (JIRA)" <ji...@apache.org> on 2011/06/17 12:15:47 UTC
[jira] [Created] (PIG-2130) Piggybank:MultiStorage is not
compressing output files
Piggybank:MultiStorage is not compressing output files
------------------------------------------------------
Key: PIG-2130
URL: https://issues.apache.org/jira/browse/PIG-2130
Project: Pig
Issue Type: Bug
Affects Versions: 0.8.0, 0.9.0
Reporter: Vivek Padmanabhan
Assignee: Vivek Padmanabhan
MultiStorage is not compressing the records while writing the output. Even though it takes a compression param, when the record is written it ignores the compression.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2130) Piggybank:MultiStorage is not
compressing output files
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13063707#comment-13063707 ]
Daniel Dai commented on PIG-2130:
---------------------------------
All tests pass. test-patch all pass. Commit to trunk first.
> Piggybank:MultiStorage is not compressing output files
> ------------------------------------------------------
>
> Key: PIG-2130
> URL: https://issues.apache.org/jira/browse/PIG-2130
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.9.0
>
> Attachments: PIG-2130_1.patch
>
>
> MultiStorage is not compressing the records while writing the output. Even though it takes a compression param, when the record is written it ignores the compression.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2130) Piggybank:MultiStorage is not
compressing output files
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13062863#comment-13062863 ]
jiraposter@reviews.apache.org commented on PIG-2130:
----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1063/
-----------------------------------------------------------
Review request for pig.
Summary
-------
MultiStorage is not compressing the records while writing the output. Even though it takes a compression param, when the record is written it ignores the compression.
As a fix enabled compressing the output.
If compression is used, then the sub diretories and output files will be having the corresponding extension.
For example, if output001.bz2 is output path and f1,f2 are the keys, the files will look like;
/tmp/output001.bz2
/tmp/output001.bz2/f1.bz2
/tmp/output001.bz2/f1.bz2/f1-0.bz2
/tmp/output001.bz2/f2.bz2
/tmp/output001.bz2/f2.bz2/f2-0.bz2
This addresses bug PIG-2130.
https://issues.apache.org/jira/browse/PIG-2130
Diffs
-----
Diff: https://reviews.apache.org/r/1063/diff
Testing
-------
Thanks,
Vivek
> Piggybank:MultiStorage is not compressing output files
> ------------------------------------------------------
>
> Key: PIG-2130
> URL: https://issues.apache.org/jira/browse/PIG-2130
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.9.0
>
> Attachments: PIG-2130_1.patch
>
>
> MultiStorage is not compressing the records while writing the output. Even though it takes a compression param, when the record is written it ignores the compression.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2130) Piggybank:MultiStorage is not
compressing output files
Posted by "Vivek Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vivek Padmanabhan updated PIG-2130:
-----------------------------------
Fix Version/s: 0.8.0
0.9.0
Status: Patch Available (was: Open)
> Piggybank:MultiStorage is not compressing output files
> ------------------------------------------------------
>
> Key: PIG-2130
> URL: https://issues.apache.org/jira/browse/PIG-2130
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.9.0, 0.8.0
>
> Attachments: PIG-2130_1.patch
>
>
> MultiStorage is not compressing the records while writing the output. Even though it takes a compression param, when the record is written it ignores the compression.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2130) Piggybank:MultiStorage is not
compressing output files
Posted by "Vivek Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051841#comment-13051841 ]
Vivek Padmanabhan commented on PIG-2130:
----------------------------------------
Please note that , if compression is used, then the subfolders and output files will be having the corresponding extension.
For example, if output001.bz2 is output path and f1,f2 are the keys, the files will look like;
/tmp/output001.bz2
/tmp/output001.bz2/f1.bz2
/tmp/output001.bz2/f1.bz2/f1-0.bz2
/tmp/output001.bz2/f2.bz2
/tmp/output001.bz2/f2.bz2/f2-0.bz2
> Piggybank:MultiStorage is not compressing output files
> ------------------------------------------------------
>
> Key: PIG-2130
> URL: https://issues.apache.org/jira/browse/PIG-2130
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Attachments: PIG-2130_1.patch
>
>
> MultiStorage is not compressing the records while writing the output. Even though it takes a compression param, when the record is written it ignores the compression.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2130) Piggybank:MultiStorage is not
compressing output files
Posted by "Olga Natkovich (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Olga Natkovich updated PIG-2130:
--------------------------------
Fix Version/s: (was: 0.8.0)
> Piggybank:MultiStorage is not compressing output files
> ------------------------------------------------------
>
> Key: PIG-2130
> URL: https://issues.apache.org/jira/browse/PIG-2130
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.9.0
>
> Attachments: PIG-2130_1.patch
>
>
> MultiStorage is not compressing the records while writing the output. Even though it takes a compression param, when the record is written it ignores the compression.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2130) Piggybank:MultiStorage is not
compressing output files
Posted by "Vivek Padmanabhan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vivek Padmanabhan updated PIG-2130:
-----------------------------------
Attachment: PIG-2130_1.patch
Attaching an initial patch
> Piggybank:MultiStorage is not compressing output files
> ------------------------------------------------------
>
> Key: PIG-2130
> URL: https://issues.apache.org/jira/browse/PIG-2130
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Attachments: PIG-2130_1.patch
>
>
> MultiStorage is not compressing the records while writing the output. Even though it takes a compression param, when the record is written it ignores the compression.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2130) Piggybank:MultiStorage is not
compressing output files
Posted by "Daniel Dai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Dai updated PIG-2130:
----------------------------
Resolution: Fixed
Fix Version/s: (was: 0.9.0)
0.10
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
This seems not an urgent issue, I don't think we need to back port to 0.9. Since we already commit the patch to trunk, close the ticket.
> Piggybank:MultiStorage is not compressing output files
> ------------------------------------------------------
>
> Key: PIG-2130
> URL: https://issues.apache.org/jira/browse/PIG-2130
> Project: Pig
> Issue Type: Bug
> Affects Versions: 0.8.0, 0.9.0
> Reporter: Vivek Padmanabhan
> Assignee: Vivek Padmanabhan
> Fix For: 0.10
>
> Attachments: PIG-2130_1.patch
>
>
> MultiStorage is not compressing the records while writing the output. Even though it takes a compression param, when the record is written it ignores the compression.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira