You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@pig.apache.org by "Joe Crobak (Created) (JIRA)" <ji...@apache.org> on 2011/10/14 15:32:12 UTC
[jira] [Created] (PIG-2319) Pig should support snappy as a value
for pig.tmpfilecompression.codec
Pig should support snappy as a value for pig.tmpfilecompression.codec
---------------------------------------------------------------------
Key: PIG-2319
URL: https://issues.apache.org/jira/browse/PIG-2319
Project: Pig
Issue Type: Improvement
Affects Versions: 0.8.1, 0.9.1
Reporter: Joe Crobak
Priority: Trivial
Utils.tmpFileCompressionCodec() hard-codes support for only "gz" and "lzo" compression. Since support for snappy was added in HADOOP-7206, it would be nice to allow this codec as well.
A future-proof solution to this problem might let the user provide a full classname (like in the hadoop settings) or the short-hand, in case the short-hand doesn't exist for a given codec.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2319) Pig should support snappy as a value
for pig.tmpfilecompression.codec
Posted by "Prashant Kommireddi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190794#comment-13190794 ]
Prashant Kommireddi commented on PIG-2319:
------------------------------------------
Seems like a good addition. I tested Snappy with Pig and it works just fine on compressing map output (uses the underlying Hadoop framework). We should extend the default Load/Store func to be able to use Snappy.
Currently LZO and GZ are hardcoded, but it would be nice to pick up the available codecs from "io.compression.codecs" and make them available. Thoughts?
> Pig should support snappy as a value for pig.tmpfilecompression.codec
> ---------------------------------------------------------------------
>
> Key: PIG-2319
> URL: https://issues.apache.org/jira/browse/PIG-2319
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.1, 0.9.1
> Reporter: Joe Crobak
>
> Utils.tmpFileCompressionCodec() hard-codes support for only "gz" and "lzo" compression. Since support for snappy was added in HADOOP-7206, it would be nice to allow this codec as well.
> A future-proof solution to this problem might let the user provide a full classname (like in the hadoop settings) or the short-hand, in case the short-hand doesn't exist for a given codec.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2319) Pig should support snappy as a value
for pig.tmpfilecompression.codec
Posted by "Prashant Kommireddi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190850#comment-13190850 ]
Prashant Kommireddi commented on PIG-2319:
------------------------------------------
TFileStorage is used for writing temporary/intermediate file and it looks only for gz and lzo. Code here will need to be modified for temporary files to be able to use other compression codecs.
> Pig should support snappy as a value for pig.tmpfilecompression.codec
> ---------------------------------------------------------------------
>
> Key: PIG-2319
> URL: https://issues.apache.org/jira/browse/PIG-2319
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.1, 0.9.1
> Reporter: Joe Crobak
>
> Utils.tmpFileCompressionCodec() hard-codes support for only "gz" and "lzo" compression. Since support for snappy was added in HADOOP-7206, it would be nice to allow this codec as well.
> A future-proof solution to this problem might let the user provide a full classname (like in the hadoop settings) or the short-hand, in case the short-hand doesn't exist for a given codec.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (PIG-2319) Pig should support snappy as a value
for pig.tmpfilecompression.codec
Posted by "Prashant Kommireddi (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashant Kommireddi updated PIG-2319:
-------------------------------------
Priority: Major (was: Trivial)
> Pig should support snappy as a value for pig.tmpfilecompression.codec
> ---------------------------------------------------------------------
>
> Key: PIG-2319
> URL: https://issues.apache.org/jira/browse/PIG-2319
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.1, 0.9.1
> Reporter: Joe Crobak
>
> Utils.tmpFileCompressionCodec() hard-codes support for only "gz" and "lzo" compression. Since support for snappy was added in HADOOP-7206, it would be nice to allow this codec as well.
> A future-proof solution to this problem might let the user provide a full classname (like in the hadoop settings) or the short-hand, in case the short-hand doesn't exist for a given codec.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2319) Pig should support snappy as a value
for pig.tmpfilecompression.codec
Posted by "Prashant Kommireddi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13464308#comment-13464308 ]
Prashant Kommireddi commented on PIG-2319:
------------------------------------------
Rakesh, this needs to be done on the Hadoop project. There is a JIRA for this https://issues.apache.org/jira/browse/HADOOP-7990
> Pig should support snappy as a value for pig.tmpfilecompression.codec
> ---------------------------------------------------------------------
>
> Key: PIG-2319
> URL: https://issues.apache.org/jira/browse/PIG-2319
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.1, 0.9.1
> Reporter: Joe Crobak
>
> Utils.tmpFileCompressionCodec() hard-codes support for only "gz" and "lzo" compression. Since support for snappy was added in HADOOP-7206, it would be nice to allow this codec as well.
> A future-proof solution to this problem might let the user provide a full classname (like in the hadoop settings) or the short-hand, in case the short-hand doesn't exist for a given codec.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2319) Pig should support snappy as a value
for pig.tmpfilecompression.codec
Posted by "Prashant Kommireddi (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192562#comment-13192562 ]
Prashant Kommireddi commented on PIG-2319:
------------------------------------------
Hi Dmitriy, I tested read a snappy compressed file with PigStorage and it works just fine.
{code}
grunt> set output.compression.enabled true;
grunt> set output.compression.codec org.apache.hadoop.io.compress.SnappyCodec;
grunt> A = load 'input';
grunt> rmf out;
grunt> STORE A INTO 'out'
{code}
Pig generates a snappy compressed file at location "out"
{code}
grunt> C = load 'out';
grunt> D = LIMIT C 10;
grunt> DUMP D;
{code}
The above successfully reads snappy compressed file, as PigStorage uses the Hadoop TextInputFormat in this case.
However, this is not the case for temporary files created by Pig between multiple MR jobs because TFile Writer is used which supports only LZO and GZ. Do you see a workaround we could find to support Snappy in this case?
> Pig should support snappy as a value for pig.tmpfilecompression.codec
> ---------------------------------------------------------------------
>
> Key: PIG-2319
> URL: https://issues.apache.org/jira/browse/PIG-2319
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.1, 0.9.1
> Reporter: Joe Crobak
>
> Utils.tmpFileCompressionCodec() hard-codes support for only "gz" and "lzo" compression. Since support for snappy was added in HADOOP-7206, it would be nice to allow this codec as well.
> A future-proof solution to this problem might let the user provide a full classname (like in the hadoop settings) or the short-hand, in case the short-hand doesn't exist for a given codec.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2319) Pig should support snappy as a value
for pig.tmpfilecompression.codec
Posted by "Rakesh Kothari (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13429320#comment-13429320 ]
Rakesh Kothari commented on PIG-2319:
-------------------------------------
Any updates on this ?
> Pig should support snappy as a value for pig.tmpfilecompression.codec
> ---------------------------------------------------------------------
>
> Key: PIG-2319
> URL: https://issues.apache.org/jira/browse/PIG-2319
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.1, 0.9.1
> Reporter: Joe Crobak
>
> Utils.tmpFileCompressionCodec() hard-codes support for only "gz" and "lzo" compression. Since support for snappy was added in HADOOP-7206, it would be nice to allow this codec as well.
> A future-proof solution to this problem might let the user provide a full classname (like in the hadoop settings) or the short-hand, in case the short-hand doesn't exist for a given codec.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2319) Pig should support snappy as a value
for pig.tmpfilecompression.codec
Posted by "Andrew Look (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489880#comment-13489880 ]
Andrew Look commented on PIG-2319:
----------------------------------
I'm working on a patch for HADOOP-7990 - any thoughts on which version of hadoop-common this should be applied to?
> Pig should support snappy as a value for pig.tmpfilecompression.codec
> ---------------------------------------------------------------------
>
> Key: PIG-2319
> URL: https://issues.apache.org/jira/browse/PIG-2319
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.1, 0.9.1
> Reporter: Joe Crobak
>
> Utils.tmpFileCompressionCodec() hard-codes support for only "gz" and "lzo" compression. Since support for snappy was added in HADOOP-7206, it would be nice to allow this codec as well.
> A future-proof solution to this problem might let the user provide a full classname (like in the hadoop settings) or the short-hand, in case the short-hand doesn't exist for a given codec.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2319) Pig should support snappy as a value
for pig.tmpfilecompression.codec
Posted by "Dmitriy V. Ryaboy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190849#comment-13190849 ]
Dmitriy V. Ryaboy commented on PIG-2319:
----------------------------------------
Ok, so it looks like we'll automatically do the right thing for storage if we have output.compression.enabled and output.compression.codec set. We don't do the same for reading.
PIG-2143 had a sketch of how to make the whole thing a more flexible implementation (see the first comment). Should be straightforward enough to allow specifying "-compression=$foo" and have the codec for $foo looked up dynamically.
> Pig should support snappy as a value for pig.tmpfilecompression.codec
> ---------------------------------------------------------------------
>
> Key: PIG-2319
> URL: https://issues.apache.org/jira/browse/PIG-2319
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.1, 0.9.1
> Reporter: Joe Crobak
>
> Utils.tmpFileCompressionCodec() hard-codes support for only "gz" and "lzo" compression. Since support for snappy was added in HADOOP-7206, it would be nice to allow this codec as well.
> A future-proof solution to this problem might let the user provide a full classname (like in the hadoop settings) or the short-hand, in case the short-hand doesn't exist for a given codec.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (PIG-2319) Pig should support snappy as a value
for pig.tmpfilecompression.codec
Posted by "Dmitriy V. Ryaboy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/PIG-2319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190847#comment-13190847 ]
Dmitriy V. Ryaboy commented on PIG-2319:
----------------------------------------
Yeah I even had a patch to do that somewhere (we cleaned up PigStorage a bunch in the fall..)
> Pig should support snappy as a value for pig.tmpfilecompression.codec
> ---------------------------------------------------------------------
>
> Key: PIG-2319
> URL: https://issues.apache.org/jira/browse/PIG-2319
> Project: Pig
> Issue Type: Improvement
> Affects Versions: 0.8.1, 0.9.1
> Reporter: Joe Crobak
>
> Utils.tmpFileCompressionCodec() hard-codes support for only "gz" and "lzo" compression. Since support for snappy was added in HADOOP-7206, it would be nice to allow this codec as well.
> A future-proof solution to this problem might let the user provide a full classname (like in the hadoop settings) or the short-hand, in case the short-hand doesn't exist for a given codec.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira