You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org> on 2011/07/06 21:08:16 UTC
[jira] [Created] (HIVE-2266) Fix compression parameters
Fix compression parameters
--------------------------
Key: HIVE-2266
URL: https://issues.apache.org/jira/browse/HIVE-2266
Project: Hive
Issue Type: Bug
Reporter: Vaibhav Aggarwal
Assignee: Vaibhav Aggarwal
There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2266) Fix compression parameters
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vaibhav Aggarwal updated HIVE-2266:
-----------------------------------
Status: Patch Available (was: Open)
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-2266.patch
>
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2266) Fix compression parameters
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060778#comment-13060778 ]
Vaibhav Aggarwal commented on HIVE-2266:
----------------------------------------
Review request:
https://reviews.apache.org/r/1017/
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2266) Fix compression parameters
Posted by "jiraposter@reviews.apache.org (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13060779#comment-13060779 ]
jiraposter@reviews.apache.org commented on HIVE-2266:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1017/
-----------------------------------------------------------
Review request for hive.
Summary
-------
Fix compression parameters for FileSinkOperator
This addresses bug HIVE-2266.
https://issues.apache.org/jira/browse/HIVE-2266
Diffs
-----
ql/src/java/org/apache/hadoop/hive/ql/io/HiveFileFormatUtils.java 404c4d3
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMRFileSink1.java 092a5cd
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 8fe5984
Diff: https://reviews.apache.org/r/1017/diff
Testing
-------
Thanks,
Vaibhav
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2266) Fix compression parameters
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vaibhav Aggarwal updated HIVE-2266:
-----------------------------------
Attachment: HIVE-2266.patch
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-2266.patch
>
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2266) Fix compression parameters
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vaibhav Aggarwal updated HIVE-2266:
-----------------------------------
Status: Patch Available (was: Open)
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-2266-2.patch, HIVE-2266.patch
>
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2266) Fix compression parameters
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072546#comment-13072546 ]
Carl Steinbach commented on HIVE-2266:
--------------------------------------
@Vaibav: Is this patch ready for review? If so, please attach the patch to this ticket and click "Submit Patch". Thanks.
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2266) Fix compression parameters
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13072549#comment-13072549 ]
Vaibhav Aggarwal commented on HIVE-2266:
----------------------------------------
Carl, I have attached the patch.
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-2266.patch
>
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2266) Fix compression parameters
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13092161#comment-13092161 ]
Vaibhav Aggarwal commented on HIVE-2266:
----------------------------------------
Hi Carl
The patch fixes compression which means that there is a difference in file size only.
Besides the file size will differ on different platforms. Hence there is no way to test this except for manual test.
This test case executes the new code and makes sure that it works.
Sent from my phone.
----- Reply message -----
From: "Carl Steinbach (JIRA)" <ji...@apache.org>
Date: Fri, Aug 26, 2011 6:13 pm
Subject: [jira] [Updated] (HIVE-2266) Fix compression parameters
To: "Aggarwal, Vaibhav" <va...@amazon.com>
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2266:
---------------------------------
Status: Open (was: Patch Available)
@Vaibhav: The compress.q test passes even without your patch in place. Can you please supply a testcase that fails without your patch?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-2266-2.patch, HIVE-2266.patch
>
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2266) Fix compression parameters
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2266:
---------------------------------
Status: Open (was: Patch Available)
@Vaibhav: The compress.q test passes even without your patch in place. Can you please supply a testcase that fails without your patch?
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-2266-2.patch, HIVE-2266.patch
>
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2266) Fix compression parameters
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vaibhav Aggarwal updated HIVE-2266:
-----------------------------------
Attachment: HIVE-2266-2.patch
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-2266-2.patch, HIVE-2266.patch
>
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2266) Fix compression parameters
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13504479#comment-13504479 ]
Harsh J commented on HIVE-2266:
-------------------------------
bq. Hadoop loads native compression libraries. I believe that they are platform dependent hence I do not assume that they always have same compression ratio. Please correct me if I am wrong here.
Compression is based on standard algorithms, which is platform independent. The native code is platform-dependent cause of the library references it has.
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-2266-2.patch, HIVE-2266.patch
>
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2266) Fix compression parameters
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vaibhav Aggarwal updated HIVE-2266:
-----------------------------------
Status: Patch Available (was: Open)
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-2266-2.patch, HIVE-2266.patch
>
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2266) Fix compression parameters
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2266:
---------------------------------
Status: Open (was: Patch Available)
@Vaibhav: The changes look good, but it needs a test. Is it feasible to add something to TestCliDriver? Thanks.
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-2266.patch
>
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2266) Fix compression parameters
Posted by "Vaibhav Aggarwal (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13095747#comment-13095747 ]
Vaibhav Aggarwal commented on HIVE-2266:
----------------------------------------
This patch attempts to fix a bug in the existing functionality in two ways:
1. In HiveFileFormatUtils.java, wrong jobconf is getting passed which is clear from the context.
2. In other cases the compression parameters are not getting set.
The only difference this patch produces from the current behavior is smaller file sizes on file system. I am not sure how to write a hive query which can verify difference in file sizes. Do you have any ideas which can help me add some quick tests for this? The current test executes though the code checking that it does not result in any Exception or Error. It does not compare file size.
> Really? Which platforms are you talking about? Can you tell me how to reproduce this interesting behavior?
Hadoop loads native compression libraries. I believe that they are platform dependent hence I do not assume that they always have same compression ratio. Please correct me if I am wrong here.
In any case I think this is a broken existing functionality in Hive which we should fix.
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-2266-2.patch, HIVE-2266.patch
>
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2266) Fix compression parameters
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2266:
---------------------------------
Status: Open (was: Patch Available)
bq. Besides the file size will differ on different platforms.
Really? Which platforms are you talking about? Can you tell me how to reproduce this interesting behavior?
bq. This test case executes the new code and makes sure that it works.
How exactly does it verify that the new code works? How do you even know that your new code is getting exercised during the execution of this testcase?
> Fix compression parameters
> --------------------------
>
> Key: HIVE-2266
> URL: https://issues.apache.org/jira/browse/HIVE-2266
> Project: Hive
> Issue Type: Bug
> Reporter: Vaibhav Aggarwal
> Assignee: Vaibhav Aggarwal
> Attachments: HIVE-2266-2.patch, HIVE-2266.patch
>
>
> There are a number of places where compression values are not set correctly in FileSinkOperator. This results in uncompressed files.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira