You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2009/08/06 09:12:14 UTC
[jira] Created: (HIVE-732) Hive to use LazyBinarySerDe
Hive to use LazyBinarySerDe
---------------------------
Key: HIVE-732
URL: https://issues.apache.org/jira/browse/HIVE-732
Project: Hadoop Hive
Issue Type: New Feature
Components: Query Processor
Reporter: Zheng Shao
Assignee: Zheng Shao
Fix For: 0.4.0
Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
We should also allow users to create tables stored as binary format.
{code}
CREATE TABLE xxx (...)
ROW FORMAT BINARY
STORED AS SEQUENCEFILE;
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-732) Hive to use LazyBinarySerDe
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-732:
----------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed. Thanks Zheng
> Hive to use LazyBinarySerDe
> ---------------------------
>
> Key: HIVE-732
> URL: https://issues.apache.org/jira/browse/HIVE-732
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.5.0
>
> Attachments: HIVE-732.1.patch, HIVE-732.3.patch, HIVE-732.4.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-732) Hive to use LazyBinarySerDe
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-732:
----------------------------
Fix Version/s: (was: 0.4.0)
0.5.0
Status: Patch Available (was: Open)
> Hive to use LazyBinarySerDe
> ---------------------------
>
> Key: HIVE-732
> URL: https://issues.apache.org/jira/browse/HIVE-732
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.5.0
>
> Attachments: HIVE-732.1.patch, HIVE-732.3.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-732) Hive to use LazyBinarySerDe
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740287#action_12740287 ]
Namit Jain commented on HIVE-732:
---------------------------------
Instead of supporting:
CREATE TABLE xxx (...)
ROW FORMAT BINARY
STORED AS SEQUENCEFILE;
shouldn't we let the user specify the serde
CREATE TABLE xxx (...)
ROW FORMAT SERDE '....';
I mean the existing syntax.
If we have more binary representations in future, how will we address that ?
> Hive to use LazyBinarySerDe
> ---------------------------
>
> Key: HIVE-732
> URL: https://issues.apache.org/jira/browse/HIVE-732
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-732.1.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-732) Hive to use LazyBinarySerDe
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-732:
----------------------------
Attachment: HIVE-732.4.patch
Included test changes.
> Hive to use LazyBinarySerDe
> ---------------------------
>
> Key: HIVE-732
> URL: https://issues.apache.org/jira/browse/HIVE-732
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.5.0
>
> Attachments: HIVE-732.1.patch, HIVE-732.3.patch, HIVE-732.4.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-732) Hive to use LazyBinarySerDe
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740389#action_12740389 ]
Zheng Shao commented on HIVE-732:
---------------------------------
"ROW FORMAT BINARY" is just a shortcut to "ROW FORMAT SERDE ...".
This shortcut will help the adoption of this serde for tables created by users, and users will enjoy the performance benefit of this over the default LazySimpleSerDe.
We can have more binary representations in future, but they will need a different shortcut (if a shortcut is needed).
> Hive to use LazyBinarySerDe
> ---------------------------
>
> Key: HIVE-732
> URL: https://issues.apache.org/jira/browse/HIVE-732
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-732.1.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HIVE-732) Hive to use LazyBinarySerDe
Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740644#action_12740644 ]
Namit Jain commented on HIVE-732:
---------------------------------
Thats fine - You need to create the patch again, there were a lot of conflicts with 691.
> Hive to use LazyBinarySerDe
> ---------------------------
>
> Key: HIVE-732
> URL: https://issues.apache.org/jira/browse/HIVE-732
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-732.1.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-732) Hive to use LazyBinarySerDe
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-732:
----------------------------
Attachment: HIVE-732.3.patch
This patch is a simple switch to use LazyBinarySerDe to improve the performance.
We can do "row format binary" later in a separate transaction if needed.
> Hive to use LazyBinarySerDe
> ---------------------------
>
> Key: HIVE-732
> URL: https://issues.apache.org/jira/browse/HIVE-732
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.5.0
>
> Attachments: HIVE-732.1.patch, HIVE-732.3.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-732) Hive to use LazyBinarySerDe
Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zheng Shao updated HIVE-732:
----------------------------
Attachment: HIVE-732.1.patch
HIVE-732.1.patch does the job. Please note that this for code review purpose - test result changes are not included yet.
> Hive to use LazyBinarySerDe
> ---------------------------
>
> Key: HIVE-732
> URL: https://issues.apache.org/jira/browse/HIVE-732
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.4.0
>
> Attachments: HIVE-732.1.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HIVE-732) Store intermediate data in binary using
LazyBinarySerDe
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-732:
--------------------------------
Summary: Store intermediate data in binary using LazyBinarySerDe (was: Hive to use LazyBinarySerDe)
> Store intermediate data in binary using LazyBinarySerDe
> -------------------------------------------------------
>
> Key: HIVE-732
> URL: https://issues.apache.org/jira/browse/HIVE-732
> Project: Hadoop Hive
> Issue Type: New Feature
> Components: Query Processor
> Reporter: Zheng Shao
> Assignee: Zheng Shao
> Fix For: 0.5.0
>
> Attachments: HIVE-732.1.patch, HIVE-732.3.patch, HIVE-732.4.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.