You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Zheng Shao (JIRA)" <ji...@apache.org> on 2009/08/06 09:12:14 UTC

[jira] Created: (HIVE-732) Hive to use LazyBinarySerDe

Hive to use LazyBinarySerDe
---------------------------

                 Key: HIVE-732
                 URL: https://issues.apache.org/jira/browse/HIVE-732
             Project: Hadoop Hive
          Issue Type: New Feature
          Components: Query Processor
            Reporter: Zheng Shao
            Assignee: Zheng Shao
             Fix For: 0.4.0


Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
We should also allow users to create tables stored as binary format.
{code}
CREATE TABLE xxx (...)
ROW FORMAT BINARY
STORED AS SEQUENCEFILE;
{code}


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-732) Hive to use LazyBinarySerDe

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Namit Jain updated HIVE-732:
----------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]
          Status: Resolved  (was: Patch Available)

Committed. Thanks Zheng

> Hive to use LazyBinarySerDe
> ---------------------------
>
>                 Key: HIVE-732
>                 URL: https://issues.apache.org/jira/browse/HIVE-732
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>             Fix For: 0.5.0
>
>         Attachments: HIVE-732.1.patch, HIVE-732.3.patch, HIVE-732.4.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-732) Hive to use LazyBinarySerDe

Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zheng Shao updated HIVE-732:
----------------------------

    Fix Version/s:     (was: 0.4.0)
                   0.5.0
           Status: Patch Available  (was: Open)

> Hive to use LazyBinarySerDe
> ---------------------------
>
>                 Key: HIVE-732
>                 URL: https://issues.apache.org/jira/browse/HIVE-732
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>             Fix For: 0.5.0
>
>         Attachments: HIVE-732.1.patch, HIVE-732.3.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HIVE-732) Hive to use LazyBinarySerDe

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740287#action_12740287 ] 

Namit Jain commented on HIVE-732:
---------------------------------

Instead of  supporting:

CREATE TABLE xxx (...)
ROW FORMAT BINARY
STORED AS SEQUENCEFILE;


shouldn't we let the user specify the serde 

CREATE TABLE xxx (...)
ROW FORMAT SERDE '....';

I mean the existing syntax.

If we have more binary representations in future, how will we address that ?



> Hive to use LazyBinarySerDe
> ---------------------------
>
>                 Key: HIVE-732
>                 URL: https://issues.apache.org/jira/browse/HIVE-732
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>             Fix For: 0.4.0
>
>         Attachments: HIVE-732.1.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-732) Hive to use LazyBinarySerDe

Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zheng Shao updated HIVE-732:
----------------------------

    Attachment: HIVE-732.4.patch

Included test changes.

> Hive to use LazyBinarySerDe
> ---------------------------
>
>                 Key: HIVE-732
>                 URL: https://issues.apache.org/jira/browse/HIVE-732
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>             Fix For: 0.5.0
>
>         Attachments: HIVE-732.1.patch, HIVE-732.3.patch, HIVE-732.4.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HIVE-732) Hive to use LazyBinarySerDe

Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740389#action_12740389 ] 

Zheng Shao commented on HIVE-732:
---------------------------------

"ROW FORMAT BINARY" is just a shortcut to "ROW FORMAT SERDE ...".
This shortcut will help the adoption of this serde for tables created by users, and users will enjoy the performance benefit of this over the default LazySimpleSerDe.

We can have more binary representations in future, but they will need a different shortcut (if a shortcut is needed).


> Hive to use LazyBinarySerDe
> ---------------------------
>
>                 Key: HIVE-732
>                 URL: https://issues.apache.org/jira/browse/HIVE-732
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>             Fix For: 0.4.0
>
>         Attachments: HIVE-732.1.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HIVE-732) Hive to use LazyBinarySerDe

Posted by "Namit Jain (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12740644#action_12740644 ] 

Namit Jain commented on HIVE-732:
---------------------------------

Thats fine - You need to create the patch again, there were a lot of conflicts with 691.



> Hive to use LazyBinarySerDe
> ---------------------------
>
>                 Key: HIVE-732
>                 URL: https://issues.apache.org/jira/browse/HIVE-732
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>             Fix For: 0.4.0
>
>         Attachments: HIVE-732.1.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-732) Hive to use LazyBinarySerDe

Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zheng Shao updated HIVE-732:
----------------------------

    Attachment: HIVE-732.3.patch

This patch is a simple switch to use LazyBinarySerDe to improve the performance.
We can do "row format binary" later in a separate transaction if needed.

> Hive to use LazyBinarySerDe
> ---------------------------
>
>                 Key: HIVE-732
>                 URL: https://issues.apache.org/jira/browse/HIVE-732
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>             Fix For: 0.5.0
>
>         Attachments: HIVE-732.1.patch, HIVE-732.3.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-732) Hive to use LazyBinarySerDe

Posted by "Zheng Shao (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Zheng Shao updated HIVE-732:
----------------------------

    Attachment: HIVE-732.1.patch

HIVE-732.1.patch does the job. Please note that this for code review purpose - test result changes are not included yet.

> Hive to use LazyBinarySerDe
> ---------------------------
>
>                 Key: HIVE-732
>                 URL: https://issues.apache.org/jira/browse/HIVE-732
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>             Fix For: 0.4.0
>
>         Attachments: HIVE-732.1.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HIVE-732) Store intermediate data in binary using LazyBinarySerDe

Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HIVE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Carl Steinbach updated HIVE-732:
--------------------------------

    Summary: Store intermediate data in binary using LazyBinarySerDe  (was: Hive to use LazyBinarySerDe)

> Store intermediate data in binary using LazyBinarySerDe
> -------------------------------------------------------
>
>                 Key: HIVE-732
>                 URL: https://issues.apache.org/jira/browse/HIVE-732
>             Project: Hadoop Hive
>          Issue Type: New Feature
>          Components: Query Processor
>            Reporter: Zheng Shao
>            Assignee: Zheng Shao
>             Fix For: 0.5.0
>
>         Attachments: HIVE-732.1.patch, HIVE-732.3.patch, HIVE-732.4.patch
>
>
> Follow-up on HIVE-640. We should use LazyBinarySerDe in several places in the code to improve the efficiency: value between map-reduce boundary, temporary tables.
> We should also allow users to create tables stored as binary format.
> {code}
> CREATE TABLE xxx (...)
> ROW FORMAT BINARY
> STORED AS SEQUENCEFILE;
> {code}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.