You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Robert Surówka (Created JIRA)" <ji...@apache.org> on 2011/11/09 20:07:51 UTC
[jira] [Created] (HIVE-2567) Some CTAS queries with * and group by
don't work.
Some CTAS queries with * and group by don't work.
-------------------------------------------------
Key: HIVE-2567
URL: https://issues.apache.org/jira/browse/HIVE-2567
Project: Hive
Issue Type: Bug
Reporter: Robert Surówka
Assignee: Robert Surówka
Priority: Minor
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2567) Some CTAS queries with * and group by
don't work.
Posted by "Robert Surówka (Assigned JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka reassigned HIVE-2567:
------------------------------------
Assignee: He Yongqiang (was: Robert Surówka)
> Some CTAS queries with * and group by don't work.
> -------------------------------------------------
>
> Key: HIVE-2567
> URL: https://issues.apache.org/jira/browse/HIVE-2567
> Project: Hive
> Issue Type: Bug
> Reporter: Robert Surówka
> Assignee: He Yongqiang
> Priority: Minor
> Attachments: HIVE-2567.1.patch, HIVE-2567.2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2567) Some CTAS queries with * and group
by don't work.
Posted by "Ning Zhang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149443#comment-13149443 ]
Ning Zhang commented on HIVE-2567:
----------------------------------
Robert, this patch doesn't apply cleanly. Can you svn up and regenerate the patch?
> Some CTAS queries with * and group by don't work.
> -------------------------------------------------
>
> Key: HIVE-2567
> URL: https://issues.apache.org/jira/browse/HIVE-2567
> Project: Hive
> Issue Type: Bug
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2567.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2567) Some CTAS queries with * and group
by don't work.
Posted by "He Yongqiang (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152598#comment-13152598 ]
He Yongqiang commented on HIVE-2567:
------------------------------------
will run tests
> Some CTAS queries with * and group by don't work.
> -------------------------------------------------
>
> Key: HIVE-2567
> URL: https://issues.apache.org/jira/browse/HIVE-2567
> Project: Hive
> Issue Type: Bug
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2567.1.patch, HIVE-2567.2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2567) Some CTAS queries with * and group by
don't work.
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2567:
---------------------------------
Attachment: HIVE-2567.1.patch
> Some CTAS queries with * and group by don't work.
> -------------------------------------------------
>
> Key: HIVE-2567
> URL: https://issues.apache.org/jira/browse/HIVE-2567
> Project: Hive
> Issue Type: Bug
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2567.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2567) Some CTAS queries with * and group
by don't work.
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149791#comment-13149791 ]
jiraposter@reviews.apache.org commented on HIVE-2567:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2792/
-----------------------------------------------------------
(Updated 2011-11-14 18:06:11.336387)
Review request for Ning Zhang and namit jain.
Changes
-------
Updated the patch to apply cleanly on current trunk
Summary
-------
To introduce proper support for the problem would require a lot of work I believe. Yet this patch seems to do the job pretty well, and with very good chance doesn't break anything :).
When user puts tablename.columnname in group by, then the name of the column from * won't be columname but tablename_columnname, to allow ctas with * and joins on tables that share column names (like in the test). In very rare cases it may lead to duplicate column anyway, as shown in the negative test.
This isn't the final resolution of the problem (as that would require substantial changes), yet at least this allows hive to support many use cases with ctas, * and group by.
I also run all the tests (with -overwrite option) for this change, the tests executed correctly and no output file for any already existing test had been changed.
This addresses bug HIVE-2567.
https://issues.apache.org/jira/browse/HIVE-2567
Diffs (updated)
-----
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1201807
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1201807
trunk/ql/src/test/queries/clientnegative/ctas_group_by_failure1.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/input48.q PRE-CREATION
trunk/ql/src/test/results/clientnegative/ctas_group_by_failure1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/input48.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/2792/diff
Testing
-------
Worked on some sample queries.
Unit tests work too.
Thanks,
Robert
> Some CTAS queries with * and group by don't work.
> -------------------------------------------------
>
> Key: HIVE-2567
> URL: https://issues.apache.org/jira/browse/HIVE-2567
> Project: Hive
> Issue Type: Bug
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2567.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2567) Some CTAS queries with * and group by
don't work.
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2567:
---------------------------------
Attachment: HIVE-2567.2.patch
> Some CTAS queries with * and group by don't work.
> -------------------------------------------------
>
> Key: HIVE-2567
> URL: https://issues.apache.org/jira/browse/HIVE-2567
> Project: Hive
> Issue Type: Bug
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2567.1.patch, HIVE-2567.2.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2567) Some CTAS queries with * and group
by don't work.
Posted by "jiraposter@reviews.apache.org (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148853#comment-13148853 ]
jiraposter@reviews.apache.org commented on HIVE-2567:
-----------------------------------------------------
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/2792/
-----------------------------------------------------------
(Updated 2011-11-11 23:47:08.652126)
Review request for Ning Zhang and namit jain.
Summary
-------
To introduce proper support for the problem would require a lot of work I believe. Yet this patch seems to do the job pretty well, and with very good chance doesn't break anything :).
When user puts tablename.columnname in group by, then the name of the column from * won't be columname but tablename_columnname, to allow ctas with * and joins on tables that share column names (like in the test). In very rare cases it may lead to duplicate column anyway, as shown in the negative test.
This isn't the final resolution of the problem (as that would require substantial changes), yet at least this allows hive to support many use cases with ctas, * and group by.
I also run all the tests (with -overwrite option) for this change, the tests executed correctly and no output file for any already existing test had been changed.
This addresses bug HIVE-2567.
https://issues.apache.org/jira/browse/HIVE-2567
Diffs
-----
trunk/metastore/src/java/org/apache/hadoop/hive/metastore/MetaStoreUtils.java 1199920
trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java 1199920
trunk/ql/src/test/queries/clientnegative/ctas_group_by_failure1.q PRE-CREATION
trunk/ql/src/test/queries/clientpositive/input48.q PRE-CREATION
trunk/ql/src/test/results/clientnegative/ctas_group_by_failure1.q.out PRE-CREATION
trunk/ql/src/test/results/clientpositive/input48.q.out PRE-CREATION
Diff: https://reviews.apache.org/r/2792/diff
Testing
-------
Worked on some sample queries.
Unit tests work too.
Thanks,
Robert
> Some CTAS queries with * and group by don't work.
> -------------------------------------------------
>
> Key: HIVE-2567
> URL: https://issues.apache.org/jira/browse/HIVE-2567
> Project: Hive
> Issue Type: Bug
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2567.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2567) Some CTAS queries with * and group by
don't work.
Posted by "Robert Surówka (Updated JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Surówka updated HIVE-2567:
---------------------------------
Status: Patch Available (was: Open)
> Some CTAS queries with * and group by don't work.
> -------------------------------------------------
>
> Key: HIVE-2567
> URL: https://issues.apache.org/jira/browse/HIVE-2567
> Project: Hive
> Issue Type: Bug
> Reporter: Robert Surówka
> Assignee: Robert Surówka
> Priority: Minor
> Attachments: HIVE-2567.1.patch
>
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira