You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Kevin Wilfong (Created) (JIRA)" <ji...@apache.org> on 2012/02/25 04:51:46 UTC
[jira] [Created] (HIVE-2825) Concatenating a partition does not
inherit location from table
Concatenating a partition does not inherit location from table
--------------------------------------------------------------
Key: HIVE-2825
URL: https://issues.apache.org/jira/browse/HIVE-2825
Project: Hive
Issue Type: Improvement
Reporter: Kevin Wilfong
Assignee: Kevin Wilfong
When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly.
See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2825) Concatenating a partition does not
inherit location from table
Posted by "Namit Jain (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Namit Jain updated HIVE-2825:
-----------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed. Thanks Kevin.
> Concatenating a partition does not inherit location from table
> --------------------------------------------------------------
>
> Key: HIVE-2825
> URL: https://issues.apache.org/jira/browse/HIVE-2825
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2825.D1995.1.patch, HIVE-2825.D1995.2.patch
>
>
> When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly.
> See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2825) Concatenating a partition does not
inherit location from table
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217979#comment-13217979 ]
Phabricator commented on HIVE-2825:
-----------------------------------
njain has commented on the revision "HIVE-2825 [jira] Concatenating a partition does not inherit location from table".
Do we need to collect stats for this ?
I mean, the number of rows etc. cannot change.
REVISION DETAIL
https://reviews.facebook.net/D1995
> Concatenating a partition does not inherit location from table
> --------------------------------------------------------------
>
> Key: HIVE-2825
> URL: https://issues.apache.org/jira/browse/HIVE-2825
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2825.D1995.1.patch
>
>
> When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly.
> See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2825) Concatenating a partition does not
inherit location from table
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2825:
------------------------------
Attachment: HIVE-2825.D1995.2.patch
kevinwilfong updated the revision "HIVE-2825 [jira] Concatenating a partition does not inherit location from table".
Reviewers: JIRA, njain
Added a test to verify that when the scheme for the table is changed, concatenating a partition changes the scheme for the partition as well. It uses a hook because the location is masked in the output files.
REVISION DETAIL
https://reviews.facebook.net/D1995
AFFECTED FILES
ql/src/test/results/clientpositive/concatenate_inherit_table_location.q.out
ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyPartitionIsSubdirectoryOfTableHook.java
ql/src/test/queries/clientpositive/concatenate_inherit_table_location.q
ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
> Concatenating a partition does not inherit location from table
> --------------------------------------------------------------
>
> Key: HIVE-2825
> URL: https://issues.apache.org/jira/browse/HIVE-2825
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2825.D1995.1.patch, HIVE-2825.D1995.2.patch
>
>
> When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly.
> See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2825) Concatenating a partition does not
inherit location from table
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218875#comment-13218875 ]
Hudson commented on HIVE-2825:
------------------------------
Integrated in Hive-trunk-h0.21 #1284 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1284/])
HIVE-2825 Concatenating a partition does not inherit location from table
(Kevin Wilfong via namit) (Revision 1294942)
Result = FAILURE
namit : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1294942
Files :
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
* /hive/trunk/ql/src/test/org/apache/hadoop/hive/ql/hooks/VerifyPartitionIsSubdirectoryOfTableHook.java
* /hive/trunk/ql/src/test/queries/clientpositive/concatenate_inherit_table_location.q
* /hive/trunk/ql/src/test/results/clientpositive/concatenate_inherit_table_location.q.out
> Concatenating a partition does not inherit location from table
> --------------------------------------------------------------
>
> Key: HIVE-2825
> URL: https://issues.apache.org/jira/browse/HIVE-2825
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2825.D1995.1.patch, HIVE-2825.D1995.2.patch
>
>
> When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly.
> See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2825) Concatenating a partition does not
inherit location from table
Posted by "Ashutosh Chauhan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ashutosh Chauhan updated HIVE-2825:
-----------------------------------
Fix Version/s: 0.9.0
> Concatenating a partition does not inherit location from table
> --------------------------------------------------------------
>
> Key: HIVE-2825
> URL: https://issues.apache.org/jira/browse/HIVE-2825
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Fix For: 0.9.0
>
> Attachments: HIVE-2825.D1995.1.patch, HIVE-2825.D1995.2.patch
>
>
> When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly.
> See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2825) Concatenating a partition does not
inherit location from table
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2825:
------------------------------
Attachment: HIVE-2825.D1995.1.patch
kevinwilfong requested code review of "HIVE-2825 [jira] Concatenating a partition does not inherit location from table".
Reviewers: JIRA
https://issues.apache.org/jira/browse/HIVE-2825
When merging a partition the location is recreated using the scheme and authority of the table's URI, so if the table's default dfs has changed, the partition's location will as well. As part of this I noticed that the auto stats gather configuration variable was being ignored for merge, so I added this as well.
When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly.
See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D1995
AFFECTED FILES
ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/4251/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Concatenating a partition does not inherit location from table
> --------------------------------------------------------------
>
> Key: HIVE-2825
> URL: https://issues.apache.org/jira/browse/HIVE-2825
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2825.D1995.1.patch
>
>
> When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly.
> See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2825) Concatenating a partition does not
inherit location from table
Posted by "Kevin Wilfong (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-2825:
--------------------------------
Status: Patch Available (was: Open)
> Concatenating a partition does not inherit location from table
> --------------------------------------------------------------
>
> Key: HIVE-2825
> URL: https://issues.apache.org/jira/browse/HIVE-2825
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2825.D1995.1.patch
>
>
> When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly.
> See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2825) Concatenating a partition does not
inherit location from table
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13217975#comment-13217975 ]
Phabricator commented on HIVE-2825:
-----------------------------------
njain has commented on the revision "HIVE-2825 [jira] Concatenating a partition does not inherit location from table".
Can you add a test for this ?
Check the location of the output via describe.
REVISION DETAIL
https://reviews.facebook.net/D1995
> Concatenating a partition does not inherit location from table
> --------------------------------------------------------------
>
> Key: HIVE-2825
> URL: https://issues.apache.org/jira/browse/HIVE-2825
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2825.D1995.1.patch
>
>
> When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly.
> See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2825) Concatenating a partition does not
inherit location from table
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13218830#comment-13218830 ]
Phabricator commented on HIVE-2825:
-----------------------------------
kevinwilfong has commented on the revision "HIVE-2825 [jira] Concatenating a partition does not inherit location from table".
I left the stats task in as the number of files should change when concatenating files.
REVISION DETAIL
https://reviews.facebook.net/D1995
> Concatenating a partition does not inherit location from table
> --------------------------------------------------------------
>
> Key: HIVE-2825
> URL: https://issues.apache.org/jira/browse/HIVE-2825
> Project: Hive
> Issue Type: Improvement
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-2825.D1995.1.patch, HIVE-2825.D1995.2.patch
>
>
> When a table is created in one dfs, a partition is added to that table, the table's dfs is changed, and the partitioned is concatenated, the partitions location remains exactly the same. Instead, it should inherit its location from the table, and be updated accordingly.
> See https://issues.apache.org/jira/browse/HIVE-1707 for an analogous change to insert overwrite.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira