You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Kevin Wilfong (JIRA)" <ji...@apache.org> on 2012/06/14 01:37:45 UTC
[jira] [Created] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Kevin Wilfong created HIVE-3134:
-----------------------------------
Summary: Drop table/index/database can result in orphaned locations
Key: HIVE-3134
URL: https://issues.apache.org/jira/browse/HIVE-3134
Project: Hive
Issue Type: Bug
Reporter: Kevin Wilfong
Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
The same applies to dropping a database with cascade and a table has a location outside the database.
I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-3134:
--------------------------------
Status: Patch Available (was: Open)
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-3134.1.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-3134:
--------------------------------
Attachment: HIVE-3134.2.patch.txt
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-3134.1.patch.txt, HIVE-3134.2.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295845#comment-13295845 ]
Carl Steinbach commented on HIVE-3134:
--------------------------------------
+1
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-3134.1.patch.txt, HIVE-3134.2.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-3134:
--------------------------------
Attachment: HIVE-3134.1.patch.txt
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-3134.1.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong reassigned HIVE-3134:
-----------------------------------
Assignee: Kevin Wilfong
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-3134.1.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294725#comment-13294725 ]
Kevin Wilfong commented on HIVE-3134:
-------------------------------------
Attached a diff for this issue here https://reviews.facebook.net/D3633
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-3134.1.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295186#comment-13295186 ]
Kevin Wilfong commented on HIVE-3134:
-------------------------------------
Responded to comments on phabricator.
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-3134.1.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-3134:
---------------------------------
Resolution: Fixed
Fix Version/s: 0.10.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks Kevin!
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Fix For: 0.10.0
>
> Attachments: HIVE-3134.1.patch.txt, HIVE-3134.2.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295847#comment-13295847 ]
Carl Steinbach commented on HIVE-3134:
--------------------------------------
Will commit if tests pass.
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-3134.1.patch.txt, HIVE-3134.2.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294800#comment-13294800 ]
Kevin Wilfong commented on HIVE-3134:
-------------------------------------
If you have permissions to write to that storage location I would imagine so, but that's an issue with/without this patch. There is certainly nothing in Hive that prevents two tables from sharing a location.
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-3134.1.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-3134:
---------------------------------
Status: Open (was: Patch Available)
@Kevin: Please see my comments on phabricator. Thanks.
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-3134.1.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Kevin Wilfong (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong updated HIVE-3134:
--------------------------------
Status: Patch Available (was: Open)
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-3134.1.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13294798#comment-13294798 ]
Carl Steinbach commented on HIVE-3134:
--------------------------------------
On a related note, what happens if you create a managed table and modify it's location to point to the storage location of another managed table? Will Hive let you do this, and if so, can you use this trick to delete someone else's table data?
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-3134.1.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-3134) Drop table/index/database can result
in orphaned locations
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-3134:
---------------------------------
Component/s: Metastore
> Drop table/index/database can result in orphaned locations
> ----------------------------------------------------------
>
> Key: HIVE-3134
> URL: https://issues.apache.org/jira/browse/HIVE-3134
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Reporter: Kevin Wilfong
> Assignee: Kevin Wilfong
> Attachments: HIVE-3134.1.patch.txt
>
>
> Today when a managed table has a partition with a location which is not a subdirectory of the table's location, when the table is dropped the partition's data is not deleted from HDFS, resulting in an orphaned directory (the data exists but nothing points to it).
> The same applies to dropping a database with cascade and a table has a location outside the database.
> I think it is safe to assume managed tables/partitions own the directories they point to, so we should clean these up.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira