You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by "Mousom Dhar Gupta (Created) (JIRA)" <ji...@apache.org> on 2012/03/26 20:18:29 UTC
[jira] [Created] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Hive error when dropping a table with large number of partitions
----------------------------------------------------------------
Key: HIVE-2907
URL: https://issues.apache.org/jira/browse/HIVE-2907
Project: Hive
Issue Type: Bug
Components: Metastore
Affects Versions: 0.9.0
Environment: General. Hive Metastore bug.
Reporter: Mousom Dhar Gupta
Priority: Minor
Fix For: 0.9.0
Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Posted by "Mousom Dhar Gupta (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mousom Dhar Gupta updated HIVE-2907:
------------------------------------
Attachment: HIVE-2907.1.patch.txt
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238894#comment-13238894 ]
Phabricator commented on HIVE-2907:
-----------------------------------
kevinwilfong has commented on the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
Could you add a negative test, in which hive.metastore.batch.retrieve.max=1; and otherwise the same as drop_table_failure3.q, this should check that you are not looping over the same partition to check for NO_DROP.
REVISION DETAIL
https://reviews.facebook.net/D2505
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2907:
------------------------------
Attachment: HIVE-2907.D2505.5.patch
mousom updated the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
Reviewers: JIRA, kevinwilfong
Fixed typo. reaturn => return
REVISION DETAIL
https://reviews.facebook.net/D2505
AFFECTED FILES
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/test/results/clientnegative/drop_table_failure3.q.out
ql/src/test/results/clientpositive/drop_table2.q.out
ql/src/test/queries/clientnegative/drop_table_failure3.q
ql/src/test/queries/clientpositive/drop_table2.q
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2907:
------------------------------
Attachment: HIVE-2907.D2505.2.patch
mousom updated the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
Reviewers: JIRA, kevinwilfong
Addressed code review comments.
REVISION DETAIL
https://reviews.facebook.net/D2505
AFFECTED FILES
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/test/results/clientnegative/drop_table_failure3.q.out
ql/src/test/results/clientpositive/drop_table2.q.out
ql/src/test/queries/clientnegative/drop_table_failure3.q
ql/src/test/queries/clientpositive/drop_table2.q
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2907:
------------------------------
Attachment: HIVE-2907.D2505.6.patch
mousom updated the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
Reviewers: JIRA, kevinwilfong
Fixed drop_table_failure3.q
REVISION DETAIL
https://reviews.facebook.net/D2505
AFFECTED FILES
conf/hive-default.xml.template
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/test/results/clientnegative/drop_table_failure3.q.out
ql/src/test/results/clientpositive/drop_table2.q.out
ql/src/test/queries/clientnegative/drop_table_failure3.q
ql/src/test/queries/clientpositive/drop_table2.q
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Posted by "Kevin Wilfong (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong resolved HIVE-2907.
---------------------------------
Resolution: Fixed
Committed. Thanks Mousom.
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280317#comment-13280317 ]
Srinivas commented on HIVE-2907:
--------------------------------
Thanks for the timely clarification.
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.10.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Kevin Wilfong (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13250791#comment-13250791 ]
Kevin Wilfong commented on HIVE-2907:
-------------------------------------
+1 tests passed, will commit.
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Phabricator (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13274818#comment-13274818 ]
Phabricator commented on HIVE-2907:
-----------------------------------
kevinwilfong has accepted the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
REVISION DETAIL
https://reviews.facebook.net/D2505
BRANCH
svn
To: JIRA, kevinwilfong, mousom
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13245473#comment-13245473 ]
Phabricator commented on HIVE-2907:
-----------------------------------
kevinwilfong has commented on the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
I'm getting conflicts trying to apply the patch, could you update your checkout and post an updated diff.
REVISION DETAIL
https://reviews.facebook.net/D2505
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280304#comment-13280304 ]
Srinivas commented on HIVE-2907:
--------------------------------
I downloaded the source code for Hive-0.9.1. However, it appears that "ObjectStore.java" is missing the fix that fetches partition-metadata in batches. So, it can still cause issues when trying to drop a table with a large number of partitions.
Proposed fix in method dropTable from ObjectStore.java
==========================================================
int partitionBatchSize = HiveConf.getIntVar(getConf(),
ConfVars.METASTORE_BATCH_RETRIEVE_MAX);
// call dropPartition on each of the table's partitions to follow the
// procedure for cleanly dropping partitions.
List<MPartition> partsToDelete = listMPartitions(dbName, tableName, partitionBatchSize);
while (true){
if (partsToDelete != null || partsToDelete.isEmpty()) {
break;
}
for (MPartition mpart : partsToDelete) {
dropPartitionCommon(mpart);
}
}
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13280308#comment-13280308 ]
Srinivas commented on HIVE-2907:
--------------------------------
Also, the method dropTable in "DDLTask.java" appears to be missing the fix proposed in the JIRA. I am looking at the Hive-0.9.0 source code base.
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2907:
------------------------------
Attachment: HIVE-2907.D2505.7.patch
mousom updated the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
Reviewers: JIRA, kevinwilfong
REVISION DETAIL
https://reviews.facebook.net/D2505
AFFECTED FILES
conf/hive-default.xml.template
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/test/results/clientnegative/drop_table_failure3.q.out
ql/src/test/results/clientpositive/drop_table2.q.out
ql/src/test/queries/clientnegative/drop_table_failure3.q
ql/src/test/queries/clientpositive/drop_table2.q
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239718#comment-13239718 ]
Phabricator commented on HIVE-2907:
-----------------------------------
kevinwilfong has accepted the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
+1 Will commit if tests pass.
REVISION DETAIL
https://reviews.facebook.net/D2505
BRANCH
svn
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Posted by "Mousom Dhar Gupta (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mousom Dhar Gupta updated HIVE-2907:
------------------------------------
Attachment: HIVE-2907.3.patch.txt
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Posted by "Kevin Wilfong (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kevin Wilfong reassigned HIVE-2907:
-----------------------------------
Assignee: Mousom Dhar Gupta
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Posted by "Mousom Dhar Gupta (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mousom Dhar Gupta updated HIVE-2907:
------------------------------------
Attachment: HIVE-2907.2.patch.txt
Here is the new patch.
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13240846#comment-13240846 ]
Phabricator commented on HIVE-2907:
-----------------------------------
kevinwilfong has requested changes to the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
I'm seeing a lot of test failures in TestNegativeCliDriver. They seem to work ok when run individually, but when I run ant test or ant test -Dtestcase=TestNegativeCliDriver they fail. Could you take a look?
REVISION DETAIL
https://reviews.facebook.net/D2505
BRANCH
svn
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13251235#comment-13251235 ]
Hudson commented on HIVE-2907:
------------------------------
Integrated in Hive-trunk-h0.21 #1367 (See [https://builds.apache.org/job/Hive-trunk-h0.21/1367/])
HIVE-2907. Hive error when dropping a table with large number of partitions. (mousom via kevinwilfong) (Revision 1311850)
Result = FAILURE
kevinwilfong : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1311850
Files :
* /hive/trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
* /hive/trunk/conf/hive-default.xml.template
* /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
* /hive/trunk/metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
* /hive/trunk/metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
* /hive/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
* /hive/trunk/ql/src/test/queries/clientnegative/drop_table_failure3.q
* /hive/trunk/ql/src/test/queries/clientpositive/drop_table2.q
* /hive/trunk/ql/src/test/results/clientnegative/drop_table_failure3.q.out
* /hive/trunk/ql/src/test/results/clientpositive/drop_table2.q.out
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Posted by "Carl Steinbach (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Carl Steinbach updated HIVE-2907:
---------------------------------
Fix Version/s: (was: 0.9.0)
0.10.0
@Srinivas: This patch was not backported to 0.9.0. The previous fixVersion was inaccurate. I have updated the field to make it clear that this patch will appear in 0.10.0. Thanks.
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Assignee: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.10.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.2.patch.txt, HIVE-2907.3.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch, HIVE-2907.D2505.5.patch, HIVE-2907.D2505.6.patch, HIVE-2907.D2505.7.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2907:
------------------------------
Attachment: HIVE-2907.D2505.1.patch
mousom requested code review of "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
Reviewers: JIRA
https://issues.apache.org/jira/browse/HIVE-2907
Hive error when dropping a table with large number of partitions.
Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
TEST PLAN
EMPTY
REVISION DETAIL
https://reviews.facebook.net/D2505
AFFECTED FILES
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/test/results/clientnegative/drop_table_failure3.q.out
ql/src/test/queries/clientnegative/drop_table_failure3.q
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
MANAGE HERALD DIFFERENTIAL RULES
https://reviews.facebook.net/herald/view/differential/
WHY DID I GET THIS EMAIL?
https://reviews.facebook.net/herald/transcript/5595/
Tip: use the X-Herald-Rules header to filter Herald messages in your client.
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238964#comment-13238964 ]
Phabricator commented on HIVE-2907:
-----------------------------------
mousom has commented on the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
May I modify the drop_table_failure3.q test itself to do that? That might serve both purposes.
INLINE COMMENTS
metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:164 Can you please make this a separate task Kevin?
Thanks
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:292 like the *.q file I added?
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:544 yup
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:1997 my bad. done.
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:292 i will add both in *.q files.
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1399 I thought about it but it's just intuitively does not make sense if we return 'ALL' when 0 is passed.And it might be a very difficult to catch bug for a client.But, on the other hand, you are right that this change might break existing clients and that has to be avoided, so I guess I will make this bug a feature :-)
REVISION DETAIL
https://reviews.facebook.net/D2505
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239099#comment-13239099 ]
Phabricator commented on HIVE-2907:
-----------------------------------
kevinwilfong has commented on the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
INLINE COMMENTS
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:464-465 Why are you checking for values.size() / 2 here? Admittedly, this works, but shouldn't it be just 2? Or shouldn't you use values.size() / 2 in line 461?
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:463 sp returned
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:506-507 Same here re values.size() / 2 vs. 2
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:505 same here re returned sp
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:512 sp returned
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:470 sp returned
REVISION DETAIL
https://reviews.facebook.net/D2505
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2907:
------------------------------
Attachment: HIVE-2907.D2505.3.patch
mousom updated the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
Reviewers: JIRA, kevinwilfong
Fixed drop_table2.q
REVISION DETAIL
https://reviews.facebook.net/D2505
AFFECTED FILES
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/test/results/clientnegative/drop_table_failure3.q.out
ql/src/test/results/clientpositive/drop_table2.q.out
ql/src/test/queries/clientnegative/drop_table_failure3.q
ql/src/test/queries/clientpositive/drop_table2.q
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HIVE-2907) Hive error when dropping a table with
large number of partitions
Posted by "Phabricator (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Phabricator updated HIVE-2907:
------------------------------
Attachment: HIVE-2907.D2505.4.patch
mousom updated the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
Reviewers: JIRA, kevinwilfong
Updated drop_table_failure3.q by setting
hive.metastore.batch.retrieve.max=1 so that it makes
sure that the code is not looping over the same partition to check for NO_DROP.
REVISION DETAIL
https://reviews.facebook.net/D2505
AFFECTED FILES
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java
metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
ql/src/test/results/clientnegative/drop_table_failure3.q.out
ql/src/test/results/clientpositive/drop_table2.q.out
ql/src/test/queries/clientnegative/drop_table_failure3.q
ql/src/test/queries/clientpositive/drop_table2.q
ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch, HIVE-2907.D2505.2.patch, HIVE-2907.D2505.3.patch, HIVE-2907.D2505.4.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Phabricator (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238791#comment-13238791 ]
Phabricator commented on HIVE-2907:
-----------------------------------
kevinwilfong has commented on the revision "HIVE-2907 [jira] Hive error when dropping a table with large number of partitions".
This looks really good, thanks for the code clean up, just some small comments.
INLINE COMMENTS
metastore/src/java/org/apache/hadoop/hive/metastore/HiveAlterHandler.java:164 It'd be great if we could use the same config here, otherwise alter table will face the same memory issues for tables with lots of partitions.
This could be a separate task too.
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1399-1401 If any client code already assumes that if 0 is max, it returns all partitions, which, given the code in HiveAlterHandler, seems possible, this will break that code. Could we treat 0 as infinity as well?
metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java:1565-1567 Same here regarding 0 => infinity
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:1997 Can you remove this line if it's not needed.
metastore/src/test/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:544 Shouldn't this be "table should not have existed"
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:292 Could you add a testcase, where you set this really low, e.g. 1, and you drop a table with multiple partitions.
Also, could you add a similar test where you make one of the partitions NO_DROP, and make sure it fails.
REVISION DETAIL
https://reviews.facebook.net/D2505
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt, HIVE-2907.D2505.1.patch
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HIVE-2907) Hive error when dropping a table
with large number of partitions
Posted by "Mousom Dhar Gupta (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HIVE-2907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238662#comment-13238662 ]
Mousom Dhar Gupta commented on HIVE-2907:
-----------------------------------------
For ObjectStore::dropTable am dropping pratitions in batches. And for DDLTask::dropTable am
getting all partition names and then getting partition metadata from those names in batches.
The batch size is determined by a config parameter (set to 1000 to start with).
Thanks
> Hive error when dropping a table with large number of partitions
> ----------------------------------------------------------------
>
> Key: HIVE-2907
> URL: https://issues.apache.org/jira/browse/HIVE-2907
> Project: Hive
> Issue Type: Bug
> Components: Metastore
> Affects Versions: 0.9.0
> Environment: General. Hive Metastore bug.
> Reporter: Mousom Dhar Gupta
> Priority: Minor
> Fix For: 0.9.0
>
> Attachments: HIVE-2907.1.patch.txt
>
> Original Estimate: 10h
> Remaining Estimate: 10h
>
> Running into an "Out Of Memory" error when trying to drop a table with 128K partitions.
> The methods dropTable in metastore/src/java/org/apache/hadoop/hive/metastore/ObjectStore.java
> and dropTable in ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java encounter out of memory errors
> when dropping tables with lots of partitions because they try to load the metadata for every partition into memory.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira