You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by GitBox <gi...@apache.org> on 2021/05/14 08:23:27 UTC
[GitHub] [hive] klcopp opened a new pull request #2274: HIVE-25115: Compaction queue entries may accumulate in "ready for cleaning" state
klcopp opened a new pull request #2274:
URL: https://github.com/apache/hive/pull/2274
See HIVE-25115
### How was this patch tested?
Unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #2274: HIVE-25115: Compaction queue entries may accumulate in "ready for cleaning" state
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #2274:
URL: https://github.com/apache/hive/pull/2274#discussion_r632418906
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -305,17 +302,21 @@ public void markCompacted(CompactionInfo info) throws MetaException {
* By filtering on minOpenTxnWaterMark, we will only cleanup after every transaction is committed, that could see
* the uncompacted deltas. This way the cleaner can clean up everything that was made obsolete by this compaction.
*/
- String s = "SELECT \"CQ_ID\", \"CQ_DATABASE\", \"CQ_TABLE\", \"CQ_PARTITION\", "
- + "\"CQ_TYPE\", \"CQ_RUN_AS\", \"CQ_HIGHEST_WRITE_ID\" FROM \"COMPACTION_QUEUE\" WHERE \"CQ_STATE\" = '"
- + READY_FOR_CLEANING + "'";
- if (minOpenTxnWaterMark > 0) {
- s = s + " AND (\"CQ_NEXT_TXN_ID\" <= " + minOpenTxnWaterMark + " OR \"CQ_NEXT_TXN_ID\" IS NULL)";
- }
+ StringBuilder sb = new StringBuilder();
Review comment:
I think, with this approach, you might end up not cleaning anything at all as you'll be always looking at latest compaction id that might be blocked by incoming txns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr commented on a change in pull request #2274: HIVE-25115: Compaction queue entries may accumulate in "ready for cleaning" state
Posted by GitBox <gi...@apache.org>.
belugabehr commented on a change in pull request #2274:
URL: https://github.com/apache/hive/pull/2274#discussion_r637098785
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
##########
@@ -383,6 +384,37 @@ public static String getFullTableName(String dbName, String tableName) {
return ret;
}
+ /**
+ * Executes simple update queries based on output from TxnUtils#buildQueryWithINClauseStrings.
+ * Example: desired queries are "delete from x where y in (1, 2, 3)" and "delete from x where y in (4, 5, 6)"
+ *
+ * @param updateQueries List of: "delete from x where y in (?,?,?)", "delete from x where y in (?,?,?)"
+ * @param updateQueryCount Number of queries to execute, in the example: 2
+ * @param ids to prepare statement with. In the example: List containing: 1, 2, 3, 4, 5, 6
+ * @param dbConn database Connection
+ * @throws SQLException
+ */
+ public static int executeUpdateQueries(List<String> updateQueries, List<Integer> updateQueryCount, List<Long> ids,
+ Connection dbConn)
+ throws SQLException {
+ int totalCount = 0;
+ int updatedCount = 0;
+ for (int i = 0; i < updateQueries.size(); i++) {
+ String query = updateQueries.get(i);
+ long insertCount = updateQueryCount.get(i);
+ LOG.debug("Going to execute update <" + query + ">");
Review comment:
`LOG.debug("Going to ... <{}>", query);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] klcopp closed pull request #2274: HIVE-25115: Compaction queue entries may accumulate in "ready for cleaning" state
Posted by GitBox <gi...@apache.org>.
klcopp closed pull request #2274:
URL: https://github.com/apache/hive/pull/2274
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr commented on a change in pull request #2274: HIVE-25115: Compaction queue entries may accumulate in "ready for cleaning" state
Posted by GitBox <gi...@apache.org>.
belugabehr commented on a change in pull request #2274:
URL: https://github.com/apache/hive/pull/2274#discussion_r637097800
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -305,16 +302,19 @@ public void markCompacted(CompactionInfo info) throws MetaException {
* By filtering on minOpenTxnWaterMark, we will only cleanup after every transaction is committed, that could see
* the uncompacted deltas. This way the cleaner can clean up everything that was made obsolete by this compaction.
*/
- String s = "SELECT \"CQ_ID\", \"CQ_DATABASE\", \"CQ_TABLE\", \"CQ_PARTITION\", "
- + "\"CQ_TYPE\", \"CQ_RUN_AS\", \"CQ_HIGHEST_WRITE_ID\" FROM \"COMPACTION_QUEUE\" WHERE \"CQ_STATE\" = '"
- + READY_FOR_CLEANING + "'";
- if (minOpenTxnWaterMark > 0) {
- s = s + " AND (\"CQ_NEXT_TXN_ID\" <= " + minOpenTxnWaterMark + " OR \"CQ_NEXT_TXN_ID\" IS NULL)";
- }
+ String s = "SELECT \"CQ_ID\", \"CQ_DATABASE\", \"CQ_TABLE\", \"CQ_PARTITION\", \"CQ_TYPE\", \"CQ_RUN_AS\"," +
+ " \"CQ_HIGHEST_WRITE_ID\", \"CQ_NEXT_TXN_ID\" FROM \"COMPACTION_QUEUE\" WHERE \"CQ_ID\" IN (" +
+ " SELECT DISTINCT \"CQ_ID\" FROM (" +
+ " SELECT MAX(\"CQ_NEXT_TXN_ID\") \"CQ_NEXT_TXN_ID\", MAX(\"CQ_ID\") \"CQ_ID\" " +
+ " FROM \"COMPACTION_QUEUE\" WHERE \"CQ_STATE\" = '" + READY_FOR_CLEANING + "'" +
+ " AND (\"CQ_NEXT_TXN_ID\" <= "+ minOpenTxnWaterMark + " OR \"CQ_NEXT_TXN_ID\" IS NULL)";
if (retentionTime > 0) {
- s = s + " AND \"CQ_COMMIT_TIME\" < (" + getEpochFn(dbProduct) + " - " + retentionTime + ")";
+ s += " AND \"CQ_COMMIT_TIME\" < (" + getEpochFn(dbProduct) + " - " + retentionTime + ")";
+ }
+ s += " GROUP BY \"CQ_DATABASE\", \"CQ_TABLE\", \"CQ_PARTITION\" ) \"X\" )";
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("Going to execute query <" + s + ">");
Review comment:
`LOG.debug("Going to... <{}>", s);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr commented on a change in pull request #2274: HIVE-25115: Compaction queue entries may accumulate in "ready for cleaning" state
Posted by GitBox <gi...@apache.org>.
belugabehr commented on a change in pull request #2274:
URL: https://github.com/apache/hive/pull/2274#discussion_r637098961
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/TxnUtils.java
##########
@@ -383,6 +384,37 @@ public static String getFullTableName(String dbName, String tableName) {
return ret;
}
+ /**
+ * Executes simple update queries based on output from TxnUtils#buildQueryWithINClauseStrings.
+ * Example: desired queries are "delete from x where y in (1, 2, 3)" and "delete from x where y in (4, 5, 6)"
+ *
+ * @param updateQueries List of: "delete from x where y in (?,?,?)", "delete from x where y in (?,?,?)"
+ * @param updateQueryCount Number of queries to execute, in the example: 2
+ * @param ids to prepare statement with. In the example: List containing: 1, 2, 3, 4, 5, 6
+ * @param dbConn database Connection
+ * @throws SQLException
+ */
+ public static int executeUpdateQueries(List<String> updateQueries, List<Integer> updateQueryCount, List<Long> ids,
+ Connection dbConn)
+ throws SQLException {
+ int totalCount = 0;
+ int updatedCount = 0;
+ for (int i = 0; i < updateQueries.size(); i++) {
+ String query = updateQueries.get(i);
+ long insertCount = updateQueryCount.get(i);
+ LOG.debug("Going to execute update <" + query + ">");
+ PreparedStatement pStmt = dbConn.prepareStatement(query);
+ for (int j = 0; j < insertCount; j++) {
+ pStmt.setLong(j + 1, ids.get(totalCount + j));
+ }
+ totalCount += insertCount;
+ int count = pStmt.executeUpdate();
+ LOG.debug("Updated " + count + " records");
Review comment:
Use anchors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] github-actions[bot] commented on pull request #2274: HIVE-25115: Compaction queue entries may accumulate in "ready for cleaning" state
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #2274:
URL: https://github.com/apache/hive/pull/2274#issuecomment-883785153
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] belugabehr commented on a change in pull request #2274: HIVE-25115: Compaction queue entries may accumulate in "ready for cleaning" state
Posted by GitBox <gi...@apache.org>.
belugabehr commented on a change in pull request #2274:
URL: https://github.com/apache/hive/pull/2274#discussion_r637098230
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -365,7 +365,29 @@ public void markCleaned(CompactionInfo info) throws MetaException {
ResultSet rs = null;
try {
dbConn = getDbConn(Connection.TRANSACTION_READ_COMMITTED);
- String s = "INSERT INTO \"COMPLETED_COMPACTIONS\"(\"CC_ID\", \"CC_DATABASE\", "
+
+ // Get all of this partition's COMPACTION_QUEUE entries in "ready for cleaning" with smaller id.
+ // TODO eventually change this to CQ_NEXT_TXN_ID (it might be null for some entires)
+ String s = "SELECT \"CQ_ID\" FROM \"COMPACTION_QUEUE\" WHERE \"CQ_DATABASE\"=? AND \"CQ_TABLE\"=? ";
+ if (info.partName != null) {
+ s += " AND \"CQ_PARTITION\" = ?";
+ }
+ s += " AND \"CQ_STATE\"='" + READY_FOR_CLEANING + "' AND \"CQ_ID\" <= " + info.id;
+ pStmt = dbConn.prepareStatement(s);
+ pStmt.setString(1, info.dbname);
+ pStmt.setString(2, info.tableName);
+ if (info.partName != null) {
+ pStmt.setString(3, info.partName);
+ }
+ LOG.debug("Going to execute query <" + s + "> for CQ_ID=" + info.id);
Review comment:
`LOG.debug("Going to execute query <{}> for CQ_ID={}", s, info.id);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on a change in pull request #2274: HIVE-25115: Compaction queue entries may accumulate in "ready for cleaning" state
Posted by GitBox <gi...@apache.org>.
deniskuzZ commented on a change in pull request #2274:
URL: https://github.com/apache/hive/pull/2274#discussion_r632397219
##########
File path: standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/txn/CompactionTxnHandler.java
##########
@@ -305,17 +302,21 @@ public void markCompacted(CompactionInfo info) throws MetaException {
* By filtering on minOpenTxnWaterMark, we will only cleanup after every transaction is committed, that could see
* the uncompacted deltas. This way the cleaner can clean up everything that was made obsolete by this compaction.
*/
- String s = "SELECT \"CQ_ID\", \"CQ_DATABASE\", \"CQ_TABLE\", \"CQ_PARTITION\", "
- + "\"CQ_TYPE\", \"CQ_RUN_AS\", \"CQ_HIGHEST_WRITE_ID\" FROM \"COMPACTION_QUEUE\" WHERE \"CQ_STATE\" = '"
- + READY_FOR_CLEANING + "'";
- if (minOpenTxnWaterMark > 0) {
- s = s + " AND (\"CQ_NEXT_TXN_ID\" <= " + minOpenTxnWaterMark + " OR \"CQ_NEXT_TXN_ID\" IS NULL)";
- }
+ StringBuilder sb = new StringBuilder();
Review comment:
It's really hard to read the query with builder constructs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org