You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "wecharyu (via GitHub)" <gi...@apache.org> on 2023/07/27 01:40:08 UTC
[GitHub] [hive] wecharyu opened a new pull request, #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
wecharyu opened a new pull request, #4517:
URL: https://github.com/apache/hive/pull/4517
### What changes were proposed in this pull request?
1. Rename `rename_partition` to `alter_partition_core`.
2. Implemented update in batches for all the tables involved in alter partitions.
### Why are the changes needed?
For the performance improvement:
1. JDO update need fetch all fields of old partitions while direct sql only need to get the specific fields.
2. JDO persistence produces many redundant queries while direct sql does not.
### Does this PR introduce _any_ user-facing change?
No.
### Is the change a dependency upgrade?
No.
### How was this patch tested?
1. passing all existing UT.
2. add a benchmark test:
```bash
java -jar ./hmsbench-jar-with-dependencies.jar -H localhost --savedata /tmp/benchdata --sanitize -N 10 -N 100 -N 1000 -o bench_results_direct.csv -C -d testbench_http --params=100 -E 'drop.*' -E 'renameTable.*' -E 'getTableObjectsByName.*' -E 'listTables.*' -E 'listPartitions.*' -E 'getPartitions.*' -E 'getPartitionsByNames.*' -E 'getPartitionNames.*' -E 'listPartition' -E 'getPartition' -E 'getNid' -E 'listDatabases' -E 'getTable' -E 'createTable' -E 'addPartitions.*'
```
- JDO
```bash
Operation Mean Med Min Max Err%
addPartition 27.77 25.05 22.09 50.14 21.88
alterPartitions.10 98.23 97.49 92.21 132.7 4.700
alterPartitions.100 771.6 780.4 713.3 858.3 4.208
alterPartitions.1000 7414 7362 7207 7740 2.050
```
- direct sql
```bash
Operation Mean Med Min Max Err%
addPartition 23.50 20.54 19.55 61.42 31.05
alterPartitions.10 71.33 70.60 67.16 86.74 3.685
alterPartitions.100 428.1 425.2 412.0 479.0 3.052
alterPartitions.1000 3986 3983 3862 4161 1.655
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] deniskuzZ commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1693323450
looks like some unrelated test failures, restarted the build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] wecharyu commented on a diff in pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1317567686
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##########
@@ -532,6 +535,66 @@ public void addPartitions(List<MPartition> parts, List<List<MPartitionPrivilege>
directSqlInsertPart.addPartitions(parts, partPrivilegesList, partColPrivilegesList);
}
+ /**
+ * Alter partitions in batch using direct SQL
+ * @param table the target table
+ * @param partNames list of partition names
+ * @param newParts list of new partitions
+ * @param queryWriteIdList valid write id list
+ * @return
+ * @throws MetaException
+ */
+ public List<Partition> alterPartitions(Table table, List<String> partNames,
+ List<Partition> newParts, String queryWriteIdList) throws MetaException {
+ String filter = "" + PARTITIONS + ".\"PART_NAME\" in (" + makeParams(partNames.size()) + ")";
Review Comment:
There should be no limit on 'IN' clause, and there are also many calls of `makeParams()` in `MetaStoreDirectSql`, for ex:
https://github.com/apache/hive/blob/920f9e535db6270a401db274eef3267d70c1fd2f/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L671-L674
Nevertheless, I think it is reasonable to query in batch just like below.
And as I know only SQL server has a parameter number limit, I will change the code to follow this rule just like add_partitions does:
https://github.com/apache/hive/blob/920f9e535db6270a401db274eef3267d70c1fd2f/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlInsertPart.java#L85-L91
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] saihemanth-cloudera commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "saihemanth-cloudera (via GitHub)" <gi...@apache.org>.
saihemanth-cloudera commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1706995245
@wecharyu - The benchmark test runs could compared with and without your patch. IMO, that could make a strong case for performance improvement. Currently, JDO vs. DirectSQL results are compared with your current patch, which doesn't give the whole story.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1428802963
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdate.java:
##########
@@ -0,0 +1,1541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the statistics update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdate {
+ private static final Logger LOG = LoggerFactory.getLogger(DirectSqlUpdate.class.getName());
+ PersistenceManager pm;
+ Configuration conf;
+ DatabaseProduct dbType;
+ int maxBatchSize;
+ SQLGenerator sqlGenerator;
+ private static final ReentrantLock derbyLock = new ReentrantLock(true);
+
+ public DirectSqlUpdate(PersistenceManager pm, Configuration conf,
+ DatabaseProduct dbType, int batchSize) {
+ this.pm = pm;
+ this.conf = conf;
+ this.dbType = dbType;
+ this.maxBatchSize = batchSize;
+ sqlGenerator = new SQLGenerator(dbType, conf);
+ }
+
+ /**
+ * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
+ * Select ... For Update to sequence operations properly. In practice that means when running
+ * with Derby database. See more notes at class level.
+ */
+ private void lockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.lock();
+ }
+ }
+
+ private void unlockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.unlock();
+ }
+ }
+
+ void rollbackDBConn(Connection dbConn) {
+ try {
+ if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
+ } catch (SQLException e) {
+ LOG.warn("Failed to rollback db connection ", e);
+ }
+ }
+
+ void closeDbConn(JDOConnection jdoConn) {
+ try {
+ if (jdoConn != null) {
+ jdoConn.close();
+ }
+ } catch (Exception e) {
+ LOG.warn("Failed to close db connection", e);
+ }
+ }
+
+ void closeStmt(Statement stmt) {
+ try {
+ if (stmt != null && !stmt.isClosed()) stmt.close();
+ } catch (SQLException e) {
+ LOG.warn("Failed to close statement ", e);
+ }
+ }
+
+ void close(ResultSet rs) {
+ try {
+ if (rs != null && !rs.isClosed()) {
+ rs.close();
+ }
+ }
+ catch(SQLException ex) {
+ LOG.warn("Failed to close statement ", ex);
+ }
+ }
+
+ static String quoteString(String input) {
+ return "'" + input + "'";
+ }
+
+ void close(ResultSet rs, Statement stmt, JDOConnection dbConn) {
+ close(rs);
+ closeStmt(stmt);
+ closeDbConn(dbConn);
+ }
+
+ private void populateInsertUpdateMap(Map<PartitionInfo, ColumnStatistics> statsPartInfoMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics>insertMap,
+ Connection dbConn, Table tbl) throws SQLException, MetaException, NoSuchObjectException {
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+ List<String> queries = new ArrayList<>();
+ Set<PartColNameInfo> selectedParts = new HashSet<>();
+
+ List<Long> partIdList = statsPartInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ prefix.append("select \"PART_ID\", \"COLUMN_NAME\" from \"PART_COL_STATS\" WHERE ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ selectedParts.add(new PartColNameInfo(rs.getLong(1), rs.getString(2)));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+
+ for (Map.Entry entry : statsPartInfoMap.entrySet()) {
+ PartitionInfo partitionInfo = (PartitionInfo) entry.getKey();
+ ColumnStatistics colStats = (ColumnStatistics) entry.getValue();
+ long partId = partitionInfo.partitionId;
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ if (!statsDesc.isSetCatName()) {
+ statsDesc.setCatName(tbl.getCatName());
+ }
+ for (ColumnStatisticsObj statisticsObj : colStats.getStatsObj()) {
+ PartColNameInfo temp = new PartColNameInfo(partId, statisticsObj.getColName());
+ if (selectedParts.contains(temp)) {
+ updateMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ } else {
+ insertMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ }
+ }
+ }
+ }
+
+ private void updatePartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement pst = null;
+ for (Map.Entry entry : updateMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+ String update = "UPDATE \"PART_COL_STATS\" SET ";
+ update += StatObjectConverter.getUpdatedColumnSql(mPartitionColumnStatistics);
+ update += " WHERE \"PART_ID\" = " + partId + " AND "
+ + " \"COLUMN_NAME\" = " + quoteString(mPartitionColumnStatistics.getColName());
+ try {
+ pst = dbConn.prepareStatement(update);
+ StatObjectConverter.initUpdatedColumnStatement(mPartitionColumnStatistics, pst);
+ LOG.debug("Going to execute update " + update);
+ int numUpdate = pst.executeUpdate();
+ if (numUpdate != 1) {
+ throw new MetaException("Invalid state of PART_COL_STATS for PART_ID " + partId);
+ }
+ } finally {
+ closeStmt(pst);
+ }
+ }
+ }
+
+ private void insertIntoPartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> insertMap,
+ long maxCsId,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement preparedStatement = null;
+ int numRows = 0;
+ String insert = "INSERT INTO \"PART_COL_STATS\" (\"CS_ID\", \"CAT_NAME\", \"DB_NAME\","
+ + "\"TABLE_NAME\", \"PARTITION_NAME\", \"COLUMN_NAME\", \"COLUMN_TYPE\", \"PART_ID\","
+ + " \"LONG_LOW_VALUE\", \"LONG_HIGH_VALUE\", \"DOUBLE_HIGH_VALUE\", \"DOUBLE_LOW_VALUE\","
+ + " \"BIG_DECIMAL_LOW_VALUE\", \"BIG_DECIMAL_HIGH_VALUE\", \"NUM_NULLS\", \"NUM_DISTINCTS\", \"BIT_VECTOR\" ,"
+ + " \"HISTOGRAM\", \"AVG_COL_LEN\", \"MAX_COL_LEN\", \"NUM_TRUES\", \"NUM_FALSES\", \"LAST_ANALYZED\", \"ENGINE\") values "
+ + "(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
+
+ try {
+ preparedStatement = dbConn.prepareStatement(insert);
+ for (Map.Entry entry : insertMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+
+ preparedStatement.setLong(1, maxCsId);
+ preparedStatement.setString(2, mPartitionColumnStatistics.getCatName());
+ preparedStatement.setString(3, mPartitionColumnStatistics.getDbName());
+ preparedStatement.setString(4, mPartitionColumnStatistics.getTableName());
+ preparedStatement.setString(5, mPartitionColumnStatistics.getPartitionName());
+ preparedStatement.setString(6, mPartitionColumnStatistics.getColName());
+ preparedStatement.setString(7, mPartitionColumnStatistics.getColType());
+ preparedStatement.setLong(8, partId);
+ preparedStatement.setObject(9, mPartitionColumnStatistics.getLongLowValue());
+ preparedStatement.setObject(10, mPartitionColumnStatistics.getLongHighValue());
+ preparedStatement.setObject(11, mPartitionColumnStatistics.getDoubleHighValue());
+ preparedStatement.setObject(12, mPartitionColumnStatistics.getDoubleLowValue());
+ preparedStatement.setString(13, mPartitionColumnStatistics.getDecimalLowValue());
+ preparedStatement.setString(14, mPartitionColumnStatistics.getDecimalHighValue());
+ preparedStatement.setObject(15, mPartitionColumnStatistics.getNumNulls());
+ preparedStatement.setObject(16, mPartitionColumnStatistics.getNumDVs());
+ preparedStatement.setObject(17, mPartitionColumnStatistics.getBitVector());
+ preparedStatement.setBytes(18, mPartitionColumnStatistics.getHistogram());
+ preparedStatement.setObject(19, mPartitionColumnStatistics.getAvgColLen());
+ preparedStatement.setObject(20, mPartitionColumnStatistics.getMaxColLen());
+ preparedStatement.setObject(21, mPartitionColumnStatistics.getNumTrues());
+ preparedStatement.setObject(22, mPartitionColumnStatistics.getNumFalses());
+ preparedStatement.setLong(23, mPartitionColumnStatistics.getLastAnalyzed());
+ preparedStatement.setString(24, mPartitionColumnStatistics.getEngine());
+
+ maxCsId++;
+ numRows++;
+ preparedStatement.addBatch();
+ if (numRows == maxBatchSize) {
+ preparedStatement.executeBatch();
+ numRows = 0;
+ }
+ }
+
+ if (numRows != 0) {
+ preparedStatement.executeBatch();
+ }
+ } finally {
+ closeStmt(preparedStatement);
+ }
+ }
+
+ private Map<Long, String> getParamValues(Connection dbConn, List<Long> partIdList) throws SQLException {
+ List<String> queries = new ArrayList<>();
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+
+ prefix.append("select \"PART_ID\", \"PARAM_VALUE\" "
+ + " from \"PARTITION_PARAMS\" where "
+ + " \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE' "
+ + " and ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ Map<Long, String> partIdToParaMap = new HashMap<>();
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ partIdToParaMap.put(rs.getLong(1), rs.getString(2));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+ return partIdToParaMap;
+ }
+
+ private void updateWriteIdForPartitions(Connection dbConn, long writeId, List<Long> partIdList) throws SQLException {
+ StringBuilder prefix = new StringBuilder();
+ List<String> queries = new ArrayList<>();
+ StringBuilder suffix = new StringBuilder();
+
+ prefix.append("UPDATE \"PARTITIONS\" set \"WRITE_ID\" = " + writeId + " where ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", false, false);
+
+ Statement statement = null;
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute update " + query);
+ statement.executeUpdate(query);
+ } finally {
+ closeStmt(statement);
+ }
+ }
+ }
+
+ private Map<String, Map<String, String>> updatePartitionParamTable(Connection dbConn,
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap,
+ String validWriteIds,
+ long writeId,
+ boolean isAcidTable)
+ throws SQLException, MetaException {
+ Map<String, Map<String, String>> result = new HashMap<>();
+ boolean areTxnStatsSupported = MetastoreConf.getBoolVar(conf, ConfVars.HIVE_TXN_STATS_ENABLED);
+ PreparedStatement statementInsert = null;
+ PreparedStatement statementDelete = null;
+ PreparedStatement statementUpdate = null;
+ String insert = "INSERT INTO \"PARTITION_PARAMS\" (\"PART_ID\", \"PARAM_KEY\", \"PARAM_VALUE\") "
+ + "VALUES( ? , 'COLUMN_STATS_ACCURATE' , ? )";
+ String delete = "DELETE from \"PARTITION_PARAMS\" "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ String update = "UPDATE \"PARTITION_PARAMS\" set \"PARAM_VALUE\" = ? "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ int numInsert = 0;
+ int numDelete = 0;
+ int numUpdate = 0;
+
+ List<Long> partIdList = partitionInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ // get the old parameters from PARTITION_PARAMS table.
+ Map<Long, String> partIdToParaMap = getParamValues(dbConn, partIdList);
+
+ try {
Review Comment:
please add new line before the `for` loop
i can see multiple places here where we don't use try-for-resources, should it be part of refactor JIRA?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1804878751
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] saihemanth-cloudera commented on a diff in pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "saihemanth-cloudera (via GitHub)" <gi...@apache.org>.
saihemanth-cloudera commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1316190670
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##########
@@ -532,6 +535,66 @@ public void addPartitions(List<MPartition> parts, List<List<MPartitionPrivilege>
directSqlInsertPart.addPartitions(parts, partPrivilegesList, partColPrivilegesList);
}
+ /**
+ * Alter partitions in batch using direct SQL
+ * @param table the target table
+ * @param partNames list of partition names
+ * @param newParts list of new partitions
+ * @param queryWriteIdList valid write id list
+ * @return
+ * @throws MetaException
+ */
+ public List<Partition> alterPartitions(Table table, List<String> partNames,
+ List<Partition> newParts, String queryWriteIdList) throws MetaException {
+ String filter = "" + PARTITIONS + ".\"PART_NAME\" in (" + makeParams(partNames.size()) + ")";
Review Comment:
I think Oracle or Postgres would error out if the partNames.size() > 1000 in the 'IN' clause.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1655508283
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [96 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1656027072
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [96 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1692967955
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [98 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1867200725
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1868499332
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1428804335
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdatePart.java:
##########
@@ -0,0 +1,1536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdatePart {
+ private static final Logger LOG = LoggerFactory.getLogger(DirectSqlUpdatePart.class.getName());
+
+ private final PersistenceManager pm;
+ private final Configuration conf;
+ private final DatabaseProduct dbType;
+ private final int maxBatchSize;
+ private final SQLGenerator sqlGenerator;
+
+ private static final ReentrantLock derbyLock = new ReentrantLock(true);
+
+ public DirectSqlUpdatePart(PersistenceManager pm, Configuration conf,
+ DatabaseProduct dbType, int batchSize) {
+ this.pm = pm;
+ this.conf = conf;
+ this.dbType = dbType;
+ this.maxBatchSize = batchSize;
+ sqlGenerator = new SQLGenerator(dbType, conf);
+ }
+
+ /**
+ * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
+ * Select ... For Update to sequence operations properly. In practice that means when running
+ * with Derby database. See more notes at class level.
+ */
+ private void lockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.lock();
+ }
+ }
+
+ private void unlockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.unlock();
+ }
+ }
+
+ void rollbackDBConn(Connection dbConn) {
+ try {
+ if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
+ } catch (SQLException e) {
+ LOG.warn("Failed to rollback db connection ", e);
+ }
+ }
+
+ void closeDbConn(JDOConnection jdoConn) {
+ try {
+ if (jdoConn != null) {
+ jdoConn.close();
+ }
+ } catch (Exception e) {
+ LOG.warn("Failed to close db connection", e);
+ }
+ }
+
+ void closeStmt(Statement stmt) {
+ try {
+ if (stmt != null && !stmt.isClosed()) stmt.close();
+ } catch (SQLException e) {
+ LOG.warn("Failed to close statement ", e);
+ }
+ }
+
+ void close(ResultSet rs) {
+ try {
+ if (rs != null && !rs.isClosed()) {
+ rs.close();
+ }
+ }
+ catch(SQLException ex) {
+ LOG.warn("Failed to close statement ", ex);
+ }
+ }
+
+ static String quoteString(String input) {
+ return "'" + input + "'";
+ }
+
+ void close(ResultSet rs, Statement stmt, JDOConnection dbConn) {
+ close(rs);
+ closeStmt(stmt);
+ closeDbConn(dbConn);
+ }
+
+ private void populateInsertUpdateMap(Map<PartitionInfo, ColumnStatistics> statsPartInfoMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics>insertMap,
+ Connection dbConn, Table tbl) throws SQLException, MetaException, NoSuchObjectException {
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+ List<String> queries = new ArrayList<>();
+ Set<PartColNameInfo> selectedParts = new HashSet<>();
+
+ List<Long> partIdList = statsPartInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ prefix.append("select \"PART_ID\", \"COLUMN_NAME\" from \"PART_COL_STATS\" WHERE ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ selectedParts.add(new PartColNameInfo(rs.getLong(1), rs.getString(2)));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+
+ for (Map.Entry entry : statsPartInfoMap.entrySet()) {
+ PartitionInfo partitionInfo = (PartitionInfo) entry.getKey();
+ ColumnStatistics colStats = (ColumnStatistics) entry.getValue();
+ long partId = partitionInfo.partitionId;
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ if (!statsDesc.isSetCatName()) {
+ statsDesc.setCatName(tbl.getCatName());
+ }
+ for (ColumnStatisticsObj statisticsObj : colStats.getStatsObj()) {
+ PartColNameInfo temp = new PartColNameInfo(partId, statisticsObj.getColName());
+ if (selectedParts.contains(temp)) {
+ updateMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ } else {
+ insertMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ }
+ }
+ }
+ }
+
+ private void updatePartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement pst = null;
+ for (Map.Entry entry : updateMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+ String update = "UPDATE \"PART_COL_STATS\" SET ";
+ update += StatObjectConverter.getUpdatedColumnSql(mPartitionColumnStatistics);
+ update += " WHERE \"PART_ID\" = " + partId + " AND "
+ + " \"COLUMN_NAME\" = " + quoteString(mPartitionColumnStatistics.getColName());
+ try {
+ pst = dbConn.prepareStatement(update);
+ StatObjectConverter.initUpdatedColumnStatement(mPartitionColumnStatistics, pst);
+ LOG.debug("Going to execute update " + update);
+ int numUpdate = pst.executeUpdate();
+ if (numUpdate != 1) {
+ throw new MetaException("Invalid state of PART_COL_STATS for PART_ID " + partId);
+ }
+ } finally {
+ closeStmt(pst);
+ }
+ }
+ }
+
+ private void insertIntoPartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> insertMap,
+ long maxCsId,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement preparedStatement = null;
+ int numRows = 0;
+ String insert = "INSERT INTO \"PART_COL_STATS\" (\"CS_ID\", \"CAT_NAME\", \"DB_NAME\","
+ + "\"TABLE_NAME\", \"PARTITION_NAME\", \"COLUMN_NAME\", \"COLUMN_TYPE\", \"PART_ID\","
+ + " \"LONG_LOW_VALUE\", \"LONG_HIGH_VALUE\", \"DOUBLE_HIGH_VALUE\", \"DOUBLE_LOW_VALUE\","
+ + " \"BIG_DECIMAL_LOW_VALUE\", \"BIG_DECIMAL_HIGH_VALUE\", \"NUM_NULLS\", \"NUM_DISTINCTS\", \"BIT_VECTOR\" ,"
+ + " \"HISTOGRAM\", \"AVG_COL_LEN\", \"MAX_COL_LEN\", \"NUM_TRUES\", \"NUM_FALSES\", \"LAST_ANALYZED\", \"ENGINE\") values "
+ + "(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
+
+ try {
+ preparedStatement = dbConn.prepareStatement(insert);
+ for (Map.Entry entry : insertMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+
+ preparedStatement.setLong(1, maxCsId);
+ preparedStatement.setString(2, mPartitionColumnStatistics.getCatName());
+ preparedStatement.setString(3, mPartitionColumnStatistics.getDbName());
+ preparedStatement.setString(4, mPartitionColumnStatistics.getTableName());
+ preparedStatement.setString(5, mPartitionColumnStatistics.getPartitionName());
+ preparedStatement.setString(6, mPartitionColumnStatistics.getColName());
+ preparedStatement.setString(7, mPartitionColumnStatistics.getColType());
+ preparedStatement.setLong(8, partId);
+ preparedStatement.setObject(9, mPartitionColumnStatistics.getLongLowValue());
+ preparedStatement.setObject(10, mPartitionColumnStatistics.getLongHighValue());
+ preparedStatement.setObject(11, mPartitionColumnStatistics.getDoubleHighValue());
+ preparedStatement.setObject(12, mPartitionColumnStatistics.getDoubleLowValue());
+ preparedStatement.setString(13, mPartitionColumnStatistics.getDecimalLowValue());
+ preparedStatement.setString(14, mPartitionColumnStatistics.getDecimalHighValue());
+ preparedStatement.setObject(15, mPartitionColumnStatistics.getNumNulls());
+ preparedStatement.setObject(16, mPartitionColumnStatistics.getNumDVs());
+ preparedStatement.setObject(17, mPartitionColumnStatistics.getBitVector());
+ preparedStatement.setBytes(18, mPartitionColumnStatistics.getHistogram());
+ preparedStatement.setObject(19, mPartitionColumnStatistics.getAvgColLen());
+ preparedStatement.setObject(20, mPartitionColumnStatistics.getMaxColLen());
+ preparedStatement.setObject(21, mPartitionColumnStatistics.getNumTrues());
+ preparedStatement.setObject(22, mPartitionColumnStatistics.getNumFalses());
+ preparedStatement.setLong(23, mPartitionColumnStatistics.getLastAnalyzed());
+ preparedStatement.setString(24, mPartitionColumnStatistics.getEngine());
+
+ maxCsId++;
+ numRows++;
+ preparedStatement.addBatch();
+ if (numRows == maxBatchSize) {
+ preparedStatement.executeBatch();
+ numRows = 0;
+ }
+ }
+
+ if (numRows != 0) {
+ preparedStatement.executeBatch();
+ }
+ } finally {
+ closeStmt(preparedStatement);
+ }
+ }
+
+ private Map<Long, String> getParamValues(Connection dbConn, List<Long> partIdList) throws SQLException {
+ List<String> queries = new ArrayList<>();
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+
+ prefix.append("select \"PART_ID\", \"PARAM_VALUE\" "
+ + " from \"PARTITION_PARAMS\" where "
+ + " \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE' "
+ + " and ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ Map<Long, String> partIdToParaMap = new HashMap<>();
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ partIdToParaMap.put(rs.getLong(1), rs.getString(2));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+ return partIdToParaMap;
+ }
+
+ private void updateWriteIdForPartitions(Connection dbConn, long writeId, List<Long> partIdList) throws SQLException {
+ StringBuilder prefix = new StringBuilder();
+ List<String> queries = new ArrayList<>();
+ StringBuilder suffix = new StringBuilder();
+
+ prefix.append("UPDATE \"PARTITIONS\" set \"WRITE_ID\" = " + writeId + " where ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", false, false);
+
+ Statement statement = null;
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute update " + query);
+ statement.executeUpdate(query);
+ } finally {
+ closeStmt(statement);
+ }
+ }
+ }
+
+ private Map<String, Map<String, String>> updatePartitionParamTable(Connection dbConn,
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap,
+ String validWriteIds,
+ long writeId,
+ boolean isAcidTable)
+ throws SQLException, MetaException {
+ Map<String, Map<String, String>> result = new HashMap<>();
+ boolean areTxnStatsSupported = MetastoreConf.getBoolVar(conf, ConfVars.HIVE_TXN_STATS_ENABLED);
+ String insert = "INSERT INTO \"PARTITION_PARAMS\" (\"PART_ID\", \"PARAM_KEY\", \"PARAM_VALUE\") "
+ + "VALUES( ? , 'COLUMN_STATS_ACCURATE' , ? )";
+ String delete = "DELETE from \"PARTITION_PARAMS\" "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ String update = "UPDATE \"PARTITION_PARAMS\" set \"PARAM_VALUE\" = ? "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ int numInsert = 0;
+ int numDelete = 0;
+ int numUpdate = 0;
+
+ List<Long> partIdList = partitionInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ // get the old parameters from PARTITION_PARAMS table.
+ Map<Long, String> partIdToParaMap = getParamValues(dbConn, partIdList);
+
+ try (PreparedStatement statementInsert = dbConn.prepareStatement(insert);
+ PreparedStatement statementDelete = dbConn.prepareStatement(delete);
+ PreparedStatement statementUpdate = dbConn.prepareStatement(update)) {
+ for (Map.Entry entry : partitionInfoMap.entrySet()) {
+ PartitionInfo partitionInfo = (PartitionInfo) entry.getKey();
+ ColumnStatistics colStats = (ColumnStatistics) entry.getValue();
+ List<String> colNames = colStats.getStatsObj().stream().map(e -> e.getColName()).collect(Collectors.toList());
+ long partWriteId = partitionInfo.writeId;
+ long partId = partitionInfo.partitionId;
+ Map<String, String> newParameter;
+
+ if (!partIdToParaMap.containsKey(partId)) {
+ newParameter = new HashMap<>();
+ newParameter.put(COLUMN_STATS_ACCURATE, "TRUE");
+ StatsSetupConst.setColumnStatsState(newParameter, colNames);
+ statementInsert.setLong(1, partId);
+ statementInsert.setString(2, newParameter.get(COLUMN_STATS_ACCURATE));
+ numInsert++;
+ statementInsert.addBatch();
+ if (numInsert == maxBatchSize) {
+ LOG.debug(" Executing insert " + insert);
+ statementInsert.executeBatch();
+ numInsert = 0;
+ }
+ } else {
+ String oldStats = partIdToParaMap.get(partId);
+
+ Map<String, String> oldParameter = new HashMap<>();
+ oldParameter.put(COLUMN_STATS_ACCURATE, oldStats);
+
+ newParameter = new HashMap<>();
+ newParameter.put(COLUMN_STATS_ACCURATE, oldStats);
+ StatsSetupConst.setColumnStatsState(newParameter, colNames);
+
+ if (isAcidTable) {
+ String errorMsg = ObjectStore.verifyStatsChangeCtx(
+ colStats.getStatsDesc().getDbName() + "." + colStats.getStatsDesc().getTableName(),
+ oldParameter, newParameter, writeId, validWriteIds, true);
+ if (errorMsg != null) {
+ throw new MetaException(errorMsg);
+ }
+ }
+
+ if (isAcidTable &&
+ (!areTxnStatsSupported || !ObjectStore.isCurrentStatsValidForTheQuery(oldParameter, partWriteId,
+ validWriteIds, true))) {
+ statementDelete.setLong(1, partId);
+ statementDelete.addBatch();
+ numDelete++;
+ if (numDelete == maxBatchSize) {
+ statementDelete.executeBatch();
+ numDelete = 0;
+ LOG.debug("Removed COLUMN_STATS_ACCURATE from the parameters of the partition "
+ + colStats.getStatsDesc().getDbName() + "." + colStats.getStatsDesc().getTableName() + "."
+ + colStats.getStatsDesc().getPartName());
+ }
+ } else {
+ statementUpdate.setString(1, newParameter.get(COLUMN_STATS_ACCURATE));
+ statementUpdate.setLong(2, partId);
+ statementUpdate.addBatch();
+ numUpdate++;
+ if (numUpdate == maxBatchSize) {
+ LOG.debug(" Executing update " + statementUpdate);
+ statementUpdate.executeBatch();
+ numUpdate = 0;
+ }
+ }
+ }
+ result.put(partitionInfo.partitionName, newParameter);
+ }
+
+ if (numInsert != 0) {
+ statementInsert.executeBatch();
+ }
+
+ if (numUpdate != 0) {
+ statementUpdate.executeBatch();
+ }
+
+ if (numDelete != 0) {
+ statementDelete.executeBatch();
+ }
+
+ if (isAcidTable) {
+ updateWriteIdForPartitions(dbConn, writeId, partIdList);
+ }
+ return result;
+ }
+ }
+
+
+ private Map<PartitionInfo, ColumnStatistics> getPartitionInfo(Connection dbConn, long tblId,
+ Map<String, ColumnStatistics> partColStatsMap)
+ throws SQLException, MetaException {
+ List<String> queries = new ArrayList<>();
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap = new HashMap<>();
+
+ List<String> partKeys = partColStatsMap.keySet().stream().map(
+ e -> quoteString(e)).collect(Collectors.toList()
+ );
+
+ prefix.append("select \"PART_ID\", \"WRITE_ID\", \"PART_NAME\" from \"PARTITIONS\" where ");
+ suffix.append(" and \"TBL_ID\" = " + tblId);
+ TxnUtils.buildQueryWithINClauseStrings(conf, queries, prefix, suffix,
+ partKeys, "\"PART_NAME\"", true, false);
+
+ for (String query : queries) {
+ // Select for update makes sure that the partitions are not modified while the stats are getting updated.
+ query = sqlGenerator.addForUpdateClause(query);
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query <" + query + ">");
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ PartitionInfo partitionInfo = new PartitionInfo(rs.getLong(1),
+ rs.getLong(2), rs.getString(3));
+ partitionInfoMap.put(partitionInfo, partColStatsMap.get(rs.getString(3)));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+ return partitionInfoMap;
+ }
+
+ private void setAnsiQuotes(Connection dbConn) throws SQLException {
+ if (sqlGenerator.getDbProduct().isMYSQL()) {
+ try (Statement stmt = dbConn.createStatement()) {
+ stmt.execute("SET @@session.sql_mode=ANSI_QUOTES");
+ }
+ }
+ }
+
+ /**
+ * Update the statistics for the given partitions. Add the notification logs also.
+ * @return map of partition key to column stats if successful, null otherwise.
+ */
+ public Map<String, Map<String, String>> updatePartitionColumnStatistics(Map<String, ColumnStatistics> partColStatsMap,
+ Table tbl, long csId,
+ String validWriteIds, long writeId,
+ List<TransactionalMetaStoreEventListener> transactionalListeners)
+ throws MetaException {
+ JDOConnection jdoConn = null;
+ Connection dbConn = null;
+ boolean committed = false;
+ try {
+ lockInternal();
+ jdoConn = pm.getDataStoreConnection();
+ dbConn = (Connection) (jdoConn.getNativeConnection());
+
+ setAnsiQuotes(dbConn);
+
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap = getPartitionInfo(dbConn, tbl.getId(), partColStatsMap);
+
+ Map<String, Map<String, String>> result =
+ updatePartitionParamTable(dbConn, partitionInfoMap, validWriteIds, writeId, TxnUtils.isAcidTable(tbl));
+
+ Map<PartColNameInfo, MPartitionColumnStatistics> insertMap = new HashMap<>();
+ Map<PartColNameInfo, MPartitionColumnStatistics> updateMap = new HashMap<>();
+ populateInsertUpdateMap(partitionInfoMap, updateMap, insertMap, dbConn, tbl);
+
+ LOG.info("Number of stats to insert " + insertMap.size() + " update " + updateMap.size());
+
+ if (insertMap.size() != 0) {
+ insertIntoPartColStatTable(insertMap, csId, dbConn);
+ }
+
+ if (updateMap.size() != 0) {
+ updatePartColStatTable(updateMap, dbConn);
+ }
+
+ if (transactionalListeners != null) {
+ UpdatePartitionColumnStatEventBatch eventBatch = new UpdatePartitionColumnStatEventBatch(null);
+ for (Map.Entry entry : result.entrySet()) {
+ Map<String, String> parameters = (Map<String, String>) entry.getValue();
+ ColumnStatistics colStats = partColStatsMap.get(entry.getKey());
+ List<String> partVals = getPartValsFromName(tbl, colStats.getStatsDesc().getPartName());
+ UpdatePartitionColumnStatEvent event = new UpdatePartitionColumnStatEvent(colStats, partVals, parameters,
+ tbl, writeId, null);
+ eventBatch.addPartColStatEvent(event);
+ }
+ MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+ EventMessage.EventType.UPDATE_PARTITION_COLUMN_STAT_BATCH, eventBatch, dbConn, sqlGenerator);
+ }
+ dbConn.commit();
+ committed = true;
+ return result;
+ } catch (Exception e) {
+ LOG.error("Unable to update Column stats for " + tbl.getTableName(), e);
+ throw new MetaException("Unable to update Column stats for " + tbl.getTableName()
+ + " due to: " + e.getMessage());
+ } finally {
+ if (!committed) {
+ rollbackDBConn(dbConn);
+ }
+ closeDbConn(jdoConn);
+ unlockInternal();
+ }
+ }
+
+ /**
+ * Gets the next CS id from sequence MPartitionColumnStatistics and increment the CS id by numStats.
+ * @return The CD id before update.
+ */
+ public long getNextCSIdForMPartitionColumnStatistics(long numStats) throws MetaException {
+ Statement statement = null;
+ ResultSet rs = null;
+ long maxCsId = 0;
+ boolean committed = false;
+ Connection dbConn = null;
+ JDOConnection jdoConn = null;
+
+ try {
+ lockInternal();
+ jdoConn = pm.getDataStoreConnection();
+ dbConn = (Connection) (jdoConn.getNativeConnection());
+
+ setAnsiQuotes(dbConn);
+
+ // This loop will be iterated at max twice. If there is no records, it will first insert and then do a select.
+ // We are not using any upsert operations as select for update and then update is required to make sure that
+ // the caller gets a reserved range for CSId not used by any other thread.
+ boolean insertDone = false;
+ while (maxCsId == 0) {
+ String query = sqlGenerator.addForUpdateClause("SELECT \"NEXT_VAL\" FROM \"SEQUENCE_TABLE\" "
+ + "WHERE \"SEQUENCE_NAME\"= "
+ + quoteString("org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics"));
+ LOG.debug("Going to execute query " + query);
+ statement = dbConn.createStatement();
Review Comment:
where do we close the statement
see https://sonarcloud.io/project/issues?resolved=false&types=BUG&pullRequest=4517&id=apache_hive&open=AYvIijb_0txSvpyOT_4y
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdatePart.java:
##########
@@ -0,0 +1,1536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdatePart {
+ private static final Logger LOG = LoggerFactory.getLogger(DirectSqlUpdatePart.class.getName());
+
+ private final PersistenceManager pm;
+ private final Configuration conf;
+ private final DatabaseProduct dbType;
+ private final int maxBatchSize;
+ private final SQLGenerator sqlGenerator;
+
+ private static final ReentrantLock derbyLock = new ReentrantLock(true);
+
+ public DirectSqlUpdatePart(PersistenceManager pm, Configuration conf,
+ DatabaseProduct dbType, int batchSize) {
+ this.pm = pm;
+ this.conf = conf;
+ this.dbType = dbType;
+ this.maxBatchSize = batchSize;
+ sqlGenerator = new SQLGenerator(dbType, conf);
+ }
+
+ /**
+ * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
+ * Select ... For Update to sequence operations properly. In practice that means when running
+ * with Derby database. See more notes at class level.
+ */
+ private void lockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.lock();
+ }
+ }
+
+ private void unlockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.unlock();
+ }
+ }
+
+ void rollbackDBConn(Connection dbConn) {
+ try {
+ if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
+ } catch (SQLException e) {
+ LOG.warn("Failed to rollback db connection ", e);
+ }
+ }
+
+ void closeDbConn(JDOConnection jdoConn) {
+ try {
+ if (jdoConn != null) {
+ jdoConn.close();
+ }
+ } catch (Exception e) {
+ LOG.warn("Failed to close db connection", e);
+ }
+ }
+
+ void closeStmt(Statement stmt) {
+ try {
+ if (stmt != null && !stmt.isClosed()) stmt.close();
+ } catch (SQLException e) {
+ LOG.warn("Failed to close statement ", e);
+ }
+ }
+
+ void close(ResultSet rs) {
+ try {
+ if (rs != null && !rs.isClosed()) {
+ rs.close();
+ }
+ }
+ catch(SQLException ex) {
+ LOG.warn("Failed to close statement ", ex);
+ }
+ }
+
+ static String quoteString(String input) {
+ return "'" + input + "'";
+ }
+
+ void close(ResultSet rs, Statement stmt, JDOConnection dbConn) {
+ close(rs);
+ closeStmt(stmt);
+ closeDbConn(dbConn);
+ }
+
+ private void populateInsertUpdateMap(Map<PartitionInfo, ColumnStatistics> statsPartInfoMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics>insertMap,
+ Connection dbConn, Table tbl) throws SQLException, MetaException, NoSuchObjectException {
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+ List<String> queries = new ArrayList<>();
+ Set<PartColNameInfo> selectedParts = new HashSet<>();
+
+ List<Long> partIdList = statsPartInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ prefix.append("select \"PART_ID\", \"COLUMN_NAME\" from \"PART_COL_STATS\" WHERE ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ selectedParts.add(new PartColNameInfo(rs.getLong(1), rs.getString(2)));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+
+ for (Map.Entry entry : statsPartInfoMap.entrySet()) {
+ PartitionInfo partitionInfo = (PartitionInfo) entry.getKey();
+ ColumnStatistics colStats = (ColumnStatistics) entry.getValue();
+ long partId = partitionInfo.partitionId;
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ if (!statsDesc.isSetCatName()) {
+ statsDesc.setCatName(tbl.getCatName());
+ }
+ for (ColumnStatisticsObj statisticsObj : colStats.getStatsObj()) {
+ PartColNameInfo temp = new PartColNameInfo(partId, statisticsObj.getColName());
+ if (selectedParts.contains(temp)) {
+ updateMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ } else {
+ insertMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ }
+ }
+ }
+ }
+
+ private void updatePartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement pst = null;
+ for (Map.Entry entry : updateMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+ String update = "UPDATE \"PART_COL_STATS\" SET ";
+ update += StatObjectConverter.getUpdatedColumnSql(mPartitionColumnStatistics);
+ update += " WHERE \"PART_ID\" = " + partId + " AND "
+ + " \"COLUMN_NAME\" = " + quoteString(mPartitionColumnStatistics.getColName());
+ try {
+ pst = dbConn.prepareStatement(update);
+ StatObjectConverter.initUpdatedColumnStatement(mPartitionColumnStatistics, pst);
+ LOG.debug("Going to execute update " + update);
+ int numUpdate = pst.executeUpdate();
+ if (numUpdate != 1) {
+ throw new MetaException("Invalid state of PART_COL_STATS for PART_ID " + partId);
+ }
+ } finally {
+ closeStmt(pst);
+ }
+ }
+ }
+
+ private void insertIntoPartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> insertMap,
+ long maxCsId,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement preparedStatement = null;
+ int numRows = 0;
+ String insert = "INSERT INTO \"PART_COL_STATS\" (\"CS_ID\", \"CAT_NAME\", \"DB_NAME\","
+ + "\"TABLE_NAME\", \"PARTITION_NAME\", \"COLUMN_NAME\", \"COLUMN_TYPE\", \"PART_ID\","
+ + " \"LONG_LOW_VALUE\", \"LONG_HIGH_VALUE\", \"DOUBLE_HIGH_VALUE\", \"DOUBLE_LOW_VALUE\","
+ + " \"BIG_DECIMAL_LOW_VALUE\", \"BIG_DECIMAL_HIGH_VALUE\", \"NUM_NULLS\", \"NUM_DISTINCTS\", \"BIT_VECTOR\" ,"
+ + " \"HISTOGRAM\", \"AVG_COL_LEN\", \"MAX_COL_LEN\", \"NUM_TRUES\", \"NUM_FALSES\", \"LAST_ANALYZED\", \"ENGINE\") values "
+ + "(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
+
+ try {
+ preparedStatement = dbConn.prepareStatement(insert);
+ for (Map.Entry entry : insertMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+
+ preparedStatement.setLong(1, maxCsId);
+ preparedStatement.setString(2, mPartitionColumnStatistics.getCatName());
+ preparedStatement.setString(3, mPartitionColumnStatistics.getDbName());
+ preparedStatement.setString(4, mPartitionColumnStatistics.getTableName());
+ preparedStatement.setString(5, mPartitionColumnStatistics.getPartitionName());
+ preparedStatement.setString(6, mPartitionColumnStatistics.getColName());
+ preparedStatement.setString(7, mPartitionColumnStatistics.getColType());
+ preparedStatement.setLong(8, partId);
+ preparedStatement.setObject(9, mPartitionColumnStatistics.getLongLowValue());
+ preparedStatement.setObject(10, mPartitionColumnStatistics.getLongHighValue());
+ preparedStatement.setObject(11, mPartitionColumnStatistics.getDoubleHighValue());
+ preparedStatement.setObject(12, mPartitionColumnStatistics.getDoubleLowValue());
+ preparedStatement.setString(13, mPartitionColumnStatistics.getDecimalLowValue());
+ preparedStatement.setString(14, mPartitionColumnStatistics.getDecimalHighValue());
+ preparedStatement.setObject(15, mPartitionColumnStatistics.getNumNulls());
+ preparedStatement.setObject(16, mPartitionColumnStatistics.getNumDVs());
+ preparedStatement.setObject(17, mPartitionColumnStatistics.getBitVector());
+ preparedStatement.setBytes(18, mPartitionColumnStatistics.getHistogram());
+ preparedStatement.setObject(19, mPartitionColumnStatistics.getAvgColLen());
+ preparedStatement.setObject(20, mPartitionColumnStatistics.getMaxColLen());
+ preparedStatement.setObject(21, mPartitionColumnStatistics.getNumTrues());
+ preparedStatement.setObject(22, mPartitionColumnStatistics.getNumFalses());
+ preparedStatement.setLong(23, mPartitionColumnStatistics.getLastAnalyzed());
+ preparedStatement.setString(24, mPartitionColumnStatistics.getEngine());
+
+ maxCsId++;
+ numRows++;
+ preparedStatement.addBatch();
+ if (numRows == maxBatchSize) {
+ preparedStatement.executeBatch();
+ numRows = 0;
+ }
+ }
+
+ if (numRows != 0) {
+ preparedStatement.executeBatch();
+ }
+ } finally {
+ closeStmt(preparedStatement);
+ }
+ }
+
+ private Map<Long, String> getParamValues(Connection dbConn, List<Long> partIdList) throws SQLException {
+ List<String> queries = new ArrayList<>();
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+
+ prefix.append("select \"PART_ID\", \"PARAM_VALUE\" "
+ + " from \"PARTITION_PARAMS\" where "
+ + " \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE' "
+ + " and ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ Map<Long, String> partIdToParaMap = new HashMap<>();
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ partIdToParaMap.put(rs.getLong(1), rs.getString(2));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+ return partIdToParaMap;
+ }
+
+ private void updateWriteIdForPartitions(Connection dbConn, long writeId, List<Long> partIdList) throws SQLException {
+ StringBuilder prefix = new StringBuilder();
+ List<String> queries = new ArrayList<>();
+ StringBuilder suffix = new StringBuilder();
+
+ prefix.append("UPDATE \"PARTITIONS\" set \"WRITE_ID\" = " + writeId + " where ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", false, false);
+
+ Statement statement = null;
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute update " + query);
+ statement.executeUpdate(query);
+ } finally {
+ closeStmt(statement);
+ }
+ }
+ }
+
+ private Map<String, Map<String, String>> updatePartitionParamTable(Connection dbConn,
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap,
+ String validWriteIds,
+ long writeId,
+ boolean isAcidTable)
+ throws SQLException, MetaException {
+ Map<String, Map<String, String>> result = new HashMap<>();
+ boolean areTxnStatsSupported = MetastoreConf.getBoolVar(conf, ConfVars.HIVE_TXN_STATS_ENABLED);
+ String insert = "INSERT INTO \"PARTITION_PARAMS\" (\"PART_ID\", \"PARAM_KEY\", \"PARAM_VALUE\") "
+ + "VALUES( ? , 'COLUMN_STATS_ACCURATE' , ? )";
+ String delete = "DELETE from \"PARTITION_PARAMS\" "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ String update = "UPDATE \"PARTITION_PARAMS\" set \"PARAM_VALUE\" = ? "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ int numInsert = 0;
+ int numDelete = 0;
+ int numUpdate = 0;
+
+ List<Long> partIdList = partitionInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ // get the old parameters from PARTITION_PARAMS table.
+ Map<Long, String> partIdToParaMap = getParamValues(dbConn, partIdList);
+
+ try (PreparedStatement statementInsert = dbConn.prepareStatement(insert);
+ PreparedStatement statementDelete = dbConn.prepareStatement(delete);
+ PreparedStatement statementUpdate = dbConn.prepareStatement(update)) {
+ for (Map.Entry entry : partitionInfoMap.entrySet()) {
+ PartitionInfo partitionInfo = (PartitionInfo) entry.getKey();
+ ColumnStatistics colStats = (ColumnStatistics) entry.getValue();
+ List<String> colNames = colStats.getStatsObj().stream().map(e -> e.getColName()).collect(Collectors.toList());
+ long partWriteId = partitionInfo.writeId;
+ long partId = partitionInfo.partitionId;
+ Map<String, String> newParameter;
+
+ if (!partIdToParaMap.containsKey(partId)) {
+ newParameter = new HashMap<>();
+ newParameter.put(COLUMN_STATS_ACCURATE, "TRUE");
+ StatsSetupConst.setColumnStatsState(newParameter, colNames);
+ statementInsert.setLong(1, partId);
+ statementInsert.setString(2, newParameter.get(COLUMN_STATS_ACCURATE));
+ numInsert++;
+ statementInsert.addBatch();
+ if (numInsert == maxBatchSize) {
+ LOG.debug(" Executing insert " + insert);
+ statementInsert.executeBatch();
+ numInsert = 0;
+ }
+ } else {
+ String oldStats = partIdToParaMap.get(partId);
+
+ Map<String, String> oldParameter = new HashMap<>();
+ oldParameter.put(COLUMN_STATS_ACCURATE, oldStats);
+
+ newParameter = new HashMap<>();
+ newParameter.put(COLUMN_STATS_ACCURATE, oldStats);
+ StatsSetupConst.setColumnStatsState(newParameter, colNames);
+
+ if (isAcidTable) {
+ String errorMsg = ObjectStore.verifyStatsChangeCtx(
+ colStats.getStatsDesc().getDbName() + "." + colStats.getStatsDesc().getTableName(),
+ oldParameter, newParameter, writeId, validWriteIds, true);
+ if (errorMsg != null) {
+ throw new MetaException(errorMsg);
+ }
+ }
+
+ if (isAcidTable &&
+ (!areTxnStatsSupported || !ObjectStore.isCurrentStatsValidForTheQuery(oldParameter, partWriteId,
+ validWriteIds, true))) {
+ statementDelete.setLong(1, partId);
+ statementDelete.addBatch();
+ numDelete++;
+ if (numDelete == maxBatchSize) {
+ statementDelete.executeBatch();
+ numDelete = 0;
+ LOG.debug("Removed COLUMN_STATS_ACCURATE from the parameters of the partition "
+ + colStats.getStatsDesc().getDbName() + "." + colStats.getStatsDesc().getTableName() + "."
+ + colStats.getStatsDesc().getPartName());
+ }
+ } else {
+ statementUpdate.setString(1, newParameter.get(COLUMN_STATS_ACCURATE));
+ statementUpdate.setLong(2, partId);
+ statementUpdate.addBatch();
+ numUpdate++;
+ if (numUpdate == maxBatchSize) {
+ LOG.debug(" Executing update " + statementUpdate);
+ statementUpdate.executeBatch();
+ numUpdate = 0;
+ }
+ }
+ }
+ result.put(partitionInfo.partitionName, newParameter);
+ }
+
+ if (numInsert != 0) {
+ statementInsert.executeBatch();
+ }
+
+ if (numUpdate != 0) {
+ statementUpdate.executeBatch();
+ }
+
+ if (numDelete != 0) {
+ statementDelete.executeBatch();
+ }
+
+ if (isAcidTable) {
+ updateWriteIdForPartitions(dbConn, writeId, partIdList);
+ }
+ return result;
+ }
+ }
+
+
+ private Map<PartitionInfo, ColumnStatistics> getPartitionInfo(Connection dbConn, long tblId,
+ Map<String, ColumnStatistics> partColStatsMap)
+ throws SQLException, MetaException {
+ List<String> queries = new ArrayList<>();
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap = new HashMap<>();
+
+ List<String> partKeys = partColStatsMap.keySet().stream().map(
+ e -> quoteString(e)).collect(Collectors.toList()
+ );
+
+ prefix.append("select \"PART_ID\", \"WRITE_ID\", \"PART_NAME\" from \"PARTITIONS\" where ");
+ suffix.append(" and \"TBL_ID\" = " + tblId);
+ TxnUtils.buildQueryWithINClauseStrings(conf, queries, prefix, suffix,
+ partKeys, "\"PART_NAME\"", true, false);
+
+ for (String query : queries) {
+ // Select for update makes sure that the partitions are not modified while the stats are getting updated.
+ query = sqlGenerator.addForUpdateClause(query);
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query <" + query + ">");
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ PartitionInfo partitionInfo = new PartitionInfo(rs.getLong(1),
+ rs.getLong(2), rs.getString(3));
+ partitionInfoMap.put(partitionInfo, partColStatsMap.get(rs.getString(3)));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+ return partitionInfoMap;
+ }
+
+ private void setAnsiQuotes(Connection dbConn) throws SQLException {
+ if (sqlGenerator.getDbProduct().isMYSQL()) {
+ try (Statement stmt = dbConn.createStatement()) {
+ stmt.execute("SET @@session.sql_mode=ANSI_QUOTES");
+ }
+ }
+ }
+
+ /**
+ * Update the statistics for the given partitions. Add the notification logs also.
+ * @return map of partition key to column stats if successful, null otherwise.
+ */
+ public Map<String, Map<String, String>> updatePartitionColumnStatistics(Map<String, ColumnStatistics> partColStatsMap,
+ Table tbl, long csId,
+ String validWriteIds, long writeId,
+ List<TransactionalMetaStoreEventListener> transactionalListeners)
+ throws MetaException {
+ JDOConnection jdoConn = null;
+ Connection dbConn = null;
+ boolean committed = false;
+ try {
+ lockInternal();
+ jdoConn = pm.getDataStoreConnection();
+ dbConn = (Connection) (jdoConn.getNativeConnection());
+
+ setAnsiQuotes(dbConn);
+
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap = getPartitionInfo(dbConn, tbl.getId(), partColStatsMap);
+
+ Map<String, Map<String, String>> result =
+ updatePartitionParamTable(dbConn, partitionInfoMap, validWriteIds, writeId, TxnUtils.isAcidTable(tbl));
+
+ Map<PartColNameInfo, MPartitionColumnStatistics> insertMap = new HashMap<>();
+ Map<PartColNameInfo, MPartitionColumnStatistics> updateMap = new HashMap<>();
+ populateInsertUpdateMap(partitionInfoMap, updateMap, insertMap, dbConn, tbl);
+
+ LOG.info("Number of stats to insert " + insertMap.size() + " update " + updateMap.size());
+
+ if (insertMap.size() != 0) {
+ insertIntoPartColStatTable(insertMap, csId, dbConn);
+ }
+
+ if (updateMap.size() != 0) {
+ updatePartColStatTable(updateMap, dbConn);
+ }
+
+ if (transactionalListeners != null) {
+ UpdatePartitionColumnStatEventBatch eventBatch = new UpdatePartitionColumnStatEventBatch(null);
+ for (Map.Entry entry : result.entrySet()) {
+ Map<String, String> parameters = (Map<String, String>) entry.getValue();
+ ColumnStatistics colStats = partColStatsMap.get(entry.getKey());
+ List<String> partVals = getPartValsFromName(tbl, colStats.getStatsDesc().getPartName());
+ UpdatePartitionColumnStatEvent event = new UpdatePartitionColumnStatEvent(colStats, partVals, parameters,
+ tbl, writeId, null);
+ eventBatch.addPartColStatEvent(event);
+ }
+ MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+ EventMessage.EventType.UPDATE_PARTITION_COLUMN_STAT_BATCH, eventBatch, dbConn, sqlGenerator);
+ }
+ dbConn.commit();
+ committed = true;
+ return result;
+ } catch (Exception e) {
+ LOG.error("Unable to update Column stats for " + tbl.getTableName(), e);
+ throw new MetaException("Unable to update Column stats for " + tbl.getTableName()
+ + " due to: " + e.getMessage());
+ } finally {
+ if (!committed) {
+ rollbackDBConn(dbConn);
+ }
+ closeDbConn(jdoConn);
+ unlockInternal();
+ }
+ }
+
+ /**
+ * Gets the next CS id from sequence MPartitionColumnStatistics and increment the CS id by numStats.
+ * @return The CD id before update.
+ */
+ public long getNextCSIdForMPartitionColumnStatistics(long numStats) throws MetaException {
+ Statement statement = null;
+ ResultSet rs = null;
+ long maxCsId = 0;
+ boolean committed = false;
+ Connection dbConn = null;
+ JDOConnection jdoConn = null;
+
+ try {
+ lockInternal();
+ jdoConn = pm.getDataStoreConnection();
+ dbConn = (Connection) (jdoConn.getNativeConnection());
+
+ setAnsiQuotes(dbConn);
+
+ // This loop will be iterated at max twice. If there is no records, it will first insert and then do a select.
+ // We are not using any upsert operations as select for update and then update is required to make sure that
+ // the caller gets a reserved range for CSId not used by any other thread.
+ boolean insertDone = false;
+ while (maxCsId == 0) {
+ String query = sqlGenerator.addForUpdateClause("SELECT \"NEXT_VAL\" FROM \"SEQUENCE_TABLE\" "
+ + "WHERE \"SEQUENCE_NAME\"= "
+ + quoteString("org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics"));
+ LOG.debug("Going to execute query " + query);
+ statement = dbConn.createStatement();
Review Comment:
where do we close the statement?
see https://sonarcloud.io/project/issues?resolved=false&types=BUG&pullRequest=4517&id=apache_hive&open=AYvIijb_0txSvpyOT_4y
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1862257872
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1808020945
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [96 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1410548941
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdate.java:
##########
@@ -0,0 +1,1541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the statistics update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdate {
+ private static final Logger LOG = LoggerFactory.getLogger(DirectSqlUpdate.class.getName());
+ PersistenceManager pm;
Review Comment:
add new line and add private modifiers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1859187595
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[97 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "nrg4878 (via GitHub)" <gi...@apache.org>.
nrg4878 commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1861011813
@VenuReddy2103 Can you please review this based on your work for AddPartitions() ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1867385188
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1863254728
@wecharyu , http://ci.hive.apache.org/job/hive-precommit/job/PR-4517/17/testReport/junit/org.apache.hadoop.hive.ql.exec/TestConcurrentDppInserts/Testing___split_17___PostProcess___testConcurrentCreationOfSamePartition/ seems related
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1865628021
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1868518460
@deniskuzZ The CI passed finally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1806325969
> @saihemanth-cloudera Here is another benchmark test for before and after this path:
>
> * before
>
> ```shell
> Operation Mean Med Min Max Err%
> addPartition 25.61 22.90 20.19 59.38 26.34
> alterPartitions.10 59.55 58.81 55.35 69.30 5.140
> alterPartitions.100 295.5 291.8 283.9 382.4 4.622
> alterPartitions.1000 3532 3558 2721 5680 18.14
> ```
>
> * after
>
> ```shell
> Operation Mean Med Min Max Err%
> addPartition 25.29 22.77 20.06 42.67 21.88
> alterPartitions.10 41.49 40.56 37.99 55.41 7.436
> alterPartitions.100 95.19 84.00 80.56 169.2 26.87
> alterPartitions.1000 1343 1332 1190 1546 5.928
> ```
numbers look great, I've removed the stale label. @wecharyu could you please rebase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1410553559
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdate.java:
##########
@@ -0,0 +1,1541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the statistics update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdate {
+ private static final Logger LOG = LoggerFactory.getLogger(DirectSqlUpdate.class.getName());
+ PersistenceManager pm;
+ Configuration conf;
+ DatabaseProduct dbType;
+ int maxBatchSize;
+ SQLGenerator sqlGenerator;
+ private static final ReentrantLock derbyLock = new ReentrantLock(true);
+
+ public DirectSqlUpdate(PersistenceManager pm, Configuration conf,
+ DatabaseProduct dbType, int batchSize) {
+ this.pm = pm;
+ this.conf = conf;
+ this.dbType = dbType;
+ this.maxBatchSize = batchSize;
+ sqlGenerator = new SQLGenerator(dbType, conf);
+ }
+
+ /**
+ * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
+ * Select ... For Update to sequence operations properly. In practice that means when running
+ * with Derby database. See more notes at class level.
+ */
+ private void lockInternal() {
Review Comment:
all this lock, unlock, rollback, close should be moved to some DirectSql interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1410570623
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##########
@@ -901,23 +968,48 @@ private List<Long> getPartitionIdsViaSqlFilter(
String catName, String dbName, String tblName, String sqlFilter,
List<? extends Object> paramsForFilter, List<String> joinsForFilter, Integer max)
throws MetaException {
+ return getPartitionFieldsViaSqlFilter(catName, dbName, tblName,
+ Arrays.asList("\"PART_ID\""), sqlFilter, paramsForFilter, joinsForFilter, max);
+ }
+
+ /**
+ * Get partition fields for the query using direct SQL queries, to avoid bazillion
+ * queries created by DN retrieving stuff for each object individually.
+ * @param catName MetaStore catalog name
+ * @param dbName MetaStore db name
+ * @param tblName MetaStore table name
+ * @param partColumns part fields want to get
+ * @param sqlFilter SQL filter to use. Better be SQL92-compliant.
+ * @param paramsForFilter params for ?-s in SQL filter text. Params must be in order.
+ * @param joinsForFilter if the filter needs additional join statement, they must be in
+ * this list. Better be SQL92-compliant.
+ * @param max The maximum number of partitions to return.
+ * @return List of partition objects.
+ */
+ public <T> List<T> getPartitionFieldsViaSqlFilter(
+ String catName, String dbName, String tblName, List<String> partColumns, String sqlFilter,
+ List<? extends Object> paramsForFilter, List<String> joinsForFilter, Integer max)
+ throws MetaException {
boolean doTrace = LOG.isDebugEnabled();
final String dbNameLcase = dbName.toLowerCase();
final String tblNameLcase = tblName.toLowerCase();
final String catNameLcase = normalizeSpace(catName).toLowerCase();
// We have to be mindful of order during filtering if we are not returning all partitions.
String orderForFilter = (max != null) ? " order by " + MetastoreConf.getVar(conf, ConfVars.PARTITION_ORDER_EXPR) : "";
+ String columns = partColumns.stream()
Review Comment:
keep it in 1 line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1868208958
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] wecharyu commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1657475823
@saihemanth-cloudera @dengzhhu653 @deniskuzZ: Could you help review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] wecharyu commented on a diff in pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1317567686
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##########
@@ -532,6 +535,66 @@ public void addPartitions(List<MPartition> parts, List<List<MPartitionPrivilege>
directSqlInsertPart.addPartitions(parts, partPrivilegesList, partColPrivilegesList);
}
+ /**
+ * Alter partitions in batch using direct SQL
+ * @param table the target table
+ * @param partNames list of partition names
+ * @param newParts list of new partitions
+ * @param queryWriteIdList valid write id list
+ * @return
+ * @throws MetaException
+ */
+ public List<Partition> alterPartitions(Table table, List<String> partNames,
+ List<Partition> newParts, String queryWriteIdList) throws MetaException {
+ String filter = "" + PARTITIONS + ".\"PART_NAME\" in (" + makeParams(partNames.size()) + ")";
Review Comment:
There should be no limit on 'IN' clause, and there are also many calls of `makeParams()` in `MetaStoreDirectSql`, for ex: https://github.com/apache/hive/blob/920f9e535db6270a401db274eef3267d70c1fd2f/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java#L671-L674
Nevertheless, I think it is reasonable to query in batch just like below.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] wecharyu commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1692930094
I have test alter partitions on MySQL and Postgres, and does not encounter db related issue. IMHO the update statement is compatible for all these dbs. CC: @nrg4878 @saihemanth-cloudera @VenuReddy2103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1866211950
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1833577139
restarted the build
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1833888251
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [97 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1410569029
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##########
@@ -203,7 +206,8 @@ public MetaStoreDirectSql(PersistenceManager pm, Configuration conf, String sche
batchSize = dbType.needsInBatching() ? 1000 : NO_BATCHING;
}
this.batchSize = batchSize;
- this.updateStat = new DirectSqlUpdateStat(pm, conf, dbType, batchSize);
+ this.areTxnStatsEnabled = MetastoreConf.getBoolVar(conf, ConfVars.HIVE_TXN_STATS_ENABLED);
Review Comment:
isTxnStatsEnabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1413028152
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java:
##########
@@ -675,6 +676,24 @@ public List<String> createInsertValuesStmt(String tblColumns, List<String> rows,
}
}
+ public String createUpdatePreparedStmt(String tableName, List<String> columnNames,
Review Comment:
Make sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1837457566
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [96 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] wecharyu commented on a diff in pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1317573199
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##########
@@ -532,6 +535,66 @@ public void addPartitions(List<MPartition> parts, List<List<MPartitionPrivilege>
directSqlInsertPart.addPartitions(parts, partPrivilegesList, partColPrivilegesList);
}
+ /**
+ * Alter partitions in batch using direct SQL
+ * @param table the target table
+ * @param partNames list of partition names
+ * @param newParts list of new partitions
+ * @param queryWriteIdList valid write id list
+ * @return
+ * @throws MetaException
+ */
+ public List<Partition> alterPartitions(Table table, List<String> partNames,
+ List<Partition> newParts, String queryWriteIdList) throws MetaException {
+ String filter = "" + PARTITIONS + ".\"PART_NAME\" in (" + makeParams(partNames.size()) + ")";
+ List<String> columns = Arrays.asList("\"PART_ID\"", "\"PART_NAME\"", "\"SD_ID\"", "\"WRITE_ID\"");
+ List<Object[]> rows = getPartitionFieldsViaSqlFilter(table.getCatName(), table.getDbName(),
+ table.getTableName(), columns, filter, partNames, Collections.emptyList(), null);
+ Map<List<String>, Long> partValuesToId = new HashMap<>();
+ Map<Long, Long> partIdToSdId = new HashMap<>();
+ Map<Long, Long> partIdToWriteId = new HashMap<>();
+ for (Object[] row : rows) {
+ Long partId = MetastoreDirectSqlUtils.extractSqlLong(row[0]);
+ Long sdId = MetastoreDirectSqlUtils.extractSqlLong(row[2]);
+ Long writeId = MetastoreDirectSqlUtils.extractSqlLong(row[3]);
+ partIdToSdId.put(partId, sdId);
+ partIdToWriteId.put(partId, writeId);
+ List<String> partValues = Warehouse.getPartValuesFromPartName((String) row[1]);
+ partValuesToId.put(partValues, partId);
+ }
+
+ boolean isTxn = TxnUtils.isTransactionalTable(table.getParameters());
+ for (Partition newPart : newParts) {
+ Long partId = partValuesToId.get(newPart.getValues());
+ boolean useOldWriteId = true;
+ // If transactional, add/update the MUPdaterTransaction
+ // for the current updater query.
+ if (isTxn) {
+ if (!areTxnStatsSupported) {
+ StatsSetupConst.setBasicStatsState(newPart.getParameters(), StatsSetupConst.FALSE);
+ } else if (queryWriteIdList != null && newPart.getWriteId() > 0) {
+ // Check concurrent INSERT case and set false to the flag.
+ if (!ObjectStore.isCurrentStatsValidForTheQuery(newPart.getParameters(),
+ partIdToWriteId.get(partId), queryWriteIdList, true)) {
+ StatsSetupConst.setBasicStatsState(newPart.getParameters(), StatsSetupConst.FALSE);
+ LOG.info("Removed COLUMN_STATS_ACCURATE from the parameters of the partition " +
+ Warehouse.getQualifiedName(newPart) + " will be made persistent.");
+ }
+ useOldWriteId = false;
+ }
+ }
+
+ if (useOldWriteId) {
+ newPart.setWriteId(partIdToWriteId.get(partId));
+ }
+ }
+
+ directSqlUpdate.alterPartitions(partValuesToId, partIdToSdId, newParts);
+ // The returned result were not utilized, so return an empty list in direct sql for performance.
+ // TODO: Change the API signature of RawStore#alterPartitions to return void instead.
+ return Collections.emptyList();
Review Comment:
Make sense, we can return `newParts` object here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1652822982
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [100 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=duplicated_lines_density&view=list) No Duplication information
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1692331822
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [97 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] wecharyu commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1694173723
@deniskuzZ Yes I have checked the sonar reported 2 bugs.
> Use try-with-resources or close this "Statement" in a "finally" clause.
Actually the related code is inherited from `DirectSqlUpdateStat.java`, and the statement will be closed in the top finally clause.
> A "NullPointerException" could be thrown; "table" is nullable here.
Since we called `ensureGetTable()`, it will throw `NoSuchObjectException` if not get the table, so the table will never be null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] nrg4878 commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "nrg4878 (via GitHub)" <gi...@apache.org>.
nrg4878 commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1686845737
@wecharyu Have you tested this against all DBs supported by HMS. When we added directSQL support for addPartition(), we did run into DB-specific issues during our testing. Do we would like to confirm that this testing has been done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] wecharyu commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1708843511
@saihemanth-cloudera Here is another benchmark test for before and after this path:
- before
```bash
Operation Mean Med Min Max Err%
addPartition 25.61 22.90 20.19 59.38 26.34
alterPartitions.10 59.55 58.81 55.35 69.30 5.140
alterPartitions.100 295.5 291.8 283.9 382.4 4.622
alterPartitions.1000 3532 3558 2721 5680 18.14
```
- after
```bash
Operation Mean Med Min Max Err%
addPartition 25.29 22.77 20.06 42.67 21.88
alterPartitions.10 41.49 40.56 37.99 55.41 7.436
alterPartitions.100 95.19 84.00 80.56 169.2 26.87
alterPartitions.1000 1343 1332 1190 1546 5.928
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1428808671
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdatePart.java:
##########
@@ -0,0 +1,1536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdatePart {
+ private static final Logger LOG = LoggerFactory.getLogger(DirectSqlUpdatePart.class.getName());
+
+ private final PersistenceManager pm;
+ private final Configuration conf;
+ private final DatabaseProduct dbType;
+ private final int maxBatchSize;
+ private final SQLGenerator sqlGenerator;
+
+ private static final ReentrantLock derbyLock = new ReentrantLock(true);
+
+ public DirectSqlUpdatePart(PersistenceManager pm, Configuration conf,
+ DatabaseProduct dbType, int batchSize) {
+ this.pm = pm;
+ this.conf = conf;
+ this.dbType = dbType;
+ this.maxBatchSize = batchSize;
+ sqlGenerator = new SQLGenerator(dbType, conf);
+ }
+
+ /**
+ * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
+ * Select ... For Update to sequence operations properly. In practice that means when running
+ * with Derby database. See more notes at class level.
+ */
+ private void lockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.lock();
+ }
+ }
+
+ private void unlockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.unlock();
+ }
+ }
+
+ void rollbackDBConn(Connection dbConn) {
+ try {
+ if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
+ } catch (SQLException e) {
+ LOG.warn("Failed to rollback db connection ", e);
+ }
+ }
+
+ void closeDbConn(JDOConnection jdoConn) {
+ try {
+ if (jdoConn != null) {
+ jdoConn.close();
+ }
+ } catch (Exception e) {
+ LOG.warn("Failed to close db connection", e);
+ }
+ }
+
+ void closeStmt(Statement stmt) {
+ try {
+ if (stmt != null && !stmt.isClosed()) stmt.close();
+ } catch (SQLException e) {
+ LOG.warn("Failed to close statement ", e);
+ }
+ }
+
+ void close(ResultSet rs) {
+ try {
+ if (rs != null && !rs.isClosed()) {
+ rs.close();
+ }
+ }
+ catch(SQLException ex) {
+ LOG.warn("Failed to close statement ", ex);
+ }
+ }
+
+ static String quoteString(String input) {
+ return "'" + input + "'";
+ }
+
+ void close(ResultSet rs, Statement stmt, JDOConnection dbConn) {
+ close(rs);
+ closeStmt(stmt);
+ closeDbConn(dbConn);
+ }
+
+ private void populateInsertUpdateMap(Map<PartitionInfo, ColumnStatistics> statsPartInfoMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics>insertMap,
+ Connection dbConn, Table tbl) throws SQLException, MetaException, NoSuchObjectException {
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+ List<String> queries = new ArrayList<>();
+ Set<PartColNameInfo> selectedParts = new HashSet<>();
+
+ List<Long> partIdList = statsPartInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ prefix.append("select \"PART_ID\", \"COLUMN_NAME\" from \"PART_COL_STATS\" WHERE ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ selectedParts.add(new PartColNameInfo(rs.getLong(1), rs.getString(2)));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+
+ for (Map.Entry entry : statsPartInfoMap.entrySet()) {
+ PartitionInfo partitionInfo = (PartitionInfo) entry.getKey();
+ ColumnStatistics colStats = (ColumnStatistics) entry.getValue();
+ long partId = partitionInfo.partitionId;
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ if (!statsDesc.isSetCatName()) {
+ statsDesc.setCatName(tbl.getCatName());
+ }
+ for (ColumnStatisticsObj statisticsObj : colStats.getStatsObj()) {
+ PartColNameInfo temp = new PartColNameInfo(partId, statisticsObj.getColName());
+ if (selectedParts.contains(temp)) {
+ updateMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ } else {
+ insertMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ }
+ }
+ }
+ }
+
+ private void updatePartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement pst = null;
+ for (Map.Entry entry : updateMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+ String update = "UPDATE \"PART_COL_STATS\" SET ";
+ update += StatObjectConverter.getUpdatedColumnSql(mPartitionColumnStatistics);
+ update += " WHERE \"PART_ID\" = " + partId + " AND "
+ + " \"COLUMN_NAME\" = " + quoteString(mPartitionColumnStatistics.getColName());
+ try {
+ pst = dbConn.prepareStatement(update);
+ StatObjectConverter.initUpdatedColumnStatement(mPartitionColumnStatistics, pst);
+ LOG.debug("Going to execute update " + update);
+ int numUpdate = pst.executeUpdate();
+ if (numUpdate != 1) {
+ throw new MetaException("Invalid state of PART_COL_STATS for PART_ID " + partId);
+ }
+ } finally {
+ closeStmt(pst);
+ }
+ }
+ }
+
+ private void insertIntoPartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> insertMap,
+ long maxCsId,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement preparedStatement = null;
+ int numRows = 0;
+ String insert = "INSERT INTO \"PART_COL_STATS\" (\"CS_ID\", \"CAT_NAME\", \"DB_NAME\","
+ + "\"TABLE_NAME\", \"PARTITION_NAME\", \"COLUMN_NAME\", \"COLUMN_TYPE\", \"PART_ID\","
+ + " \"LONG_LOW_VALUE\", \"LONG_HIGH_VALUE\", \"DOUBLE_HIGH_VALUE\", \"DOUBLE_LOW_VALUE\","
+ + " \"BIG_DECIMAL_LOW_VALUE\", \"BIG_DECIMAL_HIGH_VALUE\", \"NUM_NULLS\", \"NUM_DISTINCTS\", \"BIT_VECTOR\" ,"
+ + " \"HISTOGRAM\", \"AVG_COL_LEN\", \"MAX_COL_LEN\", \"NUM_TRUES\", \"NUM_FALSES\", \"LAST_ANALYZED\", \"ENGINE\") values "
+ + "(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
+
+ try {
+ preparedStatement = dbConn.prepareStatement(insert);
+ for (Map.Entry entry : insertMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+
+ preparedStatement.setLong(1, maxCsId);
+ preparedStatement.setString(2, mPartitionColumnStatistics.getCatName());
+ preparedStatement.setString(3, mPartitionColumnStatistics.getDbName());
+ preparedStatement.setString(4, mPartitionColumnStatistics.getTableName());
+ preparedStatement.setString(5, mPartitionColumnStatistics.getPartitionName());
+ preparedStatement.setString(6, mPartitionColumnStatistics.getColName());
+ preparedStatement.setString(7, mPartitionColumnStatistics.getColType());
+ preparedStatement.setLong(8, partId);
+ preparedStatement.setObject(9, mPartitionColumnStatistics.getLongLowValue());
+ preparedStatement.setObject(10, mPartitionColumnStatistics.getLongHighValue());
+ preparedStatement.setObject(11, mPartitionColumnStatistics.getDoubleHighValue());
+ preparedStatement.setObject(12, mPartitionColumnStatistics.getDoubleLowValue());
+ preparedStatement.setString(13, mPartitionColumnStatistics.getDecimalLowValue());
+ preparedStatement.setString(14, mPartitionColumnStatistics.getDecimalHighValue());
+ preparedStatement.setObject(15, mPartitionColumnStatistics.getNumNulls());
+ preparedStatement.setObject(16, mPartitionColumnStatistics.getNumDVs());
+ preparedStatement.setObject(17, mPartitionColumnStatistics.getBitVector());
+ preparedStatement.setBytes(18, mPartitionColumnStatistics.getHistogram());
+ preparedStatement.setObject(19, mPartitionColumnStatistics.getAvgColLen());
+ preparedStatement.setObject(20, mPartitionColumnStatistics.getMaxColLen());
+ preparedStatement.setObject(21, mPartitionColumnStatistics.getNumTrues());
+ preparedStatement.setObject(22, mPartitionColumnStatistics.getNumFalses());
+ preparedStatement.setLong(23, mPartitionColumnStatistics.getLastAnalyzed());
+ preparedStatement.setString(24, mPartitionColumnStatistics.getEngine());
+
+ maxCsId++;
+ numRows++;
+ preparedStatement.addBatch();
+ if (numRows == maxBatchSize) {
+ preparedStatement.executeBatch();
+ numRows = 0;
+ }
+ }
+
+ if (numRows != 0) {
+ preparedStatement.executeBatch();
+ }
+ } finally {
+ closeStmt(preparedStatement);
+ }
+ }
+
+ private Map<Long, String> getParamValues(Connection dbConn, List<Long> partIdList) throws SQLException {
+ List<String> queries = new ArrayList<>();
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+
+ prefix.append("select \"PART_ID\", \"PARAM_VALUE\" "
+ + " from \"PARTITION_PARAMS\" where "
+ + " \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE' "
+ + " and ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ Map<Long, String> partIdToParaMap = new HashMap<>();
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ partIdToParaMap.put(rs.getLong(1), rs.getString(2));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+ return partIdToParaMap;
+ }
+
+ private void updateWriteIdForPartitions(Connection dbConn, long writeId, List<Long> partIdList) throws SQLException {
+ StringBuilder prefix = new StringBuilder();
+ List<String> queries = new ArrayList<>();
+ StringBuilder suffix = new StringBuilder();
+
+ prefix.append("UPDATE \"PARTITIONS\" set \"WRITE_ID\" = " + writeId + " where ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", false, false);
+
+ Statement statement = null;
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute update " + query);
+ statement.executeUpdate(query);
+ } finally {
+ closeStmt(statement);
+ }
+ }
+ }
+
+ private Map<String, Map<String, String>> updatePartitionParamTable(Connection dbConn,
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap,
+ String validWriteIds,
+ long writeId,
+ boolean isAcidTable)
+ throws SQLException, MetaException {
+ Map<String, Map<String, String>> result = new HashMap<>();
+ boolean areTxnStatsSupported = MetastoreConf.getBoolVar(conf, ConfVars.HIVE_TXN_STATS_ENABLED);
+ String insert = "INSERT INTO \"PARTITION_PARAMS\" (\"PART_ID\", \"PARAM_KEY\", \"PARAM_VALUE\") "
+ + "VALUES( ? , 'COLUMN_STATS_ACCURATE' , ? )";
+ String delete = "DELETE from \"PARTITION_PARAMS\" "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ String update = "UPDATE \"PARTITION_PARAMS\" set \"PARAM_VALUE\" = ? "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ int numInsert = 0;
+ int numDelete = 0;
+ int numUpdate = 0;
+
+ List<Long> partIdList = partitionInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ // get the old parameters from PARTITION_PARAMS table.
+ Map<Long, String> partIdToParaMap = getParamValues(dbConn, partIdList);
+
+ try (PreparedStatement statementInsert = dbConn.prepareStatement(insert);
+ PreparedStatement statementDelete = dbConn.prepareStatement(delete);
+ PreparedStatement statementUpdate = dbConn.prepareStatement(update)) {
+ for (Map.Entry entry : partitionInfoMap.entrySet()) {
+ PartitionInfo partitionInfo = (PartitionInfo) entry.getKey();
+ ColumnStatistics colStats = (ColumnStatistics) entry.getValue();
+ List<String> colNames = colStats.getStatsObj().stream().map(e -> e.getColName()).collect(Collectors.toList());
+ long partWriteId = partitionInfo.writeId;
+ long partId = partitionInfo.partitionId;
+ Map<String, String> newParameter;
+
+ if (!partIdToParaMap.containsKey(partId)) {
+ newParameter = new HashMap<>();
+ newParameter.put(COLUMN_STATS_ACCURATE, "TRUE");
+ StatsSetupConst.setColumnStatsState(newParameter, colNames);
+ statementInsert.setLong(1, partId);
+ statementInsert.setString(2, newParameter.get(COLUMN_STATS_ACCURATE));
+ numInsert++;
+ statementInsert.addBatch();
+ if (numInsert == maxBatchSize) {
+ LOG.debug(" Executing insert " + insert);
+ statementInsert.executeBatch();
+ numInsert = 0;
+ }
+ } else {
+ String oldStats = partIdToParaMap.get(partId);
+
+ Map<String, String> oldParameter = new HashMap<>();
+ oldParameter.put(COLUMN_STATS_ACCURATE, oldStats);
+
+ newParameter = new HashMap<>();
+ newParameter.put(COLUMN_STATS_ACCURATE, oldStats);
+ StatsSetupConst.setColumnStatsState(newParameter, colNames);
+
+ if (isAcidTable) {
+ String errorMsg = ObjectStore.verifyStatsChangeCtx(
+ colStats.getStatsDesc().getDbName() + "." + colStats.getStatsDesc().getTableName(),
+ oldParameter, newParameter, writeId, validWriteIds, true);
+ if (errorMsg != null) {
+ throw new MetaException(errorMsg);
+ }
+ }
+
+ if (isAcidTable &&
+ (!areTxnStatsSupported || !ObjectStore.isCurrentStatsValidForTheQuery(oldParameter, partWriteId,
+ validWriteIds, true))) {
+ statementDelete.setLong(1, partId);
+ statementDelete.addBatch();
+ numDelete++;
+ if (numDelete == maxBatchSize) {
+ statementDelete.executeBatch();
+ numDelete = 0;
+ LOG.debug("Removed COLUMN_STATS_ACCURATE from the parameters of the partition "
+ + colStats.getStatsDesc().getDbName() + "." + colStats.getStatsDesc().getTableName() + "."
+ + colStats.getStatsDesc().getPartName());
+ }
+ } else {
+ statementUpdate.setString(1, newParameter.get(COLUMN_STATS_ACCURATE));
+ statementUpdate.setLong(2, partId);
+ statementUpdate.addBatch();
+ numUpdate++;
+ if (numUpdate == maxBatchSize) {
+ LOG.debug(" Executing update " + statementUpdate);
+ statementUpdate.executeBatch();
+ numUpdate = 0;
+ }
+ }
+ }
+ result.put(partitionInfo.partitionName, newParameter);
+ }
+
+ if (numInsert != 0) {
+ statementInsert.executeBatch();
+ }
+
+ if (numUpdate != 0) {
+ statementUpdate.executeBatch();
+ }
+
+ if (numDelete != 0) {
+ statementDelete.executeBatch();
+ }
+
+ if (isAcidTable) {
+ updateWriteIdForPartitions(dbConn, writeId, partIdList);
+ }
+ return result;
+ }
+ }
+
+
+ private Map<PartitionInfo, ColumnStatistics> getPartitionInfo(Connection dbConn, long tblId,
+ Map<String, ColumnStatistics> partColStatsMap)
+ throws SQLException, MetaException {
+ List<String> queries = new ArrayList<>();
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap = new HashMap<>();
+
+ List<String> partKeys = partColStatsMap.keySet().stream().map(
+ e -> quoteString(e)).collect(Collectors.toList()
+ );
+
+ prefix.append("select \"PART_ID\", \"WRITE_ID\", \"PART_NAME\" from \"PARTITIONS\" where ");
+ suffix.append(" and \"TBL_ID\" = " + tblId);
+ TxnUtils.buildQueryWithINClauseStrings(conf, queries, prefix, suffix,
+ partKeys, "\"PART_NAME\"", true, false);
+
+ for (String query : queries) {
+ // Select for update makes sure that the partitions are not modified while the stats are getting updated.
+ query = sqlGenerator.addForUpdateClause(query);
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query <" + query + ">");
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ PartitionInfo partitionInfo = new PartitionInfo(rs.getLong(1),
+ rs.getLong(2), rs.getString(3));
+ partitionInfoMap.put(partitionInfo, partColStatsMap.get(rs.getString(3)));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+ return partitionInfoMap;
+ }
+
+ private void setAnsiQuotes(Connection dbConn) throws SQLException {
+ if (sqlGenerator.getDbProduct().isMYSQL()) {
+ try (Statement stmt = dbConn.createStatement()) {
+ stmt.execute("SET @@session.sql_mode=ANSI_QUOTES");
+ }
+ }
+ }
+
+ /**
+ * Update the statistics for the given partitions. Add the notification logs also.
+ * @return map of partition key to column stats if successful, null otherwise.
+ */
+ public Map<String, Map<String, String>> updatePartitionColumnStatistics(Map<String, ColumnStatistics> partColStatsMap,
+ Table tbl, long csId,
+ String validWriteIds, long writeId,
+ List<TransactionalMetaStoreEventListener> transactionalListeners)
+ throws MetaException {
+ JDOConnection jdoConn = null;
+ Connection dbConn = null;
+ boolean committed = false;
+ try {
+ lockInternal();
+ jdoConn = pm.getDataStoreConnection();
+ dbConn = (Connection) (jdoConn.getNativeConnection());
+
+ setAnsiQuotes(dbConn);
+
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap = getPartitionInfo(dbConn, tbl.getId(), partColStatsMap);
+
+ Map<String, Map<String, String>> result =
+ updatePartitionParamTable(dbConn, partitionInfoMap, validWriteIds, writeId, TxnUtils.isAcidTable(tbl));
+
+ Map<PartColNameInfo, MPartitionColumnStatistics> insertMap = new HashMap<>();
+ Map<PartColNameInfo, MPartitionColumnStatistics> updateMap = new HashMap<>();
+ populateInsertUpdateMap(partitionInfoMap, updateMap, insertMap, dbConn, tbl);
+
+ LOG.info("Number of stats to insert " + insertMap.size() + " update " + updateMap.size());
+
+ if (insertMap.size() != 0) {
+ insertIntoPartColStatTable(insertMap, csId, dbConn);
+ }
+
+ if (updateMap.size() != 0) {
+ updatePartColStatTable(updateMap, dbConn);
+ }
+
+ if (transactionalListeners != null) {
+ UpdatePartitionColumnStatEventBatch eventBatch = new UpdatePartitionColumnStatEventBatch(null);
+ for (Map.Entry entry : result.entrySet()) {
+ Map<String, String> parameters = (Map<String, String>) entry.getValue();
+ ColumnStatistics colStats = partColStatsMap.get(entry.getKey());
+ List<String> partVals = getPartValsFromName(tbl, colStats.getStatsDesc().getPartName());
+ UpdatePartitionColumnStatEvent event = new UpdatePartitionColumnStatEvent(colStats, partVals, parameters,
+ tbl, writeId, null);
+ eventBatch.addPartColStatEvent(event);
+ }
+ MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+ EventMessage.EventType.UPDATE_PARTITION_COLUMN_STAT_BATCH, eventBatch, dbConn, sqlGenerator);
+ }
+ dbConn.commit();
+ committed = true;
+ return result;
+ } catch (Exception e) {
+ LOG.error("Unable to update Column stats for " + tbl.getTableName(), e);
+ throw new MetaException("Unable to update Column stats for " + tbl.getTableName()
+ + " due to: " + e.getMessage());
+ } finally {
+ if (!committed) {
+ rollbackDBConn(dbConn);
+ }
+ closeDbConn(jdoConn);
+ unlockInternal();
+ }
+ }
+
+ /**
+ * Gets the next CS id from sequence MPartitionColumnStatistics and increment the CS id by numStats.
+ * @return The CD id before update.
+ */
+ public long getNextCSIdForMPartitionColumnStatistics(long numStats) throws MetaException {
+ Statement statement = null;
+ ResultSet rs = null;
+ long maxCsId = 0;
+ boolean committed = false;
+ Connection dbConn = null;
+ JDOConnection jdoConn = null;
+
+ try {
+ lockInternal();
+ jdoConn = pm.getDataStoreConnection();
+ dbConn = (Connection) (jdoConn.getNativeConnection());
+
+ setAnsiQuotes(dbConn);
+
+ // This loop will be iterated at max twice. If there is no records, it will first insert and then do a select.
+ // We are not using any upsert operations as select for update and then update is required to make sure that
+ // the caller gets a reserved range for CSId not used by any other thread.
+ boolean insertDone = false;
+ while (maxCsId == 0) {
+ String query = sqlGenerator.addForUpdateClause("SELECT \"NEXT_VAL\" FROM \"SEQUENCE_TABLE\" "
+ + "WHERE \"SEQUENCE_NAME\"= "
+ + quoteString("org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics"));
+ LOG.debug("Going to execute query " + query);
+ statement = dbConn.createStatement();
Review Comment:
no change here, copy of legacy code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1858840850
@deniskuzZ Thanks for the patient review, here is the refactor jira: [HIVE-27958](https://issues.apache.org/jira/browse/HIVE-27958).
Please help take a look, and I'm willing to take it over.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1410566433
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdate.java:
##########
@@ -0,0 +1,1541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the statistics update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdate {
+ private static final Logger LOG = LoggerFactory.getLogger(DirectSqlUpdate.class.getName());
+ PersistenceManager pm;
+ Configuration conf;
+ DatabaseProduct dbType;
+ int maxBatchSize;
+ SQLGenerator sqlGenerator;
+ private static final ReentrantLock derbyLock = new ReentrantLock(true);
+
+ public DirectSqlUpdate(PersistenceManager pm, Configuration conf,
+ DatabaseProduct dbType, int batchSize) {
+ this.pm = pm;
+ this.conf = conf;
+ this.dbType = dbType;
+ this.maxBatchSize = batchSize;
+ sqlGenerator = new SQLGenerator(dbType, conf);
+ }
+
+ /**
+ * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
+ * Select ... For Update to sequence operations properly. In practice that means when running
+ * with Derby database. See more notes at class level.
+ */
+ private void lockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.lock();
+ }
+ }
+
+ private void unlockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.unlock();
+ }
+ }
+
+ void rollbackDBConn(Connection dbConn) {
+ try {
+ if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
+ } catch (SQLException e) {
+ LOG.warn("Failed to rollback db connection ", e);
+ }
+ }
+
+ void closeDbConn(JDOConnection jdoConn) {
+ try {
+ if (jdoConn != null) {
+ jdoConn.close();
+ }
+ } catch (Exception e) {
+ LOG.warn("Failed to close db connection", e);
+ }
+ }
+
+ void closeStmt(Statement stmt) {
+ try {
+ if (stmt != null && !stmt.isClosed()) stmt.close();
+ } catch (SQLException e) {
+ LOG.warn("Failed to close statement ", e);
+ }
+ }
+
+ void close(ResultSet rs) {
+ try {
+ if (rs != null && !rs.isClosed()) {
+ rs.close();
+ }
+ }
+ catch(SQLException ex) {
+ LOG.warn("Failed to close statement ", ex);
+ }
+ }
+
+ static String quoteString(String input) {
+ return "'" + input + "'";
+ }
+
+ void close(ResultSet rs, Statement stmt, JDOConnection dbConn) {
+ close(rs);
+ closeStmt(stmt);
+ closeDbConn(dbConn);
+ }
+
+ private void populateInsertUpdateMap(Map<PartitionInfo, ColumnStatistics> statsPartInfoMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics>insertMap,
+ Connection dbConn, Table tbl) throws SQLException, MetaException, NoSuchObjectException {
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+ List<String> queries = new ArrayList<>();
+ Set<PartColNameInfo> selectedParts = new HashSet<>();
+
+ List<Long> partIdList = statsPartInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ prefix.append("select \"PART_ID\", \"COLUMN_NAME\" from \"PART_COL_STATS\" WHERE ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ selectedParts.add(new PartColNameInfo(rs.getLong(1), rs.getString(2)));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+
+ for (Map.Entry entry : statsPartInfoMap.entrySet()) {
+ PartitionInfo partitionInfo = (PartitionInfo) entry.getKey();
+ ColumnStatistics colStats = (ColumnStatistics) entry.getValue();
+ long partId = partitionInfo.partitionId;
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ if (!statsDesc.isSetCatName()) {
+ statsDesc.setCatName(tbl.getCatName());
+ }
+ for (ColumnStatisticsObj statisticsObj : colStats.getStatsObj()) {
+ PartColNameInfo temp = new PartColNameInfo(partId, statisticsObj.getColName());
+ if (selectedParts.contains(temp)) {
+ updateMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ } else {
+ insertMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ }
+ }
+ }
+ }
+
+ private void updatePartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement pst = null;
+ for (Map.Entry entry : updateMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+ String update = "UPDATE \"PART_COL_STATS\" SET ";
+ update += StatObjectConverter.getUpdatedColumnSql(mPartitionColumnStatistics);
+ update += " WHERE \"PART_ID\" = " + partId + " AND "
+ + " \"COLUMN_NAME\" = " + quoteString(mPartitionColumnStatistics.getColName());
+ try {
+ pst = dbConn.prepareStatement(update);
+ StatObjectConverter.initUpdatedColumnStatement(mPartitionColumnStatistics, pst);
+ LOG.debug("Going to execute update " + update);
+ int numUpdate = pst.executeUpdate();
+ if (numUpdate != 1) {
+ throw new MetaException("Invalid state of PART_COL_STATS for PART_ID " + partId);
+ }
+ } finally {
+ closeStmt(pst);
+ }
+ }
+ }
+
+ private void insertIntoPartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> insertMap,
+ long maxCsId,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement preparedStatement = null;
+ int numRows = 0;
+ String insert = "INSERT INTO \"PART_COL_STATS\" (\"CS_ID\", \"CAT_NAME\", \"DB_NAME\","
+ + "\"TABLE_NAME\", \"PARTITION_NAME\", \"COLUMN_NAME\", \"COLUMN_TYPE\", \"PART_ID\","
+ + " \"LONG_LOW_VALUE\", \"LONG_HIGH_VALUE\", \"DOUBLE_HIGH_VALUE\", \"DOUBLE_LOW_VALUE\","
+ + " \"BIG_DECIMAL_LOW_VALUE\", \"BIG_DECIMAL_HIGH_VALUE\", \"NUM_NULLS\", \"NUM_DISTINCTS\", \"BIT_VECTOR\" ,"
+ + " \"HISTOGRAM\", \"AVG_COL_LEN\", \"MAX_COL_LEN\", \"NUM_TRUES\", \"NUM_FALSES\", \"LAST_ANALYZED\", \"ENGINE\") values "
+ + "(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
+
+ try {
+ preparedStatement = dbConn.prepareStatement(insert);
+ for (Map.Entry entry : insertMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+
+ preparedStatement.setLong(1, maxCsId);
+ preparedStatement.setString(2, mPartitionColumnStatistics.getCatName());
+ preparedStatement.setString(3, mPartitionColumnStatistics.getDbName());
+ preparedStatement.setString(4, mPartitionColumnStatistics.getTableName());
+ preparedStatement.setString(5, mPartitionColumnStatistics.getPartitionName());
+ preparedStatement.setString(6, mPartitionColumnStatistics.getColName());
+ preparedStatement.setString(7, mPartitionColumnStatistics.getColType());
+ preparedStatement.setLong(8, partId);
+ preparedStatement.setObject(9, mPartitionColumnStatistics.getLongLowValue());
+ preparedStatement.setObject(10, mPartitionColumnStatistics.getLongHighValue());
+ preparedStatement.setObject(11, mPartitionColumnStatistics.getDoubleHighValue());
+ preparedStatement.setObject(12, mPartitionColumnStatistics.getDoubleLowValue());
+ preparedStatement.setString(13, mPartitionColumnStatistics.getDecimalLowValue());
+ preparedStatement.setString(14, mPartitionColumnStatistics.getDecimalHighValue());
+ preparedStatement.setObject(15, mPartitionColumnStatistics.getNumNulls());
+ preparedStatement.setObject(16, mPartitionColumnStatistics.getNumDVs());
+ preparedStatement.setObject(17, mPartitionColumnStatistics.getBitVector());
+ preparedStatement.setBytes(18, mPartitionColumnStatistics.getHistogram());
+ preparedStatement.setObject(19, mPartitionColumnStatistics.getAvgColLen());
+ preparedStatement.setObject(20, mPartitionColumnStatistics.getMaxColLen());
+ preparedStatement.setObject(21, mPartitionColumnStatistics.getNumTrues());
+ preparedStatement.setObject(22, mPartitionColumnStatistics.getNumFalses());
+ preparedStatement.setLong(23, mPartitionColumnStatistics.getLastAnalyzed());
+ preparedStatement.setString(24, mPartitionColumnStatistics.getEngine());
+
+ maxCsId++;
+ numRows++;
+ preparedStatement.addBatch();
+ if (numRows == maxBatchSize) {
+ preparedStatement.executeBatch();
+ numRows = 0;
+ }
+ }
+
+ if (numRows != 0) {
+ preparedStatement.executeBatch();
+ }
+ } finally {
+ closeStmt(preparedStatement);
+ }
+ }
+
+ private Map<Long, String> getParamValues(Connection dbConn, List<Long> partIdList) throws SQLException {
+ List<String> queries = new ArrayList<>();
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+
+ prefix.append("select \"PART_ID\", \"PARAM_VALUE\" "
+ + " from \"PARTITION_PARAMS\" where "
+ + " \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE' "
+ + " and ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ Map<Long, String> partIdToParaMap = new HashMap<>();
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ partIdToParaMap.put(rs.getLong(1), rs.getString(2));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+ return partIdToParaMap;
+ }
+
+ private void updateWriteIdForPartitions(Connection dbConn, long writeId, List<Long> partIdList) throws SQLException {
+ StringBuilder prefix = new StringBuilder();
+ List<String> queries = new ArrayList<>();
+ StringBuilder suffix = new StringBuilder();
+
+ prefix.append("UPDATE \"PARTITIONS\" set \"WRITE_ID\" = " + writeId + " where ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", false, false);
+
+ Statement statement = null;
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute update " + query);
+ statement.executeUpdate(query);
+ } finally {
+ closeStmt(statement);
+ }
+ }
+ }
+
+ private Map<String, Map<String, String>> updatePartitionParamTable(Connection dbConn,
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap,
+ String validWriteIds,
+ long writeId,
+ boolean isAcidTable)
+ throws SQLException, MetaException {
+ Map<String, Map<String, String>> result = new HashMap<>();
+ boolean areTxnStatsSupported = MetastoreConf.getBoolVar(conf, ConfVars.HIVE_TXN_STATS_ENABLED);
+ PreparedStatement statementInsert = null;
+ PreparedStatement statementDelete = null;
+ PreparedStatement statementUpdate = null;
+ String insert = "INSERT INTO \"PARTITION_PARAMS\" (\"PART_ID\", \"PARAM_KEY\", \"PARAM_VALUE\") "
+ + "VALUES( ? , 'COLUMN_STATS_ACCURATE' , ? )";
+ String delete = "DELETE from \"PARTITION_PARAMS\" "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ String update = "UPDATE \"PARTITION_PARAMS\" set \"PARAM_VALUE\" = ? "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ int numInsert = 0;
+ int numDelete = 0;
+ int numUpdate = 0;
+
+ List<Long> partIdList = partitionInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ // get the old parameters from PARTITION_PARAMS table.
+ Map<Long, String> partIdToParaMap = getParamValues(dbConn, partIdList);
+
+ try {
+ statementInsert = dbConn.prepareStatement(insert);
+ statementDelete = dbConn.prepareStatement(delete);
+ statementUpdate = dbConn.prepareStatement(update);
+ for (Map.Entry entry : partitionInfoMap.entrySet()) {
+ PartitionInfo partitionInfo = (PartitionInfo) entry.getKey();
+ ColumnStatistics colStats = (ColumnStatistics) entry.getValue();
+ List<String> colNames = colStats.getStatsObj().stream().map(e -> e.getColName()).collect(Collectors.toList());
+ long partWriteId = partitionInfo.writeId;
+ long partId = partitionInfo.partitionId;
+ Map<String, String> newParameter;
+
+ if (!partIdToParaMap.containsKey(partId)) {
+ newParameter = new HashMap<>();
+ newParameter.put(COLUMN_STATS_ACCURATE, "TRUE");
+ StatsSetupConst.setColumnStatsState(newParameter, colNames);
+ statementInsert.setLong(1, partId);
+ statementInsert.setString(2, newParameter.get(COLUMN_STATS_ACCURATE));
+ numInsert++;
+ statementInsert.addBatch();
+ if (numInsert == maxBatchSize) {
+ LOG.debug(" Executing insert " + insert);
+ statementInsert.executeBatch();
+ numInsert = 0;
+ }
+ } else {
+ String oldStats = partIdToParaMap.get(partId);
+
+ Map<String, String> oldParameter = new HashMap<>();
+ oldParameter.put(COLUMN_STATS_ACCURATE, oldStats);
+
+ newParameter = new HashMap<>();
+ newParameter.put(COLUMN_STATS_ACCURATE, oldStats);
+ StatsSetupConst.setColumnStatsState(newParameter, colNames);
+
+ if (isAcidTable) {
+ String errorMsg = ObjectStore.verifyStatsChangeCtx(
+ colStats.getStatsDesc().getDbName() + "." + colStats.getStatsDesc().getTableName(),
+ oldParameter, newParameter, writeId, validWriteIds, true);
+ if (errorMsg != null) {
+ throw new MetaException(errorMsg);
+ }
+ }
+
+ if (isAcidTable &&
+ (!areTxnStatsSupported || !ObjectStore.isCurrentStatsValidForTheQuery(oldParameter, partWriteId,
+ validWriteIds, true))) {
+ statementDelete.setLong(1, partId);
+ statementDelete.addBatch();
+ numDelete++;
+ if (numDelete == maxBatchSize) {
+ statementDelete.executeBatch();
+ numDelete = 0;
+ LOG.debug("Removed COLUMN_STATS_ACCURATE from the parameters of the partition "
+ + colStats.getStatsDesc().getDbName() + "." + colStats.getStatsDesc().getTableName() + "."
+ + colStats.getStatsDesc().getPartName());
+ }
+ } else {
+ statementUpdate.setString(1, newParameter.get(COLUMN_STATS_ACCURATE));
+ statementUpdate.setLong(2, partId);
+ statementUpdate.addBatch();
+ numUpdate++;
+ if (numUpdate == maxBatchSize) {
+ LOG.debug(" Executing update " + statementUpdate);
+ statementUpdate.executeBatch();
+ numUpdate = 0;
+ }
+ }
+ }
+ result.put(partitionInfo.partitionName, newParameter);
+ }
+
+ if (numInsert != 0) {
+ statementInsert.executeBatch();
+ }
+
+ if (numUpdate != 0) {
+ statementUpdate.executeBatch();
+ }
+
+ if (numDelete != 0) {
+ statementDelete.executeBatch();
+ }
+
+ if (isAcidTable) {
+ updateWriteIdForPartitions(dbConn, writeId, partIdList);
+ }
+ return result;
+ } finally {
+ closeStmt(statementInsert);
+ closeStmt(statementUpdate);
+ closeStmt(statementDelete);
+ }
+ }
+
+ private static class PartitionInfo {
Review Comment:
add `final` and move to the end of the class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1862878678
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[90 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1428806834
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdatePart.java:
##########
@@ -0,0 +1,1536 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdatePart {
+ private static final Logger LOG = LoggerFactory.getLogger(DirectSqlUpdatePart.class.getName());
+
+ private final PersistenceManager pm;
+ private final Configuration conf;
+ private final DatabaseProduct dbType;
+ private final int maxBatchSize;
+ private final SQLGenerator sqlGenerator;
+
+ private static final ReentrantLock derbyLock = new ReentrantLock(true);
+
+ public DirectSqlUpdatePart(PersistenceManager pm, Configuration conf,
+ DatabaseProduct dbType, int batchSize) {
+ this.pm = pm;
+ this.conf = conf;
+ this.dbType = dbType;
+ this.maxBatchSize = batchSize;
+ sqlGenerator = new SQLGenerator(dbType, conf);
+ }
+
+ /**
+ * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
+ * Select ... For Update to sequence operations properly. In practice that means when running
+ * with Derby database. See more notes at class level.
+ */
+ private void lockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.lock();
+ }
+ }
+
+ private void unlockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.unlock();
+ }
+ }
+
+ void rollbackDBConn(Connection dbConn) {
+ try {
+ if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
+ } catch (SQLException e) {
+ LOG.warn("Failed to rollback db connection ", e);
+ }
+ }
+
+ void closeDbConn(JDOConnection jdoConn) {
+ try {
+ if (jdoConn != null) {
+ jdoConn.close();
+ }
+ } catch (Exception e) {
+ LOG.warn("Failed to close db connection", e);
+ }
+ }
+
+ void closeStmt(Statement stmt) {
+ try {
+ if (stmt != null && !stmt.isClosed()) stmt.close();
+ } catch (SQLException e) {
+ LOG.warn("Failed to close statement ", e);
+ }
+ }
+
+ void close(ResultSet rs) {
+ try {
+ if (rs != null && !rs.isClosed()) {
+ rs.close();
+ }
+ }
+ catch(SQLException ex) {
+ LOG.warn("Failed to close statement ", ex);
+ }
+ }
+
+ static String quoteString(String input) {
+ return "'" + input + "'";
+ }
+
+ void close(ResultSet rs, Statement stmt, JDOConnection dbConn) {
+ close(rs);
+ closeStmt(stmt);
+ closeDbConn(dbConn);
+ }
+
+ private void populateInsertUpdateMap(Map<PartitionInfo, ColumnStatistics> statsPartInfoMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics>insertMap,
+ Connection dbConn, Table tbl) throws SQLException, MetaException, NoSuchObjectException {
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+ List<String> queries = new ArrayList<>();
+ Set<PartColNameInfo> selectedParts = new HashSet<>();
+
+ List<Long> partIdList = statsPartInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ prefix.append("select \"PART_ID\", \"COLUMN_NAME\" from \"PART_COL_STATS\" WHERE ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ selectedParts.add(new PartColNameInfo(rs.getLong(1), rs.getString(2)));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+
+ for (Map.Entry entry : statsPartInfoMap.entrySet()) {
+ PartitionInfo partitionInfo = (PartitionInfo) entry.getKey();
+ ColumnStatistics colStats = (ColumnStatistics) entry.getValue();
+ long partId = partitionInfo.partitionId;
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ if (!statsDesc.isSetCatName()) {
+ statsDesc.setCatName(tbl.getCatName());
+ }
+ for (ColumnStatisticsObj statisticsObj : colStats.getStatsObj()) {
+ PartColNameInfo temp = new PartColNameInfo(partId, statisticsObj.getColName());
+ if (selectedParts.contains(temp)) {
+ updateMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ } else {
+ insertMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ }
+ }
+ }
+ }
+
+ private void updatePartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement pst = null;
+ for (Map.Entry entry : updateMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+ String update = "UPDATE \"PART_COL_STATS\" SET ";
+ update += StatObjectConverter.getUpdatedColumnSql(mPartitionColumnStatistics);
+ update += " WHERE \"PART_ID\" = " + partId + " AND "
+ + " \"COLUMN_NAME\" = " + quoteString(mPartitionColumnStatistics.getColName());
+ try {
+ pst = dbConn.prepareStatement(update);
+ StatObjectConverter.initUpdatedColumnStatement(mPartitionColumnStatistics, pst);
+ LOG.debug("Going to execute update " + update);
+ int numUpdate = pst.executeUpdate();
+ if (numUpdate != 1) {
+ throw new MetaException("Invalid state of PART_COL_STATS for PART_ID " + partId);
+ }
+ } finally {
+ closeStmt(pst);
+ }
+ }
+ }
+
+ private void insertIntoPartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> insertMap,
+ long maxCsId,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement preparedStatement = null;
+ int numRows = 0;
+ String insert = "INSERT INTO \"PART_COL_STATS\" (\"CS_ID\", \"CAT_NAME\", \"DB_NAME\","
+ + "\"TABLE_NAME\", \"PARTITION_NAME\", \"COLUMN_NAME\", \"COLUMN_TYPE\", \"PART_ID\","
+ + " \"LONG_LOW_VALUE\", \"LONG_HIGH_VALUE\", \"DOUBLE_HIGH_VALUE\", \"DOUBLE_LOW_VALUE\","
+ + " \"BIG_DECIMAL_LOW_VALUE\", \"BIG_DECIMAL_HIGH_VALUE\", \"NUM_NULLS\", \"NUM_DISTINCTS\", \"BIT_VECTOR\" ,"
+ + " \"HISTOGRAM\", \"AVG_COL_LEN\", \"MAX_COL_LEN\", \"NUM_TRUES\", \"NUM_FALSES\", \"LAST_ANALYZED\", \"ENGINE\") values "
+ + "(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
+
+ try {
+ preparedStatement = dbConn.prepareStatement(insert);
+ for (Map.Entry entry : insertMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+
+ preparedStatement.setLong(1, maxCsId);
+ preparedStatement.setString(2, mPartitionColumnStatistics.getCatName());
+ preparedStatement.setString(3, mPartitionColumnStatistics.getDbName());
+ preparedStatement.setString(4, mPartitionColumnStatistics.getTableName());
+ preparedStatement.setString(5, mPartitionColumnStatistics.getPartitionName());
+ preparedStatement.setString(6, mPartitionColumnStatistics.getColName());
+ preparedStatement.setString(7, mPartitionColumnStatistics.getColType());
+ preparedStatement.setLong(8, partId);
+ preparedStatement.setObject(9, mPartitionColumnStatistics.getLongLowValue());
+ preparedStatement.setObject(10, mPartitionColumnStatistics.getLongHighValue());
+ preparedStatement.setObject(11, mPartitionColumnStatistics.getDoubleHighValue());
+ preparedStatement.setObject(12, mPartitionColumnStatistics.getDoubleLowValue());
+ preparedStatement.setString(13, mPartitionColumnStatistics.getDecimalLowValue());
+ preparedStatement.setString(14, mPartitionColumnStatistics.getDecimalHighValue());
+ preparedStatement.setObject(15, mPartitionColumnStatistics.getNumNulls());
+ preparedStatement.setObject(16, mPartitionColumnStatistics.getNumDVs());
+ preparedStatement.setObject(17, mPartitionColumnStatistics.getBitVector());
+ preparedStatement.setBytes(18, mPartitionColumnStatistics.getHistogram());
+ preparedStatement.setObject(19, mPartitionColumnStatistics.getAvgColLen());
+ preparedStatement.setObject(20, mPartitionColumnStatistics.getMaxColLen());
+ preparedStatement.setObject(21, mPartitionColumnStatistics.getNumTrues());
+ preparedStatement.setObject(22, mPartitionColumnStatistics.getNumFalses());
+ preparedStatement.setLong(23, mPartitionColumnStatistics.getLastAnalyzed());
+ preparedStatement.setString(24, mPartitionColumnStatistics.getEngine());
+
+ maxCsId++;
+ numRows++;
+ preparedStatement.addBatch();
+ if (numRows == maxBatchSize) {
+ preparedStatement.executeBatch();
+ numRows = 0;
+ }
+ }
+
+ if (numRows != 0) {
+ preparedStatement.executeBatch();
+ }
+ } finally {
+ closeStmt(preparedStatement);
+ }
+ }
+
+ private Map<Long, String> getParamValues(Connection dbConn, List<Long> partIdList) throws SQLException {
+ List<String> queries = new ArrayList<>();
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+
+ prefix.append("select \"PART_ID\", \"PARAM_VALUE\" "
+ + " from \"PARTITION_PARAMS\" where "
+ + " \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE' "
+ + " and ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ Map<Long, String> partIdToParaMap = new HashMap<>();
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ partIdToParaMap.put(rs.getLong(1), rs.getString(2));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+ return partIdToParaMap;
+ }
+
+ private void updateWriteIdForPartitions(Connection dbConn, long writeId, List<Long> partIdList) throws SQLException {
+ StringBuilder prefix = new StringBuilder();
+ List<String> queries = new ArrayList<>();
+ StringBuilder suffix = new StringBuilder();
+
+ prefix.append("UPDATE \"PARTITIONS\" set \"WRITE_ID\" = " + writeId + " where ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", false, false);
+
+ Statement statement = null;
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute update " + query);
+ statement.executeUpdate(query);
+ } finally {
+ closeStmt(statement);
+ }
+ }
+ }
+
+ private Map<String, Map<String, String>> updatePartitionParamTable(Connection dbConn,
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap,
+ String validWriteIds,
+ long writeId,
+ boolean isAcidTable)
+ throws SQLException, MetaException {
+ Map<String, Map<String, String>> result = new HashMap<>();
+ boolean areTxnStatsSupported = MetastoreConf.getBoolVar(conf, ConfVars.HIVE_TXN_STATS_ENABLED);
+ String insert = "INSERT INTO \"PARTITION_PARAMS\" (\"PART_ID\", \"PARAM_KEY\", \"PARAM_VALUE\") "
+ + "VALUES( ? , 'COLUMN_STATS_ACCURATE' , ? )";
+ String delete = "DELETE from \"PARTITION_PARAMS\" "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ String update = "UPDATE \"PARTITION_PARAMS\" set \"PARAM_VALUE\" = ? "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ int numInsert = 0;
+ int numDelete = 0;
+ int numUpdate = 0;
+
+ List<Long> partIdList = partitionInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ // get the old parameters from PARTITION_PARAMS table.
+ Map<Long, String> partIdToParaMap = getParamValues(dbConn, partIdList);
+
+ try (PreparedStatement statementInsert = dbConn.prepareStatement(insert);
+ PreparedStatement statementDelete = dbConn.prepareStatement(delete);
+ PreparedStatement statementUpdate = dbConn.prepareStatement(update)) {
+ for (Map.Entry entry : partitionInfoMap.entrySet()) {
+ PartitionInfo partitionInfo = (PartitionInfo) entry.getKey();
+ ColumnStatistics colStats = (ColumnStatistics) entry.getValue();
+ List<String> colNames = colStats.getStatsObj().stream().map(e -> e.getColName()).collect(Collectors.toList());
+ long partWriteId = partitionInfo.writeId;
+ long partId = partitionInfo.partitionId;
+ Map<String, String> newParameter;
+
+ if (!partIdToParaMap.containsKey(partId)) {
+ newParameter = new HashMap<>();
+ newParameter.put(COLUMN_STATS_ACCURATE, "TRUE");
+ StatsSetupConst.setColumnStatsState(newParameter, colNames);
+ statementInsert.setLong(1, partId);
+ statementInsert.setString(2, newParameter.get(COLUMN_STATS_ACCURATE));
+ numInsert++;
+ statementInsert.addBatch();
+ if (numInsert == maxBatchSize) {
+ LOG.debug(" Executing insert " + insert);
+ statementInsert.executeBatch();
+ numInsert = 0;
+ }
+ } else {
+ String oldStats = partIdToParaMap.get(partId);
+
+ Map<String, String> oldParameter = new HashMap<>();
+ oldParameter.put(COLUMN_STATS_ACCURATE, oldStats);
+
+ newParameter = new HashMap<>();
+ newParameter.put(COLUMN_STATS_ACCURATE, oldStats);
+ StatsSetupConst.setColumnStatsState(newParameter, colNames);
+
+ if (isAcidTable) {
+ String errorMsg = ObjectStore.verifyStatsChangeCtx(
+ colStats.getStatsDesc().getDbName() + "." + colStats.getStatsDesc().getTableName(),
+ oldParameter, newParameter, writeId, validWriteIds, true);
+ if (errorMsg != null) {
+ throw new MetaException(errorMsg);
+ }
+ }
+
+ if (isAcidTable &&
+ (!areTxnStatsSupported || !ObjectStore.isCurrentStatsValidForTheQuery(oldParameter, partWriteId,
+ validWriteIds, true))) {
+ statementDelete.setLong(1, partId);
+ statementDelete.addBatch();
+ numDelete++;
+ if (numDelete == maxBatchSize) {
+ statementDelete.executeBatch();
+ numDelete = 0;
+ LOG.debug("Removed COLUMN_STATS_ACCURATE from the parameters of the partition "
+ + colStats.getStatsDesc().getDbName() + "." + colStats.getStatsDesc().getTableName() + "."
+ + colStats.getStatsDesc().getPartName());
+ }
+ } else {
+ statementUpdate.setString(1, newParameter.get(COLUMN_STATS_ACCURATE));
+ statementUpdate.setLong(2, partId);
+ statementUpdate.addBatch();
+ numUpdate++;
+ if (numUpdate == maxBatchSize) {
+ LOG.debug(" Executing update " + statementUpdate);
+ statementUpdate.executeBatch();
+ numUpdate = 0;
+ }
+ }
+ }
+ result.put(partitionInfo.partitionName, newParameter);
+ }
+
+ if (numInsert != 0) {
+ statementInsert.executeBatch();
+ }
+
+ if (numUpdate != 0) {
+ statementUpdate.executeBatch();
+ }
+
+ if (numDelete != 0) {
+ statementDelete.executeBatch();
+ }
+
+ if (isAcidTable) {
+ updateWriteIdForPartitions(dbConn, writeId, partIdList);
+ }
+ return result;
+ }
+ }
+
+
+ private Map<PartitionInfo, ColumnStatistics> getPartitionInfo(Connection dbConn, long tblId,
+ Map<String, ColumnStatistics> partColStatsMap)
+ throws SQLException, MetaException {
+ List<String> queries = new ArrayList<>();
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap = new HashMap<>();
+
+ List<String> partKeys = partColStatsMap.keySet().stream().map(
+ e -> quoteString(e)).collect(Collectors.toList()
+ );
+
+ prefix.append("select \"PART_ID\", \"WRITE_ID\", \"PART_NAME\" from \"PARTITIONS\" where ");
+ suffix.append(" and \"TBL_ID\" = " + tblId);
+ TxnUtils.buildQueryWithINClauseStrings(conf, queries, prefix, suffix,
+ partKeys, "\"PART_NAME\"", true, false);
+
+ for (String query : queries) {
+ // Select for update makes sure that the partitions are not modified while the stats are getting updated.
+ query = sqlGenerator.addForUpdateClause(query);
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query <" + query + ">");
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ PartitionInfo partitionInfo = new PartitionInfo(rs.getLong(1),
+ rs.getLong(2), rs.getString(3));
+ partitionInfoMap.put(partitionInfo, partColStatsMap.get(rs.getString(3)));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+ return partitionInfoMap;
+ }
+
+ private void setAnsiQuotes(Connection dbConn) throws SQLException {
+ if (sqlGenerator.getDbProduct().isMYSQL()) {
+ try (Statement stmt = dbConn.createStatement()) {
+ stmt.execute("SET @@session.sql_mode=ANSI_QUOTES");
+ }
+ }
+ }
+
+ /**
+ * Update the statistics for the given partitions. Add the notification logs also.
+ * @return map of partition key to column stats if successful, null otherwise.
+ */
+ public Map<String, Map<String, String>> updatePartitionColumnStatistics(Map<String, ColumnStatistics> partColStatsMap,
+ Table tbl, long csId,
+ String validWriteIds, long writeId,
+ List<TransactionalMetaStoreEventListener> transactionalListeners)
+ throws MetaException {
+ JDOConnection jdoConn = null;
+ Connection dbConn = null;
+ boolean committed = false;
+ try {
+ lockInternal();
+ jdoConn = pm.getDataStoreConnection();
+ dbConn = (Connection) (jdoConn.getNativeConnection());
+
+ setAnsiQuotes(dbConn);
+
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap = getPartitionInfo(dbConn, tbl.getId(), partColStatsMap);
+
+ Map<String, Map<String, String>> result =
+ updatePartitionParamTable(dbConn, partitionInfoMap, validWriteIds, writeId, TxnUtils.isAcidTable(tbl));
+
+ Map<PartColNameInfo, MPartitionColumnStatistics> insertMap = new HashMap<>();
+ Map<PartColNameInfo, MPartitionColumnStatistics> updateMap = new HashMap<>();
+ populateInsertUpdateMap(partitionInfoMap, updateMap, insertMap, dbConn, tbl);
+
+ LOG.info("Number of stats to insert " + insertMap.size() + " update " + updateMap.size());
+
+ if (insertMap.size() != 0) {
+ insertIntoPartColStatTable(insertMap, csId, dbConn);
+ }
+
+ if (updateMap.size() != 0) {
+ updatePartColStatTable(updateMap, dbConn);
+ }
+
+ if (transactionalListeners != null) {
+ UpdatePartitionColumnStatEventBatch eventBatch = new UpdatePartitionColumnStatEventBatch(null);
+ for (Map.Entry entry : result.entrySet()) {
+ Map<String, String> parameters = (Map<String, String>) entry.getValue();
+ ColumnStatistics colStats = partColStatsMap.get(entry.getKey());
+ List<String> partVals = getPartValsFromName(tbl, colStats.getStatsDesc().getPartName());
+ UpdatePartitionColumnStatEvent event = new UpdatePartitionColumnStatEvent(colStats, partVals, parameters,
+ tbl, writeId, null);
+ eventBatch.addPartColStatEvent(event);
+ }
+ MetaStoreListenerNotifier.notifyEventWithDirectSql(transactionalListeners,
+ EventMessage.EventType.UPDATE_PARTITION_COLUMN_STAT_BATCH, eventBatch, dbConn, sqlGenerator);
+ }
+ dbConn.commit();
+ committed = true;
+ return result;
+ } catch (Exception e) {
+ LOG.error("Unable to update Column stats for " + tbl.getTableName(), e);
+ throw new MetaException("Unable to update Column stats for " + tbl.getTableName()
+ + " due to: " + e.getMessage());
+ } finally {
+ if (!committed) {
+ rollbackDBConn(dbConn);
+ }
+ closeDbConn(jdoConn);
+ unlockInternal();
+ }
+ }
+
+ /**
+ * Gets the next CS id from sequence MPartitionColumnStatistics and increment the CS id by numStats.
+ * @return The CD id before update.
+ */
+ public long getNextCSIdForMPartitionColumnStatistics(long numStats) throws MetaException {
+ Statement statement = null;
+ ResultSet rs = null;
+ long maxCsId = 0;
+ boolean committed = false;
+ Connection dbConn = null;
+ JDOConnection jdoConn = null;
+
+ try {
+ lockInternal();
+ jdoConn = pm.getDataStoreConnection();
+ dbConn = (Connection) (jdoConn.getNativeConnection());
+
+ setAnsiQuotes(dbConn);
+
+ // This loop will be iterated at max twice. If there is no records, it will first insert and then do a select.
+ // We are not using any upsert operations as select for update and then update is required to make sure that
+ // the caller gets a reserved range for CSId not used by any other thread.
+ boolean insertDone = false;
+ while (maxCsId == 0) {
+ String query = sqlGenerator.addForUpdateClause("SELECT \"NEXT_VAL\" FROM \"SEQUENCE_TABLE\" "
+ + "WHERE \"SEQUENCE_NAME\"= "
+ + quoteString("org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics"));
+ LOG.debug("Going to execute query " + query);
+ statement = dbConn.createStatement();
+ rs = statement.executeQuery(query);
+ if (rs.next()) {
+ maxCsId = rs.getLong(1);
+ } else if (insertDone) {
+ throw new MetaException("Invalid state of SEQUENCE_TABLE for MPartitionColumnStatistics");
+ } else {
+ insertDone = true;
+ closeStmt(statement);
+ statement = dbConn.createStatement();
Review Comment:
why do we recreate the statement here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1860323906
> @deniskuzZ Thanks for the patient review, here is the refactor jira: [HIVE-27958](https://issues.apache.org/jira/browse/HIVE-27958). Please help take a look, and I'm willing to take it over.
oh my, sorry @wecharyu , but looks like we need to rebase again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1864486971
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[90 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1410548941
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdate.java:
##########
@@ -0,0 +1,1541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the statistics update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdate {
+ private static final Logger LOG = LoggerFactory.getLogger(DirectSqlUpdate.class.getName());
+ PersistenceManager pm;
Review Comment:
add new line
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1410557709
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdate.java:
##########
@@ -0,0 +1,1541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the statistics update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdate {
Review Comment:
should we rename to DirectSqlUpdatePart to be consistent with DirectSqlInsertPart?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1413043224
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdate.java:
##########
@@ -0,0 +1,1541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the statistics update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdate {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1866803205
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] saihemanth-cloudera commented on a diff in pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "saihemanth-cloudera (via GitHub)" <gi...@apache.org>.
saihemanth-cloudera commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1316187598
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##########
@@ -200,7 +202,8 @@ public MetaStoreDirectSql(PersistenceManager pm, Configuration conf, String sche
batchSize = dbType.needsInBatching() ? 1000 : NO_BATCHING;
}
this.batchSize = batchSize;
- this.updateStat = new DirectSqlUpdateStat(pm, conf, dbType, batchSize);
+ this.areTxnStatsSupported = MetastoreConf.getBoolVar(conf, ConfVars.HIVE_TXN_STATS_ENABLED);
Review Comment:
Nit: should this variable be 'areTxnStatsEnabled'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1708926581
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [3 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [96 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] saihemanth-cloudera commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "saihemanth-cloudera (via GitHub)" <gi...@apache.org>.
saihemanth-cloudera commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1686842101
@VenuReddy2103
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1709467501
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [96 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1712333546
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [1 Bug](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [104 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1865241251
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1867998326
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1867750234
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1868258135
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ merged PR #4517:
URL: https://github.com/apache/hive/pull/4517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1428801804
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdate.java:
##########
@@ -0,0 +1,1541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the statistics update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdate {
+ private static final Logger LOG = LoggerFactory.getLogger(DirectSqlUpdate.class.getName());
+ PersistenceManager pm;
+ Configuration conf;
+ DatabaseProduct dbType;
+ int maxBatchSize;
+ SQLGenerator sqlGenerator;
+ private static final ReentrantLock derbyLock = new ReentrantLock(true);
+
+ public DirectSqlUpdate(PersistenceManager pm, Configuration conf,
+ DatabaseProduct dbType, int batchSize) {
+ this.pm = pm;
+ this.conf = conf;
+ this.dbType = dbType;
+ this.maxBatchSize = batchSize;
+ sqlGenerator = new SQLGenerator(dbType, conf);
+ }
+
+ /**
+ * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
+ * Select ... For Update to sequence operations properly. In practice that means when running
+ * with Derby database. See more notes at class level.
+ */
+ private void lockInternal() {
Review Comment:
sure, @wecharyu, could you please create ticket for that
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] wecharyu commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1687336748
@nrg4878 I have only test this patch on MySQL DB so far, let me add the test for other DBs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] wecharyu commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1683271360
Gentle ping @saihemanth-cloudera @dengzhhu653, could you take a look when you have a moment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] sonarcloud[bot] commented on pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1693414842
Kudos, SonarCloud Quality Gate passed! [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG) [2 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=BUG)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=VULNERABILITY)
[](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT) [2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&types=SECURITY_HOTSPOT)
[](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL) [98 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&types=CODE_SMELL)
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=coverage&view=list) No Coverage information
[](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4517&metric=duplicated_lines_density&view=list) No Duplication information
 The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
Read more [here](https://docs.sonarcloud.io/appendices/scanner-environment/)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
[GitHub] [hive] saihemanth-cloudera commented on a diff in pull request #4517: HIVE-27530: Implement direct SQL for alter partitions to improve performance
Posted by "saihemanth-cloudera (via GitHub)" <gi...@apache.org>.
saihemanth-cloudera commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1316203471
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
##########
@@ -532,6 +535,66 @@ public void addPartitions(List<MPartition> parts, List<List<MPartitionPrivilege>
directSqlInsertPart.addPartitions(parts, partPrivilegesList, partColPrivilegesList);
}
+ /**
+ * Alter partitions in batch using direct SQL
+ * @param table the target table
+ * @param partNames list of partition names
+ * @param newParts list of new partitions
+ * @param queryWriteIdList valid write id list
+ * @return
+ * @throws MetaException
+ */
+ public List<Partition> alterPartitions(Table table, List<String> partNames,
+ List<Partition> newParts, String queryWriteIdList) throws MetaException {
+ String filter = "" + PARTITIONS + ".\"PART_NAME\" in (" + makeParams(partNames.size()) + ")";
+ List<String> columns = Arrays.asList("\"PART_ID\"", "\"PART_NAME\"", "\"SD_ID\"", "\"WRITE_ID\"");
+ List<Object[]> rows = getPartitionFieldsViaSqlFilter(table.getCatName(), table.getDbName(),
+ table.getTableName(), columns, filter, partNames, Collections.emptyList(), null);
+ Map<List<String>, Long> partValuesToId = new HashMap<>();
+ Map<Long, Long> partIdToSdId = new HashMap<>();
+ Map<Long, Long> partIdToWriteId = new HashMap<>();
+ for (Object[] row : rows) {
+ Long partId = MetastoreDirectSqlUtils.extractSqlLong(row[0]);
+ Long sdId = MetastoreDirectSqlUtils.extractSqlLong(row[2]);
+ Long writeId = MetastoreDirectSqlUtils.extractSqlLong(row[3]);
+ partIdToSdId.put(partId, sdId);
+ partIdToWriteId.put(partId, writeId);
+ List<String> partValues = Warehouse.getPartValuesFromPartName((String) row[1]);
+ partValuesToId.put(partValues, partId);
+ }
+
+ boolean isTxn = TxnUtils.isTransactionalTable(table.getParameters());
+ for (Partition newPart : newParts) {
+ Long partId = partValuesToId.get(newPart.getValues());
+ boolean useOldWriteId = true;
+ // If transactional, add/update the MUPdaterTransaction
+ // for the current updater query.
+ if (isTxn) {
+ if (!areTxnStatsSupported) {
+ StatsSetupConst.setBasicStatsState(newPart.getParameters(), StatsSetupConst.FALSE);
+ } else if (queryWriteIdList != null && newPart.getWriteId() > 0) {
+ // Check concurrent INSERT case and set false to the flag.
+ if (!ObjectStore.isCurrentStatsValidForTheQuery(newPart.getParameters(),
+ partIdToWriteId.get(partId), queryWriteIdList, true)) {
+ StatsSetupConst.setBasicStatsState(newPart.getParameters(), StatsSetupConst.FALSE);
+ LOG.info("Removed COLUMN_STATS_ACCURATE from the parameters of the partition " +
+ Warehouse.getQualifiedName(newPart) + " will be made persistent.");
+ }
+ useOldWriteId = false;
+ }
+ }
+
+ if (useOldWriteId) {
+ newPart.setWriteId(partIdToWriteId.get(partId));
+ }
+ }
+
+ directSqlUpdate.alterPartitions(partValuesToId, partIdToSdId, newParts);
+ // The returned result were not utilized, so return an empty list in direct sql for performance.
+ // TODO: Change the API signature of RawStore#alterPartitions to return void instead.
+ return Collections.emptyList();
Review Comment:
It would be nice to return the updated altered partitioned result set. It would be used by clients immediately after the alter operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1858858431
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[97 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1860972653
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517) **Quality Gate passed**
The SonarCloud Quality Gate passed, but some issues were introduced.
[102 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
[2 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4517&resolved=false&inNewCodePeriod=true)
No data about Coverage
No data about Duplication
[See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4517)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1864364009
> @wecharyu , http://ci.hive.apache.org/job/hive-precommit/job/PR-4517/17/testReport/junit/org.apache.hadoop.hive.ql.exec/TestConcurrentDppInserts/Testing___split_17___PostProcess___testConcurrentCreationOfSamePartition/ seems related, failed in multiple runs
This test is add_partition operation, which should be unrelated to this patch. It seems a flaky test that would fail occasionally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on PR #4517:
URL: https://github.com/apache/hive/pull/4517#issuecomment-1807868603
@deniskuzZ Fixed the conflict.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1410545851
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DatabaseProduct.java:
##########
@@ -675,6 +676,24 @@ public List<String> createInsertValuesStmt(String tblColumns, List<String> rows,
}
}
+ public String createUpdatePreparedStmt(String tableName, List<String> columnNames,
Review Comment:
why it's in DatabaseProduct, they are generic and not db specific? should we move them to TxnUtils and mark them as static?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1410559604
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdate.java:
##########
@@ -0,0 +1,1541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the statistics update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdate {
+ private static final Logger LOG = LoggerFactory.getLogger(DirectSqlUpdate.class.getName());
+ PersistenceManager pm;
+ Configuration conf;
+ DatabaseProduct dbType;
+ int maxBatchSize;
+ SQLGenerator sqlGenerator;
+ private static final ReentrantLock derbyLock = new ReentrantLock(true);
+
+ public DirectSqlUpdate(PersistenceManager pm, Configuration conf,
+ DatabaseProduct dbType, int batchSize) {
+ this.pm = pm;
+ this.conf = conf;
+ this.dbType = dbType;
+ this.maxBatchSize = batchSize;
+ sqlGenerator = new SQLGenerator(dbType, conf);
+ }
+
+ /**
+ * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
+ * Select ... For Update to sequence operations properly. In practice that means when running
+ * with Derby database. See more notes at class level.
+ */
+ private void lockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.lock();
+ }
+ }
+
+ private void unlockInternal() {
+ if(dbType.isDERBY()) {
+ derbyLock.unlock();
+ }
+ }
+
+ void rollbackDBConn(Connection dbConn) {
+ try {
+ if (dbConn != null && !dbConn.isClosed()) dbConn.rollback();
+ } catch (SQLException e) {
+ LOG.warn("Failed to rollback db connection ", e);
+ }
+ }
+
+ void closeDbConn(JDOConnection jdoConn) {
+ try {
+ if (jdoConn != null) {
+ jdoConn.close();
+ }
+ } catch (Exception e) {
+ LOG.warn("Failed to close db connection", e);
+ }
+ }
+
+ void closeStmt(Statement stmt) {
+ try {
+ if (stmt != null && !stmt.isClosed()) stmt.close();
+ } catch (SQLException e) {
+ LOG.warn("Failed to close statement ", e);
+ }
+ }
+
+ void close(ResultSet rs) {
+ try {
+ if (rs != null && !rs.isClosed()) {
+ rs.close();
+ }
+ }
+ catch(SQLException ex) {
+ LOG.warn("Failed to close statement ", ex);
+ }
+ }
+
+ static String quoteString(String input) {
+ return "'" + input + "'";
+ }
+
+ void close(ResultSet rs, Statement stmt, JDOConnection dbConn) {
+ close(rs);
+ closeStmt(stmt);
+ closeDbConn(dbConn);
+ }
+
+ private void populateInsertUpdateMap(Map<PartitionInfo, ColumnStatistics> statsPartInfoMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Map<PartColNameInfo, MPartitionColumnStatistics>insertMap,
+ Connection dbConn, Table tbl) throws SQLException, MetaException, NoSuchObjectException {
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+ List<String> queries = new ArrayList<>();
+ Set<PartColNameInfo> selectedParts = new HashSet<>();
+
+ List<Long> partIdList = statsPartInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ prefix.append("select \"PART_ID\", \"COLUMN_NAME\" from \"PART_COL_STATS\" WHERE ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ selectedParts.add(new PartColNameInfo(rs.getLong(1), rs.getString(2)));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+
+ for (Map.Entry entry : statsPartInfoMap.entrySet()) {
+ PartitionInfo partitionInfo = (PartitionInfo) entry.getKey();
+ ColumnStatistics colStats = (ColumnStatistics) entry.getValue();
+ long partId = partitionInfo.partitionId;
+ ColumnStatisticsDesc statsDesc = colStats.getStatsDesc();
+ if (!statsDesc.isSetCatName()) {
+ statsDesc.setCatName(tbl.getCatName());
+ }
+ for (ColumnStatisticsObj statisticsObj : colStats.getStatsObj()) {
+ PartColNameInfo temp = new PartColNameInfo(partId, statisticsObj.getColName());
+ if (selectedParts.contains(temp)) {
+ updateMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ } else {
+ insertMap.put(temp, StatObjectConverter.
+ convertToMPartitionColumnStatistics(null, statsDesc, statisticsObj, colStats.getEngine()));
+ }
+ }
+ }
+ }
+
+ private void updatePartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> updateMap,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement pst = null;
+ for (Map.Entry entry : updateMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+ String update = "UPDATE \"PART_COL_STATS\" SET ";
+ update += StatObjectConverter.getUpdatedColumnSql(mPartitionColumnStatistics);
+ update += " WHERE \"PART_ID\" = " + partId + " AND "
+ + " \"COLUMN_NAME\" = " + quoteString(mPartitionColumnStatistics.getColName());
+ try {
+ pst = dbConn.prepareStatement(update);
+ StatObjectConverter.initUpdatedColumnStatement(mPartitionColumnStatistics, pst);
+ LOG.debug("Going to execute update " + update);
+ int numUpdate = pst.executeUpdate();
+ if (numUpdate != 1) {
+ throw new MetaException("Invalid state of PART_COL_STATS for PART_ID " + partId);
+ }
+ } finally {
+ closeStmt(pst);
+ }
+ }
+ }
+
+ private void insertIntoPartColStatTable(Map<PartColNameInfo, MPartitionColumnStatistics> insertMap,
+ long maxCsId,
+ Connection dbConn) throws SQLException, MetaException, NoSuchObjectException {
+ PreparedStatement preparedStatement = null;
+ int numRows = 0;
+ String insert = "INSERT INTO \"PART_COL_STATS\" (\"CS_ID\", \"CAT_NAME\", \"DB_NAME\","
+ + "\"TABLE_NAME\", \"PARTITION_NAME\", \"COLUMN_NAME\", \"COLUMN_TYPE\", \"PART_ID\","
+ + " \"LONG_LOW_VALUE\", \"LONG_HIGH_VALUE\", \"DOUBLE_HIGH_VALUE\", \"DOUBLE_LOW_VALUE\","
+ + " \"BIG_DECIMAL_LOW_VALUE\", \"BIG_DECIMAL_HIGH_VALUE\", \"NUM_NULLS\", \"NUM_DISTINCTS\", \"BIT_VECTOR\" ,"
+ + " \"HISTOGRAM\", \"AVG_COL_LEN\", \"MAX_COL_LEN\", \"NUM_TRUES\", \"NUM_FALSES\", \"LAST_ANALYZED\", \"ENGINE\") values "
+ + "(?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)";
+
+ try {
+ preparedStatement = dbConn.prepareStatement(insert);
+ for (Map.Entry entry : insertMap.entrySet()) {
+ PartColNameInfo partColNameInfo = (PartColNameInfo) entry.getKey();
+ Long partId = partColNameInfo.partitionId;
+ MPartitionColumnStatistics mPartitionColumnStatistics = (MPartitionColumnStatistics) entry.getValue();
+
+ preparedStatement.setLong(1, maxCsId);
+ preparedStatement.setString(2, mPartitionColumnStatistics.getCatName());
+ preparedStatement.setString(3, mPartitionColumnStatistics.getDbName());
+ preparedStatement.setString(4, mPartitionColumnStatistics.getTableName());
+ preparedStatement.setString(5, mPartitionColumnStatistics.getPartitionName());
+ preparedStatement.setString(6, mPartitionColumnStatistics.getColName());
+ preparedStatement.setString(7, mPartitionColumnStatistics.getColType());
+ preparedStatement.setLong(8, partId);
+ preparedStatement.setObject(9, mPartitionColumnStatistics.getLongLowValue());
+ preparedStatement.setObject(10, mPartitionColumnStatistics.getLongHighValue());
+ preparedStatement.setObject(11, mPartitionColumnStatistics.getDoubleHighValue());
+ preparedStatement.setObject(12, mPartitionColumnStatistics.getDoubleLowValue());
+ preparedStatement.setString(13, mPartitionColumnStatistics.getDecimalLowValue());
+ preparedStatement.setString(14, mPartitionColumnStatistics.getDecimalHighValue());
+ preparedStatement.setObject(15, mPartitionColumnStatistics.getNumNulls());
+ preparedStatement.setObject(16, mPartitionColumnStatistics.getNumDVs());
+ preparedStatement.setObject(17, mPartitionColumnStatistics.getBitVector());
+ preparedStatement.setBytes(18, mPartitionColumnStatistics.getHistogram());
+ preparedStatement.setObject(19, mPartitionColumnStatistics.getAvgColLen());
+ preparedStatement.setObject(20, mPartitionColumnStatistics.getMaxColLen());
+ preparedStatement.setObject(21, mPartitionColumnStatistics.getNumTrues());
+ preparedStatement.setObject(22, mPartitionColumnStatistics.getNumFalses());
+ preparedStatement.setLong(23, mPartitionColumnStatistics.getLastAnalyzed());
+ preparedStatement.setString(24, mPartitionColumnStatistics.getEngine());
+
+ maxCsId++;
+ numRows++;
+ preparedStatement.addBatch();
+ if (numRows == maxBatchSize) {
+ preparedStatement.executeBatch();
+ numRows = 0;
+ }
+ }
+
+ if (numRows != 0) {
+ preparedStatement.executeBatch();
+ }
+ } finally {
+ closeStmt(preparedStatement);
+ }
+ }
+
+ private Map<Long, String> getParamValues(Connection dbConn, List<Long> partIdList) throws SQLException {
+ List<String> queries = new ArrayList<>();
+ StringBuilder prefix = new StringBuilder();
+ StringBuilder suffix = new StringBuilder();
+ Statement statement = null;
+ ResultSet rs = null;
+
+ prefix.append("select \"PART_ID\", \"PARAM_VALUE\" "
+ + " from \"PARTITION_PARAMS\" where "
+ + " \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE' "
+ + " and ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", true, false);
+
+ Map<Long, String> partIdToParaMap = new HashMap<>();
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute query " + query);
+ rs = statement.executeQuery(query);
+ while (rs.next()) {
+ partIdToParaMap.put(rs.getLong(1), rs.getString(2));
+ }
+ } finally {
+ close(rs, statement, null);
+ }
+ }
+ return partIdToParaMap;
+ }
+
+ private void updateWriteIdForPartitions(Connection dbConn, long writeId, List<Long> partIdList) throws SQLException {
+ StringBuilder prefix = new StringBuilder();
+ List<String> queries = new ArrayList<>();
+ StringBuilder suffix = new StringBuilder();
+
+ prefix.append("UPDATE \"PARTITIONS\" set \"WRITE_ID\" = " + writeId + " where ");
+ TxnUtils.buildQueryWithINClause(conf, queries, prefix, suffix,
+ partIdList, "\"PART_ID\"", false, false);
+
+ Statement statement = null;
+ for (String query : queries) {
+ try {
+ statement = dbConn.createStatement();
+ LOG.debug("Going to execute update " + query);
+ statement.executeUpdate(query);
+ } finally {
+ closeStmt(statement);
+ }
+ }
+ }
+
+ private Map<String, Map<String, String>> updatePartitionParamTable(Connection dbConn,
+ Map<PartitionInfo, ColumnStatistics> partitionInfoMap,
+ String validWriteIds,
+ long writeId,
+ boolean isAcidTable)
+ throws SQLException, MetaException {
+ Map<String, Map<String, String>> result = new HashMap<>();
+ boolean areTxnStatsSupported = MetastoreConf.getBoolVar(conf, ConfVars.HIVE_TXN_STATS_ENABLED);
+ PreparedStatement statementInsert = null;
+ PreparedStatement statementDelete = null;
+ PreparedStatement statementUpdate = null;
+ String insert = "INSERT INTO \"PARTITION_PARAMS\" (\"PART_ID\", \"PARAM_KEY\", \"PARAM_VALUE\") "
+ + "VALUES( ? , 'COLUMN_STATS_ACCURATE' , ? )";
+ String delete = "DELETE from \"PARTITION_PARAMS\" "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ String update = "UPDATE \"PARTITION_PARAMS\" set \"PARAM_VALUE\" = ? "
+ + " where \"PART_ID\" = ? "
+ + " and \"PARAM_KEY\" = 'COLUMN_STATS_ACCURATE'";
+ int numInsert = 0;
+ int numDelete = 0;
+ int numUpdate = 0;
+
+ List<Long> partIdList = partitionInfoMap.keySet().stream().map(
+ e -> e.partitionId).collect(Collectors.toList()
+ );
+
+ // get the old parameters from PARTITION_PARAMS table.
+ Map<Long, String> partIdToParaMap = getParamValues(dbConn, partIdList);
+
+ try {
Review Comment:
could we. use try-with-resources
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1410548941
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdate.java:
##########
@@ -0,0 +1,1541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the statistics update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdate {
+ private static final Logger LOG = LoggerFactory.getLogger(DirectSqlUpdate.class.getName());
+ PersistenceManager pm;
Review Comment:
add new line and add private final modifiers
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-27530: Implement direct SQL for alter partitions to improve performance [hive]
Posted by "wecharyu (via GitHub)" <gi...@apache.org>.
wecharyu commented on code in PR #4517:
URL: https://github.com/apache/hive/pull/4517#discussion_r1413040933
##########
standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/DirectSqlUpdate.java:
##########
@@ -0,0 +1,1541 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.metastore;
+
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.hadoop.hive.common.StatsSetupConst;
+import org.apache.hadoop.hive.metastore.api.ColumnStatistics;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsDesc;
+import org.apache.hadoop.hive.metastore.api.ColumnStatisticsObj;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.api.NoSuchObjectException;
+import org.apache.hadoop.hive.metastore.api.Order;
+import org.apache.hadoop.hive.metastore.api.Partition;
+import org.apache.hadoop.hive.metastore.api.SerDeInfo;
+import org.apache.hadoop.hive.metastore.api.SkewedInfo;
+import org.apache.hadoop.hive.metastore.api.StorageDescriptor;
+import org.apache.hadoop.hive.metastore.api.Table;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf;
+import org.apache.hadoop.hive.metastore.conf.MetastoreConf.ConfVars;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEvent;
+import org.apache.hadoop.hive.metastore.events.UpdatePartitionColumnStatEventBatch;
+import org.apache.hadoop.hive.metastore.messaging.EventMessage;
+import org.apache.hadoop.hive.metastore.model.MColumnDescriptor;
+import org.apache.hadoop.hive.metastore.model.MPartitionColumnStatistics;
+import org.apache.hadoop.hive.metastore.model.MStringList;
+import org.apache.hadoop.hive.metastore.tools.SQLGenerator;
+import org.apache.hadoop.hive.metastore.txn.TxnUtils;
+import org.datanucleus.ExecutionContext;
+import org.datanucleus.api.jdo.JDOPersistenceManager;
+import org.datanucleus.metadata.AbstractClassMetaData;
+import org.datanucleus.metadata.IdentityType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.apache.hadoop.conf.Configuration;
+
+import javax.jdo.PersistenceManager;
+import javax.jdo.datastore.JDOConnection;
+import java.sql.Connection;
+import java.sql.PreparedStatement;
+import java.sql.ResultSet;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.locks.ReentrantLock;
+import java.util.stream.Collectors;
+
+import static org.apache.hadoop.hive.common.StatsSetupConst.COLUMN_STATS_ACCURATE;
+import static org.apache.hadoop.hive.metastore.HMSHandler.getPartValsFromName;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.executeWithArray;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlInt;
+import static org.apache.hadoop.hive.metastore.MetastoreDirectSqlUtils.extractSqlLong;
+
+/**
+ * This class contains the optimizations for MetaStore that rely on direct SQL access to
+ * the underlying database. It should use ANSI SQL and be compatible with common databases
+ * such as MySQL (note that MySQL doesn't use full ANSI mode by default), Postgres, etc.
+ *
+ * This class separates out the statistics update part from MetaStoreDirectSql class.
+ */
+class DirectSqlUpdate {
+ private static final Logger LOG = LoggerFactory.getLogger(DirectSqlUpdate.class.getName());
+ PersistenceManager pm;
+ Configuration conf;
+ DatabaseProduct dbType;
+ int maxBatchSize;
+ SQLGenerator sqlGenerator;
+ private static final ReentrantLock derbyLock = new ReentrantLock(true);
+
+ public DirectSqlUpdate(PersistenceManager pm, Configuration conf,
+ DatabaseProduct dbType, int batchSize) {
+ this.pm = pm;
+ this.conf = conf;
+ this.dbType = dbType;
+ this.maxBatchSize = batchSize;
+ sqlGenerator = new SQLGenerator(dbType, conf);
+ }
+
+ /**
+ * {@link #lockInternal()} and {@link #unlockInternal()} are used to serialize those operations that require
+ * Select ... For Update to sequence operations properly. In practice that means when running
+ * with Derby database. See more notes at class level.
+ */
+ private void lockInternal() {
Review Comment:
This class was renamed from `DirectSqlUpdateStats`, and these methods were not changed.
I agree that they should be moved to a common class because there are duplicate implements among `TxnHandler`, `SqlRetryHandler` and this class. But it seems not related to this patch, can we make a new patch for this change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org