You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "abstractdog (via GitHub)" <gi...@apache.org> on 2025/01/16 08:42:50 UTC
[PR] [DRAFT] HIVE-28095: Hive Query History [hive]
abstractdog opened a new pull request, #5613:
URL: https://github.com/apache/hive/pull/5613
DRAFT: I'm filling description as we speak
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://cwiki.apache.org/confluence/display/Hive/HowToContribute
2. Ensure that you have created an issue on the Hive project JIRA: https://issues.apache.org/jira/projects/HIVE/summary
3. Ensure you have added or run the appropriate tests for your PR:
4. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP]HIVE-XXXXX: Your PR title ...'.
5. Be sure to keep the PR description updated to reflect all changes.
6. Please write your PR title to summarize what this PR proposes.
7. If possible, provide a concise example to reproduce the issue for a faster review.
-->
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### Is the change a dependency upgrade?
<!--
If yes, please attach a file with output from mvn dependency:tree to validate a complete upgrade of dependency.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930769852
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java:
##########
@@ -236,6 +237,10 @@ public Set<String> getTabAliases() {
return aliasToTabs.keySet();
}
+ public Collection<String> getTableAliasValues() {
Review Comment:
not used, please remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931178435
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
Review Comment:
if memory option is on and user submits a simple query, would he see it in a history after completion, or only after the batch size is reached?
what if it not reached in 1 day, should we add time-based `flush` as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931638463
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void handleQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ QueryHistoryRecord record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ persistRecordsIfNeeded();
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ persistWithBatchSize(maxBatchSize);
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(), maxBatchSize);
+ }
+ }
+
+ private void persistWithBatchSize(int batchSize) {
+ CountDownLatch latch = new CountDownLatch(1);
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ repository.persist(queryHistoryQueue);
+ latch.countDown();
+ });
+ if (batchSize == 0) {
+ waitForAllRecordsToBePersisted(latch);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted(CountDownLatch queueEmptyCondition) {
+ try {
+ queueEmptyCondition.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ boolean needToPersist() {
+ boolean needToPersistAccordingToQueueSize = queryHistoryQueue.size() >= maxBatchSize;
Review Comment:
ack, eventually, this will be the batch size used for flushing
changing "hive.query.history.persist.max.batch.size" accordingly to "hive.query.history.persist.batch.size"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930403501
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -7136,6 +7156,11 @@ public boolean isWebUiQueryInfoCacheEnabled() {
return isWebUiEnabled() && this.getIntVar(ConfVars.HIVE_SERVER2_WEBUI_MAX_HISTORIC_QUERIES) > 0;
}
+ public boolean isQueryHistoryExplainEnabled() {
Review Comment:
the method is used just in `Compiler#explainOutput` + it reads as explain history not 'explain plan'. I would inline and use config directly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930492394
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/queryhistory/repository/TestIcebergRepository.java:
##########
@@ -0,0 +1,115 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one§
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchemaTestUtils;
+import org.apache.iceberg.mr.mapred.Container;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.queryhistory.QueryHistoryService;
+import org.apache.hadoop.hive.ql.queryhistory.repository.IcebergRepositoryForTest;
+import org.apache.hadoop.hive.ql.queryhistory.schema.DummyQueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.IcebergQueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileInputFormat;
+import org.apache.hadoop.mapred.InputFormat;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hadoop.mapred.Reporter;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.Queue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+public class TestIcebergRepository {
Review Comment:
why iceberg, is't it implementation details, maybe TestQueryHistoryRepository
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930421202
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -838,6 +845,9 @@ public void storageHandlerCommit(Properties commitProperties, Operation operatio
}
HiveIcebergOutputCommitter committer = getOutputCommitter();
try {
+ if (context != null) {
+ committer.commitTask(context);
Review Comment:
can't we get `HiveIcebergOutputCommitter` from `TaskAttemptContext context`?
````
OutputCommitter commiter = getOutputCommitter(context.getConfiguration());
committer.commitTask(context);
....
public static OutputCommitter getOutputCommitter(Configuration conf) {
return (OutputCommitter)ReflectionUtils.newInstance(this.getClass("mapred.output.committer.class", FileOutputCommitter.class, OutputCommitter.class), conf);
}
````
see `FileOutputFormatContainer# getOutputCommitter(TaskAttemptContext context)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930879388
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
Review Comment:
why is it protected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931069266
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/AbstractQueryHistoryRepository.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.TableType;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.parse.PartitionTransform;
+import org.apache.hadoop.hive.ql.parse.TransformSpec;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public abstract class AbstractQueryHistoryRepository implements QueryHistoryRepository {
+ protected Logger LOG = LoggerFactory.getLogger(getClass());
+ @VisibleForTesting
+ HiveConf conf;
+ protected QueryHistorySchema schema;
+
+ public void init(HiveConf conf, QueryHistorySchema schema) {
+ this.conf = conf;
+ this.schema = schema;
+
+ try (Hive hive = Hive.get(conf)) {
+ Database database = initDatabase(hive);
+ Table table = initTable(hive, database);
+ postInitTable(table);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ protected Database initDatabase(Hive hive) {
+ Database db;
+ try {
+ db = hive.getDatabase(QUERY_HISTORY_DB_NAME);
+ if (db == null) {
+ LOG.warn("Database ({}) for query history table hasn't been found, auto-creating one", QUERY_HISTORY_DB_NAME);
+ String location = getDatabaseLocation(QUERY_HISTORY_DB_NAME);
+ db = new Database(QUERY_HISTORY_DB_NAME, QUERY_HISTORY_DB_COMMENT,
+ location, null);
+ hive.createDatabase(db, false);
+ }
+ return db;
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private String getDatabaseLocation(String databaseName) {
+ String warehouseLocation = conf.get(HiveConf.ConfVars.HIVE_METASTORE_WAREHOUSE_EXTERNAL.varname);
+ if (warehouseLocation == null) {
+ warehouseLocation = conf.get(HiveConf.ConfVars.METASTORE_WAREHOUSE.varname);
+ }
+ Preconditions.checkNotNull(
+ warehouseLocation, "Warehouse location is not set: 'hive.metastore.warehouse.external.dir' and " +
+ "'hive.metastore.warehouse.dir' are both null");
+ return String.format("%s/%s.db", warehouseLocation, databaseName.toLowerCase());
+ }
+
+ protected Table initTable(Hive hive, Database db) {
+ Table table;
+ try {
+ table = hive.getTable(QUERY_HISTORY_DB_NAME, QUERY_HISTORY_TABLE_NAME, null, false);
+ if (table == null) {
+ LOG.info("Query history table ({}) isn't created yet", QUERY_HISTORY_TABLE_NAME);
+ table = createTable(hive, db);
+ }
+ return table;
+ } catch (HiveException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ protected abstract void postInitTable(Table table) throws Exception;
+
+ /**
+ * Supposed to create the query history table in metastore. It's only called from
+ * initTable if it doesn't exist yet.
+ */
+ protected abstract Table createTable(Hive hive, Database db) throws HiveException;
+
+ /**
+ * While creating the table, getInitialTable is supposed to return a common table object,
+ * which contains anything general that's not specific to QueryHistoryRepository subclasses.
+ */
+ protected Table getInitialTable() {
+ Table table = new Table(QUERY_HISTORY_DB_NAME, QUERY_HISTORY_TABLE_NAME);
+ table.setProperty("EXTERNAL", "TRUE");
+ table.setTableType(TableType.EXTERNAL_TABLE);
+
+ List<FieldSchema> partCols = new ArrayList<>();
+
+ Arrays.stream(QueryHistorySchema.Field.values()).filter(QueryHistorySchema.Field::isPartitioningCol)
Review Comment:
why don't you add `List<FieldSchema> getPartCols()` in QueryHistorySchema?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931100674
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/IcebergRepository.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Calendar;
+import java.util.GregorianCalendar;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Queue;
+import java.util.UUID;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.io.HiveFileFormatUtils;
+import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.HiveStorageHandler;
+import org.apache.hadoop.hive.ql.metadata.HiveUtils;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.security.authorization.HiveCustomStorageHandlerUtils;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.apache.hadoop.hive.serde2.Serializer;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TaskAttemptID;
+import org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IcebergRepository extends AbstractQueryHistoryRepository implements QueryHistoryRepository {
+ private static final Logger LOG = LoggerFactory.getLogger(IcebergRepository.class);
+ private static final String ICEBERG_STORAGE_HANDLER = "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler";
+
+ private HiveOutputFormat<?, ?> outputFormat;
+ private Serializer serializer;
+ @VisibleForTesting
+ HiveStorageHandler storageHandler;
+ @VisibleForTesting
+ TableDesc tableDesc;
+
+ @Override
Review Comment:
why override?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930389025
##########
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:
##########
@@ -755,6 +753,13 @@ int closeInProcess(boolean destroyed) {
// is called to stop the query if it is running, clean query results, and release resources.
@Override
public void close() {
+ if (driverContext.getConf().getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_ENABLED)) {
+ if (driverState.isClosed() || driverState.isDestroyed()){
+ LOG.warn("Driver instance {} already closed or destroyed, prevent handling query history", this);
+ } else{
+ QueryHistoryService.handle(driverContext);
Review Comment:
logQuery
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931026799
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/schema/QueryHistoryRecord.java:
##########
@@ -0,0 +1,358 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Strings;
+import org.apache.hadoop.hive.common.type.Timestamp;
+import org.apache.hadoop.hive.ql.QueryProperties;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema.Field;
+import org.apache.hadoop.hive.serde2.Serializer;
+import org.apache.hadoop.io.Writable;
+import org.apache.hive.common.util.HiveVersionInfo;
+
+public class QueryHistoryRecord {
+ private long estimatedSizeInMemoryBytes = 0;
+
+ public QueryHistoryRecord(){
+ set(Field.QUERY_HISTORY_SCHEMA_VERSION, QueryHistorySchema.CURRENT_VERSION);
+ set(Field.HIVE_VERSION, HiveVersionInfo.getVersion());
+ }
+
+ protected final Map<String, Object> record = new HashMap<>();
+
+ public void setQueryId(String queryId) {
+ set(QueryHistorySchema.Field.QUERY_ID, queryId);
+ }
+
+ public void setSessionId(String sessionId) {
+ set(QueryHistorySchema.Field.SESSION_ID, sessionId);
+ }
+
+ public void setOperationId(String operationId) {
+ set(QueryHistorySchema.Field.OPERATION_ID, operationId);
+ }
+
+ public void setExecutionEngine(String executionEngine) {
+ set(QueryHistorySchema.Field.EXECUTION_ENGINE, executionEngine);
+ }
+
+ public void setExecutionMode(String executionMode) {
+ set(QueryHistorySchema.Field.EXECUTION_MODE, executionMode);
+ }
+
+ public void setDagId(String dagId) {
+ set(QueryHistorySchema.Field.TEZ_DAG_ID, dagId);
+ }
+
+ public void setClusterId(String clusterId) {
+ set(Field.CLUSTER_ID, clusterId);
+ }
+
+ public void setTezApplicationId(String applicationId) {
+ set(QueryHistorySchema.Field.TEZ_APP_ID, applicationId);
+ }
+
+ public void setTezSessionId(String sessionId) {
+ set(QueryHistorySchema.Field.TEZ_SESSION_ID, sessionId);
+ }
+
+ public void setSessionType(String sessionType) {
+ set(QueryHistorySchema.Field.SESSION_TYPE, sessionType);
+ }
+
+ public void setClientProtocol(int clientProtocol) {
+ set(QueryHistorySchema.Field.HIVERSERVER2_PROTOCOL_VERSION, clientProtocol);
+ }
+
+ public void setClusterUser(String username) {
+ set(QueryHistorySchema.Field.CLUSTER_USER, username);
+ }
+
+ public void setEndUser(String userNameConnection) {
+ set(QueryHistorySchema.Field.END_USER, userNameConnection);
+ }
+
+ public void setQuerySql(String querySql) {
+ set(QueryHistorySchema.Field.SQL, querySql);
+ }
+
+ public void setCurrentDatabase(String currentDatabase) {
+ set(QueryHistorySchema.Field.DB_NAME, currentDatabase);
+ }
+
+ public void setTezAmAddress(String amAddress) {
+ set(QueryHistorySchema.Field.TEZ_COORDINATOR, amAddress);
+ }
+
+ public void setQueryState(String state) {
+ set(QueryHistorySchema.Field.QUERY_STATE, state);
+ }
+
+ public void setQueryType(QueryProperties.QueryType type) {
+ set(QueryHistorySchema.Field.QUERY_TYPE, type == null ? QueryProperties.QueryType.OTHER.getName() :
+ type.getName());
+ }
+
+ public void setDdlType(String ddlType) {
+ set(Field.DDL_TYPE, ddlType);
+ }
+
+ public void setServerAddress(String serverAddress) {
+ set(QueryHistorySchema.Field.SERVER_ADDRESS, serverAddress);
+ }
+
+ public void setServerPort(int serverPort) {
+ set(QueryHistorySchema.Field.SERVER_PORT, serverPort);
+ }
+
+ public void setClientAddress(String userIpAddress) {
+ set(QueryHistorySchema.Field.CLIENT_ADDRESS, userIpAddress);
+ }
+
+ public void setQueryStartTime(long beginTimeEpochMillis) {
+ set(Field.START_TIME_UTC, Timestamp.ofEpochMilli(beginTimeEpochMillis));
+ set(Field.START_TIME, Timestamp.ofEpochMilli(beginTimeEpochMillis, ZoneId.systemDefault()));
+ }
+
+ public void setQueryEndTime(long endTimeEpochMillis) {
+ set(Field.END_TIME_UTC, Timestamp.ofEpochMilli(endTimeEpochMillis));
+ set(Field.END_TIME, Timestamp.ofEpochMilli(endTimeEpochMillis, ZoneId.systemDefault()));
+ }
+
+ public void setTotalTime(long elapsedTime) {
+ set(Field.TOTAL_TIME_MS, elapsedTime);
+ }
+
+ public void setPlanningDuration(long duration) {
+ set(Field.PLANNING_DURATION, duration);
+ }
+
+ public void setPlanningStartTime(long startTime) {
+ set(Field.PLANNING_START_TIME, timeToNullDefault(startTime));
+ }
+
+ public void setPreparePlanDuration(long duration) {
+ set(Field.PREPARE_PLAN_DURATION, duration);
+ }
+
+ public void setPreparePlanStartTime(long startTime) {
+ set(Field.PREPARE_PLAN_START_TIME, timeToNullDefault(startTime));
+ }
+
+ public void setGetSessionDuration(long duration) {
+ set(Field.GET_SESSION_DURATION, duration);
+ }
+
+ public void setGetSessionStartTime(long startTime) {
+ set(Field.GET_SESSION_START_TIME, timeToNullDefault(startTime));
+ }
+
+ public void setExecutionDuration(long duration) {
+ set(Field.EXECUTION_DURATION, duration);
+ }
+
+ public void setExecutionStartTime(long startTime) {
+ set(Field.EXECUTION_START_TIME, timeToNullDefault(startTime));
+ }
+
+ public void setFailureReason(String errorMessage) {
+ set(Field.FAILURE_REASON, errorMessage);
+ }
+
+ public void setNumRowsFetched(int totalRows) {
+ set(Field.NUM_ROWS_FETCHED, totalRows);
+ }
+
+ public void setPlan(String explainPlan) {
+ set(Field.PLAN, explainPlan);
+ }
+
+
+ public void setTablesQueried(List<String> tablesQueried) {
+ set(Field.TABLES_QUERIED, String.join(",", tablesQueried));
+ }
+
+ public void setExecSummary(String summary) {
+ set(Field.EXEC_SUMMARY, summary);
+ }
+
+ public void setConfigurationOptionsChanged(Map<String, String> overriddenConfigurations) {
+ set(Field.CONFIGURATION_OPTIONS_CHANGED, Joiner.on(";").withKeyValueSeparator("=").join(overriddenConfigurations));
+ }
+
+ public void setTotalNumberOfTasks(int numTasks) {
+ set(Field.TOTAL_LAUNCHED_TASKS, numTasks);
+ }
+
+ public void setNumberOfSucceededTasks(int numSucceededTasks) {
+ set(Field.NUM_SUCCEEDED_TASKS, numSucceededTasks);
+ }
+
+ public void setNumberOfKilledTasks(int numKilledTasks) {
+ set(Field.NUM_KILLED_TASKS, numKilledTasks);
+ }
+
+ public void setNumberOfFailedTasks(int numFailedTasks) {
+ set(Field.NUM_FAILED_TASKS, numFailedTasks);
+ }
+
+ public void setNodeUsedCount(int nodeUsedCount) {
+ set(Field.NODE_USED_COUNT, nodeUsedCount);
+ }
+
+ public void setNodeTotalCount(int nodeTotalCount) {
+ set(Field.NODE_TOTAL_COUNT, nodeTotalCount);
+ }
+
+ public void setReduceInputGroups(long reduceInputGroups) {
+ set(Field.REDUCE_INPUT_GROUPS, reduceInputGroups);
+ }
+
+ public void setReduceInputRecords(long reduceInputRecords) {
+ set(Field.REDUCE_INPUT_RECORDS, reduceInputRecords);
+ }
+
+ public void setSpilledRecords(long spilledRecords) {
+ set(Field.SPILLED_RECORDS, spilledRecords);
+ }
+
+ public void setNumShuffledInputs(long numShuffledInputs) {
+ set(Field.NUM_SHUFFLED_INPUTS, numShuffledInputs);
+ }
+
+ public void setNumFailedShuffleInputs(long numFailedShuffleInputs) {
+ set(Field.NUM_FAILED_SHUFFLE_INPUTS, numFailedShuffleInputs);
+ }
+
+ public void setTaskDurationMillis(long taskDurationMillis) {
+ set(Field.TASK_DURATION_MILLIS, taskDurationMillis);
+ }
+
+ public void setInputRecordsProcessed(long inputRecordsProcessed) {
+ set(Field.INPUT_RECORDS_PROCESSED, inputRecordsProcessed);
+ }
+
+ public void setInputSplitLengthBytes(long inputSplitLengthBytes) {
+ set(Field.INPUT_SPLIT_LENGTH_BYTES, inputSplitLengthBytes);
+ }
+
+ public void setOutputRecords(long outputRecords) {
+ set(Field.OUTPUT_RECORDS, outputRecords);
+ }
+
+ public void setOutputBytesPhysical(long outputBytesPhysical) {
+ set(Field.OUTPUT_BYTES_PHYSICAL, outputBytesPhysical);
+ }
+
+ public void setShuffleChunkCount(long shuffleChunkCount) {
+ set(Field.SHUFFLE_CHUNK_COUNT, shuffleChunkCount);
+ }
+
+ public void setShuffleBytes(long shuffleBytes) {
+ set(Field.SHUFFLE_BYTES, shuffleBytes);
+ }
+
+ public void setShuffleBytesDiskDirect(long shuffleBytesDiskDirect) {
+ set(Field.SHUFFLE_BYTES_DISK_DIRECT, shuffleBytesDiskDirect);
+ }
+
+ public void setShufflePhaseTime(long shufflePhaseTime) {
+ set(Field.SHUFFLE_PHASE_TIME, shufflePhaseTime);
+ }
+
+ public void setMergePhaseTime(long phaseTime) {
+ set(Field.MERGE_PHASE_TIME, phaseTime);
+ }
+
+ public Object get(Field field) {
+ return record.get(field.name);
+ }
+
+ private void set(Field field, Object value) {
+ record.put(field.name, value);
+ }
+
+ public String toString() {
+ return String.format("QueryHistoryRecord [queryId: %s, sessionId: %s]", get(QueryHistorySchema.Field.QUERY_ID),
+ get(QueryHistorySchema.Field.SESSION_ID));
+ }
+
+ /**
+ * Utility toString kind of method for debugging.
+ * @return string representation of the underlying map
+ */
+ public String toLongString() {
Review Comment:
unused method + class has unused import
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2619004026
@deniskuzZ : addressed your most recent comments in https://github.com/apache/hive/pull/5613/commits/8c9f675651974eea065ec057c67ffca8b3b2626a
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1938987529
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2092,4 +2110,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
Review Comment:
should it be abstract?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939196213
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -330,6 +335,19 @@ public void analyze(ASTNode ast, Context ctx) throws SemanticException {
initCtx(ctx);
init(true);
analyzeInternal(ast);
+ postAnalyze();
+ }
+
+ /**
+ * This method is called after analyzeInternal, assuming all the semantic context is present (analyzeInternal is
+ * overridden by the subclasses of BaseSemanticAnalyzer).
+ * The reason why postAnalyze is private is that it's not supposed to be overridden.
+ */
+ private void postAnalyze() {
+ if (queryProperties == null) {
+ queryProperties = new QueryProperties();
+ }
+ queryProperties.setUsedTables(CacheTableHelper.getUniqueNames(ctx.getParsedTables()));
Review Comment:
yes, please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939579849
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
i don't see the change
````
I would expect to see something like
(Delete/Merge/Update)SemanticAnalyzer#getQueryType(){
return QueryType.DML;
}
ColumnStatsSemanticAnalyzer#getQueryType(){
return QueryType.STATS;
}
DDLSemanticAnalyzerFactory.getAnalyzer(
analyzer.setQueryType(QueryType.DDL);
}
etc
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1937636492
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/queryhistory/schema/IcebergQueryHistoryRecord.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+
+/**
+ * Convenient class for creating a QueryHistoryRecord from an iceberg Container.
+ */
+public class IcebergQueryHistoryRecord extends QueryHistoryRecord {
Review Comment:
this specific to single test class (TestIcebergRepository), so if this is overengineering, then QueryHistoryRecordBuilder is also overengineering, let me refactor this to a simple method in the test class
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/queryhistory/schema/IcebergQueryHistoryRecord.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+
+/**
+ * Convenient class for creating a QueryHistoryRecord from an iceberg Container.
+ */
+public class IcebergQueryHistoryRecord extends QueryHistoryRecord {
Review Comment:
this is specific to single test class (TestIcebergRepository), so if this is overengineering, then QueryHistoryRecordBuilder is also overengineering, let me refactor this to a simple method in the test class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939128911
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2092,4 +2110,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
+ (queryProperties.isDML() ?
+ QueryType.DML : null));
+ // common confusion whether CTAS is DML or DDL, let's pick DDL here
+ if (queryProperties.isCTAS()) {
+ queryProperties.setQueryType(QueryType.DDL);
+ }
+ if (queryProperties.getQueryType() != null) {
+ return; //already figured out
+ }
+ queryProperties.setQueryType(rootTasks.stream().findFirst().map(t -> {
+ String type = t.getType().toString();
+ // a MAPRED stage could mean an INSERT query also
+ if (queryProperties != null && type.equalsIgnoreCase("MAPRED")) {
+ return queryProperties.isDML() ? QueryType.DML : QueryType.QUERY;
+ }
+ return null;
+ }).orElseGet(() -> {
+ // in case of a semantic exception (e.g. a table not found or something else)
+ // the root AST Node can still imply if this is a query, try to fall back to that
+ // instead of ""
+ if (tree.getText().equalsIgnoreCase("TOK_QUERY")) {
Review Comment:
Generally it is better to compare the constant with the nullable value like
```
"TOK_QUERY".equalsIgnoreCase(tree.getText())
```
It is null safe.
Another option is
```
StringUtils.equalsIgnoreCase(tree.getText(), "TOK_QUERY")
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1941995705
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -330,6 +332,17 @@ public void analyze(ASTNode ast, Context ctx) throws SemanticException {
initCtx(ctx);
init(true);
analyzeInternal(ast);
+ postAnalyze();
+ }
+
+ /**
+ * This method is called after analyzeInternal, assuming all the semantic context is present (analyzeInternal is
+ * overridden by the subclasses of BaseSemanticAnalyzer).
+ * The reason why postAnalyze is private is that it's not supposed to be overridden.
+ */
+ private void postAnalyze() {
Review Comment:
if the build fails, please set the `usedTables` first and then `queryType`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1941870009
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java:
##########
@@ -786,6 +786,6 @@ private static Map<String, TypeInfo> collectStructFieldNames(ASTNode structTypeN
* @param node the ASTNode
*/
public static String getNodeName(ASTNode node) {
- return node == null ? "" : node.getText().substring("TOK_".length());
+ return node == null || node.getText() == null ? "" : node.getText().substring("TOK_".length());
Review Comment:
how about
````
return Optional.ofNullable(node)
.map(ASTNode::getText).map(text -> text.substring("TOK_".length()))
.orElse("");
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2596290094
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613) **Quality Gate passed**
Issues
 [104 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923835663
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.persist.QueryHistoryPersistor;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ private QueryHistoryPersistor queryHistoryPersistor;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+ private int maxBatchSize = 0;
+ private int maxQueueSizeInMemoryBytes = 0;
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService persistor thread").build());
+
+ private ServiceContext serviceContext;
+
+ public static QueryHistoryService start(HiveConf inputConf, ServiceContext serviceContext) {
Review Comment:
why do we create service instance in a start method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923528482
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java:
##########
@@ -27,17 +29,25 @@
import org.apache.hadoop.hive.metastore.api.TxnType;
import org.apache.hadoop.hive.ql.cache.results.CacheUsage;
import org.apache.hadoop.hive.ql.cache.results.QueryResultsCache.CacheEntry;
+import org.apache.hadoop.hive.ql.ddl.DDLSemanticAnalyzerFactory;
import org.apache.hadoop.hive.ql.exec.FetchTask;
import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.Task;
import org.apache.hadoop.hive.ql.exec.tez.TezRuntimeContext;
import org.apache.hadoop.hive.ql.exec.tez.TezTask;
import org.apache.hadoop.hive.ql.lockmgr.HiveTxnManager;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+import org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer;
+import org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer;
import org.apache.hadoop.hive.ql.plan.mapper.StatsSource;
/**
* Context for the procedure managed by the Driver.
*/
public class DriverContext {
+ public static final String DRIVER_DEFAULT_USER_NAME_PROP = "hive.driver.default.user.name";
Review Comment:
can we shorten the var name (remove `DRIVER_`), it's already in DriverContext
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923540112
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java:
##########
@@ -27,17 +29,25 @@
import org.apache.hadoop.hive.metastore.api.TxnType;
import org.apache.hadoop.hive.ql.cache.results.CacheUsage;
import org.apache.hadoop.hive.ql.cache.results.QueryResultsCache.CacheEntry;
+import org.apache.hadoop.hive.ql.ddl.DDLSemanticAnalyzerFactory;
import org.apache.hadoop.hive.ql.exec.FetchTask;
import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.Task;
import org.apache.hadoop.hive.ql.exec.tez.TezRuntimeContext;
import org.apache.hadoop.hive.ql.exec.tez.TezTask;
import org.apache.hadoop.hive.ql.lockmgr.HiveTxnManager;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
+import org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer;
+import org.apache.hadoop.hive.ql.parse.RewriteSemanticAnalyzer;
import org.apache.hadoop.hive.ql.plan.mapper.StatsSource;
/**
* Context for the procedure managed by the Driver.
*/
public class DriverContext {
+ public static final String DRIVER_DEFAULT_USER_NAME_PROP = "hive.driver.default.user.name";
Review Comment:
yeah, let me check
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923556771
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java:
##########
@@ -247,4 +275,107 @@ public String getQueryErrorMessage() {
public void setQueryErrorMessage(String queryErrorMessage) {
this.queryErrorMessage = queryErrorMessage;
}
+
+ public long getQueryStartTime() {
+ return getQueryInfo() != null ? getQueryInfo().getBeginTime() : getQueryDisplay().getQueryStartTime();
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY: a simple fetch task is a QUERY
+ * UNKNOWN: if we can't determine the type of the query:
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param sem the semantic analyzer which already analyzed the query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(BaseSemanticAnalyzer sem, ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = sem.getAllRootTasks();
+ QueryProperties props = sem.getQueryProperties();
+ queryType = isAnalyze(props) ? "STATS" : (isDML(props) ? "DML" : null);
+ // common confusion whether CTAS is DML or DDL, let's pick DDL here
+ if (props != null && props.isCTAS()) {
+ queryType = "DDL";
+ }
+ if (queryType != null) {
+ return; //already figured out
+ }
+ queryType = rootTasks.stream().findFirst().map(t -> {
+ String type = t.getType().toString();
+ // a MAPRED stage could mean an INSERT query also
+ if (props != null && type.equalsIgnoreCase("MAPRED")) {
+ return props.isInsert ? "DML" : "QUERY";
+ }
+ return type;
+ }).orElseGet(() -> {
+ // in case of a semantic exception (e.g. a table not found or something else)
+ // the root AST Node can still imply if this is a query, try to fall back to that
+ // instead of "UNKNOWN"
+ if (tree.getText().equalsIgnoreCase("TOK_QUERY")){
+ return "QUERY";
+ }
+ // if there is a fetch task, this is a query
+ if (sem.getFetchTask() != null) {
+ return "QUERY";
+ }
+ // from this point, best efforts come
+ // whether it's a DDL?
+ if (DDLSemanticAnalyzerFactory.handles(tree)) {
+ return "DDL";
+ }
+ // whether it's DML? UPDATE/DELETE/MERGE queries are handled with an instance of RewriteSemanticAnalyzer
+ if (RewriteSemanticAnalyzer.class.isAssignableFrom(sem.getClass())) {
+ return "DML";
+ }
+ return "UNKNOWN";
+ });
+
+ // in case of DDL queries, the DDL type can be figured out from the tree root token, e.g.:
+ // CREATETABLE, ALTERTABLE_ADDPARTS, SHOWDATABASES, SHOWTABLES, etc.
+ this.ddlType = "DDL".equalsIgnoreCase(queryType) ? tree.getText().substring("TOK_".length()) : "";
+ }
+
+ /**
+ * The return value of either isAnalyzeCommand() or isAnalyzeRewrite() is always true for analyze commands:
+ * isAnalyzeCommand=true for "compute statistics",
+ * isAnalyzeRewrite=true for "compute statistics for columns".
+ * @return
+ */
+ private boolean isAnalyze(QueryProperties queryProperties) {
+ return queryProperties != null && (queryProperties.isAnalyzeCommand() || queryProperties.isAnalyzeRewrite());
+ }
+
+ /**
+ * DML (Data Manipulation Language) operations are operations that modify data,
+ * queryProperties.isInsert seems to reflect this very well: true for INSERT [OVERWRITE], UPDATE, DELETE, MERGE
+ * @return
+ */
+ private boolean isDML(QueryProperties queryProperties) {
+ return queryProperties != null && queryProperties.isInsert;
+ }
+
+ public boolean isQueryHistoryExplainEnabled() {
+ return conf.getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_ENABLED) &&
+ conf.getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_EXPLAIN_PLAN_ENABLED);
+ }
+
+ public void setExplainPlan(String explainPlan) {
+ this.explainPlan = explainPlan;
+ }
+
+ public String getExplainPlan() {
+ return explainPlan;
+ }
+
+ public static String getUserNameFromConf(HiveConf conf) {
Review Comment:
why do you need it here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923544375
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java:
##########
@@ -137,13 +152,26 @@ public void setPlan(QueryPlan plan) {
this.runtimeContext = Utilities.getFirstTezTask(plan.getRootTasks())
.map(TezTask::getRuntimeContext)
.orElse(null);
+ this.queryProperties = plan.getQueryProperties();
}
}
+ public String getQueryType() {
+ return queryType == null ? "UNKNOWN" : queryType;
Review Comment:
a query must have a type under ideal circumstances, and if we failed to figure out, that's where "UNKNOWN" comes into the picture, whereas ddlType is not applicable at all for non-DDL queries (so not "unknown", just "nothing"), does this make sense? do you think it's better to queryType=="" as the dummy value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923569593
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java:
##########
@@ -355,6 +376,16 @@ public void clear() {
multiDestQuery = false;
filterWithSubQuery = false;
+
+ tablesQueried.clear();
}
+ /**
+ * This method extracts all query related information from the root QB from its state available after semantic
+ * analysis.
+ * @param qb the root QB of the analyzer
+ */
+ public void extractInfoFromQueryBlock(QB qb) {
+ qb.iterate(qbIt -> tablesQueried.addAll(qbIt.getTableAliasValues()));
Review Comment:
does it include the target table as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923756268
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java:
##########
@@ -458,6 +460,18 @@ private void printSummary(boolean success, Map<String, Progress> progressMap) {
}
}
+ public void endSummary() {
+ console.endSummary();
+ }
+
+ /**
+ * Collecting execution summary might be a bit expensive, only do it if needed for query history.
+ */
+ private boolean shouldCollectSummaryString() {
+ return hiveConf.getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_ENABLED) &&
Review Comment:
can we eagerly initialize var in TezJobMonitor constructor instead of evaluation from conf on every method call?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923762215
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##########
@@ -263,7 +263,7 @@
*/
@SuppressWarnings({"deprecation", "rawtypes"})
-public class Hive {
+public class Hive implements AutoCloseable {
Review Comment:
should it be Closeable or AutoCloseable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1927410504
##########
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:
##########
@@ -755,6 +753,13 @@ int closeInProcess(boolean destroyed) {
// is called to stop the query if it is running, clean query results, and release resources.
@Override
public void close() {
+ if (driverContext.getConf().getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_ENABLED)) {
+ if (driverState.isClosed() || driverState.isDestroyed()){
+ LOG.warn("Driver instance {} already closed or destroyed, prevent handling query history", this);
+ } else{
+ QueryHistoryService.handle(driverContext);
Review Comment:
renamed to handleQuery, I don't know better at the moment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2604381278
> What is the use case for that service? can't I check the query history in HUE or DAS (removed for some storage reason), etc Please take a look at #5319 which is being worked on by [rtrivedi12](https://github.com/rtrivedi12) I think it provides some extra details for an active queries cc @nrg4878
looks like #5319 is completely different, it uses a well-known SHOW PROCESSLIST for the live queries (live==recent==present in hs2 memory), whereas Query History Service is meant to be a scalable historical query service, scalable in a sense that it uses the iceberg table format
HUE/DAS might work from different sources, like the protobuf history, which's data source also created by a query hook, but this service aims to redesign the way of persisting data while trying to use the same or similar field names that has already been implemented by impala
the current HiveProtoLoggingHook contains much information about the storage details (e.g. rolling over files and stuff), which makes it look a bit less modern when compared to e.g. iceberg format, by which we win everything (in terms of performance for instance) that we achieve by working on integrating iceberg into our product
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923507011
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -5734,6 +5734,26 @@ public static enum ConfVars {
+ "a background update happens periodically to report the actual state of the query"),
HIVE_SCHEDULED_QUERIES_CREATE_AS_ENABLED("hive.scheduled.queries.create.as.enabled", true,
"This option sets the default behaviour of newly created scheduled queries."),
+ HIVE_QUERY_HISTORY_SERVICE_ENABLED("hive.query.history.service.enabled", true,
+ "Whether to start QueryHistoryService in HS2"),
+ HIVE_QUERY_HISTORY_SERVICE_EXPLAIN_PLAN_ENABLED("hive.query.history.service.explain.plan.enabled", true,
+ "Whether to collect and store explain plan in the query history. Default is true."),
+ HIVE_QUERY_HISTORY_SERVICE_EXEC_SUMMARY_ENABLED("hive.query.history.service.exec.summary.enabled", true,
+ "Whether to collect and store execution summary in the query history. Default is true."),
+ HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE("hive.query.history.service.persist.max.batch.size", 200,
+ "The maximum amount of records held in memory " +
+ "before query history service persists them to the target table. " +
+ "A small value (like 1-5) will lead to more real-time behavior with the price of small files. " +
+ "Set this to 0 to wait for the records to be persisted synchronously (not recommended in production)."),
+ HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_MEMORY_BYTES("hive.query.history.service.persist.max.memory.bytes",
+ 100 * 1024 * 1024, // 100MB
Review Comment:
I thought in the whole HS2, it's not, but we can reduce it to 20MB maybe?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923533397
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java:
##########
@@ -137,13 +152,26 @@ public void setPlan(QueryPlan plan) {
this.runtimeContext = Utilities.getFirstTezTask(plan.getRootTasks())
.map(TezTask::getRuntimeContext)
.orElse(null);
+ this.queryProperties = plan.getQueryProperties();
}
}
+ public String getQueryType() {
+ return queryType == null ? "UNKNOWN" : queryType;
Review Comment:
why `null` handling is different from the `getDdlType`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923523558
##########
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:
##########
@@ -558,11 +560,7 @@ private void setTriggerContext(String queryId) {
// query info is created by SQLOperation which will have start time of the operation. When JDBC Statement is not
// used queryInfo will be null, in which case we take creation of Driver instance as query start time (which is also
// the time when query display object is created)
- if (driverContext.getQueryInfo() != null) {
- queryStartTime = driverContext.getQueryInfo().getBeginTime();
- } else {
- queryStartTime = driverContext.getQueryDisplay().getQueryStartTime();
- }
+ queryStartTime = driverContext.getQueryStartTime();
Review Comment:
not sure why we need this change, but now above comment doesn't make any sence
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923526846
##########
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:
##########
@@ -755,6 +753,13 @@ int closeInProcess(boolean destroyed) {
// is called to stop the query if it is running, clean query results, and release resources.
@Override
public void close() {
+ if (driverContext.getConf().getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_ENABLED)) {
+ if (driverState.isClosed() || driverState.isDestroyed()){
+ LOG.warn("Driver instance {} already closed or destroyed, prevent handling query history", this);
+ } else{
+ QueryHistoryService.handle(driverContext);
Review Comment:
the method name is wage, what does it do `handle`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923539124
##########
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:
##########
@@ -755,6 +753,13 @@ int closeInProcess(boolean destroyed) {
// is called to stop the query if it is running, clean query results, and release resources.
@Override
public void close() {
+ if (driverContext.getConf().getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_ENABLED)) {
+ if (driverState.isClosed() || driverState.isDestroyed()){
+ LOG.warn("Driver instance {} already closed or destroyed, prevent handling query history", this);
+ } else{
+ QueryHistoryService.handle(driverContext);
Review Comment:
this is the main entrypoint of the service, where it gets the driverContext, and handles it (convert to QueryHistoryRecord, queue to persist), what name would make more sense here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923544375
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java:
##########
@@ -137,13 +152,26 @@ public void setPlan(QueryPlan plan) {
this.runtimeContext = Utilities.getFirstTezTask(plan.getRootTasks())
.map(TezTask::getRuntimeContext)
.orElse(null);
+ this.queryProperties = plan.getQueryProperties();
}
}
+ public String getQueryType() {
+ return queryType == null ? "UNKNOWN" : queryType;
Review Comment:
a query must have a type under ideal circumstances, and if we failed to figure out, that's where "UNKNOWN" comes into the picture, whereas ddlType is not applicable at all for non-DDL queries, does this make sense? do you think it's better to queryType=="" as the dummy value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923536118
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java:
##########
@@ -137,13 +152,26 @@ public void setPlan(QueryPlan plan) {
this.runtimeContext = Utilities.getFirstTezTask(plan.getRootTasks())
.map(TezTask::getRuntimeContext)
.orElse(null);
+ this.queryProperties = plan.getQueryProperties();
}
}
+ public String getQueryType() {
Review Comment:
i think those methods do not belong here, and maybe `QueryInfo` would be a. better place
cc @kasakrisz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923832602
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.persist.QueryHistoryPersistor;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ private QueryHistoryPersistor queryHistoryPersistor;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+ private int maxBatchSize = 0;
+ private int maxQueueSizeInMemoryBytes = 0;
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService persistor thread").build());
+
+ private ServiceContext serviceContext;
+
+ public static QueryHistoryService start(HiveConf inputConf, ServiceContext serviceContext) {
+ HiveConf conf = new HiveConf(inputConf);
+ // we can only hit this codepath if the INSTANCE is already initialized while calling start(...)
+ // for convenient test compatibility, this call is allowed to be idempotent (LOG.warn instead of exception)
+ if (INSTANCE != null) {
+ LOG.warn("There is already a QueryHistoryService instance ({}), returning existing instance", INSTANCE);
Review Comment:
why do we need to log this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923845729
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/persist/AbstractQueryHistoryPersistor.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory.persist;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.TableType;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.parse.PartitionTransform;
+import org.apache.hadoop.hive.ql.parse.TransformSpec;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public abstract class AbstractQueryHistoryPersistor implements QueryHistoryPersistor {
Review Comment:
usually such classes are called Repository (`QueryHistoryRepository`,`AbstractQueryHistoryRepository`)
````
As per [Eric Evansā book Domain-Driven Design](https://www.pearson.com/en-us/subject-catalog/p/domain-driven-design-tackling-complexity-in-the-heart-of-software/P200000009375/9780321125217), the ārepository is a mechanism for encapsulating storage, retrieval, and search behavior, which emulates a collection of objects.ā
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923849983
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##########
@@ -263,7 +263,7 @@
*/
@SuppressWarnings({"deprecation", "rawtypes"})
-public class Hive {
+public class Hive implements AutoCloseable {
Review Comment:
I used it in a try-with-resources block, hence it looked to be a good candidate for AutoCloseable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1925128843
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java:
##########
@@ -247,4 +275,107 @@ public String getQueryErrorMessage() {
public void setQueryErrorMessage(String queryErrorMessage) {
this.queryErrorMessage = queryErrorMessage;
}
+
+ public long getQueryStartTime() {
+ return getQueryInfo() != null ? getQueryInfo().getBeginTime() : getQueryDisplay().getQueryStartTime();
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY: a simple fetch task is a QUERY
+ * UNKNOWN: if we can't determine the type of the query:
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param sem the semantic analyzer which already analyzed the query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(BaseSemanticAnalyzer sem, ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = sem.getAllRootTasks();
+ QueryProperties props = sem.getQueryProperties();
+ queryType = isAnalyze(props) ? "STATS" : (isDML(props) ? "DML" : null);
+ // common confusion whether CTAS is DML or DDL, let's pick DDL here
+ if (props != null && props.isCTAS()) {
+ queryType = "DDL";
+ }
+ if (queryType != null) {
+ return; //already figured out
+ }
+ queryType = rootTasks.stream().findFirst().map(t -> {
+ String type = t.getType().toString();
+ // a MAPRED stage could mean an INSERT query also
+ if (props != null && type.equalsIgnoreCase("MAPRED")) {
+ return props.isInsert ? "DML" : "QUERY";
+ }
+ return type;
+ }).orElseGet(() -> {
+ // in case of a semantic exception (e.g. a table not found or something else)
+ // the root AST Node can still imply if this is a query, try to fall back to that
+ // instead of "UNKNOWN"
+ if (tree.getText().equalsIgnoreCase("TOK_QUERY")){
+ return "QUERY";
+ }
+ // if there is a fetch task, this is a query
+ if (sem.getFetchTask() != null) {
+ return "QUERY";
+ }
+ // from this point, best efforts come
+ // whether it's a DDL?
+ if (DDLSemanticAnalyzerFactory.handles(tree)) {
+ return "DDL";
+ }
+ // whether it's DML? UPDATE/DELETE/MERGE queries are handled with an instance of RewriteSemanticAnalyzer
+ if (RewriteSemanticAnalyzer.class.isAssignableFrom(sem.getClass())) {
+ return "DML";
+ }
+ return "UNKNOWN";
+ });
+
+ // in case of DDL queries, the DDL type can be figured out from the tree root token, e.g.:
+ // CREATETABLE, ALTERTABLE_ADDPARTS, SHOWDATABASES, SHOWTABLES, etc.
+ this.ddlType = "DDL".equalsIgnoreCase(queryType) ? tree.getText().substring("TOK_".length()) : "";
+ }
+
+ /**
+ * The return value of either isAnalyzeCommand() or isAnalyzeRewrite() is always true for analyze commands:
+ * isAnalyzeCommand=true for "compute statistics",
+ * isAnalyzeRewrite=true for "compute statistics for columns".
+ * @return
+ */
+ private boolean isAnalyze(QueryProperties queryProperties) {
+ return queryProperties != null && (queryProperties.isAnalyzeCommand() || queryProperties.isAnalyzeRewrite());
+ }
+
+ /**
+ * DML (Data Manipulation Language) operations are operations that modify data,
+ * queryProperties.isInsert seems to reflect this very well: true for INSERT [OVERWRITE], UPDATE, DELETE, MERGE
+ * @return
+ */
+ private boolean isDML(QueryProperties queryProperties) {
+ return queryProperties != null && queryProperties.isInsert;
+ }
+
+ public boolean isQueryHistoryExplainEnabled() {
+ return conf.getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_ENABLED) &&
Review Comment:
+ moved the convenience method to the HiveConf (which already has similar ones like isWebUiQueryInfoCacheEnabled)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931555728
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -5734,6 +5734,26 @@ public static enum ConfVars {
+ "a background update happens periodically to report the actual state of the query"),
HIVE_SCHEDULED_QUERIES_CREATE_AS_ENABLED("hive.scheduled.queries.create.as.enabled", true,
"This option sets the default behaviour of newly created scheduled queries."),
+ HIVE_QUERY_HISTORY_SERVICE_ENABLED("hive.query.history.service.enabled", true,
+ "Whether to start QueryHistoryService in HS2"),
+ HIVE_QUERY_HISTORY_SERVICE_EXPLAIN_PLAN_ENABLED("hive.query.history.service.explain.plan.enabled", true,
+ "Whether to collect and store explain plan in the query history. Default is true."),
+ HIVE_QUERY_HISTORY_SERVICE_EXEC_SUMMARY_ENABLED("hive.query.history.service.exec.summary.enabled", true,
+ "Whether to collect and store execution summary in the query history. Default is true."),
+ HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE("hive.query.history.service.persist.max.batch.size", 200,
+ "The maximum amount of records held in memory " +
+ "before query history service persists them to the target table. " +
+ "A small value (like 1-5) will lead to more real-time behavior with the price of small files. " +
+ "Set this to 0 to wait for the records to be persisted synchronously (not recommended in production)."),
Review Comment:
flush is always async (for a reason, the thread used by the persistExecutor has the session state set up for iceberg writes), the question is whether we wait for it or not
if I want to separate with, I'll have to share the condition between those, actually I'm fine with that
```
CountDownLatch latch = new CountDownLatch(1);
persistRecordsIfNeeded(latch);
maybeWaitForRecordsToBePersisted(maxBatchSize == 0, latch);
private void maybeWaitForRecordsToBePersisted(boolean shouldWait, CountDownLatch condition) {
if (shouldWait) {
try {
condition.await();
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
throw new RuntimeException(e);
}
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1934052718
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/IcebergRepository.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Calendar;
+import java.util.GregorianCalendar;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Queue;
+import java.util.UUID;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.io.HiveFileFormatUtils;
+import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.HiveStorageHandler;
+import org.apache.hadoop.hive.ql.metadata.HiveUtils;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.security.authorization.HiveCustomStorageHandlerUtils;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.apache.hadoop.hive.serde2.Serializer;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TaskAttemptID;
+import org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IcebergRepository extends AbstractQueryHistoryRepository implements QueryHistoryRepository {
+ private static final Logger LOG = LoggerFactory.getLogger(IcebergRepository.class);
+ private static final String ICEBERG_STORAGE_HANDLER = "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler";
+
+ private HiveOutputFormat<?, ?> outputFormat;
+ private Serializer serializer;
+ @VisibleForTesting
+ HiveStorageHandler storageHandler;
+ @VisibleForTesting
+ TableDesc tableDesc;
+
+ @Override
Review Comment:
it has just super call
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930898383
##########
service/src/java/org/apache/hive/service/servlet/QueriesRESTfulAPIServlet.java:
##########
@@ -33,6 +33,7 @@
import com.fasterxml.jackson.databind.SerializerProvider;
import com.fasterxml.jackson.databind.module.SimpleModule;
+
Review Comment:
revert this class, no changes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931030159
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/QueryHistoryRepository.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Queue;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+
+public interface QueryHistoryRepository {
+ String QUERY_HISTORY_DB_NAME = "sys";
+ String QUERY_HISTORY_TABLE_NAME = "query_history";
+ String QUERY_HISTORY_DB_TABLE_NAME =
+ String.format("%s.%s", QUERY_HISTORY_DB_NAME, QUERY_HISTORY_TABLE_NAME);
+
+ String QUERY_HISTORY_DB_COMMENT = "Hive SYS database";
Review Comment:
do we need this or could just use null
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930675845
##########
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:
##########
@@ -755,6 +746,13 @@ int closeInProcess(boolean destroyed) {
// is called to stop the query if it is running, clean query results, and release resources.
@Override
public void close() {
+ if (driverContext.getConf().getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_ENABLED)) {
Review Comment:
could we extract this block into a method say `logQueryHistory`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931209157
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -5734,6 +5734,26 @@ public static enum ConfVars {
+ "a background update happens periodically to report the actual state of the query"),
HIVE_SCHEDULED_QUERIES_CREATE_AS_ENABLED("hive.scheduled.queries.create.as.enabled", true,
"This option sets the default behaviour of newly created scheduled queries."),
+ HIVE_QUERY_HISTORY_SERVICE_ENABLED("hive.query.history.service.enabled", true,
+ "Whether to start QueryHistoryService in HS2"),
+ HIVE_QUERY_HISTORY_SERVICE_EXPLAIN_PLAN_ENABLED("hive.query.history.service.explain.plan.enabled", true,
+ "Whether to collect and store explain plan in the query history. Default is true."),
+ HIVE_QUERY_HISTORY_SERVICE_EXEC_SUMMARY_ENABLED("hive.query.history.service.exec.summary.enabled", true,
+ "Whether to collect and store execution summary in the query history. Default is true."),
+ HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE("hive.query.history.service.persist.max.batch.size", 200,
+ "The maximum amount of records held in memory " +
+ "before query history service persists them to the target table. " +
+ "A small value (like 1-5) will lead to more real-time behavior with the price of small files. " +
+ "Set this to 0 to wait for the records to be persisted synchronously (not recommended in production)."),
Review Comment:
it seems like 2 responsibilities here:
1. controls the needToPersist()
2. controls the async/sync flush
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931150150
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void handleQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ QueryHistoryRecord record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ persistRecordsIfNeeded();
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ persistWithBatchSize(maxBatchSize);
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(), maxBatchSize);
+ }
+ }
+
+ private void persistWithBatchSize(int batchSize) {
+ CountDownLatch latch = new CountDownLatch(1);
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ repository.persist(queryHistoryQueue);
+ latch.countDown();
+ });
+ if (batchSize == 0) {
+ waitForAllRecordsToBePersisted(latch);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted(CountDownLatch queueEmptyCondition) {
+ try {
+ queueEmptyCondition.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ boolean needToPersist() {
+ boolean needToPersistAccordingToQueueSize = queryHistoryQueue.size() >= maxBatchSize;
+ if (needToPersistAccordingToQueueSize) {
+ LOG.info("Need to persist query history records as queue size ({}) >= batch size ({})",
+ queryHistoryQueue.size(), maxBatchSize);
+ return true;
+ }
+
+ // whether ignore the consumed memory or not
+ if (maxQueueSizeInMemoryBytes == 0) {
+ return false;
+ }
+
+ long queueMemorySize =
+ queryHistoryQueue.stream().mapToLong(QueryHistoryRecord::getEstimatedSizeInMemoryBytes).sum();
+ boolean needToPersistAccordingToQueueMemorySize = queueMemorySize > maxQueueSizeInMemoryBytes;
+ if (needToPersistAccordingToQueueMemorySize) {
+ LOG.info("Need to persist query history records as queue size in memory ({} bytes) is greater than the limit " +
+ "({} bytes)",
+ queueMemorySize, maxQueueSizeInMemoryBytes);
+ }
+ return needToPersistAccordingToQueueMemorySize;
+ }
+
+ public void stop() {
+ LOG.info("Stopping QueryHistoryService, leftover records to persist: {}", queryHistoryQueue.size());
+ persistWithBatchSize(0); // persist sync
+ persistExecutor.shutdownNow();
+ }
+
+ private QueryHistoryRecord createRecord(DriverContext driverContext) {
+ return new QueryHistoryRecordEnricher(driverContext, serviceContext, SessionState.get(),
+ SessionState.getPerfLogger()).createRecord();
+ }
+
+ @VisibleForTesting
+ public QueryHistoryRepository getRepository(){
+ return repository;
+ }
+
+ @Override
+ public String toString() {
Review Comment:
why do you need `toString` on a service class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1927425373
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -600,6 +601,11 @@ default void storageHandlerCommit(Properties commitProperties, Operation operati
throw new UnsupportedOperationException();
}
+ default void storageHandlerCommit(Properties commitProperties, Operation operation, TaskAttemptContext context)
Review Comment:
copying response from another comment:
_we cannot depend on iceberg in hive-exec in compile-time, there is no iceberg* dependency in hive-exec/pom.xml, that's what's hidden by the storagehandler layer, I think, even if I really like iceberg's api, I'm limited to use the storagehandler_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2612783611
> > > > > What is the use case for that service? can't I check the query history in HUE or DAS (removed for some storage reason), etc Please take a look at #5319 which is being worked on by [rtrivedi12](https://github.com/rtrivedi12) I think it provides some extra details for an active queries cc @nrg4878
> > > >
> > > >
> > > > looks like #5319 is completely different, it uses a well-known SHOW PROCESSLIST for the live queries (live==recent==present in hs2 memory), whereas Query History Service is meant to be a scalable historical query service, scalable in a sense that it uses the iceberg table format
> > > > HUE/DAS might work from different sources, like the protobuf history, which's data source also created by a query hook, but this service aims to redesign the way of persisting data while trying to use the same or similar field names that has already been implemented by impala
> > > > the current HiveProtoLoggingHook contains much information about the storage details (e.g. rolling over files and stuff), which makes it look a bit less modern when compared to e.g. iceberg format, by which we win everything (in terms of performance for instance) that we achieve by working on integrating iceberg into our product
> > >
> > >
> > > is that supposed to do the same thing as Impala profile: https://github.com/apache/impala/blob/fdc43466350db4437b3e917d4ff24dac58af63c3/testdata/impala-profiles/impala_profile_log_tpcds_compute_stats_v2_default.expected.txt#L1445?
> >
> >
> > if you mean the corresponding impala table, that's implemented in: https://issues.apache.org/jira/browse/IMPALA-12426 I can see different upstream commits for that, this is the closest one: [apache/impala@711a9f2](https://github.com/apache/impala/commit/711a9f2bad84f92dc4af61d49ae115f0dc4239da)
> > their table is sys.impala_query_log
>
> have you checked how did they implement it, maybe used some tricks to improve perf/make it a background activity, etc?
in detail no, we discussed high-level problems, like what the schema would look like and how the table is supposed to be compacted (it's not automatically, it should be taken care of by the platform)
when it comes to performance details, what I tried to achieve is:
1. transform the query data to query history record - this is sync I admit, let me add extra logging:
```
long start = Time.monotonicNow();
QueryHistoryRecord record = createRecord(driverContext);
LOG.debug("Created history record (in {}ms): {}", Time.monotonicNow() - start, record);
```
2. the rest of the work is done async
3. set maxBatchSize to 100 by default and defined memory limit too, so every 100 query records should be written in one batch...I felt this a good tradeoff between too many small files vs. too rare writes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1927056806
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.persist.QueryHistoryPersistor;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ private QueryHistoryPersistor queryHistoryPersistor;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+ private int maxBatchSize = 0;
+ private int maxQueueSizeInMemoryBytes = 0;
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService persistor thread").build());
+
+ private ServiceContext serviceContext;
+
+ public static QueryHistoryService start(HiveConf inputConf, ServiceContext serviceContext) {
+ HiveConf conf = new HiveConf(inputConf);
+ // we can only hit this codepath if the INSTANCE is already initialized while calling start(...)
+ // for convenient test compatibility, this call is allowed to be idempotent (LOG.warn instead of exception)
+ if (INSTANCE != null) {
+ LOG.warn("There is already a QueryHistoryService instance ({}), returning existing instance", INSTANCE);
+ return INSTANCE;
+ }
+
+ INSTANCE = createServiceInstance(serviceContext, conf);
+ return INSTANCE;
+ }
+
+ private static QueryHistoryService createServiceInstance(ServiceContext serviceContext, HiveConf conf) {
+ LOG.info("Starting QueryHistoryService");
+ QueryHistoryService queryHistoryService = new QueryHistoryService();
+
+ initService(serviceContext, conf, queryHistoryService);
+ return queryHistoryService;
+ }
+
+ @VisibleForTesting
+ static void initService(ServiceContext serviceContext, HiveConf conf, QueryHistoryService queryHistoryService) {
+ queryHistoryService.queryHistoryPersistor = createPersistor(conf);
+ queryHistoryService.queryHistoryPersistor.init(conf, new QueryHistorySchema());
+ queryHistoryService.maxBatchSize = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE);
+ queryHistoryService.maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_MEMORY_BYTES);
+
+ queryHistoryService.printConfigInformation();
+ queryHistoryService.serviceContext = serviceContext;
+
+ try {
+ queryHistoryService.persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the persistor,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
Review Comment:
SessionState is bound to a thread, and persistExecutor is a single-threaded executor by design, so this task inside it takes care of connecting the conf present in QueryHistoryService to be present for all the repository tasks,that's what I meant by "localize this conf for the thread that will be running the repository"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1938318348
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.Record;
+import org.apache.hadoop.hive.ql.queryhistory.schema.Schema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<Record> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService flushExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+ private final ScheduledExecutorService periodicFlushExecutor = Executors.newScheduledThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService periodic flush check").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int batchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final int flushInterval;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ batchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_MAX_MEMORY_BYTES);
+ flushInterval = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_FLUSH_INTERVAL_SECONDS);
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [batchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ batchSize, maxQueueSizeInMemoryBytes);
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ INSTANCE = this;
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for flushing records.
+ */
+ public void start() {
+ repository.init(conf, new Schema());
+
+ try {
+ flushExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ if (flushInterval > 0) {
+ // only flush, don't wait, this is a background task
+ periodicFlushExecutor.scheduleAtFixedRate(this::doFlush, 0, flushInterval, TimeUnit.SECONDS);
+ }
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ private void printConfigInformation() {
+ if (batchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be flushed synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be flushed after every {} queries",
+ batchSize, batchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "flush decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be flushed when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+
+ if (flushInterval == 0) {
+ LOG.info("Query History flush interval size is 0: the service won't check for records to flush periodically");
+ } else {
+ LOG.info("Query History flush interval is {}s: service will attempt to flush records in every {}s",
+ flushInterval, flushInterval);
+ }
+ }
+
+ private QueryHistoryRepository createRepository(HiveConf conf) {
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void logQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ Record record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ CompletableFuture<?> flushFuture = flushRecordsIfNeeded();
+ maybeWaitForRecordsToBeFlushed(batchSize == 0, flushFuture);
+ }
+
+ private CompletableFuture<?> flushRecordsIfNeeded() {
+ if (needToFlush()) {
+ return doFlush();
+ } else {
+ LOG.debug("Not flushing history records yet, current queue size: {}, batchSize: {}",
+ queryHistoryQueue.size(), batchSize);
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+
+ private void maybeWaitForRecordsToBeFlushed(boolean shouldWait, CompletableFuture<?> flushFuture) {
Review Comment:
yes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1937584605
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.Record;
+import org.apache.hadoop.hive.ql.queryhistory.schema.Schema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<Record> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService flushExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+ private final ScheduledExecutorService periodicFlushExecutor = Executors.newScheduledThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService periodic flush check").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int batchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final int flushInterval;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ batchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_MAX_MEMORY_BYTES);
+ flushInterval = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_FLUSH_INTERVAL_SECONDS);
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [batchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ batchSize, maxQueueSizeInMemoryBytes);
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ INSTANCE = this;
Review Comment:
how about
````
private static Singleton instance;
private QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
...
}
public static QueryHistoryService newInstance(HiveConf inputConf, ServiceContext serviceContext) {
if (instance == null) {
instance = new QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext);
}
return instance;
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1937611329
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.Record;
+import org.apache.hadoop.hive.ql.queryhistory.schema.Schema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<Record> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService flushExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+ private final ScheduledExecutorService periodicFlushExecutor = Executors.newScheduledThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService periodic flush check").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int batchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final int flushInterval;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ batchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_MAX_MEMORY_BYTES);
+ flushInterval = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_FLUSH_INTERVAL_SECONDS);
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [batchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ batchSize, maxQueueSizeInMemoryBytes);
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ INSTANCE = this;
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for flushing records.
+ */
+ public void start() {
+ repository.init(conf, new Schema());
+
+ try {
+ flushExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ if (flushInterval > 0) {
+ // only flush, don't wait, this is a background task
+ periodicFlushExecutor.scheduleAtFixedRate(this::doFlush, 0, flushInterval, TimeUnit.SECONDS);
+ }
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ private void printConfigInformation() {
+ if (batchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be flushed synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be flushed after every {} queries",
+ batchSize, batchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "flush decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be flushed when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+
+ if (flushInterval == 0) {
+ LOG.info("Query History flush interval size is 0: the service won't check for records to flush periodically");
+ } else {
+ LOG.info("Query History flush interval is {}s: service will attempt to flush records in every {}s",
+ flushInterval, flushInterval);
+ }
+ }
+
+ private QueryHistoryRepository createRepository(HiveConf conf) {
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void logQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ Record record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ CompletableFuture<?> flushFuture = flushRecordsIfNeeded();
+ maybeWaitForRecordsToBeFlushed(batchSize == 0, flushFuture);
+ }
+
+ private CompletableFuture<?> flushRecordsIfNeeded() {
+ if (needToFlush()) {
+ return doFlush();
+ } else {
+ LOG.debug("Not flushing history records yet, current queue size: {}, batchSize: {}",
+ queryHistoryQueue.size(), batchSize);
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+
+ private void maybeWaitForRecordsToBeFlushed(boolean shouldWait, CompletableFuture<?> flushFuture) {
Review Comment:
can we drop `maybe` from the name, just `wait`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1938437539
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java:
##########
@@ -74,6 +101,12 @@ public class QueryProperties {
private boolean isMaterializedView;
private boolean isView;
+ private QueryType queryType;
+ private String ddlType;
+
+ // set of queried tables, aliases are resolved to real table names
+ private List<String> tablesQueried = new ArrayList<>();
Review Comment:
right, it's also written tables, let me rename the field also to used_tables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1938436319
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java:
##########
@@ -20,6 +20,7 @@
import java.io.IOException;
import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
Review Comment:
realized this was refactored by the IDE, now it's correct alphabetical ordering point of view, can I leave it like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1938439057
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2047,8 +2070,8 @@ public void startAnalysis() {
/**
* Called when we end analysis of a query.
*/
- public void endAnalysis() {
- // Nothing to do
+ public void endAnalysis(ASTNode tree) {
+ setQueryType(tree); // at this point we know the query type for sure
Review Comment:
we cannot, analyzeInternal is overriden by all the semantic analyzer children, and they are not supposed to be aware of this thing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1938443193
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void handleQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ QueryHistoryRecord record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ persistRecordsIfNeeded();
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ persistWithBatchSize(maxBatchSize);
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(), maxBatchSize);
+ }
+ }
+
+ private void persistWithBatchSize(int batchSize) {
+ CountDownLatch latch = new CountDownLatch(1);
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ repository.persist(queryHistoryQueue);
+ latch.countDown();
+ });
+ if (batchSize == 0) {
+ waitForAllRecordsToBePersisted(latch);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted(CountDownLatch queueEmptyCondition) {
+ try {
+ queueEmptyCondition.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ boolean needToPersist() {
+ boolean needToPersistAccordingToQueueSize = queryHistoryQueue.size() >= maxBatchSize;
+ if (needToPersistAccordingToQueueSize) {
+ LOG.info("Need to persist query history records as queue size ({}) >= batch size ({})",
+ queryHistoryQueue.size(), maxBatchSize);
+ return true;
+ }
+
+ // whether ignore the consumed memory or not
+ if (maxQueueSizeInMemoryBytes == 0) {
+ return false;
+ }
+
+ long queueMemorySize =
+ queryHistoryQueue.stream().mapToLong(QueryHistoryRecord::getEstimatedSizeInMemoryBytes).sum();
+ boolean needToPersistAccordingToQueueMemorySize = queueMemorySize > maxQueueSizeInMemoryBytes;
+ if (needToPersistAccordingToQueueMemorySize) {
+ LOG.info("Need to persist query history records as queue size in memory ({} bytes) is greater than the limit " +
+ "({} bytes)",
+ queueMemorySize, maxQueueSizeInMemoryBytes);
+ }
+ return needToPersistAccordingToQueueMemorySize;
+ }
+
+ public void stop() {
+ LOG.info("Stopping QueryHistoryService, leftover records to persist: {}", queryHistoryQueue.size());
+ persistWithBatchSize(0); // persist sync
+ persistExecutor.shutdownNow();
+ }
+
+ private QueryHistoryRecord createRecord(DriverContext driverContext) {
+ return new QueryHistoryRecordEnricher(driverContext, serviceContext, SessionState.get(),
+ SessionState.getPerfLogger()).createRecord();
+ }
+
+ @VisibleForTesting
+ public QueryHistoryRepository getRepository(){
Review Comment:
double-checked, used in TestIcebergRepository
```
IcebergRepository repository = (IcebergRepository) service.getRepository();
```
this is needed there as I'm persisting records there directly (without considering the different strategies)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939151452
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
IMHO this should be a utility method only if it is also used outside of SAs.
BTW can we override it in subclasses of SA? IIUC the type of SA has an impact on the query type.
* Lots of query types are handled by `SemanticAnalyzer` like queries (DQL) and inserts.
* but others like Update/Delete/Merge are handled by `RewriteSemanticAnalyzer` So `RewriteSemanticAnalyzer. setQueryType` always sets `QueryType.DML`
* Stats goes to `ColumnStatsSemanticAnalyzer`
* DDL: everything created by `DDLSemanticAnalyzerFactory` always sets `QueryType.DDL`. Create MV is still in `SemanticAnalyzer` so that must be handled there.
* etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939610218
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
we still need massive `setQueryType` in base SA?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2635458909
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613) **Quality Gate passed**
Issues
 [87 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1940511126
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
I am not sure why we are using `CalcitePlanner` for DDL, but if so, we can set `queryType` and `ddlType` as individual properties instead of `if-else` in BaseSA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939016082
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
I think it should be done something like
````
driverContext.setQueryType(
Utils.getQueryType(tree, sem);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2629313016
> @abstractdog, are we reporting queueName here? do we need fixes from https://issues.apache.org/jira/secure/attachment/12933964/HIVE-20282.01.patch
no, yarn queue name is not included in the initial implementation of query history
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939758977
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
https://github.com/apache/hive/pull/5613/commits/18f9546279df6d476eb11a99775f854de89390bb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939201455
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2092,4 +2110,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
Review Comment:
wait, this is not even needed anymore (it was only needed when I made the recursive QB traversal in the BaseSemanticAnalyzer), removing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2631143259
thanks for the comments @kasakrisz , @deniskuzZ : I addressed them in https://github.com/apache/hive/pull/5613/commits/9fc0b07baf13c167917788335e1fd31690d7f4b3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930686285
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java:
##########
@@ -74,6 +101,12 @@ public class QueryProperties {
private boolean isMaterializedView;
private boolean isView;
+ private QueryType queryType;
+ private String ddlType;
+
+ // set of queried tables, aliases are resolved to real table names
+ private List<String> tablesQueried = new ArrayList<>();
Review Comment:
is it only read tables, not write?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930686285
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java:
##########
@@ -74,6 +101,12 @@ public class QueryProperties {
private boolean isMaterializedView;
private boolean isView;
+ private QueryType queryType;
+ private String ddlType;
+
+ // set of queried tables, aliases are resolved to real table names
+ private List<String> tablesQueried = new ArrayList<>();
Review Comment:
is it only read tables, not write? maybe rename to "underlying/usedTables"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930492394
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/queryhistory/repository/TestIcebergRepository.java:
##########
@@ -0,0 +1,115 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one§
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchemaTestUtils;
+import org.apache.iceberg.mr.mapred.Container;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.queryhistory.QueryHistoryService;
+import org.apache.hadoop.hive.ql.queryhistory.repository.IcebergRepositoryForTest;
+import org.apache.hadoop.hive.ql.queryhistory.schema.DummyQueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.IcebergQueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileInputFormat;
+import org.apache.hadoop.mapred.InputFormat;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hadoop.mapred.Reporter;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.Queue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+public class TestIcebergRepository {
Review Comment:
why iceberg, isn't it implementation details, maybe TestQueryHistoryRepository
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930421202
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -838,6 +845,9 @@ public void storageHandlerCommit(Properties commitProperties, Operation operatio
}
HiveIcebergOutputCommitter committer = getOutputCommitter();
try {
+ if (context != null) {
+ committer.commitTask(context);
Review Comment:
can't we get `HiveIcebergOutputCommitter` from `TaskAttemptContext context`?
````
OutputCommitter commiter = getOutputCommitter(context.getConfiguration());
committer.commitTask(context);
....
public static OutputCommitter getOutputCommitter(Configuration conf) {
return (OutputCommitter)ReflectionUtils.newInstance(this.getClass("mapred.output.committer.class", FileOutputCommitter.class, OutputCommitter.class), conf);
}
````
see `FileOutputFormatContainer# getOutputCommitter(TaskAttemptContext context)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930421202
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -838,6 +845,9 @@ public void storageHandlerCommit(Properties commitProperties, Operation operatio
}
HiveIcebergOutputCommitter committer = getOutputCommitter();
try {
+ if (context != null) {
+ committer.commitTask(context);
Review Comment:
can't we expose `HiveIcebergOutputCommitter` via get method?
````
OutputCommitter commiter = table.getStorageHandler().getOutputCommitter();
committer.commitTask(context);
````
or better `new JobConf(context.getConfiguration()).getOutputCommitter()`, see `FileOutputFormatContainer# getOutputCommitter(TaskAttemptContext context)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930568274
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/queryhistory/repository/IcebergRepositoryForTest.java:
##########
@@ -0,0 +1,34 @@
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one§
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.queryhistory.repository.IcebergRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+
+/*
+ * A test iceberg repository which has the same behavior as parent but without any metastore locking
+ * (for easier unit test's sake).
+ */
+public class IcebergRepositoryForTest extends IcebergRepository {
Review Comment:
why do we need this? simply set that config in test setup.
It's not very clear what you test - production code or test util in (TestIcebergRepository)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930568274
##########
itests/util/src/main/java/org/apache/hadoop/hive/ql/queryhistory/repository/IcebergRepositoryForTest.java:
##########
@@ -0,0 +1,34 @@
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one§
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.queryhistory.repository.IcebergRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+
+/*
+ * A test iceberg repository which has the same behavior as parent but without any metastore locking
+ * (for easier unit test's sake).
+ */
+public class IcebergRepositoryForTest extends IcebergRepository {
Review Comment:
why do we need this? simply set that config in test setup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930905233
##########
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java:
##########
@@ -1176,6 +1176,9 @@ public static class LogHelper {
protected Logger LOG;
protected boolean isSilent;
+ private final StringBuilder summary = new StringBuilder();
+ private boolean collectSummary;
Review Comment:
ok, `querySummary` & `collectQuerySummary`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923899930
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/persist/IcebergPersistor.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.persist;
+
+import java.util.Calendar;
+import java.util.GregorianCalendar;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Queue;
+import java.util.UUID;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.io.HiveFileFormatUtils;
+import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.HiveStorageHandler;
+import org.apache.hadoop.hive.ql.metadata.HiveUtils;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.security.authorization.HiveCustomStorageHandlerUtils;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.apache.hadoop.hive.serde2.Serializer;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TaskAttemptID;
+import org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IcebergPersistor extends AbstractQueryHistoryPersistor implements QueryHistoryPersistor {
+ private static final Logger LOG = LoggerFactory.getLogger(IcebergPersistor.class);
+ private static final String ICEBERG_STORAGE_HANDLER = "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler";
+
+ private HiveOutputFormat<?, ?> outputFormat;
+ private Serializer serializer;
+ @VisibleForTesting
+ HiveStorageHandler storageHandler;
+ @VisibleForTesting
+ TableDesc tableDesc;
+
+ @Override
+ public void init(HiveConf conf, QueryHistorySchema schema) {
+ super.init(conf, schema);
+ }
+
+ @Override
+ protected Table createTable(Hive hive, Database db) throws HiveException {
Review Comment:
why can't we use Hive SQL?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923583067
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java:
##########
@@ -355,6 +376,16 @@ public void clear() {
multiDestQuery = false;
filterWithSubQuery = false;
+
+ tablesQueried.clear();
}
+ /**
+ * This method extracts all query related information from the root QB from its state available after semantic
+ * analysis.
+ * @param qb the root QB of the analyzer
+ */
+ public void extractInfoFromQueryBlock(QB qb) {
+ qb.iterate(qbIt -> tablesQueried.addAll(qbIt.getTableAliasValues()));
Review Comment:
it supposed to, let me check and add unit test
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java:
##########
@@ -355,6 +376,16 @@ public void clear() {
multiDestQuery = false;
filterWithSubQuery = false;
+
+ tablesQueried.clear();
}
+ /**
+ * This method extracts all query related information from the root QB from its state available after semantic
+ * analysis.
+ * @param qb the root QB of the analyzer
+ */
+ public void extractInfoFromQueryBlock(QB qb) {
+ qb.iterate(qbIt -> tablesQueried.addAll(qbIt.getTableAliasValues()));
Review Comment:
it is supposed to, let me check and add unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923582358
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java:
##########
@@ -337,13 +337,14 @@ public int execute() {
&& (HiveConf.getBoolVar(conf, HiveConf.ConfVars.TEZ_EXEC_SUMMARY) ||
Utilities.isPerfOrAboveLogging(conf))) {
for (CounterGroup group : runtimeContext.counters) {
- LOG.info(group.getDisplayName() +":");
- for (TezCounter counter: group) {
- LOG.info(" "+counter.getDisplayName()+": "+counter.getValue());
+ monitor.printInfo(group.getDisplayName() + ":");
+ for (TezCounter counter : group) {
+ monitor.printInfo(" " + counter.getDisplayName() + ": " + counter.getValue());
}
}
}
updateNumRows();
+ monitor.endSummary();
Review Comment:
not sure what it does, but maybe we should call it finally?
what if `updateNumRows` throws an exception, previously we added the LOGS incrementally, what happens now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931098507
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/schema/QueryHistoryRecord.java:
##########
@@ -0,0 +1,358 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Strings;
+import org.apache.hadoop.hive.common.type.Timestamp;
+import org.apache.hadoop.hive.ql.QueryProperties;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema.Field;
+import org.apache.hadoop.hive.serde2.Serializer;
+import org.apache.hadoop.io.Writable;
+import org.apache.hive.common.util.HiveVersionInfo;
+
+public class QueryHistoryRecord {
+ private long estimatedSizeInMemoryBytes = 0;
+
+ public QueryHistoryRecord(){
+ set(Field.QUERY_HISTORY_SCHEMA_VERSION, QueryHistorySchema.CURRENT_VERSION);
+ set(Field.HIVE_VERSION, HiveVersionInfo.getVersion());
+ }
+
+ protected final Map<String, Object> record = new HashMap<>();
+
+ public void setQueryId(String queryId) {
+ set(QueryHistorySchema.Field.QUERY_ID, queryId);
+ }
+
+ public void setSessionId(String sessionId) {
+ set(QueryHistorySchema.Field.SESSION_ID, sessionId);
+ }
+
+ public void setOperationId(String operationId) {
+ set(QueryHistorySchema.Field.OPERATION_ID, operationId);
+ }
+
+ public void setExecutionEngine(String executionEngine) {
+ set(QueryHistorySchema.Field.EXECUTION_ENGINE, executionEngine);
+ }
+
+ public void setExecutionMode(String executionMode) {
+ set(QueryHistorySchema.Field.EXECUTION_MODE, executionMode);
+ }
+
+ public void setDagId(String dagId) {
+ set(QueryHistorySchema.Field.TEZ_DAG_ID, dagId);
+ }
+
+ public void setClusterId(String clusterId) {
+ set(Field.CLUSTER_ID, clusterId);
+ }
+
+ public void setTezApplicationId(String applicationId) {
+ set(QueryHistorySchema.Field.TEZ_APP_ID, applicationId);
+ }
+
+ public void setTezSessionId(String sessionId) {
+ set(QueryHistorySchema.Field.TEZ_SESSION_ID, sessionId);
+ }
+
+ public void setSessionType(String sessionType) {
+ set(QueryHistorySchema.Field.SESSION_TYPE, sessionType);
+ }
+
+ public void setClientProtocol(int clientProtocol) {
+ set(QueryHistorySchema.Field.HIVERSERVER2_PROTOCOL_VERSION, clientProtocol);
+ }
+
+ public void setClusterUser(String username) {
+ set(QueryHistorySchema.Field.CLUSTER_USER, username);
+ }
+
+ public void setEndUser(String userNameConnection) {
+ set(QueryHistorySchema.Field.END_USER, userNameConnection);
+ }
+
+ public void setQuerySql(String querySql) {
+ set(QueryHistorySchema.Field.SQL, querySql);
+ }
+
+ public void setCurrentDatabase(String currentDatabase) {
+ set(QueryHistorySchema.Field.DB_NAME, currentDatabase);
+ }
+
+ public void setTezAmAddress(String amAddress) {
+ set(QueryHistorySchema.Field.TEZ_COORDINATOR, amAddress);
+ }
+
+ public void setQueryState(String state) {
+ set(QueryHistorySchema.Field.QUERY_STATE, state);
+ }
+
+ public void setQueryType(QueryProperties.QueryType type) {
+ set(QueryHistorySchema.Field.QUERY_TYPE, type == null ? QueryProperties.QueryType.OTHER.getName() :
+ type.getName());
+ }
+
+ public void setDdlType(String ddlType) {
+ set(Field.DDL_TYPE, ddlType);
+ }
+
+ public void setServerAddress(String serverAddress) {
+ set(QueryHistorySchema.Field.SERVER_ADDRESS, serverAddress);
+ }
+
+ public void setServerPort(int serverPort) {
+ set(QueryHistorySchema.Field.SERVER_PORT, serverPort);
+ }
+
+ public void setClientAddress(String userIpAddress) {
+ set(QueryHistorySchema.Field.CLIENT_ADDRESS, userIpAddress);
+ }
+
+ public void setQueryStartTime(long beginTimeEpochMillis) {
+ set(Field.START_TIME_UTC, Timestamp.ofEpochMilli(beginTimeEpochMillis));
+ set(Field.START_TIME, Timestamp.ofEpochMilli(beginTimeEpochMillis, ZoneId.systemDefault()));
+ }
+
+ public void setQueryEndTime(long endTimeEpochMillis) {
+ set(Field.END_TIME_UTC, Timestamp.ofEpochMilli(endTimeEpochMillis));
+ set(Field.END_TIME, Timestamp.ofEpochMilli(endTimeEpochMillis, ZoneId.systemDefault()));
+ }
+
+ public void setTotalTime(long elapsedTime) {
+ set(Field.TOTAL_TIME_MS, elapsedTime);
+ }
+
+ public void setPlanningDuration(long duration) {
+ set(Field.PLANNING_DURATION, duration);
+ }
+
+ public void setPlanningStartTime(long startTime) {
+ set(Field.PLANNING_START_TIME, timeToNullDefault(startTime));
+ }
+
+ public void setPreparePlanDuration(long duration) {
+ set(Field.PREPARE_PLAN_DURATION, duration);
+ }
+
+ public void setPreparePlanStartTime(long startTime) {
+ set(Field.PREPARE_PLAN_START_TIME, timeToNullDefault(startTime));
+ }
+
+ public void setGetSessionDuration(long duration) {
+ set(Field.GET_SESSION_DURATION, duration);
+ }
+
+ public void setGetSessionStartTime(long startTime) {
+ set(Field.GET_SESSION_START_TIME, timeToNullDefault(startTime));
+ }
+
+ public void setExecutionDuration(long duration) {
+ set(Field.EXECUTION_DURATION, duration);
+ }
+
+ public void setExecutionStartTime(long startTime) {
+ set(Field.EXECUTION_START_TIME, timeToNullDefault(startTime));
+ }
+
+ public void setFailureReason(String errorMessage) {
+ set(Field.FAILURE_REASON, errorMessage);
+ }
+
+ public void setNumRowsFetched(int totalRows) {
+ set(Field.NUM_ROWS_FETCHED, totalRows);
+ }
+
+ public void setPlan(String explainPlan) {
+ set(Field.PLAN, explainPlan);
+ }
+
+
+ public void setTablesQueried(List<String> tablesQueried) {
+ set(Field.TABLES_QUERIED, String.join(",", tablesQueried));
+ }
+
+ public void setExecSummary(String summary) {
+ set(Field.EXEC_SUMMARY, summary);
+ }
+
+ public void setConfigurationOptionsChanged(Map<String, String> overriddenConfigurations) {
+ set(Field.CONFIGURATION_OPTIONS_CHANGED, Joiner.on(";").withKeyValueSeparator("=").join(overriddenConfigurations));
+ }
+
+ public void setTotalNumberOfTasks(int numTasks) {
+ set(Field.TOTAL_LAUNCHED_TASKS, numTasks);
+ }
+
+ public void setNumberOfSucceededTasks(int numSucceededTasks) {
+ set(Field.NUM_SUCCEEDED_TASKS, numSucceededTasks);
+ }
+
+ public void setNumberOfKilledTasks(int numKilledTasks) {
+ set(Field.NUM_KILLED_TASKS, numKilledTasks);
+ }
+
+ public void setNumberOfFailedTasks(int numFailedTasks) {
+ set(Field.NUM_FAILED_TASKS, numFailedTasks);
+ }
+
+ public void setNodeUsedCount(int nodeUsedCount) {
+ set(Field.NODE_USED_COUNT, nodeUsedCount);
+ }
+
+ public void setNodeTotalCount(int nodeTotalCount) {
+ set(Field.NODE_TOTAL_COUNT, nodeTotalCount);
+ }
+
+ public void setReduceInputGroups(long reduceInputGroups) {
+ set(Field.REDUCE_INPUT_GROUPS, reduceInputGroups);
+ }
+
+ public void setReduceInputRecords(long reduceInputRecords) {
+ set(Field.REDUCE_INPUT_RECORDS, reduceInputRecords);
+ }
+
+ public void setSpilledRecords(long spilledRecords) {
+ set(Field.SPILLED_RECORDS, spilledRecords);
+ }
+
+ public void setNumShuffledInputs(long numShuffledInputs) {
+ set(Field.NUM_SHUFFLED_INPUTS, numShuffledInputs);
+ }
+
+ public void setNumFailedShuffleInputs(long numFailedShuffleInputs) {
+ set(Field.NUM_FAILED_SHUFFLE_INPUTS, numFailedShuffleInputs);
+ }
+
+ public void setTaskDurationMillis(long taskDurationMillis) {
+ set(Field.TASK_DURATION_MILLIS, taskDurationMillis);
+ }
+
+ public void setInputRecordsProcessed(long inputRecordsProcessed) {
+ set(Field.INPUT_RECORDS_PROCESSED, inputRecordsProcessed);
+ }
+
+ public void setInputSplitLengthBytes(long inputSplitLengthBytes) {
+ set(Field.INPUT_SPLIT_LENGTH_BYTES, inputSplitLengthBytes);
+ }
+
+ public void setOutputRecords(long outputRecords) {
+ set(Field.OUTPUT_RECORDS, outputRecords);
+ }
+
+ public void setOutputBytesPhysical(long outputBytesPhysical) {
+ set(Field.OUTPUT_BYTES_PHYSICAL, outputBytesPhysical);
+ }
+
+ public void setShuffleChunkCount(long shuffleChunkCount) {
+ set(Field.SHUFFLE_CHUNK_COUNT, shuffleChunkCount);
+ }
+
+ public void setShuffleBytes(long shuffleBytes) {
+ set(Field.SHUFFLE_BYTES, shuffleBytes);
+ }
+
+ public void setShuffleBytesDiskDirect(long shuffleBytesDiskDirect) {
+ set(Field.SHUFFLE_BYTES_DISK_DIRECT, shuffleBytesDiskDirect);
+ }
+
+ public void setShufflePhaseTime(long shufflePhaseTime) {
+ set(Field.SHUFFLE_PHASE_TIME, shufflePhaseTime);
+ }
+
+ public void setMergePhaseTime(long phaseTime) {
+ set(Field.MERGE_PHASE_TIME, phaseTime);
+ }
+
+ public Object get(Field field) {
+ return record.get(field.name);
+ }
+
+ private void set(Field field, Object value) {
+ record.put(field.name, value);
+ }
+
+ public String toString() {
+ return String.format("QueryHistoryRecord [queryId: %s, sessionId: %s]", get(QueryHistorySchema.Field.QUERY_ID),
+ get(QueryHistorySchema.Field.SESSION_ID));
+ }
+
+ /**
+ * Utility toString kind of method for debugging.
+ * @return string representation of the underlying map
+ */
+ public String toLongString() {
+ return String.format("QueryHistoryRecord [%s]", record);
+ }
+
+ public Writable serialize(Serializer serializer) throws Exception {
+ return serializer.serialize(toSerializable(), QueryHistorySchema.INSPECTOR);
+ }
+
+ Object[] toSerializable() {
Review Comment:
could be replaced with `record.values().toArray()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931106818
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/IcebergRepository.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Calendar;
+import java.util.GregorianCalendar;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Queue;
+import java.util.UUID;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.io.HiveFileFormatUtils;
+import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.HiveStorageHandler;
+import org.apache.hadoop.hive.ql.metadata.HiveUtils;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.security.authorization.HiveCustomStorageHandlerUtils;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.apache.hadoop.hive.serde2.Serializer;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TaskAttemptID;
+import org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IcebergRepository extends AbstractQueryHistoryRepository implements QueryHistoryRepository {
+ private static final Logger LOG = LoggerFactory.getLogger(IcebergRepository.class);
+ private static final String ICEBERG_STORAGE_HANDLER = "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler";
+
+ private HiveOutputFormat<?, ?> outputFormat;
+ private Serializer serializer;
+ @VisibleForTesting
+ HiveStorageHandler storageHandler;
+ @VisibleForTesting
+ TableDesc tableDesc;
+
+ @Override
+ public void init(HiveConf conf, QueryHistorySchema schema) {
+ super.init(conf, schema);
+ }
+
+ @Override
+ protected Table createTable(Hive hive, Database db) throws HiveException {
+ LOG.info("Creating iceberg Query History table...");
+
+ Table table = getInitialTable();
+
+ // iceberg specific
+ table.setProperty(org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.META_TABLE_STORAGE,
+ ICEBERG_STORAGE_HANDLER);
+ table.setProperty("table_type", "ICEBERG");
+ table.setProperty("write.format.default", "orc");
+ table.setProperty(hive_metastoreConstants.META_TABLE_NAME, QUERY_HISTORY_DB_TABLE_NAME);
+
+ table.setFields(schema.getFields());
+
+ hive.createTable(table, false);
+
+ table = hive.getTable(QUERY_HISTORY_DB_NAME, QUERY_HISTORY_TABLE_NAME);
+ table.setProperty("iceberg.mr.table.location", table.getDataLocation().toString());
+
+ return table;
+ }
+
+ @Override
+ protected void postInitTable(Table table) throws Exception {
+ this.tableDesc = Utilities.getTableDesc(table);
+
+ Map<String, String> map = new HashMap<>();
+ this.storageHandler = HiveUtils.getStorageHandler(conf, tableDesc.getProperties()
Review Comment:
can't we use `table.getStorageHandler()`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931619355
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
Review Comment:
eh, very good catch, I was thinking about this a lot...currently, there is no explicit flash HQL for this, so it depends on the currently implemented strategies, what about keeping it simple: create an executor with a configurable period (1 hour default?), and it runs flush (no matter how old the records are in the queue)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931059708
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/AbstractQueryHistoryRepository.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.TableType;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.parse.PartitionTransform;
+import org.apache.hadoop.hive.ql.parse.TransformSpec;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public abstract class AbstractQueryHistoryRepository implements QueryHistoryRepository {
+ protected Logger LOG = LoggerFactory.getLogger(getClass());
+ @VisibleForTesting
+ HiveConf conf;
+ protected QueryHistorySchema schema;
+
+ public void init(HiveConf conf, QueryHistorySchema schema) {
+ this.conf = conf;
+ this.schema = schema;
+
+ try (Hive hive = Hive.get(conf)) {
+ Database database = initDatabase(hive);
+ Table table = initTable(hive, database);
+ postInitTable(table);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ protected Database initDatabase(Hive hive) {
+ Database db;
+ try {
+ db = hive.getDatabase(QUERY_HISTORY_DB_NAME);
+ if (db == null) {
+ LOG.warn("Database ({}) for query history table hasn't been found, auto-creating one", QUERY_HISTORY_DB_NAME);
+ String location = getDatabaseLocation(QUERY_HISTORY_DB_NAME);
+ db = new Database(QUERY_HISTORY_DB_NAME, QUERY_HISTORY_DB_COMMENT,
+ location, null);
+ hive.createDatabase(db, false);
+ }
+ return db;
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private String getDatabaseLocation(String databaseName) {
Review Comment:
should we use `new Warehouse(conf).getDefaultExternalDatabasePath()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931132684
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/schema/QueryHistorySchema.java:
##########
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+
+public class QueryHistorySchema {
+ public static final int CURRENT_VERSION = 1;
+
+ // how many bytes are consumed by the "short" fields in a record
+ // rough estimate (once per JVM), which is much efficient than counting all the fields
+ public static final long BASE_RECORD_SIZE_IN_MEMORY_BYTES = calculateSimpleFieldsSize();
+
+ private static int calculateSimpleFieldsSize() {
+ int baseSize = 0;
+ for (QueryHistorySchema.Field field : QueryHistorySchema.Field.values()) {
+ switch (field.getType()) {
+ case "string":
+ // consider an average 20 chars length
+ // longer fields are going to be considered again
+ baseSize += 20 * Character.BYTES;
+ break;
+ case "int":
+ baseSize += 4;
+ break;
+ case "bigint":
+ // stored as long
+ baseSize += 8;
+ break;
+ case "timestamp":
+ baseSize += 16;
+ break;
+ }
+ }
+ return baseSize;
+ }
+
+ public interface QueryHistorySchemaField {
+ String getName();
+ String getType();
+ String getDescription();
+ boolean isPartitioningCol();
+ }
+
+ public static ObjectInspector INSPECTOR = null;
+
+ static {
+ INSPECTOR = createInspector(Field.values());
+ }
+
+ private final ArrayList<FieldSchema> fields;
+
+ public QueryHistorySchema() {
+ this.fields = new ArrayList<>();
+
+ Arrays.asList(Field.values())
+ .forEach(field -> this.fields.add(new FieldSchema(field.name, field.type, field.description)));
+ }
+
+ @VisibleForTesting
+ static ObjectInspector createInspector(QueryHistorySchemaField[] fields) {
Review Comment:
can't we use `PrimitiveObjectInspectorFactory#getPrimitiveWritableObjectInspector`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931178435
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
Review Comment:
if memory option is on and user submits a simple query, would he see it in a history after completion, or only after the batch size is reached?
what if it not reached in 1 day, should we add time-based push as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931793316
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/QueryHistoryRepository.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Queue;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+
+public interface QueryHistoryRepository {
+ String QUERY_HISTORY_DB_NAME = "sys";
+ String QUERY_HISTORY_TABLE_NAME = "query_history";
+ String QUERY_HISTORY_DB_TABLE_NAME =
+ String.format("%s.%s", QUERY_HISTORY_DB_NAME, QUERY_HISTORY_TABLE_NAME);
+
+ String QUERY_HISTORY_DB_COMMENT = "Hive SYS database";
Review Comment:
up to you, but i don't think we use this comment anywhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931070141
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/AbstractQueryHistoryRepository.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.TableType;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.parse.PartitionTransform;
+import org.apache.hadoop.hive.ql.parse.TransformSpec;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public abstract class AbstractQueryHistoryRepository implements QueryHistoryRepository {
+ protected Logger LOG = LoggerFactory.getLogger(getClass());
+ @VisibleForTesting
+ HiveConf conf;
+ protected QueryHistorySchema schema;
+
+ public void init(HiveConf conf, QueryHistorySchema schema) {
+ this.conf = conf;
+ this.schema = schema;
+
+ try (Hive hive = Hive.get(conf)) {
+ Database database = initDatabase(hive);
+ Table table = initTable(hive, database);
+ postInitTable(table);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ protected Database initDatabase(Hive hive) {
+ Database db;
+ try {
+ db = hive.getDatabase(QUERY_HISTORY_DB_NAME);
+ if (db == null) {
+ LOG.warn("Database ({}) for query history table hasn't been found, auto-creating one", QUERY_HISTORY_DB_NAME);
+ String location = getDatabaseLocation(QUERY_HISTORY_DB_NAME);
+ db = new Database(QUERY_HISTORY_DB_NAME, QUERY_HISTORY_DB_COMMENT,
+ location, null);
+ hive.createDatabase(db, false);
+ }
+ return db;
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private String getDatabaseLocation(String databaseName) {
+ String warehouseLocation = conf.get(HiveConf.ConfVars.HIVE_METASTORE_WAREHOUSE_EXTERNAL.varname);
+ if (warehouseLocation == null) {
+ warehouseLocation = conf.get(HiveConf.ConfVars.METASTORE_WAREHOUSE.varname);
+ }
+ Preconditions.checkNotNull(
+ warehouseLocation, "Warehouse location is not set: 'hive.metastore.warehouse.external.dir' and " +
+ "'hive.metastore.warehouse.dir' are both null");
+ return String.format("%s/%s.db", warehouseLocation, databaseName.toLowerCase());
+ }
+
+ protected Table initTable(Hive hive, Database db) {
+ Table table;
+ try {
+ table = hive.getTable(QUERY_HISTORY_DB_NAME, QUERY_HISTORY_TABLE_NAME, null, false);
+ if (table == null) {
+ LOG.info("Query history table ({}) isn't created yet", QUERY_HISTORY_TABLE_NAME);
+ table = createTable(hive, db);
+ }
+ return table;
+ } catch (HiveException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ protected abstract void postInitTable(Table table) throws Exception;
+
+ /**
+ * Supposed to create the query history table in metastore. It's only called from
+ * initTable if it doesn't exist yet.
+ */
+ protected abstract Table createTable(Hive hive, Database db) throws HiveException;
+
+ /**
+ * While creating the table, getInitialTable is supposed to return a common table object,
+ * which contains anything general that's not specific to QueryHistoryRepository subclasses.
+ */
+ protected Table getInitialTable() {
Review Comment:
`createTableObject()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2626692319
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613) **Quality Gate passed**
Issues
 [103 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923558017
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java:
##########
@@ -40,7 +40,7 @@ private DriverFactory() {
}
public static IDriver newDriver(HiveConf conf) {
- return newDriver(getNewQueryState(conf), null);
+ return newDriver(getNewQueryState(conf), QueryInfo.getFromConf(conf));
Review Comment:
I am not sure we should change this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923559360
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryInfo.java:
##########
@@ -27,9 +31,10 @@ public class QueryInfo {
private final long beginTime;
private final String sessionId;
private final String operationId;
+ private final Instant beginTimeUTC;
Review Comment:
why do we need time vars for several TZ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923759119
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java:
##########
@@ -513,4 +527,12 @@ ProgressMonitor progressMonitor(DAGStatus status, Map<String, Progress> progress
}
return TezProgressMonitor.NULL;
}
+
+ public String getSummary() {
Review Comment:
do we need all these console wrapper methods, why not expose the Console object and operate on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923858932
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.persist.QueryHistoryPersistor;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ private QueryHistoryPersistor queryHistoryPersistor;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+ private int maxBatchSize = 0;
+ private int maxQueueSizeInMemoryBytes = 0;
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService persistor thread").build());
+
+ private ServiceContext serviceContext;
+
+ public static QueryHistoryService start(HiveConf inputConf, ServiceContext serviceContext) {
+ HiveConf conf = new HiveConf(inputConf);
+ // we can only hit this codepath if the INSTANCE is already initialized while calling start(...)
+ // for convenient test compatibility, this call is allowed to be idempotent (LOG.warn instead of exception)
+ if (INSTANCE != null) {
+ LOG.warn("There is already a QueryHistoryService instance ({}), returning existing instance", INSTANCE);
Review Comment:
hm, we don't, let me remove
in the early stages, the singleton design changed quite often, but I'm ready to refactor this are and get rid of the message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923958010
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -5734,6 +5734,26 @@ public static enum ConfVars {
+ "a background update happens periodically to report the actual state of the query"),
HIVE_SCHEDULED_QUERIES_CREATE_AS_ENABLED("hive.scheduled.queries.create.as.enabled", true,
"This option sets the default behaviour of newly created scheduled queries."),
+ HIVE_QUERY_HISTORY_SERVICE_ENABLED("hive.query.history.service.enabled", true,
Review Comment:
probably that is something we should test to understand the impact and defaults setup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923958010
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -5734,6 +5734,26 @@ public static enum ConfVars {
+ "a background update happens periodically to report the actual state of the query"),
HIVE_SCHEDULED_QUERIES_CREATE_AS_ENABLED("hive.scheduled.queries.create.as.enabled", true,
"This option sets the default behaviour of newly created scheduled queries."),
+ HIVE_QUERY_HISTORY_SERVICE_ENABLED("hive.query.history.service.enabled", true,
Review Comment:
probably that is something we should to understand the impact and defaults setup
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog merged PR #5613:
URL: https://github.com/apache/hive/pull/5613
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931146364
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void handleQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ QueryHistoryRecord record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ persistRecordsIfNeeded();
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ persistWithBatchSize(maxBatchSize);
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(), maxBatchSize);
+ }
+ }
+
+ private void persistWithBatchSize(int batchSize) {
+ CountDownLatch latch = new CountDownLatch(1);
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ repository.persist(queryHistoryQueue);
+ latch.countDown();
+ });
+ if (batchSize == 0) {
+ waitForAllRecordsToBePersisted(latch);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted(CountDownLatch queueEmptyCondition) {
+ try {
+ queueEmptyCondition.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ boolean needToPersist() {
+ boolean needToPersistAccordingToQueueSize = queryHistoryQueue.size() >= maxBatchSize;
Review Comment:
why var is called `maxBatchSize`, not simply `batchSize`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931141544
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
Review Comment:
why don't we do this in `AbstractRepository#init`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1938436319
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java:
##########
@@ -20,6 +20,7 @@
import java.io.IOException;
import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
Review Comment:
realized this was refactored by the IDE, now it's correct alphabetical ordering point of view, can I leave it like this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1938443193
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void handleQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ QueryHistoryRecord record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ persistRecordsIfNeeded();
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ persistWithBatchSize(maxBatchSize);
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(), maxBatchSize);
+ }
+ }
+
+ private void persistWithBatchSize(int batchSize) {
+ CountDownLatch latch = new CountDownLatch(1);
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ repository.persist(queryHistoryQueue);
+ latch.countDown();
+ });
+ if (batchSize == 0) {
+ waitForAllRecordsToBePersisted(latch);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted(CountDownLatch queueEmptyCondition) {
+ try {
+ queueEmptyCondition.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ boolean needToPersist() {
+ boolean needToPersistAccordingToQueueSize = queryHistoryQueue.size() >= maxBatchSize;
+ if (needToPersistAccordingToQueueSize) {
+ LOG.info("Need to persist query history records as queue size ({}) >= batch size ({})",
+ queryHistoryQueue.size(), maxBatchSize);
+ return true;
+ }
+
+ // whether ignore the consumed memory or not
+ if (maxQueueSizeInMemoryBytes == 0) {
+ return false;
+ }
+
+ long queueMemorySize =
+ queryHistoryQueue.stream().mapToLong(QueryHistoryRecord::getEstimatedSizeInMemoryBytes).sum();
+ boolean needToPersistAccordingToQueueMemorySize = queueMemorySize > maxQueueSizeInMemoryBytes;
+ if (needToPersistAccordingToQueueMemorySize) {
+ LOG.info("Need to persist query history records as queue size in memory ({} bytes) is greater than the limit " +
+ "({} bytes)",
+ queueMemorySize, maxQueueSizeInMemoryBytes);
+ }
+ return needToPersistAccordingToQueueMemorySize;
+ }
+
+ public void stop() {
+ LOG.info("Stopping QueryHistoryService, leftover records to persist: {}", queryHistoryQueue.size());
+ persistWithBatchSize(0); // persist sync
+ persistExecutor.shutdownNow();
+ }
+
+ private QueryHistoryRecord createRecord(DriverContext driverContext) {
+ return new QueryHistoryRecordEnricher(driverContext, serviceContext, SessionState.get(),
+ SessionState.getPerfLogger()).createRecord();
+ }
+
+ @VisibleForTesting
+ public QueryHistoryRepository getRepository(){
Review Comment:
double-checked, used in TestIcebergRepository
```
IcebergRepository repository = (IcebergRepository) service.getRepository();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2610773663
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613) **Quality Gate passed**
Issues
 [96 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1927395431
##########
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java:
##########
@@ -1176,6 +1176,9 @@ public static class LogHelper {
protected Logger LOG;
protected boolean isSilent;
+ private final StringBuilder summary = new StringBuilder();
+ private boolean collectSummary;
Review Comment:
no, this is the query summary which is printed at the end of the query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923764427
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -600,6 +601,11 @@ default void storageHandlerCommit(Properties commitProperties, Operation operati
throw new UnsupportedOperationException();
}
+ default void storageHandlerCommit(Properties commitProperties, Operation operation, TaskAttemptContext context)
Review Comment:
1. javadoc is missing
2. why do we need this for query history?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923948658
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/queryhistory/schema/ExampleQueryHistoryRecord.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import com.google.common.collect.Sets;
+
+import java.time.Instant;
+import java.util.Collections;
+import java.util.Map;
+import java.util.Set;
+
+public class ExampleQueryHistoryRecord extends QueryHistoryRecord {
Review Comment:
dummy doesn't mean empty it's a kinda stub (see DummyPartition)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1924080570
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/persist/IcebergPersistor.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.persist;
+
+import java.util.Calendar;
+import java.util.GregorianCalendar;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Queue;
+import java.util.UUID;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.io.HiveFileFormatUtils;
+import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.HiveStorageHandler;
+import org.apache.hadoop.hive.ql.metadata.HiveUtils;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.security.authorization.HiveCustomStorageHandlerUtils;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.apache.hadoop.hive.serde2.Serializer;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TaskAttemptID;
+import org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IcebergPersistor extends AbstractQueryHistoryPersistor implements QueryHistoryPersistor {
+ private static final Logger LOG = LoggerFactory.getLogger(IcebergPersistor.class);
+ private static final String ICEBERG_STORAGE_HANDLER = "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler";
+
+ private HiveOutputFormat<?, ?> outputFormat;
+ private Serializer serializer;
+ @VisibleForTesting
+ HiveStorageHandler storageHandler;
+ @VisibleForTesting
+ TableDesc tableDesc;
+
+ @Override
+ public void init(HiveConf conf, QueryHistorySchema schema) {
+ super.init(conf, schema);
+ }
+
+ @Override
+ protected Table createTable(Hive hive, Database db) throws HiveException {
Review Comment:
not a problem, but I think it might be simpler? not 100% sure, need to think about it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2604307835
What is the use case for that service? can't I check the query history in HUE or DAS (removed for some storage reason), etc
Please take a look at https://github.com/apache/hive/pull/5319 which is being worked on by [rtrivedi12](https://github.com/rtrivedi12)
It provides much more details
cc @nrg4878
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923504775
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -838,6 +845,9 @@ public void storageHandlerCommit(Properties commitProperties, Operation operatio
}
HiveIcebergOutputCommitter committer = getOutputCommitter();
try {
+ if (context != null) {
+ committer.commitTask(context);
Review Comment:
this happens in HS2, where we don't have the actual execution (tez) tasks
we simply write out iceberg data from HS2 from a single place
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923593262
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryInfo.java:
##########
@@ -27,9 +31,10 @@ public class QueryInfo {
private final long beginTime;
private final String sessionId;
private final String operationId;
+ private final Instant beginTimeUTC;
Review Comment:
we need this in query history where we store start/end time in server's timezone and in utc for convenience reasons, but I can achieve it without storing it in a field, simply:
```
public Instant getBeginTimeUTC() {
return Instant.ofEpochMilli(beginTime);
}
```
this way maybe I can get away without the jackson dependency too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923472116
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -5734,6 +5734,26 @@ public static enum ConfVars {
+ "a background update happens periodically to report the actual state of the query"),
HIVE_SCHEDULED_QUERIES_CREATE_AS_ENABLED("hive.scheduled.queries.create.as.enabled", true,
"This option sets the default behaviour of newly created scheduled queries."),
+ HIVE_QUERY_HISTORY_SERVICE_ENABLED("hive.query.history.service.enabled", true,
+ "Whether to start QueryHistoryService in HS2"),
+ HIVE_QUERY_HISTORY_SERVICE_EXPLAIN_PLAN_ENABLED("hive.query.history.service.explain.plan.enabled", true,
+ "Whether to collect and store explain plan in the query history. Default is true."),
+ HIVE_QUERY_HISTORY_SERVICE_EXEC_SUMMARY_ENABLED("hive.query.history.service.exec.summary.enabled", true,
+ "Whether to collect and store execution summary in the query history. Default is true."),
+ HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE("hive.query.history.service.persist.max.batch.size", 200,
+ "The maximum amount of records held in memory " +
+ "before query history service persists them to the target table. " +
+ "A small value (like 1-5) will lead to more real-time behavior with the price of small files. " +
+ "Set this to 0 to wait for the records to be persisted synchronously (not recommended in production)."),
+ HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_MEMORY_BYTES("hive.query.history.service.persist.max.memory.bytes",
+ 100 * 1024 * 1024, // 100MB
Review Comment:
100MB seems a lot
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923544780
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java:
##########
@@ -247,4 +275,107 @@ public String getQueryErrorMessage() {
public void setQueryErrorMessage(String queryErrorMessage) {
this.queryErrorMessage = queryErrorMessage;
}
+
+ public long getQueryStartTime() {
+ return getQueryInfo() != null ? getQueryInfo().getBeginTime() : getQueryDisplay().getQueryStartTime();
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY: a simple fetch task is a QUERY
+ * UNKNOWN: if we can't determine the type of the query:
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param sem the semantic analyzer which already analyzed the query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(BaseSemanticAnalyzer sem, ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = sem.getAllRootTasks();
Review Comment:
all this check logic should be in some util class, similar to `AcidUtils#getTxnType` + instead of Strings would be nice to introduce enums
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923534667
##########
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:
##########
@@ -94,7 +96,7 @@ public Driver(HiveConf conf) {
// Pass lineageState when a driver instantiates another Driver to run or compile another query
public Driver(HiveConf conf, Context ctx, LineageState lineageState) {
- this(QueryState.getNewQueryState(conf, lineageState), null);
+ this(QueryState.getNewQueryState(conf, lineageState), QueryInfo.getFromConf(conf));
Review Comment:
in the beginning of the implementation, I ran full precommit tests with enabling this service by default and enabling everywhere it's possible (all qtests for example), which led to some problems that are not visible otherwise
one those problems were that the Driver instance can be created here with null QueryInfo, which led to NPEs in both prod and test code (as far as I can remember), so I decided to create an empty one here, which looks much more valid then null (so actually, the query history service was the one that showed QueryInfo==null can cause problems)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930573011
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/queryhistory/repository/TestIcebergRepository.java:
##########
@@ -0,0 +1,115 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one§
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchemaTestUtils;
+import org.apache.iceberg.mr.mapred.Container;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.queryhistory.QueryHistoryService;
+import org.apache.hadoop.hive.ql.queryhistory.repository.IcebergRepositoryForTest;
+import org.apache.hadoop.hive.ql.queryhistory.schema.DummyQueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.IcebergQueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileInputFormat;
+import org.apache.hadoop.mapred.InputFormat;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hadoop.mapred.Reporter;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.Queue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+public class TestIcebergRepository {
+ private static final Logger LOG = LoggerFactory.getLogger(TestIcebergRepository.class);
+ private static final ServiceContext serviceContext = new ServiceContext(() -> DummyQueryHistoryRecord.SERVER_HOST,
+ () -> DummyQueryHistoryRecord.SERVER_PORT);
+ private final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ /*
+ * This unit test asserts that the created record is persisted as expected and the values made their way to the
+ * iceberg table.
+ */
+ @Test
+ public void testPersistRecord() throws Exception {
+ HiveConf conf = new HiveConf();
+
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+ conf.setIntVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE, 0);
+ conf.setVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS, IcebergRepositoryForTest.class.getName());
Review Comment:
please replace it with a production impl and set needed config ("iceberg.engine.hive.lock-enabled", "false")
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930842192
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.persist.QueryHistoryPersistor;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ private QueryHistoryPersistor queryHistoryPersistor;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+ private int maxBatchSize = 0;
+ private int maxQueueSizeInMemoryBytes = 0;
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService persistor thread").build());
+
+ private ServiceContext serviceContext;
+
+ public static QueryHistoryService start(HiveConf inputConf, ServiceContext serviceContext) {
+ HiveConf conf = new HiveConf(inputConf);
+ // we can only hit this codepath if the INSTANCE is already initialized while calling start(...)
+ // for convenient test compatibility, this call is allowed to be idempotent (LOG.warn instead of exception)
+ if (INSTANCE != null) {
+ LOG.warn("There is already a QueryHistoryService instance ({}), returning existing instance", INSTANCE);
+ return INSTANCE;
+ }
+
+ INSTANCE = createServiceInstance(serviceContext, conf);
+ return INSTANCE;
+ }
+
+ private static QueryHistoryService createServiceInstance(ServiceContext serviceContext, HiveConf conf) {
+ LOG.info("Starting QueryHistoryService");
+ QueryHistoryService queryHistoryService = new QueryHistoryService();
+
+ initService(serviceContext, conf, queryHistoryService);
+ return queryHistoryService;
+ }
+
+ @VisibleForTesting
+ static void initService(ServiceContext serviceContext, HiveConf conf, QueryHistoryService queryHistoryService) {
+ queryHistoryService.queryHistoryPersistor = createPersistor(conf);
+ queryHistoryService.queryHistoryPersistor.init(conf, new QueryHistorySchema());
+ queryHistoryService.maxBatchSize = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE);
+ queryHistoryService.maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_MEMORY_BYTES);
+
+ queryHistoryService.printConfigInformation();
+ queryHistoryService.serviceContext = serviceContext;
+
+ try {
+ queryHistoryService.persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the persistor,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
Review Comment:
this is what i meant
````
private static class QueryHistoryThread extends Thread {
public QueryHistoryThread(Runnable runnable, HiveConf conf) {
super(runnable);
HiveConf jobConf = new HiveConf(conf);
// this session won't need tez, let's not mess with starting a TezSessionState
jobConf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
SessionState.start(jobConf);
}
}
public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
...
persistExecutor = Executors.newFixedThreadPool(1,
new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true)
.setThreadFactory(r -> new QueryHistoryThread(r, conf)).build());
repository = createRepository(conf);
...
````
that won't need hacks in `start()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930842192
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.persist.QueryHistoryPersistor;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ private QueryHistoryPersistor queryHistoryPersistor;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+ private int maxBatchSize = 0;
+ private int maxQueueSizeInMemoryBytes = 0;
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService persistor thread").build());
+
+ private ServiceContext serviceContext;
+
+ public static QueryHistoryService start(HiveConf inputConf, ServiceContext serviceContext) {
+ HiveConf conf = new HiveConf(inputConf);
+ // we can only hit this codepath if the INSTANCE is already initialized while calling start(...)
+ // for convenient test compatibility, this call is allowed to be idempotent (LOG.warn instead of exception)
+ if (INSTANCE != null) {
+ LOG.warn("There is already a QueryHistoryService instance ({}), returning existing instance", INSTANCE);
+ return INSTANCE;
+ }
+
+ INSTANCE = createServiceInstance(serviceContext, conf);
+ return INSTANCE;
+ }
+
+ private static QueryHistoryService createServiceInstance(ServiceContext serviceContext, HiveConf conf) {
+ LOG.info("Starting QueryHistoryService");
+ QueryHistoryService queryHistoryService = new QueryHistoryService();
+
+ initService(serviceContext, conf, queryHistoryService);
+ return queryHistoryService;
+ }
+
+ @VisibleForTesting
+ static void initService(ServiceContext serviceContext, HiveConf conf, QueryHistoryService queryHistoryService) {
+ queryHistoryService.queryHistoryPersistor = createPersistor(conf);
+ queryHistoryService.queryHistoryPersistor.init(conf, new QueryHistorySchema());
+ queryHistoryService.maxBatchSize = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE);
+ queryHistoryService.maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_MEMORY_BYTES);
+
+ queryHistoryService.printConfigInformation();
+ queryHistoryService.serviceContext = serviceContext;
+
+ try {
+ queryHistoryService.persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the persistor,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
Review Comment:
this is what i meant
````
private static class QueryHistoryThread extends Thread {
public QueryHistoryThread(Runnable runnable, HiveConf conf) {
super(runnable);
HiveConf jobConf = new HiveConf(conf);
// this session won't need tez, let's not mess with starting a TezSessionState
jobConf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
SessionState.start(jobConf);
}
}
public void start() {
repository.init(conf, new QueryHistorySchema());
persistExecutor = Executors.newFixedThreadPool(1,
new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true)
.setThreadFactory(r -> new QueryHistoryThread(r, conf)).build());
}
````
that won't need hacks in `start()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930842192
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.persist.QueryHistoryPersistor;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ private QueryHistoryPersistor queryHistoryPersistor;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+ private int maxBatchSize = 0;
+ private int maxQueueSizeInMemoryBytes = 0;
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService persistor thread").build());
+
+ private ServiceContext serviceContext;
+
+ public static QueryHistoryService start(HiveConf inputConf, ServiceContext serviceContext) {
+ HiveConf conf = new HiveConf(inputConf);
+ // we can only hit this codepath if the INSTANCE is already initialized while calling start(...)
+ // for convenient test compatibility, this call is allowed to be idempotent (LOG.warn instead of exception)
+ if (INSTANCE != null) {
+ LOG.warn("There is already a QueryHistoryService instance ({}), returning existing instance", INSTANCE);
+ return INSTANCE;
+ }
+
+ INSTANCE = createServiceInstance(serviceContext, conf);
+ return INSTANCE;
+ }
+
+ private static QueryHistoryService createServiceInstance(ServiceContext serviceContext, HiveConf conf) {
+ LOG.info("Starting QueryHistoryService");
+ QueryHistoryService queryHistoryService = new QueryHistoryService();
+
+ initService(serviceContext, conf, queryHistoryService);
+ return queryHistoryService;
+ }
+
+ @VisibleForTesting
+ static void initService(ServiceContext serviceContext, HiveConf conf, QueryHistoryService queryHistoryService) {
+ queryHistoryService.queryHistoryPersistor = createPersistor(conf);
+ queryHistoryService.queryHistoryPersistor.init(conf, new QueryHistorySchema());
+ queryHistoryService.maxBatchSize = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE);
+ queryHistoryService.maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_MEMORY_BYTES);
+
+ queryHistoryService.printConfigInformation();
+ queryHistoryService.serviceContext = serviceContext;
+
+ try {
+ queryHistoryService.persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the persistor,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
Review Comment:
this is what i meant
````
private static class QueryHistoryThread extends Thread {
public QueryHistoryThread(Runnable runnable, HiveConf conf) {
super(runnable);
HiveConf jobConf = new HiveConf(conf);
// this session won't need tez, let's not mess with starting a TezSessionState
jobConf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
SessionState.start(jobConf);
}
}
public QueryHistoryService start() {
repository.init(conf, new QueryHistorySchema());
....
persistExecutor = Executors.newFixedThreadPool(1,
new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true)
.setThreadFactory(r -> new QueryHistoryThread(r, conf)).build());
....
}
````
that won't need hacks in `start()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930492394
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/queryhistory/repository/TestIcebergRepository.java:
##########
@@ -0,0 +1,115 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one§
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchemaTestUtils;
+import org.apache.iceberg.mr.mapred.Container;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.queryhistory.QueryHistoryService;
+import org.apache.hadoop.hive.ql.queryhistory.repository.IcebergRepositoryForTest;
+import org.apache.hadoop.hive.ql.queryhistory.schema.DummyQueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.IcebergQueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileInputFormat;
+import org.apache.hadoop.mapred.InputFormat;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hadoop.mapred.Reporter;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.Queue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+public class TestIcebergRepository {
Review Comment:
maybe TestIcebergQueryHistoryRepository
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931113257
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/IcebergRepository.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Calendar;
+import java.util.GregorianCalendar;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Queue;
+import java.util.UUID;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.io.HiveFileFormatUtils;
+import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.HiveStorageHandler;
+import org.apache.hadoop.hive.ql.metadata.HiveUtils;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.security.authorization.HiveCustomStorageHandlerUtils;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.apache.hadoop.hive.serde2.Serializer;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TaskAttemptID;
+import org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IcebergRepository extends AbstractQueryHistoryRepository implements QueryHistoryRepository {
+ private static final Logger LOG = LoggerFactory.getLogger(IcebergRepository.class);
+ private static final String ICEBERG_STORAGE_HANDLER = "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler";
+
+ private HiveOutputFormat<?, ?> outputFormat;
+ private Serializer serializer;
+ @VisibleForTesting
+ HiveStorageHandler storageHandler;
+ @VisibleForTesting
+ TableDesc tableDesc;
+
+ @Override
+ public void init(HiveConf conf, QueryHistorySchema schema) {
+ super.init(conf, schema);
+ }
+
+ @Override
+ protected Table createTable(Hive hive, Database db) throws HiveException {
+ LOG.info("Creating iceberg Query History table...");
+
+ Table table = getInitialTable();
+
+ // iceberg specific
+ table.setProperty(org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.META_TABLE_STORAGE,
+ ICEBERG_STORAGE_HANDLER);
+ table.setProperty("table_type", "ICEBERG");
+ table.setProperty("write.format.default", "orc");
+ table.setProperty(hive_metastoreConstants.META_TABLE_NAME, QUERY_HISTORY_DB_TABLE_NAME);
+
+ table.setFields(schema.getFields());
+
+ hive.createTable(table, false);
+
+ table = hive.getTable(QUERY_HISTORY_DB_NAME, QUERY_HISTORY_TABLE_NAME);
+ table.setProperty("iceberg.mr.table.location", table.getDataLocation().toString());
+
+ return table;
+ }
+
+ @Override
+ protected void postInitTable(Table table) throws Exception {
+ this.tableDesc = Utilities.getTableDesc(table);
+
+ Map<String, String> map = new HashMap<>();
+ this.storageHandler = HiveUtils.getStorageHandler(conf, tableDesc.getProperties()
+ .getProperty(org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.META_TABLE_STORAGE));
+ storageHandler.configureOutputJobProperties(tableDesc, map);
+
+ //need to obtain a serialized table for later usage
+ map.forEach((key, value) -> conf.set(key, value));
+
+ conf.set(hive_metastoreConstants.META_TABLE_NAME, QUERY_HISTORY_DB_TABLE_NAME);
+
+ HiveCustomStorageHandlerUtils.setWriteOperation(conf, QUERY_HISTORY_DB_TABLE_NAME, Context.Operation.OTHER);
+ this.outputFormat = HiveFileFormatUtils.getHiveOutputFormat(conf, tableDesc);
+
+ this.serializer = tableDesc.getDeserializer(conf);
+ }
+
+ @Override
+ public void persist(Queue<QueryHistoryRecord> records) {
+ int numRecords = records.size();
+ LOG.info("Persisting {} records", numRecords);
+ if (numRecords == 0) {
+ return;
+ }
+
+ try {
+ prepareConfForWrite();
+ JobConf jobConf = new JobConf(conf);
+
+ RecordWriter writer = outputFormat.getHiveRecordWriter(jobConf, null, null, false, null, null);
+ while (!records.isEmpty()) {
+ writeRecord(writer, records.poll());
+ }
+
+ writer.close(false);
+ storageHandler.storageHandlerCommit(HiveConf.getProperties(conf), Context.Operation.OTHER,
Review Comment:
````
jobConf.getOutputCommitter().commitTask(
new TaskAttemptContextImpl(jobConf, TaskAttemptID.forName(conf.get("mapred.task.id")), null)
);
storageHandler.storageHandlerCommit(HiveConf.getProperties(conf), Context.Operation.OTHER)
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931178435
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
Review Comment:
if memory option is on and user submits a simple query, would he see it in a history after completion, or only after the batch size is reached?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931560420
##########
service/src/java/org/apache/hive/service/server/HiveServer2.java:
##########
@@ -325,6 +327,11 @@ public synchronized void init(HiveConf hiveConf) {
scheduledQueryService = ScheduledQueryExecutionService.startScheduledQueryExecutorService(hiveConf);
}
+ if (hiveConf.getBoolVar(ConfVars.HIVE_QUERY_HISTORY_ENABLED)) {
+ queryHistoryService = new QueryHistoryService(hiveConf, serviceContext).start();
Review Comment:
sorry, I don't get this, what's your point here?
there is a constructor and a fluent start call, that starts the actual "running" things inside
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1932097397
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/IcebergRepository.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Calendar;
+import java.util.GregorianCalendar;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Queue;
+import java.util.UUID;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.io.HiveFileFormatUtils;
+import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.HiveStorageHandler;
+import org.apache.hadoop.hive.ql.metadata.HiveUtils;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.security.authorization.HiveCustomStorageHandlerUtils;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.apache.hadoop.hive.serde2.Serializer;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TaskAttemptID;
+import org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IcebergRepository extends AbstractQueryHistoryRepository implements QueryHistoryRepository {
+ private static final Logger LOG = LoggerFactory.getLogger(IcebergRepository.class);
+ private static final String ICEBERG_STORAGE_HANDLER = "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler";
+
+ private HiveOutputFormat<?, ?> outputFormat;
+ private Serializer serializer;
+ @VisibleForTesting
+ HiveStorageHandler storageHandler;
+ @VisibleForTesting
+ TableDesc tableDesc;
+
+ @Override
Review Comment:
overrides AbstractRepository.init
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1937584605
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.Record;
+import org.apache.hadoop.hive.ql.queryhistory.schema.Schema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<Record> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService flushExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+ private final ScheduledExecutorService periodicFlushExecutor = Executors.newScheduledThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService periodic flush check").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int batchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final int flushInterval;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ batchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_MAX_MEMORY_BYTES);
+ flushInterval = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_FLUSH_INTERVAL_SECONDS);
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [batchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ batchSize, maxQueueSizeInMemoryBytes);
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ INSTANCE = this;
Review Comment:
how about
````
private static Singleton instance;
private QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
...
}
public static QueryHistoryService newInstance(HiveConf inputConf, ServiceContext serviceContext) {
if (instance == null) {
instance = new QueryHistoryService(inputConf, serviceContext);
}
return instance;
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1937613702
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.Record;
+import org.apache.hadoop.hive.ql.queryhistory.schema.Schema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<Record> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService flushExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+ private final ScheduledExecutorService periodicFlushExecutor = Executors.newScheduledThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService periodic flush check").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int batchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final int flushInterval;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ batchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_MAX_MEMORY_BYTES);
+ flushInterval = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_FLUSH_INTERVAL_SECONDS);
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [batchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ batchSize, maxQueueSizeInMemoryBytes);
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ INSTANCE = this;
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for flushing records.
+ */
+ public void start() {
+ repository.init(conf, new Schema());
+
+ try {
+ flushExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ if (flushInterval > 0) {
+ // only flush, don't wait, this is a background task
+ periodicFlushExecutor.scheduleAtFixedRate(this::doFlush, 0, flushInterval, TimeUnit.SECONDS);
+ }
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ private void printConfigInformation() {
+ if (batchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be flushed synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be flushed after every {} queries",
+ batchSize, batchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "flush decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be flushed when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+
+ if (flushInterval == 0) {
+ LOG.info("Query History flush interval size is 0: the service won't check for records to flush periodically");
+ } else {
+ LOG.info("Query History flush interval is {}s: service will attempt to flush records in every {}s",
+ flushInterval, flushInterval);
+ }
+ }
+
+ private QueryHistoryRepository createRepository(HiveConf conf) {
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void logQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ Record record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ CompletableFuture<?> flushFuture = flushRecordsIfNeeded();
+ maybeWaitForRecordsToBeFlushed(batchSize == 0, flushFuture);
+ }
+
+ private CompletableFuture<?> flushRecordsIfNeeded() {
+ if (needToFlush()) {
+ return doFlush();
+ } else {
+ LOG.debug("Not flushing history records yet, current queue size: {}, batchSize: {}",
+ queryHistoryQueue.size(), batchSize);
+ return CompletableFuture.completedFuture(null);
+ }
+ }
+
+ private void maybeWaitForRecordsToBeFlushed(boolean shouldWait, CompletableFuture<?> flushFuture) {
Review Comment:
you mean waitForRecordsToBeFlushed? the rest of the name is ok?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1937645384
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/queryhistory/schema/IcebergQueryHistoryRecord.java:
##########
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+
+/**
+ * Convenient class for creating a QueryHistoryRecord from an iceberg Container.
+ */
+public class IcebergQueryHistoryRecord extends QueryHistoryRecord {
Review Comment:
hm, wait, for this method I need to a) open the method:
```
private void set(Field field, Object value) {
record.put(field.name, value);
}
```
or b) define QueryHistoryRecordBuilder in the Record class, which is not okay, it cannot contain iceberg-specific stuff
what do you suggest here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939579849
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
i don't see the change. I would expect to see something like
````
BaseSemanticAnalyzer
private QueryType quryType = QueryType.DEFAULT;
(Delete/Merge/Update)SemanticAnalyzer#getQueryType(){
return QueryType.DML;
}
ColumnStatsSemanticAnalyzer#getQueryType(){
return QueryType.STATS;
}
DDLSemanticAnalyzerFactory.getAnalyzer(
analyzer.setQueryType(QueryType.DDL);
}
etc
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939579849
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
i don't see the change. I would expect to see something like
````
(Delete/Merge/Update)SemanticAnalyzer#getQueryType(){
return QueryType.DML;
}
ColumnStatsSemanticAnalyzer#getQueryType(){
return QueryType.STATS;
}
DDLSemanticAnalyzerFactory.getAnalyzer(
analyzer.setQueryType(QueryType.DDL);
}
etc
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939245618
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
+1 on @kasakrisz suggestion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939610218
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
we still need massive `setQueryType` in base SA?
queryProperties.setQueryType(sem.getQueryType())
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
we still need massive `setQueryType` in base SA?
````
queryProperties.setQueryType(sem.getQueryType())
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2631361998
@abstractdog I have 1 remaining concern reg the `queryType`, however, if @kasakrisz is ok with that - we can merge
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2630646919
> @abstractdog, what happens to the in-memory records when HS2 is shut down? Should we add flush in `ShutdownHookManager.addShutdownHook(...)`
no, I tried it on a cluster (Kubernetes, simply deleted the HS2 pod), and addShutdownHook was duplicate handling of this thing as HiveServer2.stop() flawlessly took care of flushing the records:
```
if (queryHistoryService != null) {
try {
LOG.info("Calling QueryHistoryService.close from HiveServer2.stop");
queryHistoryService.stop();
} catch (Exception e) {
LOG.error("Error stopping queryHistoryService", e);
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930675845
##########
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:
##########
@@ -755,6 +746,13 @@ int closeInProcess(boolean destroyed) {
// is called to stop the query if it is running, clean query results, and release resources.
@Override
public void close() {
+ if (driverContext.getConf().getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_ENABLED)) {
Review Comment:
could we extract this block into a method say `updateQueryHistory`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931137159
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/IcebergRepository.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Calendar;
+import java.util.GregorianCalendar;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Queue;
+import java.util.UUID;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.io.HiveFileFormatUtils;
+import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.HiveStorageHandler;
+import org.apache.hadoop.hive.ql.metadata.HiveUtils;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.security.authorization.HiveCustomStorageHandlerUtils;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.apache.hadoop.hive.serde2.Serializer;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TaskAttemptID;
+import org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IcebergRepository extends AbstractQueryHistoryRepository implements QueryHistoryRepository {
+ private static final Logger LOG = LoggerFactory.getLogger(IcebergRepository.class);
+ private static final String ICEBERG_STORAGE_HANDLER = "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler";
+
+ private HiveOutputFormat<?, ?> outputFormat;
+ private Serializer serializer;
+ @VisibleForTesting
+ HiveStorageHandler storageHandler;
+ @VisibleForTesting
+ TableDesc tableDesc;
+
+ @Override
+ public void init(HiveConf conf, QueryHistorySchema schema) {
+ super.init(conf, schema);
+ }
+
+ @Override
+ protected Table createTable(Hive hive, Database db) throws HiveException {
+ LOG.info("Creating iceberg Query History table...");
+
+ Table table = getInitialTable();
+
+ // iceberg specific
+ table.setProperty(org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.META_TABLE_STORAGE,
+ ICEBERG_STORAGE_HANDLER);
+ table.setProperty("table_type", "ICEBERG");
+ table.setProperty("write.format.default", "orc");
+ table.setProperty(hive_metastoreConstants.META_TABLE_NAME, QUERY_HISTORY_DB_TABLE_NAME);
+
+ table.setFields(schema.getFields());
+
+ hive.createTable(table, false);
+
+ table = hive.getTable(QUERY_HISTORY_DB_NAME, QUERY_HISTORY_TABLE_NAME);
+ table.setProperty("iceberg.mr.table.location", table.getDataLocation().toString());
+
+ return table;
+ }
+
+ @Override
+ protected void postInitTable(Table table) throws Exception {
+ this.tableDesc = Utilities.getTableDesc(table);
+
+ Map<String, String> map = new HashMap<>();
+ this.storageHandler = HiveUtils.getStorageHandler(conf, tableDesc.getProperties()
+ .getProperty(org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.META_TABLE_STORAGE));
+ storageHandler.configureOutputJobProperties(tableDesc, map);
+
+ //need to obtain a serialized table for later usage
+ map.forEach((key, value) -> conf.set(key, value));
+
+ conf.set(hive_metastoreConstants.META_TABLE_NAME, QUERY_HISTORY_DB_TABLE_NAME);
+
+ HiveCustomStorageHandlerUtils.setWriteOperation(conf, QUERY_HISTORY_DB_TABLE_NAME, Context.Operation.OTHER);
+ this.outputFormat = HiveFileFormatUtils.getHiveOutputFormat(conf, tableDesc);
+
+ this.serializer = tableDesc.getDeserializer(conf);
+ }
+
+ @Override
+ public void persist(Queue<QueryHistoryRecord> records) {
+ int numRecords = records.size();
+ LOG.info("Persisting {} records", numRecords);
+ if (numRecords == 0) {
+ return;
+ }
+
+ try {
+ prepareConfForWrite();
+ JobConf jobConf = new JobConf(conf);
+
+ RecordWriter writer = outputFormat.getHiveRecordWriter(jobConf, null, null, false, null, null);
+ while (!records.isEmpty()) {
+ writeRecord(writer, records.poll());
+ }
+
+ writer.close(false);
+ storageHandler.storageHandlerCommit(HiveConf.getProperties(conf), Context.Operation.OTHER,
+ new TaskAttemptContextImpl(jobConf, TaskAttemptID.forName(conf.get("mapred.task.id")), null));
+
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void writeRecord(RecordWriter writer, QueryHistoryRecord record) throws Exception {
+ Writable w = record.serialize(serializer);
+ writer.write(w);
+ }
+
+ private void prepareConfForWrite() {
+ GregorianCalendar gc = new GregorianCalendar();
+ String queryHistoryId = "queryhistory_" + String.format("%1$4d%2$02d%3$02d%4$02d%5$02d%6$02d",
+ gc.get(Calendar.YEAR), gc.get(Calendar.MONTH) + 1, gc.get(Calendar.DAY_OF_MONTH), gc.get(Calendar.HOUR_OF_DAY),
+ gc.get(Calendar.MINUTE), gc.get(Calendar.SECOND)) + "_" + UUID.randomUUID();
+ conf.set(HiveConf.ConfVars.HIVE_QUERY_ID.varname, queryHistoryId);
+ SessionState.getSessionConf().set(HiveConf.ConfVars.HIVE_QUERY_ID.varname, queryHistoryId);
+
+ long currentTime = System.currentTimeMillis();
+ String jobIdPostFix = String.format("%d_0000", currentTime);
+ conf.set("mapred.task.id", String.format("attempt_%s_r_000000_0", jobIdPostFix));
+ conf.set("iceberg.mr.output.tables", tableDesc.getTableName());
+
+ QueryState queryState = new QueryState.Builder().withGenerateNewQueryId(false).withHiveConf(conf).build();
+ SessionState.get().addQueryState(queryHistoryId, queryState);
+
+ String jobId = String.format("job_%s", jobIdPostFix);
+ SessionStateUtil.addCommitInfo(SessionState.getSessionConf(), tableDesc.getTableName(), jobId, 1,
+ new HashMap(tableDesc.getProperties()));
Review Comment:
`Maps.fromProperties(tableDesc.getProperties())`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931178435
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
Review Comment:
if memory option is on and user submits a simple query, would he see it in a history after completion, or only after the batch size is reached?
what if it's not reached in 1 day or so, should we add time-based `flush` as well?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931887410
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void handleQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ QueryHistoryRecord record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ persistRecordsIfNeeded();
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ persistWithBatchSize(maxBatchSize);
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(), maxBatchSize);
+ }
+ }
+
+ private void persistWithBatchSize(int batchSize) {
+ CountDownLatch latch = new CountDownLatch(1);
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ repository.persist(queryHistoryQueue);
+ latch.countDown();
+ });
+ if (batchSize == 0) {
+ waitForAllRecordsToBePersisted(latch);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted(CountDownLatch queueEmptyCondition) {
+ try {
+ queueEmptyCondition.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ boolean needToPersist() {
Review Comment:
right! and that's by design, so I'm removing synchronized
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930754248
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
please extract logic of `getQueryType(tree)` into `ParseUtils`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930887465
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void handleQuery(DriverContext driverContext) {
Review Comment:
`logQuery()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930669734
##########
ql/src/java/org/apache/hadoop/hive/ql/Compiler.java:
##########
@@ -223,7 +223,7 @@ private BaseSemanticAnalyzer analyze() throws Exception {
sem.startAnalysis();
sem.analyze(tree, context);
} finally {
- sem.endAnalysis();
+ sem.endAnalysis(tree);
Review Comment:
can we store it as property in SA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931075826
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/schema/QueryHistorySchema.java:
##########
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+
+public class QueryHistorySchema {
Review Comment:
should we remove `QueryHistory` from `QueryHistorySchema`, `QueryHistorySchemaField`, `QueryHistoryRecord`, `QueryHistoryRecordEnricher` since it's used only within `queryhistory` package?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1932162116
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
Review Comment:
adding it with default 3600s and a unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930715520
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java:
##########
@@ -688,6 +689,13 @@ DAGClient submit(DAG dag, Ref<TezSessionState> sessionStateRef) throws Exception
return new SyncDagClient(dagClient);
}
+ private DAGClient submitInternal(DAG dag, TezSessionState sessionState) throws TezException, IOException {
+ TezClient tezClient = sessionState.getSession();
+ DAGClient dagClient = tezClient.submitDAG(dag);
+ runtimeContext.init(tezClient);
Review Comment:
can we refactor
````
private DAGClient submitInternal(DAG dag, TezSessionState sessionState) throws TezException, IOException {
runtimeContext.init(sessionState);
return sessionState.getSession().submitDAG(dag);
}
TezRuntimeContext.class
public void init(TezSessionState sessionState) {
this.amAddress = sessionState.getAppMasterUri();
}
TezSessionState.class
String getAppMasterUri() {
return Optional.of(getSession()).map(
session -> session.getAmHost() + ":" + session.getAmPort())
.get();
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930743110
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2047,8 +2070,8 @@ public void startAnalysis() {
/**
* Called when we end analysis of a query.
*/
- public void endAnalysis() {
- // Nothing to do
+ public void endAnalysis(ASTNode tree) {
+ setQueryType(tree); // at this point we know the query type for sure
Review Comment:
can't we set the queryType in `analyzeInternal(ast)`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930773968
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -330,6 +335,24 @@ public void analyze(ASTNode ast, Context ctx) throws SemanticException {
initCtx(ctx);
init(true);
analyzeInternal(ast);
+ postAnalyze();
+ }
+
+ /**
+ * This method is called after analyzeInternal, assuming all the semantic context is present (analyzeInternal is
+ * overridden by the subclasses of BaseSemanticAnalyzer).
+ * The reason why postAnalyze is private is that it's not supposed to be overridden.
+ */
+ private void postAnalyze() {
+ if (queryProperties == null) {
+ queryProperties = new QueryProperties();
+ }
+ queryProperties.setTablesQueried(ctx.getParsedTables().stream()
Review Comment:
use `CacheTableHelper#getUniqueNames()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931149602
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void handleQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ QueryHistoryRecord record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ persistRecordsIfNeeded();
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ persistWithBatchSize(maxBatchSize);
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(), maxBatchSize);
+ }
+ }
+
+ private void persistWithBatchSize(int batchSize) {
+ CountDownLatch latch = new CountDownLatch(1);
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ repository.persist(queryHistoryQueue);
+ latch.countDown();
+ });
+ if (batchSize == 0) {
+ waitForAllRecordsToBePersisted(latch);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted(CountDownLatch queueEmptyCondition) {
+ try {
+ queueEmptyCondition.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ boolean needToPersist() {
+ boolean needToPersistAccordingToQueueSize = queryHistoryQueue.size() >= maxBatchSize;
+ if (needToPersistAccordingToQueueSize) {
+ LOG.info("Need to persist query history records as queue size ({}) >= batch size ({})",
+ queryHistoryQueue.size(), maxBatchSize);
+ return true;
+ }
+
+ // whether ignore the consumed memory or not
+ if (maxQueueSizeInMemoryBytes == 0) {
+ return false;
+ }
+
+ long queueMemorySize =
+ queryHistoryQueue.stream().mapToLong(QueryHistoryRecord::getEstimatedSizeInMemoryBytes).sum();
+ boolean needToPersistAccordingToQueueMemorySize = queueMemorySize > maxQueueSizeInMemoryBytes;
+ if (needToPersistAccordingToQueueMemorySize) {
+ LOG.info("Need to persist query history records as queue size in memory ({} bytes) is greater than the limit " +
+ "({} bytes)",
+ queueMemorySize, maxQueueSizeInMemoryBytes);
+ }
+ return needToPersistAccordingToQueueMemorySize;
+ }
+
+ public void stop() {
+ LOG.info("Stopping QueryHistoryService, leftover records to persist: {}", queryHistoryQueue.size());
+ persistWithBatchSize(0); // persist sync
+ persistExecutor.shutdownNow();
+ }
+
+ private QueryHistoryRecord createRecord(DriverContext driverContext) {
+ return new QueryHistoryRecordEnricher(driverContext, serviceContext, SessionState.get(),
+ SessionState.getPerfLogger()).createRecord();
+ }
+
+ @VisibleForTesting
+ public QueryHistoryRepository getRepository(){
Review Comment:
Intellij reports it as unused
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931059708
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/AbstractQueryHistoryRepository.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.TableType;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.parse.PartitionTransform;
+import org.apache.hadoop.hive.ql.parse.TransformSpec;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public abstract class AbstractQueryHistoryRepository implements QueryHistoryRepository {
+ protected Logger LOG = LoggerFactory.getLogger(getClass());
+ @VisibleForTesting
+ HiveConf conf;
+ protected QueryHistorySchema schema;
+
+ public void init(HiveConf conf, QueryHistorySchema schema) {
+ this.conf = conf;
+ this.schema = schema;
+
+ try (Hive hive = Hive.get(conf)) {
+ Database database = initDatabase(hive);
+ Table table = initTable(hive, database);
+ postInitTable(table);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ protected Database initDatabase(Hive hive) {
+ Database db;
+ try {
+ db = hive.getDatabase(QUERY_HISTORY_DB_NAME);
+ if (db == null) {
+ LOG.warn("Database ({}) for query history table hasn't been found, auto-creating one", QUERY_HISTORY_DB_NAME);
+ String location = getDatabaseLocation(QUERY_HISTORY_DB_NAME);
+ db = new Database(QUERY_HISTORY_DB_NAME, QUERY_HISTORY_DB_COMMENT,
+ location, null);
+ hive.createDatabase(db, false);
+ }
+ return db;
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private String getDatabaseLocation(String databaseName) {
Review Comment:
can we use `new Warehouse(conf).getDefaultExternalDatabasePath()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931817069
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void handleQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ QueryHistoryRecord record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ persistRecordsIfNeeded();
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ persistWithBatchSize(maxBatchSize);
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(), maxBatchSize);
+ }
+ }
+
+ private void persistWithBatchSize(int batchSize) {
+ CountDownLatch latch = new CountDownLatch(1);
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ repository.persist(queryHistoryQueue);
+ latch.countDown();
+ });
+ if (batchSize == 0) {
+ waitForAllRecordsToBePersisted(latch);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted(CountDownLatch queueEmptyCondition) {
+ try {
+ queueEmptyCondition.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ boolean needToPersist() {
+ boolean needToPersistAccordingToQueueSize = queryHistoryQueue.size() >= maxBatchSize;
+ if (needToPersistAccordingToQueueSize) {
+ LOG.info("Need to persist query history records as queue size ({}) >= batch size ({})",
+ queryHistoryQueue.size(), maxBatchSize);
+ return true;
+ }
+
+ // whether ignore the consumed memory or not
+ if (maxQueueSizeInMemoryBytes == 0) {
+ return false;
+ }
+
+ long queueMemorySize =
+ queryHistoryQueue.stream().mapToLong(QueryHistoryRecord::getEstimatedSizeInMemoryBytes).sum();
+ boolean needToPersistAccordingToQueueMemorySize = queueMemorySize > maxQueueSizeInMemoryBytes;
+ if (needToPersistAccordingToQueueMemorySize) {
+ LOG.info("Need to persist query history records as queue size in memory ({} bytes) is greater than the limit " +
+ "({} bytes)",
+ queueMemorySize, maxQueueSizeInMemoryBytes);
+ }
+ return needToPersistAccordingToQueueMemorySize;
+ }
+
+ public void stop() {
+ LOG.info("Stopping QueryHistoryService, leftover records to persist: {}", queryHistoryQueue.size());
+ persistWithBatchSize(0); // persist sync
+ persistExecutor.shutdownNow();
+ }
+
+ private QueryHistoryRecord createRecord(DriverContext driverContext) {
+ return new QueryHistoryRecordEnricher(driverContext, serviceContext, SessionState.get(),
Review Comment:
ok
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931815495
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/AbstractQueryHistoryRepository.java:
##########
@@ -0,0 +1,130 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.TableType;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.parse.PartitionTransform;
+import org.apache.hadoop.hive.ql.parse.TransformSpec;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public abstract class AbstractQueryHistoryRepository implements QueryHistoryRepository {
+ protected Logger LOG = LoggerFactory.getLogger(getClass());
+ @VisibleForTesting
+ HiveConf conf;
+ protected QueryHistorySchema schema;
+
+ public void init(HiveConf conf, QueryHistorySchema schema) {
+ this.conf = conf;
+ this.schema = schema;
+
+ try (Hive hive = Hive.get(conf)) {
+ Database database = initDatabase(hive);
+ Table table = initTable(hive, database);
+ postInitTable(table);
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ protected Database initDatabase(Hive hive) {
+ Database db;
+ try {
+ db = hive.getDatabase(QUERY_HISTORY_DB_NAME);
+ if (db == null) {
+ LOG.warn("Database ({}) for query history table hasn't been found, auto-creating one", QUERY_HISTORY_DB_NAME);
+ String location = getDatabaseLocation(QUERY_HISTORY_DB_NAME);
+ db = new Database(QUERY_HISTORY_DB_NAME, QUERY_HISTORY_DB_COMMENT,
+ location, null);
+ hive.createDatabase(db, false);
+ }
+ return db;
+ } catch (Exception e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private String getDatabaseLocation(String databaseName) {
Review Comment:
in that case please initialize `warehouse` in `init` method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1932031463
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/QueryHistoryRepository.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Queue;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+
+public interface QueryHistoryRepository {
+ String QUERY_HISTORY_DB_NAME = "sys";
+ String QUERY_HISTORY_TABLE_NAME = "query_history";
+ String QUERY_HISTORY_DB_TABLE_NAME =
+ String.format("%s.%s", QUERY_HISTORY_DB_NAME, QUERY_HISTORY_TABLE_NAME);
+
+ String QUERY_HISTORY_DB_COMMENT = "Hive SYS database";
Review Comment:
sys db is created via schema scripts, it should already exist
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1927424022
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -838,6 +845,9 @@ public void storageHandlerCommit(Properties commitProperties, Operation operatio
}
HiveIcebergOutputCommitter committer = getOutputCommitter();
try {
+ if (context != null) {
+ committer.commitTask(context);
Review Comment:
we cannot depend on iceberg in hive-exec in compile-time, there is no iceberg* dependency in hive-exec/pom.xml, that's what's hidden by the storagehandler layer, I think, even if I really like iceberg's api, I'm limited to use the storagehandler
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923852769
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -600,6 +601,11 @@ default void storageHandlerCommit(Properties commitProperties, Operation operati
throw new UnsupportedOperationException();
}
+ default void storageHandlerCommit(Properties commitProperties, Operation operation, TaskAttemptContext context)
Review Comment:
1. okay, let me add
2. because of the fact that we write and commit from HS2, where we have no tez tasks, but we emulate that we have a single writer task that has to be committed, let me know if you know other ways to achieve the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923830727
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/FromDriverContextUpdater.java:
##########
@@ -0,0 +1,158 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one§
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryInfo;
+import org.apache.hadoop.hive.ql.QueryProperties;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.tez.TezRuntimeContext;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.tez.common.counters.DAGCounter;
+import org.apache.tez.common.counters.TaskCounter;
+
+public class FromDriverContextUpdater {
Review Comment:
yeah, was thinking a lot about this, and finally all the magic can be done in 300 lines, so it's fine to have them in a single class, let me refactor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923594488
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java:
##########
@@ -247,4 +275,107 @@ public String getQueryErrorMessage() {
public void setQueryErrorMessage(String queryErrorMessage) {
this.queryErrorMessage = queryErrorMessage;
}
+
+ public long getQueryStartTime() {
+ return getQueryInfo() != null ? getQueryInfo().getBeginTime() : getQueryDisplay().getQueryStartTime();
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY: a simple fetch task is a QUERY
+ * UNKNOWN: if we can't determine the type of the query:
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param sem the semantic analyzer which already analyzed the query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(BaseSemanticAnalyzer sem, ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = sem.getAllRootTasks();
+ QueryProperties props = sem.getQueryProperties();
+ queryType = isAnalyze(props) ? "STATS" : (isDML(props) ? "DML" : null);
+ // common confusion whether CTAS is DML or DDL, let's pick DDL here
+ if (props != null && props.isCTAS()) {
+ queryType = "DDL";
+ }
+ if (queryType != null) {
+ return; //already figured out
+ }
+ queryType = rootTasks.stream().findFirst().map(t -> {
+ String type = t.getType().toString();
+ // a MAPRED stage could mean an INSERT query also
+ if (props != null && type.equalsIgnoreCase("MAPRED")) {
+ return props.isInsert ? "DML" : "QUERY";
+ }
+ return type;
+ }).orElseGet(() -> {
+ // in case of a semantic exception (e.g. a table not found or something else)
+ // the root AST Node can still imply if this is a query, try to fall back to that
+ // instead of "UNKNOWN"
+ if (tree.getText().equalsIgnoreCase("TOK_QUERY")){
+ return "QUERY";
+ }
+ // if there is a fetch task, this is a query
+ if (sem.getFetchTask() != null) {
+ return "QUERY";
+ }
+ // from this point, best efforts come
+ // whether it's a DDL?
+ if (DDLSemanticAnalyzerFactory.handles(tree)) {
+ return "DDL";
+ }
+ // whether it's DML? UPDATE/DELETE/MERGE queries are handled with an instance of RewriteSemanticAnalyzer
+ if (RewriteSemanticAnalyzer.class.isAssignableFrom(sem.getClass())) {
+ return "DML";
+ }
+ return "UNKNOWN";
+ });
+
+ // in case of DDL queries, the DDL type can be figured out from the tree root token, e.g.:
+ // CREATETABLE, ALTERTABLE_ADDPARTS, SHOWDATABASES, SHOWTABLES, etc.
+ this.ddlType = "DDL".equalsIgnoreCase(queryType) ? tree.getText().substring("TOK_".length()) : "";
+ }
+
+ /**
+ * The return value of either isAnalyzeCommand() or isAnalyzeRewrite() is always true for analyze commands:
+ * isAnalyzeCommand=true for "compute statistics",
+ * isAnalyzeRewrite=true for "compute statistics for columns".
+ * @return
+ */
+ private boolean isAnalyze(QueryProperties queryProperties) {
+ return queryProperties != null && (queryProperties.isAnalyzeCommand() || queryProperties.isAnalyzeRewrite());
+ }
+
+ /**
+ * DML (Data Manipulation Language) operations are operations that modify data,
+ * queryProperties.isInsert seems to reflect this very well: true for INSERT [OVERWRITE], UPDATE, DELETE, MERGE
+ * @return
+ */
+ private boolean isDML(QueryProperties queryProperties) {
+ return queryProperties != null && queryProperties.isInsert;
+ }
+
+ public boolean isQueryHistoryExplainEnabled() {
+ return conf.getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_ENABLED) &&
+ conf.getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_EXPLAIN_PLAN_ENABLED);
+ }
+
+ public void setExplainPlan(String explainPlan) {
+ this.explainPlan = explainPlan;
+ }
+
+ public String getExplainPlan() {
+ return explainPlan;
+ }
+
+ public static String getUserNameFromConf(HiveConf conf) {
Review Comment:
pff, leftover, not used anywhere, let me delete :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923587402
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java:
##########
@@ -688,6 +689,13 @@ DAGClient submit(DAG dag, Ref<TezSessionState> sessionStateRef) throws Exception
return new SyncDagClient(dagClient);
}
+ private DAGClient submitInternal(DAG dag, TezSessionState sessionState) throws TezException, IOException {
+ TezClient tezClient = sessionState.getSession();
+ DAGClient dagClient = tezClient.submitDAG(dag);
+ runtimeContext.init(tezClient);
Review Comment:
this is the initialization of the RuntimeContext that I introduced in HIVE-28623...when the DAG is submitted, we have the info for storing runtime information in that context object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923585216
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezTask.java:
##########
@@ -337,13 +337,14 @@ public int execute() {
&& (HiveConf.getBoolVar(conf, HiveConf.ConfVars.TEZ_EXEC_SUMMARY) ||
Utilities.isPerfOrAboveLogging(conf))) {
for (CounterGroup group : runtimeContext.counters) {
- LOG.info(group.getDisplayName() +":");
- for (TezCounter counter: group) {
- LOG.info(" "+counter.getDisplayName()+": "+counter.getValue());
+ monitor.printInfo(group.getDisplayName() + ":");
+ for (TezCounter counter : group) {
+ monitor.printInfo(" " + counter.getDisplayName() + ": " + counter.getValue());
}
}
}
updateNumRows();
+ monitor.endSummary();
Review Comment:
good catch, I'll change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2629380130
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613) **Quality Gate passed**
Issues
 [105 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1938440680
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
this logic knows too much about the analyzer, refactoring it would need parameters like:
QueryProperties, SemanticAnalyzer this.getClass(), ASTNode tree, List<Task<? extends Serializable>> rootTasks
which makes me thing that this belongs to the semantic analyzer instance much more than a util (however I agree that the analyzer class is loaded to hell already)
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
this logic knows too much about the analyzer, refactoring it would need parameters like:
QueryProperties, SemanticAnalyzer this.getClass(), ASTNode tree, List<Task<? extends Serializable>> rootTasks
which makes me think that this belongs to the semantic analyzer instance much more than a util (however I agree that the analyzer class is loaded to hell already)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2626441605
@deniskuzZ : when you have time, please let me know if this can be approved
FYI design doc is here: https://docs.google.com/document/d/1jH-F0cEFMI7_aHz4710RdHC7c_kU6NZPnlw4UhHtVGk/edit?tab=t.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1941930906
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -330,6 +332,17 @@ public void analyze(ASTNode ast, Context ctx) throws SemanticException {
initCtx(ctx);
init(true);
analyzeInternal(ast);
+ postAnalyze();
+ }
+
+ /**
+ * This method is called after analyzeInternal, assuming all the semantic context is present (analyzeInternal is
+ * overridden by the subclasses of BaseSemanticAnalyzer).
+ * The reason why postAnalyze is private is that it's not supposed to be overridden.
+ */
+ private void postAnalyze() {
Review Comment:
why do we need `postAnalyze`? can't we move `setUsedTables` to `endAnalysis`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2630194487
latest NPE in related qtest must be a consequence of the refactors, I'm checking it:
https://ci.hive.apache.org/blue/organizations/jenkins/hive-precommit/detail/PR-5613/10/tests/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939196213
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -330,6 +335,19 @@ public void analyze(ASTNode ast, Context ctx) throws SemanticException {
initCtx(ctx);
init(true);
analyzeInternal(ast);
+ postAnalyze();
+ }
+
+ /**
+ * This method is called after analyzeInternal, assuming all the semantic context is present (analyzeInternal is
+ * overridden by the subclasses of BaseSemanticAnalyzer).
+ * The reason why postAnalyze is private is that it's not supposed to be overridden.
+ */
+ private void postAnalyze() {
+ if (queryProperties == null) {
+ queryProperties = new QueryProperties();
+ }
+ queryProperties.setUsedTables(CacheTableHelper.getUniqueNames(ctx.getParsedTables()));
Review Comment:
yes, just a line break please
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923518380
##########
ql/src/java/org/apache/hadoop/hive/ql/Compiler.java:
##########
@@ -224,6 +224,7 @@ private BaseSemanticAnalyzer analyze() throws Exception {
sem.analyze(tree, context);
} finally {
sem.endAnalysis();
+ driverContext.setQueryType(sem, tree);
Review Comment:
the service works for failed queries too, where e.g. SematicException happened, that's where the finally block kicks in, making the service create a query history record containing the exception too
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923522667
##########
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:
##########
@@ -94,7 +96,7 @@ public Driver(HiveConf conf) {
// Pass lineageState when a driver instantiates another Driver to run or compile another query
public Driver(HiveConf conf, Context ctx, LineageState lineageState) {
- this(QueryState.getNewQueryState(conf, lineageState), null);
+ this(QueryState.getNewQueryState(conf, lineageState), QueryInfo.getFromConf(conf));
Review Comment:
why is this change needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1925134762
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryInfo.java:
##########
@@ -40,10 +45,17 @@ public QueryInfo(String state, String userName, String executionEngine, String s
this.userName = userName;
this.executionEngine = executionEngine;
this.beginTime = System.currentTimeMillis();
+ this.beginTimeUTC = Instant.ofEpochMilli(beginTime);
this.sessionId = sessionId;
this.operationId = operationId;
}
+ public static QueryInfo getFromConf(HiveConf conf) {
+ return new QueryInfo("INITIALIZED", conf.get(DriverContext.DRIVER_DEFAULT_USER_NAME_PROP),
Review Comment:
double-check, hive-exec doesn't depend on hive-service (actually it's other way around), so QueryInfo cannot have a reference on OperationState...I'm afraid refactoring this is beyond the scope of this ticket, created jira about it: https://issues.apache.org/jira/browse/HIVE-28721
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryInfo.java:
##########
@@ -40,10 +45,17 @@ public QueryInfo(String state, String userName, String executionEngine, String s
this.userName = userName;
this.executionEngine = executionEngine;
this.beginTime = System.currentTimeMillis();
+ this.beginTimeUTC = Instant.ofEpochMilli(beginTime);
this.sessionId = sessionId;
this.operationId = operationId;
}
+ public static QueryInfo getFromConf(HiveConf conf) {
+ return new QueryInfo("INITIALIZED", conf.get(DriverContext.DRIVER_DEFAULT_USER_NAME_PROP),
Review Comment:
double-checked, hive-exec doesn't depend on hive-service (actually it's other way around), so QueryInfo cannot have a reference on OperationState...I'm afraid refactoring this is beyond the scope of this ticket, created jira about it: https://issues.apache.org/jira/browse/HIVE-28721
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1926732011
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -15713,6 +15714,7 @@ private void copyInfoToQueryProperties(QueryProperties queryProperties) {
queryProperties.setNoScanAnalyzeCommand(qb.getParseInfo().isNoScanAnalyzeCommand());
queryProperties.setAnalyzeRewrite(qb.isAnalyzeRewrite());
queryProperties.setCTAS(qb.getTableDesc() != null);
+ queryProperties.setInsert(qb.getParseInfo().hasInsertTables());
Review Comment:
okay, I just brought this from a downstream-only commit which isn't that clear...so qb.getParseInfo().hasInsertTables() is true in all these cases:
```
INSERT [OVERWRITE], UPDATE, DELETE, MERGE
```
that's because all the UPDATE/DELETE/MERGE operations are rewritten to INSERTs, so isInsert brings much confusion here (whether it was an INSERT or it was rewritten to an insert), so...hm, let me rewrite this property to **isDML**, because that's the point here I want to exactly catch, if someone needs isInsert for only INSERT INTO queries, it can be implemented later, I don't have to bother with this now I think
let me add a unit test also to TestQueryProperties
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1926873907
##########
ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDTFGetSplits.java:
##########
@@ -69,6 +69,7 @@
import org.apache.hadoop.hive.metastore.api.FieldSchema;
import org.apache.hadoop.hive.ql.Context;
import org.apache.hadoop.hive.ql.Driver;
+import org.apache.hadoop.hive.ql.QueryInfo;
Review Comment:
yeah
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923566174
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java:
##########
@@ -20,6 +20,7 @@
import java.io.IOException;
import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
Review Comment:
no changes here, could we revert?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923956191
##########
ql/src/test/results/clientpositive/llap/show_functions.q.out:
##########
@@ -218,6 +218,12 @@ hash
hex
histogram_numeric
hour
+iceberg_bucket
Review Comment:
because of the test-scoped iceberg handler dependency
it was needed to be able to run query_history.q in the itests/qtest module (and not in the separate iceberg related modules
https://github.com/apache/hive/pull/5613/files#diff-c0bf514f60161dcf9a3b90898274ad5a3e550f0a1d0169ca8b0521a84382cb27R289
```
<dependency>
<groupId>org.apache.hive</groupId>
<artifactId>hive-iceberg-handler</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1942409655
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -330,6 +332,17 @@ public void analyze(ASTNode ast, Context ctx) throws SemanticException {
initCtx(ctx);
init(true);
analyzeInternal(ast);
+ postAnalyze();
+ }
+
+ /**
+ * This method is called after analyzeInternal, assuming all the semantic context is present (analyzeInternal is
+ * overridden by the subclasses of BaseSemanticAnalyzer).
+ * The reason why postAnalyze is private is that it's not supposed to be overridden.
+ */
+ private void postAnalyze() {
Review Comment:
THX
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2651599761
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613) **Quality Gate passed**
Issues
 [95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2636982797
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613) **Quality Gate passed**
Issues
 [85 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2637584905
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613) **Quality Gate passed**
Issues
 [85 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2652730704
precommit is green, about to merge soon
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1941978862
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -330,6 +332,17 @@ public void analyze(ASTNode ast, Context ctx) throws SemanticException {
initCtx(ctx);
init(true);
analyzeInternal(ast);
+ postAnalyze();
+ }
+
+ /**
+ * This method is called after analyzeInternal, assuming all the semantic context is present (analyzeInternal is
+ * overridden by the subclasses of BaseSemanticAnalyzer).
+ * The reason why postAnalyze is private is that it's not supposed to be overridden.
+ */
+ private void postAnalyze() {
Review Comment:
good catch! it works properly in endAnalysis too
handled in: https://github.com/apache/hive/pull/5613/commits/812bb17b47d6f6f10deb706158e3959b38ddaea2
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/ParseUtils.java:
##########
@@ -786,6 +786,6 @@ private static Map<String, TypeInfo> collectStructFieldNames(ASTNode structTypeN
* @param node the ASTNode
*/
public static String getNodeName(ASTNode node) {
- return node == null ? "" : node.getText().substring("TOK_".length());
+ return node == null || node.getText() == null ? "" : node.getText().substring("TOK_".length());
Review Comment:
https://github.com/apache/hive/pull/5613/commits/812bb17b47d6f6f10deb706158e3959b38ddaea2
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2621697817
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613) **Quality Gate passed**
Issues
 [103 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923779057
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/QB.java:
##########
@@ -506,4 +511,15 @@ public void addSubqExprAlias(ASTNode expressionTree, SemanticAnalyzer semanticAn
aliasToSubqExpr.put(alias, qbexpr);
addAlias(alias);
}
+
+ public void iterate(QBVisitor qbVisitor) {
Review Comment:
not sure we need this in order to extract table names, see `ctx.getParsedTables()`
cc @kasakrisz
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1926732011
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
##########
@@ -15713,6 +15714,7 @@ private void copyInfoToQueryProperties(QueryProperties queryProperties) {
queryProperties.setNoScanAnalyzeCommand(qb.getParseInfo().isNoScanAnalyzeCommand());
queryProperties.setAnalyzeRewrite(qb.isAnalyzeRewrite());
queryProperties.setCTAS(qb.getTableDesc() != null);
+ queryProperties.setInsert(qb.getParseInfo().hasInsertTables());
Review Comment:
okay, I just brought this from a downstream-only commit which isn't that clear...so qb.getParseInfo().hasInsertTables() is true in all these cases:
```
INSERT [OVERWRITE], UPDATE, DELETE, MERGE
```
that's because all the UPDATE/DELETE/MERGE operations are rewritten to INSERTs, so isInsert contains much confusion here, so...hm, let me rewrite this to isDML, because that's the point here I want to exactly catch, if someone needs isInsert for only INSERT INTO queries, it can be implemented later, I don't have to bother with this now I think
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2612234559
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613) **Quality Gate passed**
Issues
 [97 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923893630
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.persist.QueryHistoryPersistor;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ private QueryHistoryPersistor queryHistoryPersistor;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+ private int maxBatchSize = 0;
+ private int maxQueueSizeInMemoryBytes = 0;
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService persistor thread").build());
+
+ private ServiceContext serviceContext;
+
+ public static QueryHistoryService start(HiveConf inputConf, ServiceContext serviceContext) {
+ HiveConf conf = new HiveConf(inputConf);
+ // we can only hit this codepath if the INSTANCE is already initialized while calling start(...)
+ // for convenient test compatibility, this call is allowed to be idempotent (LOG.warn instead of exception)
+ if (INSTANCE != null) {
+ LOG.warn("There is already a QueryHistoryService instance ({}), returning existing instance", INSTANCE);
+ return INSTANCE;
+ }
+
+ INSTANCE = createServiceInstance(serviceContext, conf);
+ return INSTANCE;
+ }
+
+ private static QueryHistoryService createServiceInstance(ServiceContext serviceContext, HiveConf conf) {
+ LOG.info("Starting QueryHistoryService");
+ QueryHistoryService queryHistoryService = new QueryHistoryService();
+
+ initService(serviceContext, conf, queryHistoryService);
+ return queryHistoryService;
+ }
+
+ @VisibleForTesting
+ static void initService(ServiceContext serviceContext, HiveConf conf, QueryHistoryService queryHistoryService) {
+ queryHistoryService.queryHistoryPersistor = createPersistor(conf);
+ queryHistoryService.queryHistoryPersistor.init(conf, new QueryHistorySchema());
+ queryHistoryService.maxBatchSize = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE);
+ queryHistoryService.maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_MEMORY_BYTES);
+
+ queryHistoryService.printConfigInformation();
+ queryHistoryService.serviceContext = serviceContext;
+
+ try {
+ queryHistoryService.persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the persistor,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+ LOG.info("QueryHistoryService started [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ queryHistoryService.maxBatchSize, queryHistoryService.maxQueueSizeInMemoryBytes);
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ public static QueryHistoryPersistor createPersistor(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryPersistor) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSISTOR_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private static void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryPersistor.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryPersistor.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void handle(DriverContext driverContext) {
+ if (INSTANCE == null){
+ // we can only hit this codepath if hive.query.history.service.enabled=true but the INSTANCE is not initialized,
+ // which is unlikely to happen under a production HiveServer2, as it initializes the service properly
+ LOG.warn("QueryHistoryService is called but not initialized, this is only fine under testing circumstances.");
+ return;
+ }
+ INSTANCE.handleRecord(driverContext);
+ }
+
+ @VisibleForTesting
+ protected QueryHistoryService() {
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ queryHistoryPersistor.persist(queryHistoryQueue);
+ });
+ if (maxBatchSize == 0) {
+ waitForAllRecordsToBePersisted();
+ }
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(),
+ maxBatchSize);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted() {
+ while (!queryHistoryQueue.isEmpty()){
+ try {
+ LOG.info("Waiting for {} records to persist synchronously", queryHistoryQueue.size());
+ Thread.sleep(500);
Review Comment:
that is a red flag and anti-pattern, please use completablefuture/notify etc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923503507
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/queryhistory/schema/ExampleQueryHistoryRecord.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import com.google.common.collect.Sets;
+
+import java.time.Instant;
+import java.util.Collections;
+import java.util.Map;
+import java.util.Set;
+
+public class ExampleQueryHistoryRecord extends QueryHistoryRecord {
Review Comment:
agreed, but **Dummy** is an empty thing, here **Example** reflects that I intentionally filled all the fields in this record, does it make sense? but as we're not having a DummyQueryHistoryRecord yet, I'm ready to rename
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923512881
##########
ql/src/java/org/apache/hadoop/hive/ql/Compiler.java:
##########
@@ -224,6 +224,7 @@ private BaseSemanticAnalyzer analyze() throws Exception {
sem.analyze(tree, context);
} finally {
sem.endAnalysis();
+ driverContext.setQueryType(sem, tree);
Review Comment:
why do we need to set QueryType in a finally?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923534667
##########
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:
##########
@@ -94,7 +96,7 @@ public Driver(HiveConf conf) {
// Pass lineageState when a driver instantiates another Driver to run or compile another query
public Driver(HiveConf conf, Context ctx, LineageState lineageState) {
- this(QueryState.getNewQueryState(conf, lineageState), null);
+ this(QueryState.getNewQueryState(conf, lineageState), QueryInfo.getFromConf(conf));
Review Comment:
in the beginning of the implementation, I ran full precommit tests with enabling this service by default and enabling everywhere it's possible (all qtests for example), which led to some problems that are not visible otherwise
one of those problems were that the Driver instance can be created here with null QueryInfo, which led to NPEs in both prod and test code (as far as I can remember), so I decided to create an empty one here, which looks much more valid then null (so actually, the query history service was the one that showed QueryInfo==null can cause problems)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923554450
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java:
##########
@@ -247,4 +275,107 @@ public String getQueryErrorMessage() {
public void setQueryErrorMessage(String queryErrorMessage) {
this.queryErrorMessage = queryErrorMessage;
}
+
+ public long getQueryStartTime() {
+ return getQueryInfo() != null ? getQueryInfo().getBeginTime() : getQueryDisplay().getQueryStartTime();
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY: a simple fetch task is a QUERY
+ * UNKNOWN: if we can't determine the type of the query:
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param sem the semantic analyzer which already analyzed the query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(BaseSemanticAnalyzer sem, ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = sem.getAllRootTasks();
+ QueryProperties props = sem.getQueryProperties();
+ queryType = isAnalyze(props) ? "STATS" : (isDML(props) ? "DML" : null);
+ // common confusion whether CTAS is DML or DDL, let's pick DDL here
+ if (props != null && props.isCTAS()) {
+ queryType = "DDL";
+ }
+ if (queryType != null) {
+ return; //already figured out
+ }
+ queryType = rootTasks.stream().findFirst().map(t -> {
+ String type = t.getType().toString();
+ // a MAPRED stage could mean an INSERT query also
+ if (props != null && type.equalsIgnoreCase("MAPRED")) {
+ return props.isInsert ? "DML" : "QUERY";
+ }
+ return type;
+ }).orElseGet(() -> {
+ // in case of a semantic exception (e.g. a table not found or something else)
+ // the root AST Node can still imply if this is a query, try to fall back to that
+ // instead of "UNKNOWN"
+ if (tree.getText().equalsIgnoreCase("TOK_QUERY")){
+ return "QUERY";
+ }
+ // if there is a fetch task, this is a query
+ if (sem.getFetchTask() != null) {
+ return "QUERY";
+ }
+ // from this point, best efforts come
+ // whether it's a DDL?
+ if (DDLSemanticAnalyzerFactory.handles(tree)) {
+ return "DDL";
+ }
+ // whether it's DML? UPDATE/DELETE/MERGE queries are handled with an instance of RewriteSemanticAnalyzer
+ if (RewriteSemanticAnalyzer.class.isAssignableFrom(sem.getClass())) {
+ return "DML";
+ }
+ return "UNKNOWN";
+ });
+
+ // in case of DDL queries, the DDL type can be figured out from the tree root token, e.g.:
+ // CREATETABLE, ALTERTABLE_ADDPARTS, SHOWDATABASES, SHOWTABLES, etc.
+ this.ddlType = "DDL".equalsIgnoreCase(queryType) ? tree.getText().substring("TOK_".length()) : "";
+ }
+
+ /**
+ * The return value of either isAnalyzeCommand() or isAnalyzeRewrite() is always true for analyze commands:
+ * isAnalyzeCommand=true for "compute statistics",
+ * isAnalyzeRewrite=true for "compute statistics for columns".
+ * @return
+ */
+ private boolean isAnalyze(QueryProperties queryProperties) {
+ return queryProperties != null && (queryProperties.isAnalyzeCommand() || queryProperties.isAnalyzeRewrite());
+ }
+
+ /**
+ * DML (Data Manipulation Language) operations are operations that modify data,
+ * queryProperties.isInsert seems to reflect this very well: true for INSERT [OVERWRITE], UPDATE, DELETE, MERGE
+ * @return
+ */
+ private boolean isDML(QueryProperties queryProperties) {
+ return queryProperties != null && queryProperties.isInsert;
+ }
+
+ public boolean isQueryHistoryExplainEnabled() {
+ return conf.getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_ENABLED) &&
Review Comment:
why do you need to evaluate it every time from conf + not even sure it should be here
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923730527
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverFactory.java:
##########
@@ -40,7 +40,7 @@ private DriverFactory() {
}
public static IDriver newDriver(HiveConf conf) {
- return newDriver(getNewQueryState(conf), null);
+ return newDriver(getNewQueryState(conf), QueryInfo.getFromConf(conf));
Review Comment:
checked, we need this
we use this method extensively, mostly in tests, without this part the attached qtest is ruined, because query_history.start_time becomes NULL
<img width="697" alt="Screenshot 2025-01-21 at 14 23 24" src="https://github.com/user-attachments/assets/a2b359bc-1d25-47ee-a2f6-3ae80cfce2ed" />
I think any codepath that worked with a null QueryInfo should work properly with a non-null one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923535003
##########
ql/src/java/org/apache/hadoop/hive/ql/Driver.java:
##########
@@ -755,6 +753,13 @@ int closeInProcess(boolean destroyed) {
// is called to stop the query if it is running, clean query results, and release resources.
@Override
public void close() {
+ if (driverContext.getConf().getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_ENABLED)) {
+ if (driverState.isClosed() || driverState.isDestroyed()){
+ LOG.warn("Driver instance {} already closed or destroyed, prevent handling query history", this);
+ } else{
Review Comment:
ack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923813753
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/FromDriverContextUpdater.java:
##########
@@ -0,0 +1,158 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one§
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
Review Comment:
why do we create a separate package for that, there is already `history` package?
````
HiveHistory. Logs information such as query, query plan, runtime statistics into a file. Each session uses a new object, which creates a new file.
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923825674
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/FromDriverContextUpdater.java:
##########
@@ -0,0 +1,158 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one§
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryInfo;
+import org.apache.hadoop.hive.ql.QueryProperties;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.tez.TezRuntimeContext;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.tez.common.counters.DAGCounter;
+import org.apache.tez.common.counters.TaskCounter;
+
+public class FromDriverContextUpdater {
Review Comment:
can we create `QueryHistoryRecordEnricher#createRecord` and move all updater logic there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923825674
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/FromDriverContextUpdater.java:
##########
@@ -0,0 +1,158 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one§
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryInfo;
+import org.apache.hadoop.hive.ql.QueryProperties;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.tez.TezRuntimeContext;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.tez.common.counters.DAGCounter;
+import org.apache.tez.common.counters.TaskCounter;
+
+public class FromDriverContextUpdater {
Review Comment:
can we create `QueryHistoryRecordEnricher#createRecord` and move all updater logic there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1925128843
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java:
##########
@@ -247,4 +275,107 @@ public String getQueryErrorMessage() {
public void setQueryErrorMessage(String queryErrorMessage) {
this.queryErrorMessage = queryErrorMessage;
}
+
+ public long getQueryStartTime() {
+ return getQueryInfo() != null ? getQueryInfo().getBeginTime() : getQueryDisplay().getQueryStartTime();
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY: a simple fetch task is a QUERY
+ * UNKNOWN: if we can't determine the type of the query:
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param sem the semantic analyzer which already analyzed the query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(BaseSemanticAnalyzer sem, ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = sem.getAllRootTasks();
+ QueryProperties props = sem.getQueryProperties();
+ queryType = isAnalyze(props) ? "STATS" : (isDML(props) ? "DML" : null);
+ // common confusion whether CTAS is DML or DDL, let's pick DDL here
+ if (props != null && props.isCTAS()) {
+ queryType = "DDL";
+ }
+ if (queryType != null) {
+ return; //already figured out
+ }
+ queryType = rootTasks.stream().findFirst().map(t -> {
+ String type = t.getType().toString();
+ // a MAPRED stage could mean an INSERT query also
+ if (props != null && type.equalsIgnoreCase("MAPRED")) {
+ return props.isInsert ? "DML" : "QUERY";
+ }
+ return type;
+ }).orElseGet(() -> {
+ // in case of a semantic exception (e.g. a table not found or something else)
+ // the root AST Node can still imply if this is a query, try to fall back to that
+ // instead of "UNKNOWN"
+ if (tree.getText().equalsIgnoreCase("TOK_QUERY")){
+ return "QUERY";
+ }
+ // if there is a fetch task, this is a query
+ if (sem.getFetchTask() != null) {
+ return "QUERY";
+ }
+ // from this point, best efforts come
+ // whether it's a DDL?
+ if (DDLSemanticAnalyzerFactory.handles(tree)) {
+ return "DDL";
+ }
+ // whether it's DML? UPDATE/DELETE/MERGE queries are handled with an instance of RewriteSemanticAnalyzer
+ if (RewriteSemanticAnalyzer.class.isAssignableFrom(sem.getClass())) {
+ return "DML";
+ }
+ return "UNKNOWN";
+ });
+
+ // in case of DDL queries, the DDL type can be figured out from the tree root token, e.g.:
+ // CREATETABLE, ALTERTABLE_ADDPARTS, SHOWDATABASES, SHOWTABLES, etc.
+ this.ddlType = "DDL".equalsIgnoreCase(queryType) ? tree.getText().substring("TOK_".length()) : "";
+ }
+
+ /**
+ * The return value of either isAnalyzeCommand() or isAnalyzeRewrite() is always true for analyze commands:
+ * isAnalyzeCommand=true for "compute statistics",
+ * isAnalyzeRewrite=true for "compute statistics for columns".
+ * @return
+ */
+ private boolean isAnalyze(QueryProperties queryProperties) {
+ return queryProperties != null && (queryProperties.isAnalyzeCommand() || queryProperties.isAnalyzeRewrite());
+ }
+
+ /**
+ * DML (Data Manipulation Language) operations are operations that modify data,
+ * queryProperties.isInsert seems to reflect this very well: true for INSERT [OVERWRITE], UPDATE, DELETE, MERGE
+ * @return
+ */
+ private boolean isDML(QueryProperties queryProperties) {
+ return queryProperties != null && queryProperties.isInsert;
+ }
+
+ public boolean isQueryHistoryExplainEnabled() {
+ return conf.getBoolVar(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_ENABLED) &&
Review Comment:
moved the convenience method to the HiveConf (which already has similar ones like isWebUiQueryInfoCacheEnabled)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1927375178
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.persist.QueryHistoryPersistor;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ private QueryHistoryPersistor queryHistoryPersistor;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+ private int maxBatchSize = 0;
+ private int maxQueueSizeInMemoryBytes = 0;
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService persistor thread").build());
+
+ private ServiceContext serviceContext;
+
+ public static QueryHistoryService start(HiveConf inputConf, ServiceContext serviceContext) {
+ HiveConf conf = new HiveConf(inputConf);
+ // we can only hit this codepath if the INSTANCE is already initialized while calling start(...)
+ // for convenient test compatibility, this call is allowed to be idempotent (LOG.warn instead of exception)
+ if (INSTANCE != null) {
+ LOG.warn("There is already a QueryHistoryService instance ({}), returning existing instance", INSTANCE);
+ return INSTANCE;
+ }
+
+ INSTANCE = createServiceInstance(serviceContext, conf);
+ return INSTANCE;
+ }
+
+ private static QueryHistoryService createServiceInstance(ServiceContext serviceContext, HiveConf conf) {
+ LOG.info("Starting QueryHistoryService");
+ QueryHistoryService queryHistoryService = new QueryHistoryService();
+
+ initService(serviceContext, conf, queryHistoryService);
+ return queryHistoryService;
+ }
+
+ @VisibleForTesting
+ static void initService(ServiceContext serviceContext, HiveConf conf, QueryHistoryService queryHistoryService) {
+ queryHistoryService.queryHistoryPersistor = createPersistor(conf);
+ queryHistoryService.queryHistoryPersistor.init(conf, new QueryHistorySchema());
+ queryHistoryService.maxBatchSize = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE);
+ queryHistoryService.maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_MEMORY_BYTES);
+
+ queryHistoryService.printConfigInformation();
+ queryHistoryService.serviceContext = serviceContext;
+
+ try {
+ queryHistoryService.persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the persistor,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+ LOG.info("QueryHistoryService started [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ queryHistoryService.maxBatchSize, queryHistoryService.maxQueueSizeInMemoryBytes);
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ public static QueryHistoryPersistor createPersistor(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryPersistor) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSISTOR_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private static void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryPersistor.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryPersistor.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void handle(DriverContext driverContext) {
+ if (INSTANCE == null){
+ // we can only hit this codepath if hive.query.history.service.enabled=true but the INSTANCE is not initialized,
+ // which is unlikely to happen under a production HiveServer2, as it initializes the service properly
+ LOG.warn("QueryHistoryService is called but not initialized, this is only fine under testing circumstances.");
+ return;
+ }
+ INSTANCE.handleRecord(driverContext);
+ }
+
+ @VisibleForTesting
+ protected QueryHistoryService() {
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ queryHistoryPersistor.persist(queryHistoryQueue);
+ });
+ if (maxBatchSize == 0) {
+ waitForAllRecordsToBePersisted();
+ }
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(),
+ maxBatchSize);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted() {
+ while (!queryHistoryQueue.isEmpty()){
+ try {
+ LOG.info("Waiting for {} records to persist synchronously", queryHistoryQueue.size());
+ Thread.sleep(500);
Review Comment:
okay, I usually use CountDownLatch for this purpose, let me try it in this scenario
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923883701
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.persist.QueryHistoryPersistor;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ private QueryHistoryPersistor queryHistoryPersistor;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+ private int maxBatchSize = 0;
+ private int maxQueueSizeInMemoryBytes = 0;
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService persistor thread").build());
+
+ private ServiceContext serviceContext;
+
+ public static QueryHistoryService start(HiveConf inputConf, ServiceContext serviceContext) {
+ HiveConf conf = new HiveConf(inputConf);
+ // we can only hit this codepath if the INSTANCE is already initialized while calling start(...)
+ // for convenient test compatibility, this call is allowed to be idempotent (LOG.warn instead of exception)
+ if (INSTANCE != null) {
+ LOG.warn("There is already a QueryHistoryService instance ({}), returning existing instance", INSTANCE);
+ return INSTANCE;
+ }
+
+ INSTANCE = createServiceInstance(serviceContext, conf);
+ return INSTANCE;
+ }
+
+ private static QueryHistoryService createServiceInstance(ServiceContext serviceContext, HiveConf conf) {
+ LOG.info("Starting QueryHistoryService");
+ QueryHistoryService queryHistoryService = new QueryHistoryService();
+
+ initService(serviceContext, conf, queryHistoryService);
+ return queryHistoryService;
+ }
+
+ @VisibleForTesting
+ static void initService(ServiceContext serviceContext, HiveConf conf, QueryHistoryService queryHistoryService) {
+ queryHistoryService.queryHistoryPersistor = createPersistor(conf);
+ queryHistoryService.queryHistoryPersistor.init(conf, new QueryHistorySchema());
+ queryHistoryService.maxBatchSize = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE);
+ queryHistoryService.maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_MEMORY_BYTES);
+
+ queryHistoryService.printConfigInformation();
+ queryHistoryService.serviceContext = serviceContext;
+
+ try {
+ queryHistoryService.persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the persistor,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
Review Comment:
why don't you add this to `Thread#run`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2629053342
@abstractdog, are we reporting queueName here? do we need fixes from https://issues.apache.org/jira/secure/attachment/12933964/HIVE-20282.01.patch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931620649
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/QueryHistoryRepository.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Queue;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+
+public interface QueryHistoryRepository {
+ String QUERY_HISTORY_DB_NAME = "sys";
+ String QUERY_HISTORY_TABLE_NAME = "query_history";
+ String QUERY_HISTORY_DB_TABLE_NAME =
+ String.format("%s.%s", QUERY_HISTORY_DB_NAME, QUERY_HISTORY_TABLE_NAME);
+
+ String QUERY_HISTORY_DB_COMMENT = "Hive SYS database";
+
+ // an artificial hive query id for an artificial session for creating the iceberg query history table
+ // this is needed because the partition_transform_spec is handed over through SessionState resource which
+ // needs a QueryState instance (acquired by a query id)
+ String QUERY_ID_FOR_TABLE_CREATION = "query_history_service_query_id_for_table_creation";
+
+ void init(HiveConf conf, QueryHistorySchema schema);
+
+ void persist(Queue<QueryHistoryRecord> records);
Review Comment:
yeah, it gets a queue that should be flushed actually, I'm fine with renaming...in this case let me rename the rest of the "persist" occurrences where it makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2618075683
> since it's an iceberg table, should we compact it (could have a huge amount of small files over time)?
theoretically, yes, it can have many small files eventually, that's what I tried to balance with the different flushing strategies
my final decision was to not take care of this (which became more valid as we disabled this service by default) in the scope of this jira, because there were challenges that I didn't want to cope with now, considerations were:
1. the platform (any kind of downstream data platform having hive) might want to take care of compacting this iceberg table automatically, or the user can do it (thank god it's an iceberg table so that any compaction implementation can be utilized here that we already have)
2. the platform I'm working on is like this (you know it I think, this explanation is for the wider audience): HS2 is the permanent component (where this service runs), and the expensive compute nodes are ephemeral/autoscaled, so whether to run compaction - that spin up computes - looked like a decision I didn't want to make this time: not to mention on that particular platform, it's told to be a separate component taking care of compacting tables (in the future/present, IDK)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930421202
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -838,6 +845,9 @@ public void storageHandlerCommit(Properties commitProperties, Operation operatio
}
HiveIcebergOutputCommitter committer = getOutputCommitter();
try {
+ if (context != null) {
+ committer.commitTask(context);
Review Comment:
can't we get `HiveIcebergOutputCommitter` from `TaskAttemptContext context`?
````
OutputCommitter commiter = getOutputCommitter(context.getConfiguration());
committer.commitTask(context);
....
public static OutputCommitter getOutputCommitter(Configuration conf) {
return (OutputCommitter)ReflectionUtils.newInstance(this.getClass("mapred.output.committer.class", FileOutputCommitter.class, OutputCommitter.class), conf);
}
````
see `FileOutputFormatContainer# getOutputCommitter(TaskAttemptContext context)`
or expose `getOutputCommitter()` via `HiveStorageHandler` interface
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930891865
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void handleQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ QueryHistoryRecord record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ persistRecordsIfNeeded();
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ persistWithBatchSize(maxBatchSize);
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(), maxBatchSize);
+ }
+ }
+
+ private void persistWithBatchSize(int batchSize) {
+ CountDownLatch latch = new CountDownLatch(1);
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ repository.persist(queryHistoryQueue);
+ latch.countDown();
+ });
+ if (batchSize == 0) {
+ waitForAllRecordsToBePersisted(latch);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted(CountDownLatch queueEmptyCondition) {
+ try {
+ queueEmptyCondition.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ boolean needToPersist() {
+ boolean needToPersistAccordingToQueueSize = queryHistoryQueue.size() >= maxBatchSize;
+ if (needToPersistAccordingToQueueSize) {
+ LOG.info("Need to persist query history records as queue size ({}) >= batch size ({})",
+ queryHistoryQueue.size(), maxBatchSize);
+ return true;
+ }
+
+ // whether ignore the consumed memory or not
+ if (maxQueueSizeInMemoryBytes == 0) {
+ return false;
+ }
+
+ long queueMemorySize =
+ queryHistoryQueue.stream().mapToLong(QueryHistoryRecord::getEstimatedSizeInMemoryBytes).sum();
+ boolean needToPersistAccordingToQueueMemorySize = queueMemorySize > maxQueueSizeInMemoryBytes;
+ if (needToPersistAccordingToQueueMemorySize) {
+ LOG.info("Need to persist query history records as queue size in memory ({} bytes) is greater than the limit " +
+ "({} bytes)",
+ queueMemorySize, maxQueueSizeInMemoryBytes);
+ }
+ return needToPersistAccordingToQueueMemorySize;
+ }
+
+ public void stop() {
+ LOG.info("Stopping QueryHistoryService, leftover records to persist: {}", queryHistoryQueue.size());
+ persistWithBatchSize(0); // persist sync
+ persistExecutor.shutdownNow();
+ }
+
+ private QueryHistoryRecord createRecord(DriverContext driverContext) {
+ return new QueryHistoryRecordEnricher(driverContext, serviceContext, SessionState.get(),
Review Comment:
why do you need to pass `SessionState sessionState`, `PerfLogger perfLogger` to enricher? both of them could be. accessed. from SessionState
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931113257
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/IcebergRepository.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Calendar;
+import java.util.GregorianCalendar;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Queue;
+import java.util.UUID;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.io.HiveFileFormatUtils;
+import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.HiveStorageHandler;
+import org.apache.hadoop.hive.ql.metadata.HiveUtils;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.security.authorization.HiveCustomStorageHandlerUtils;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.apache.hadoop.hive.serde2.Serializer;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TaskAttemptID;
+import org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IcebergRepository extends AbstractQueryHistoryRepository implements QueryHistoryRepository {
+ private static final Logger LOG = LoggerFactory.getLogger(IcebergRepository.class);
+ private static final String ICEBERG_STORAGE_HANDLER = "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler";
+
+ private HiveOutputFormat<?, ?> outputFormat;
+ private Serializer serializer;
+ @VisibleForTesting
+ HiveStorageHandler storageHandler;
+ @VisibleForTesting
+ TableDesc tableDesc;
+
+ @Override
+ public void init(HiveConf conf, QueryHistorySchema schema) {
+ super.init(conf, schema);
+ }
+
+ @Override
+ protected Table createTable(Hive hive, Database db) throws HiveException {
+ LOG.info("Creating iceberg Query History table...");
+
+ Table table = getInitialTable();
+
+ // iceberg specific
+ table.setProperty(org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.META_TABLE_STORAGE,
+ ICEBERG_STORAGE_HANDLER);
+ table.setProperty("table_type", "ICEBERG");
+ table.setProperty("write.format.default", "orc");
+ table.setProperty(hive_metastoreConstants.META_TABLE_NAME, QUERY_HISTORY_DB_TABLE_NAME);
+
+ table.setFields(schema.getFields());
+
+ hive.createTable(table, false);
+
+ table = hive.getTable(QUERY_HISTORY_DB_NAME, QUERY_HISTORY_TABLE_NAME);
+ table.setProperty("iceberg.mr.table.location", table.getDataLocation().toString());
+
+ return table;
+ }
+
+ @Override
+ protected void postInitTable(Table table) throws Exception {
+ this.tableDesc = Utilities.getTableDesc(table);
+
+ Map<String, String> map = new HashMap<>();
+ this.storageHandler = HiveUtils.getStorageHandler(conf, tableDesc.getProperties()
+ .getProperty(org.apache.hadoop.hive.metastore.api.hive_metastoreConstants.META_TABLE_STORAGE));
+ storageHandler.configureOutputJobProperties(tableDesc, map);
+
+ //need to obtain a serialized table for later usage
+ map.forEach((key, value) -> conf.set(key, value));
+
+ conf.set(hive_metastoreConstants.META_TABLE_NAME, QUERY_HISTORY_DB_TABLE_NAME);
+
+ HiveCustomStorageHandlerUtils.setWriteOperation(conf, QUERY_HISTORY_DB_TABLE_NAME, Context.Operation.OTHER);
+ this.outputFormat = HiveFileFormatUtils.getHiveOutputFormat(conf, tableDesc);
+
+ this.serializer = tableDesc.getDeserializer(conf);
+ }
+
+ @Override
+ public void persist(Queue<QueryHistoryRecord> records) {
+ int numRecords = records.size();
+ LOG.info("Persisting {} records", numRecords);
+ if (numRecords == 0) {
+ return;
+ }
+
+ try {
+ prepareConfForWrite();
+ JobConf jobConf = new JobConf(conf);
+
+ RecordWriter writer = outputFormat.getHiveRecordWriter(jobConf, null, null, false, null, null);
+ while (!records.isEmpty()) {
+ writeRecord(writer, records.poll());
+ }
+
+ writer.close(false);
+ storageHandler.storageHandlerCommit(HiveConf.getProperties(conf), Context.Operation.OTHER,
Review Comment:
````
jobConf.getOutputCommitter().commitTask(
new TaskAttemptContextImpl(jobConf, TaskAttemptID.forName(conf.get("mapred.task.id")), null)
);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931559594
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
Review Comment:
yeah, this is printed only once after hiveserver2 was started, I believe we should log the major lifecycle points of a service as long as it's within a controllable level
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931638463
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void handleQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ QueryHistoryRecord record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ persistRecordsIfNeeded();
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ persistWithBatchSize(maxBatchSize);
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(), maxBatchSize);
+ }
+ }
+
+ private void persistWithBatchSize(int batchSize) {
+ CountDownLatch latch = new CountDownLatch(1);
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ repository.persist(queryHistoryQueue);
+ latch.countDown();
+ });
+ if (batchSize == 0) {
+ waitForAllRecordsToBePersisted(latch);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted(CountDownLatch queueEmptyCondition) {
+ try {
+ queueEmptyCondition.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ boolean needToPersist() {
+ boolean needToPersistAccordingToQueueSize = queryHistoryQueue.size() >= maxBatchSize;
Review Comment:
ack, eventually, this will be the batch size used for flushing
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930557190
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/queryhistory/schema/TestQueryHistorySchema.java:
##########
@@ -0,0 +1,101 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.StandardStructObjectInspector;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TestQueryHistorySchema {
+
+ @Test
+ public void testSchemaFieldDataTypes() {
Review Comment:
what do we test here? list of supported type inspectors? it seems redundant to me
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931797435
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void handleQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ QueryHistoryRecord record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ persistRecordsIfNeeded();
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ persistWithBatchSize(maxBatchSize);
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(), maxBatchSize);
+ }
+ }
+
+ private void persistWithBatchSize(int batchSize) {
+ CountDownLatch latch = new CountDownLatch(1);
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ repository.persist(queryHistoryQueue);
+ latch.countDown();
+ });
+ if (batchSize == 0) {
+ waitForAllRecordsToBePersisted(latch);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted(CountDownLatch queueEmptyCondition) {
+ try {
+ queueEmptyCondition.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ boolean needToPersist() {
+ boolean needToPersistAccordingToQueueSize = queryHistoryQueue.size() >= maxBatchSize;
+ if (needToPersistAccordingToQueueSize) {
+ LOG.info("Need to persist query history records as queue size ({}) >= batch size ({})",
+ queryHistoryQueue.size(), maxBatchSize);
+ return true;
+ }
+
+ // whether ignore the consumed memory or not
+ if (maxQueueSizeInMemoryBytes == 0) {
+ return false;
+ }
+
+ long queueMemorySize =
+ queryHistoryQueue.stream().mapToLong(QueryHistoryRecord::getEstimatedSizeInMemoryBytes).sum();
+ boolean needToPersistAccordingToQueueMemorySize = queueMemorySize > maxQueueSizeInMemoryBytes;
+ if (needToPersistAccordingToQueueMemorySize) {
+ LOG.info("Need to persist query history records as queue size in memory ({} bytes) is greater than the limit " +
+ "({} bytes)",
+ queueMemorySize, maxQueueSizeInMemoryBytes);
+ }
+ return needToPersistAccordingToQueueMemorySize;
+ }
+
+ public void stop() {
+ LOG.info("Stopping QueryHistoryService, leftover records to persist: {}", queryHistoryQueue.size());
+ persistWithBatchSize(0); // persist sync
+ persistExecutor.shutdownNow();
+ }
+
+ private QueryHistoryRecord createRecord(DriverContext driverContext) {
+ return new QueryHistoryRecordEnricher(driverContext, serviceContext, SessionState.get(),
+ SessionState.getPerfLogger()).createRecord();
+ }
+
+ @VisibleForTesting
+ public QueryHistoryRepository getRepository(){
+ return repository;
+ }
+
+ @Override
+ public String toString() {
Review Comment:
`toString` is usually used for domain objects not services.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939026203
##########
ql/src/java/org/apache/hadoop/hive/ql/DriverContext.java:
##########
@@ -19,25 +19,33 @@
package org.apache.hadoop.hive.ql;
import java.io.DataInput;
+import java.io.Serializable;
+import java.util.List;
-import org.apache.hadoop.hive.common.ValidTxnList;
-import org.apache.hadoop.hive.common.ValidWriteIdList;
import org.apache.hadoop.hive.conf.HiveConf;
import org.apache.hadoop.hive.metastore.api.Schema;
import org.apache.hadoop.hive.metastore.api.TxnType;
import org.apache.hadoop.hive.ql.cache.results.CacheUsage;
import org.apache.hadoop.hive.ql.cache.results.QueryResultsCache.CacheEntry;
+import org.apache.hadoop.hive.ql.ddl.DDLSemanticAnalyzerFactory;
import org.apache.hadoop.hive.ql.exec.FetchTask;
import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.exec.Task;
import org.apache.hadoop.hive.ql.exec.tez.TezRuntimeContext;
import org.apache.hadoop.hive.ql.exec.tez.TezTask;
import org.apache.hadoop.hive.ql.lockmgr.HiveTxnManager;
+import org.apache.hadoop.hive.ql.parse.ASTNode;
Review Comment:
unused imports after refactor, please check other classes as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2630499475
hi @abstractdog, added 3 new comments + 1 old (https://github.com/apache/hive/pull/5613#discussion_r1939016082) remaining
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939610218
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
we still need massive `setQueryType` in base SA?
````
driverContext.setQueryType(sem.getQueryType())
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939620152
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
1. `setQueryType` shouldn't be public
2. AcidExportSemanticAnalyzer also extends RewriteSemanticAnalyzer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939610218
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
we still need massive `setQueryType` in base SA?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923948658
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/queryhistory/schema/ExampleQueryHistoryRecord.java:
##########
@@ -0,0 +1,154 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import com.google.common.collect.Sets;
+
+import java.time.Instant;
+import java.util.Collections;
+import java.util.Map;
+import java.util.Set;
+
+public class ExampleQueryHistoryRecord extends QueryHistoryRecord {
Review Comment:
dummy doesn't mean empty it's something similar to stub in Hive (see DummyPartition)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1925385367
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java:
##########
@@ -355,6 +376,16 @@ public void clear() {
multiDestQuery = false;
filterWithSubQuery = false;
+
+ tablesQueried.clear();
}
+ /**
+ * This method extracts all query related information from the root QB from its state available after semantic
+ * analysis.
+ * @param qb the root QB of the analyzer
+ */
+ public void extractInfoFromQueryBlock(QB qb) {
+ qb.iterate(qbIt -> tablesQueried.addAll(qbIt.getTableAliasValues()));
Review Comment:
ctx. getParsedTables() seems to work, let me change to that, so I can remove the whole QB visitor thing that I created just for this purpose
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1925385367
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryProperties.java:
##########
@@ -355,6 +376,16 @@ public void clear() {
multiDestQuery = false;
filterWithSubQuery = false;
+
+ tablesQueried.clear();
}
+ /**
+ * This method extracts all query related information from the root QB from its state available after semantic
+ * analysis.
+ * @param qb the root QB of the analyzer
+ */
+ public void extractInfoFromQueryBlock(QB qb) {
+ qb.iterate(qbIt -> tablesQueried.addAll(qbIt.getTableAliasValues()));
Review Comment:
ctx. getParsedTables(), so I can remove the whole QB visitor that I created just for this purpose
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923862551
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/persist/AbstractQueryHistoryPersistor.java:
##########
@@ -0,0 +1,138 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory.persist;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.TableType;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.parse.PartitionTransform;
+import org.apache.hadoop.hive.ql.parse.TransformSpec;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+
+public abstract class AbstractQueryHistoryPersistor implements QueryHistoryPersistor {
Review Comment:
fine, I never really like the word "persistor" honestly
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923855052
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java:
##########
@@ -458,6 +460,18 @@ private void printSummary(boolean success, Map<String, Progress> progressMap) {
}
}
+ public void endSummary() {
+ console.endSummary();
Review Comment:
yeah, it's safe, just marks that collection can be stopped IF it has started
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923853697
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/TezJobMonitor.java:
##########
@@ -513,4 +527,12 @@ ProgressMonitor progressMonitor(DAGStatus status, Map<String, Progress> progress
}
return TezProgressMonitor.NULL;
}
+
+ public String getSummary() {
Review Comment:
hm, let me check, makes sense
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923589527
##########
ql/src/java/org/apache/hadoop/hive/ql/QueryPlan.java:
##########
@@ -20,6 +20,7 @@
import java.io.IOException;
import java.io.Serializable;
+import java.nio.charset.StandardCharsets;
Review Comment:
yeah, noise, I'll remove
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923514311
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -5734,6 +5734,26 @@ public static enum ConfVars {
+ "a background update happens periodically to report the actual state of the query"),
HIVE_SCHEDULED_QUERIES_CREATE_AS_ENABLED("hive.scheduled.queries.create.as.enabled", true,
"This option sets the default behaviour of newly created scheduled queries."),
+ HIVE_QUERY_HISTORY_SERVICE_ENABLED("hive.query.history.service.enabled", true,
+ "Whether to start QueryHistoryService in HS2"),
+ HIVE_QUERY_HISTORY_SERVICE_EXPLAIN_PLAN_ENABLED("hive.query.history.service.explain.plan.enabled", true,
+ "Whether to collect and store explain plan in the query history. Default is true."),
+ HIVE_QUERY_HISTORY_SERVICE_EXEC_SUMMARY_ENABLED("hive.query.history.service.exec.summary.enabled", true,
+ "Whether to collect and store execution summary in the query history. Default is true."),
+ HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE("hive.query.history.service.persist.max.batch.size", 200,
+ "The maximum amount of records held in memory " +
+ "before query history service persists them to the target table. " +
+ "A small value (like 1-5) will lead to more real-time behavior with the price of small files. " +
+ "Set this to 0 to wait for the records to be persisted synchronously (not recommended in production)."),
Review Comment:
take a look at QueryHistoryService:
```
if (needToPersist()) {
LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
persistExecutor.submit(() -> {
queryHistoryPersistor.persist(queryHistoryQueue);
});
if (maxBatchSize == 0) {
waitForAllRecordsToBePersisted();
}
}
```
so if maxBatchSize > 0, this method returns immediately after submitting the record to be persisted to the persistExecutor (which way is the desired one for production I believe to unblock query path)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923839337
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.persist.QueryHistoryPersistor;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ private QueryHistoryPersistor queryHistoryPersistor;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+ private int maxBatchSize = 0;
+ private int maxQueueSizeInMemoryBytes = 0;
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService persistor thread").build());
+
+ private ServiceContext serviceContext;
+
+ public static QueryHistoryService start(HiveConf inputConf, ServiceContext serviceContext) {
+ HiveConf conf = new HiveConf(inputConf);
+ // we can only hit this codepath if the INSTANCE is already initialized while calling start(...)
+ // for convenient test compatibility, this call is allowed to be idempotent (LOG.warn instead of exception)
+ if (INSTANCE != null) {
+ LOG.warn("There is already a QueryHistoryService instance ({}), returning existing instance", INSTANCE);
+ return INSTANCE;
+ }
+
+ INSTANCE = createServiceInstance(serviceContext, conf);
+ return INSTANCE;
+ }
+
+ private static QueryHistoryService createServiceInstance(ServiceContext serviceContext, HiveConf conf) {
+ LOG.info("Starting QueryHistoryService");
+ QueryHistoryService queryHistoryService = new QueryHistoryService();
+
+ initService(serviceContext, conf, queryHistoryService);
+ return queryHistoryService;
+ }
+
+ @VisibleForTesting
+ static void initService(ServiceContext serviceContext, HiveConf conf, QueryHistoryService queryHistoryService) {
Review Comment:
why use static method? can't we use a singleton constructor with an instance `init` method?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923764427
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -600,6 +601,11 @@ default void storageHandlerCommit(Properties commitProperties, Operation operati
throw new UnsupportedOperationException();
}
+ default void storageHandlerCommit(Properties commitProperties, Operation operation, TaskAttemptContext context)
Review Comment:
1. javadoc is missing
2. why do we need this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1923964719
##########
ql/src/java/org/apache/hadoop/hive/ql/Compiler.java:
##########
@@ -224,6 +224,7 @@ private BaseSemanticAnalyzer analyze() throws Exception {
sem.analyze(tree, context);
} finally {
sem.endAnalysis();
+ driverContext.setQueryType(sem, tree);
Review Comment:
makes sense, just don't really like the way we set queryType
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2652226407
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613) **Quality Gate passed**
Issues
 [95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1927425373
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -600,6 +601,11 @@ default void storageHandlerCommit(Properties commitProperties, Operation operati
throw new UnsupportedOperationException();
}
+ default void storageHandlerCommit(Properties commitProperties, Operation operation, TaskAttemptContext context)
Review Comment:
copying response from another comment:
```
we cannot depend on iceberg in hive-exec in compile-time, there is no iceberg* dependency in hive-exec/pom.xml, that's what's hidden by the storagehandler layer, I think, even if I really like iceberg's api, I'm limited to use the storagehandler
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1927425373
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/HiveStorageHandler.java:
##########
@@ -600,6 +601,11 @@ default void storageHandlerCommit(Properties commitProperties, Operation operati
throw new UnsupportedOperationException();
}
+ default void storageHandlerCommit(Properties commitProperties, Operation operation, TaskAttemptContext context)
Review Comment:
copying response from another comment:
```we cannot depend on iceberg in hive-exec in compile-time, there is no iceberg* dependency in hive-exec/pom.xml, that's what's hidden by the storagehandler layer, I think, even if I really like iceberg's api, I'm limited to use the storagehandler
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931787671
##########
service/src/java/org/apache/hive/service/server/HiveServer2.java:
##########
@@ -325,6 +327,11 @@ public synchronized void init(HiveConf hiveConf) {
scheduledQueryService = ScheduledQueryExecutionService.startScheduledQueryExecutorService(hiveConf);
}
+ if (hiveConf.getBoolVar(ConfVars.HIVE_QUERY_HISTORY_ENABLED)) {
+ queryHistoryService = new QueryHistoryService(hiveConf, serviceContext).start();
Review Comment:
start() return type should be VOID, please split the constructor and start()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1934063488
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/IcebergRepository.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Calendar;
+import java.util.GregorianCalendar;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Queue;
+import java.util.UUID;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.Database;
+import org.apache.hadoop.hive.metastore.api.hive_metastoreConstants;
+import org.apache.hadoop.hive.ql.Context;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.exec.FileSinkOperator.RecordWriter;
+import org.apache.hadoop.hive.ql.exec.Utilities;
+import org.apache.hadoop.hive.ql.io.HiveFileFormatUtils;
+import org.apache.hadoop.hive.ql.io.HiveOutputFormat;
+import org.apache.hadoop.hive.ql.metadata.Hive;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.metadata.HiveStorageHandler;
+import org.apache.hadoop.hive.ql.metadata.HiveUtils;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.plan.TableDesc;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.security.authorization.HiveCustomStorageHandlerUtils;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.hive.ql.session.SessionStateUtil;
+import org.apache.hadoop.hive.serde2.Serializer;
+import org.apache.hadoop.io.Writable;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.TaskAttemptID;
+import org.apache.tez.mapreduce.hadoop.mapred.TaskAttemptContextImpl;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class IcebergRepository extends AbstractQueryHistoryRepository implements QueryHistoryRepository {
+ private static final Logger LOG = LoggerFactory.getLogger(IcebergRepository.class);
+ private static final String ICEBERG_STORAGE_HANDLER = "org.apache.iceberg.mr.hive.HiveIcebergStorageHandler";
+
+ private HiveOutputFormat<?, ?> outputFormat;
+ private Serializer serializer;
+ @VisibleForTesting
+ HiveStorageHandler storageHandler;
+ @VisibleForTesting
+ TableDesc tableDesc;
+
+ @Override
Review Comment:
okay, but does it make a difference? it overrides, as theoretically, it might add further init steps, which it doesn't do at the moment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930743110
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2047,8 +2070,8 @@ public void startAnalysis() {
/**
* Called when we end analysis of a query.
*/
- public void endAnalysis() {
- // Nothing to do
+ public void endAnalysis(ASTNode tree) {
+ setQueryType(tree); // at this point we know the query type for sure
Review Comment:
can't we set the queryType in `analyzeInternal(ast)` so we don't need to pass `ast` in `endAnalysis`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930784201
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
Review Comment:
should this be under info? I would be more interested to see LOG entry if service started, not just created
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930863290
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
Review Comment:
remove this and set in constructor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931781223
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/schema/QueryHistorySchema.java:
##########
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+
+public class QueryHistorySchema {
+ public static final int CURRENT_VERSION = 1;
+
+ // how many bytes are consumed by the "short" fields in a record
+ // rough estimate (once per JVM), which is much efficient than counting all the fields
+ public static final long BASE_RECORD_SIZE_IN_MEMORY_BYTES = calculateSimpleFieldsSize();
+
+ private static int calculateSimpleFieldsSize() {
+ int baseSize = 0;
+ for (QueryHistorySchema.Field field : QueryHistorySchema.Field.values()) {
+ switch (field.getType()) {
+ case "string":
+ // consider an average 20 chars length
+ // longer fields are going to be considered again
+ baseSize += 20 * Character.BYTES;
+ break;
+ case "int":
+ baseSize += 4;
+ break;
+ case "bigint":
+ // stored as long
+ baseSize += 8;
+ break;
+ case "timestamp":
+ baseSize += 16;
+ break;
+ }
+ }
+ return baseSize;
+ }
+
+ public interface QueryHistorySchemaField {
+ String getName();
+ String getType();
+ String getDescription();
+ boolean isPartitioningCol();
+ }
+
+ public static ObjectInspector INSPECTOR = null;
+
+ static {
+ INSPECTOR = createInspector(Field.values());
+ }
+
+ private final ArrayList<FieldSchema> fields;
+
+ public QueryHistorySchema() {
+ this.fields = new ArrayList<>();
+
+ Arrays.asList(Field.values())
+ .forEach(field -> this.fields.add(new FieldSchema(field.name, field.type, field.description)));
+ }
+
+ @VisibleForTesting
+ static ObjectInspector createInspector(QueryHistorySchemaField[] fields) {
Review Comment:
hm, right, and this justifies that TestQueryHistorySchema became totally useless
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930669734
##########
ql/src/java/org/apache/hadoop/hive/ql/Compiler.java:
##########
@@ -223,7 +223,7 @@ private BaseSemanticAnalyzer analyze() throws Exception {
sem.startAnalysis();
sem.analyze(tree, context);
} finally {
- sem.endAnalysis();
+ sem.endAnalysis(tree);
Review Comment:
can we store it as property in SA
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930842192
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.persist.QueryHistoryPersistor;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ private QueryHistoryPersistor queryHistoryPersistor;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+ private int maxBatchSize = 0;
+ private int maxQueueSizeInMemoryBytes = 0;
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService persistor thread").build());
+
+ private ServiceContext serviceContext;
+
+ public static QueryHistoryService start(HiveConf inputConf, ServiceContext serviceContext) {
+ HiveConf conf = new HiveConf(inputConf);
+ // we can only hit this codepath if the INSTANCE is already initialized while calling start(...)
+ // for convenient test compatibility, this call is allowed to be idempotent (LOG.warn instead of exception)
+ if (INSTANCE != null) {
+ LOG.warn("There is already a QueryHistoryService instance ({}), returning existing instance", INSTANCE);
+ return INSTANCE;
+ }
+
+ INSTANCE = createServiceInstance(serviceContext, conf);
+ return INSTANCE;
+ }
+
+ private static QueryHistoryService createServiceInstance(ServiceContext serviceContext, HiveConf conf) {
+ LOG.info("Starting QueryHistoryService");
+ QueryHistoryService queryHistoryService = new QueryHistoryService();
+
+ initService(serviceContext, conf, queryHistoryService);
+ return queryHistoryService;
+ }
+
+ @VisibleForTesting
+ static void initService(ServiceContext serviceContext, HiveConf conf, QueryHistoryService queryHistoryService) {
+ queryHistoryService.queryHistoryPersistor = createPersistor(conf);
+ queryHistoryService.queryHistoryPersistor.init(conf, new QueryHistorySchema());
+ queryHistoryService.maxBatchSize = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE);
+ queryHistoryService.maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_MEMORY_BYTES);
+
+ queryHistoryService.printConfigInformation();
+ queryHistoryService.serviceContext = serviceContext;
+
+ try {
+ queryHistoryService.persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the persistor,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
Review Comment:
this is what i meant
````
private static class QueryHistoryThread extends Thread {
public QueryHistoryThread(Runnable runnable, HiveConf conf) {
super(runnable);
HiveConf jobConf = new HiveConf(conf);
// this session won't need tez, let's not mess with starting a TezSessionState
jobConf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
SessionState.start(jobConf);
}
}
public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
....
persistExecutor = Executors.newFixedThreadPool(1,
new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true)
.setThreadFactory(r -> new QueryHistoryThread(r, conf)).build());
repository = createRepository(conf);
....
}
````
that won't need hacks in `start()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931093035
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/schema/QueryHistoryRecord.java:
##########
@@ -0,0 +1,358 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import java.time.Instant;
+import java.time.ZoneId;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+import com.google.common.base.Joiner;
+import com.google.common.base.Strings;
+import org.apache.hadoop.hive.common.type.Timestamp;
+import org.apache.hadoop.hive.ql.QueryProperties;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema.Field;
+import org.apache.hadoop.hive.serde2.Serializer;
+import org.apache.hadoop.io.Writable;
+import org.apache.hive.common.util.HiveVersionInfo;
+
+public class QueryHistoryRecord {
+ private long estimatedSizeInMemoryBytes = 0;
+
+ public QueryHistoryRecord(){
+ set(Field.QUERY_HISTORY_SCHEMA_VERSION, QueryHistorySchema.CURRENT_VERSION);
+ set(Field.HIVE_VERSION, HiveVersionInfo.getVersion());
+ }
+
+ protected final Map<String, Object> record = new HashMap<>();
+
+ public void setQueryId(String queryId) {
+ set(QueryHistorySchema.Field.QUERY_ID, queryId);
+ }
+
+ public void setSessionId(String sessionId) {
+ set(QueryHistorySchema.Field.SESSION_ID, sessionId);
+ }
+
+ public void setOperationId(String operationId) {
+ set(QueryHistorySchema.Field.OPERATION_ID, operationId);
+ }
+
+ public void setExecutionEngine(String executionEngine) {
+ set(QueryHistorySchema.Field.EXECUTION_ENGINE, executionEngine);
+ }
+
+ public void setExecutionMode(String executionMode) {
+ set(QueryHistorySchema.Field.EXECUTION_MODE, executionMode);
+ }
+
+ public void setDagId(String dagId) {
+ set(QueryHistorySchema.Field.TEZ_DAG_ID, dagId);
+ }
+
+ public void setClusterId(String clusterId) {
+ set(Field.CLUSTER_ID, clusterId);
+ }
+
+ public void setTezApplicationId(String applicationId) {
+ set(QueryHistorySchema.Field.TEZ_APP_ID, applicationId);
+ }
+
+ public void setTezSessionId(String sessionId) {
+ set(QueryHistorySchema.Field.TEZ_SESSION_ID, sessionId);
+ }
+
+ public void setSessionType(String sessionType) {
+ set(QueryHistorySchema.Field.SESSION_TYPE, sessionType);
+ }
+
+ public void setClientProtocol(int clientProtocol) {
+ set(QueryHistorySchema.Field.HIVERSERVER2_PROTOCOL_VERSION, clientProtocol);
+ }
+
+ public void setClusterUser(String username) {
+ set(QueryHistorySchema.Field.CLUSTER_USER, username);
+ }
+
+ public void setEndUser(String userNameConnection) {
+ set(QueryHistorySchema.Field.END_USER, userNameConnection);
+ }
+
+ public void setQuerySql(String querySql) {
+ set(QueryHistorySchema.Field.SQL, querySql);
+ }
+
+ public void setCurrentDatabase(String currentDatabase) {
+ set(QueryHistorySchema.Field.DB_NAME, currentDatabase);
+ }
+
+ public void setTezAmAddress(String amAddress) {
+ set(QueryHistorySchema.Field.TEZ_COORDINATOR, amAddress);
+ }
+
+ public void setQueryState(String state) {
+ set(QueryHistorySchema.Field.QUERY_STATE, state);
+ }
+
+ public void setQueryType(QueryProperties.QueryType type) {
+ set(QueryHistorySchema.Field.QUERY_TYPE, type == null ? QueryProperties.QueryType.OTHER.getName() :
+ type.getName());
+ }
+
+ public void setDdlType(String ddlType) {
+ set(Field.DDL_TYPE, ddlType);
+ }
+
+ public void setServerAddress(String serverAddress) {
+ set(QueryHistorySchema.Field.SERVER_ADDRESS, serverAddress);
+ }
+
+ public void setServerPort(int serverPort) {
+ set(QueryHistorySchema.Field.SERVER_PORT, serverPort);
+ }
+
+ public void setClientAddress(String userIpAddress) {
+ set(QueryHistorySchema.Field.CLIENT_ADDRESS, userIpAddress);
+ }
+
+ public void setQueryStartTime(long beginTimeEpochMillis) {
+ set(Field.START_TIME_UTC, Timestamp.ofEpochMilli(beginTimeEpochMillis));
+ set(Field.START_TIME, Timestamp.ofEpochMilli(beginTimeEpochMillis, ZoneId.systemDefault()));
+ }
+
+ public void setQueryEndTime(long endTimeEpochMillis) {
+ set(Field.END_TIME_UTC, Timestamp.ofEpochMilli(endTimeEpochMillis));
+ set(Field.END_TIME, Timestamp.ofEpochMilli(endTimeEpochMillis, ZoneId.systemDefault()));
+ }
+
+ public void setTotalTime(long elapsedTime) {
+ set(Field.TOTAL_TIME_MS, elapsedTime);
+ }
+
+ public void setPlanningDuration(long duration) {
+ set(Field.PLANNING_DURATION, duration);
+ }
+
+ public void setPlanningStartTime(long startTime) {
+ set(Field.PLANNING_START_TIME, timeToNullDefault(startTime));
+ }
+
+ public void setPreparePlanDuration(long duration) {
+ set(Field.PREPARE_PLAN_DURATION, duration);
+ }
+
+ public void setPreparePlanStartTime(long startTime) {
+ set(Field.PREPARE_PLAN_START_TIME, timeToNullDefault(startTime));
+ }
+
+ public void setGetSessionDuration(long duration) {
+ set(Field.GET_SESSION_DURATION, duration);
+ }
+
+ public void setGetSessionStartTime(long startTime) {
+ set(Field.GET_SESSION_START_TIME, timeToNullDefault(startTime));
+ }
+
+ public void setExecutionDuration(long duration) {
+ set(Field.EXECUTION_DURATION, duration);
+ }
+
+ public void setExecutionStartTime(long startTime) {
+ set(Field.EXECUTION_START_TIME, timeToNullDefault(startTime));
+ }
+
+ public void setFailureReason(String errorMessage) {
+ set(Field.FAILURE_REASON, errorMessage);
+ }
+
+ public void setNumRowsFetched(int totalRows) {
+ set(Field.NUM_ROWS_FETCHED, totalRows);
+ }
+
+ public void setPlan(String explainPlan) {
+ set(Field.PLAN, explainPlan);
+ }
+
+
+ public void setTablesQueried(List<String> tablesQueried) {
+ set(Field.TABLES_QUERIED, String.join(",", tablesQueried));
+ }
+
+ public void setExecSummary(String summary) {
+ set(Field.EXEC_SUMMARY, summary);
+ }
+
+ public void setConfigurationOptionsChanged(Map<String, String> overriddenConfigurations) {
+ set(Field.CONFIGURATION_OPTIONS_CHANGED, Joiner.on(";").withKeyValueSeparator("=").join(overriddenConfigurations));
+ }
+
+ public void setTotalNumberOfTasks(int numTasks) {
+ set(Field.TOTAL_LAUNCHED_TASKS, numTasks);
+ }
+
+ public void setNumberOfSucceededTasks(int numSucceededTasks) {
+ set(Field.NUM_SUCCEEDED_TASKS, numSucceededTasks);
+ }
+
+ public void setNumberOfKilledTasks(int numKilledTasks) {
+ set(Field.NUM_KILLED_TASKS, numKilledTasks);
+ }
+
+ public void setNumberOfFailedTasks(int numFailedTasks) {
+ set(Field.NUM_FAILED_TASKS, numFailedTasks);
+ }
+
+ public void setNodeUsedCount(int nodeUsedCount) {
+ set(Field.NODE_USED_COUNT, nodeUsedCount);
+ }
+
+ public void setNodeTotalCount(int nodeTotalCount) {
+ set(Field.NODE_TOTAL_COUNT, nodeTotalCount);
+ }
+
+ public void setReduceInputGroups(long reduceInputGroups) {
+ set(Field.REDUCE_INPUT_GROUPS, reduceInputGroups);
+ }
+
+ public void setReduceInputRecords(long reduceInputRecords) {
+ set(Field.REDUCE_INPUT_RECORDS, reduceInputRecords);
+ }
+
+ public void setSpilledRecords(long spilledRecords) {
+ set(Field.SPILLED_RECORDS, spilledRecords);
+ }
+
+ public void setNumShuffledInputs(long numShuffledInputs) {
+ set(Field.NUM_SHUFFLED_INPUTS, numShuffledInputs);
+ }
+
+ public void setNumFailedShuffleInputs(long numFailedShuffleInputs) {
+ set(Field.NUM_FAILED_SHUFFLE_INPUTS, numFailedShuffleInputs);
+ }
+
+ public void setTaskDurationMillis(long taskDurationMillis) {
+ set(Field.TASK_DURATION_MILLIS, taskDurationMillis);
+ }
+
+ public void setInputRecordsProcessed(long inputRecordsProcessed) {
+ set(Field.INPUT_RECORDS_PROCESSED, inputRecordsProcessed);
+ }
+
+ public void setInputSplitLengthBytes(long inputSplitLengthBytes) {
+ set(Field.INPUT_SPLIT_LENGTH_BYTES, inputSplitLengthBytes);
+ }
+
+ public void setOutputRecords(long outputRecords) {
+ set(Field.OUTPUT_RECORDS, outputRecords);
+ }
+
+ public void setOutputBytesPhysical(long outputBytesPhysical) {
+ set(Field.OUTPUT_BYTES_PHYSICAL, outputBytesPhysical);
+ }
+
+ public void setShuffleChunkCount(long shuffleChunkCount) {
+ set(Field.SHUFFLE_CHUNK_COUNT, shuffleChunkCount);
+ }
+
+ public void setShuffleBytes(long shuffleBytes) {
+ set(Field.SHUFFLE_BYTES, shuffleBytes);
+ }
+
+ public void setShuffleBytesDiskDirect(long shuffleBytesDiskDirect) {
+ set(Field.SHUFFLE_BYTES_DISK_DIRECT, shuffleBytesDiskDirect);
+ }
+
+ public void setShufflePhaseTime(long shufflePhaseTime) {
+ set(Field.SHUFFLE_PHASE_TIME, shufflePhaseTime);
+ }
+
+ public void setMergePhaseTime(long phaseTime) {
+ set(Field.MERGE_PHASE_TIME, phaseTime);
+ }
+
+ public Object get(Field field) {
+ return record.get(field.name);
+ }
+
+ private void set(Field field, Object value) {
+ record.put(field.name, value);
+ }
+
+ public String toString() {
+ return String.format("QueryHistoryRecord [queryId: %s, sessionId: %s]", get(QueryHistorySchema.Field.QUERY_ID),
+ get(QueryHistorySchema.Field.SESSION_ID));
+ }
+
+ /**
+ * Utility toString kind of method for debugging.
+ * @return string representation of the underlying map
+ */
+ public String toLongString() {
+ return String.format("QueryHistoryRecord [%s]", record);
+ }
+
+ public Writable serialize(Serializer serializer) throws Exception {
Review Comment:
shouldn't be in domain object
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931174594
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ maxBatchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_PERSIST_MAX_MEMORY_BYTES);
+
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [maxBatchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ maxBatchSize, maxQueueSizeInMemoryBytes);
+ }
+
+ /**
+ * Start method actually starts the service in a sense that it initializes the repository (which might involve
+ * database operations) and starts the executor that's responsible for persisting records.
+ * @return the service instance
+ */
+ public QueryHistoryService start() {
+ repository.init(conf, new QueryHistorySchema());
+
+ try {
+ persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the repository,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
+ LOG.info("Session for QueryHistoryService started, sessionId: {}", session.getSessionId());
+ });
+ } catch (Exception e) {
+ throw new RuntimeException("Failed to start QueryHistoryService", e);
+ }
+
+ return this;
+ }
+
+ public static QueryHistoryService getInstance() {
+ if (INSTANCE == null) {
+ throw new RuntimeException("QueryHistoryService is not present when asked for the singleton instance");
+ }
+ return INSTANCE;
+ }
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ public static void setInstance(QueryHistoryService queryHistoryService) {
+ INSTANCE = queryHistoryService;
+ }
+
+ private void printConfigInformation() {
+ if (maxBatchSize == 0) {
+ LOG.info("Query History batch size is 0: every record will be persisted synchronously after the query" +
+ " (not recommended)");
+ } else {
+ LOG.info("Query History batch size is {}: records will be persisted after every {} queries",
+ maxBatchSize, maxBatchSize);
+ }
+ if (maxQueueSizeInMemoryBytes == 0) {
+ LOG.info("Query History max queue memory byte size is 0: the service won't check the consumed memory to make " +
+ "persist decision (not recommended)");
+ } else {
+ LOG.info("Query History max queue memory byte size is {}: records will be persisted when the queue consumes " +
+ "more than {} bytes in the memory",
+ maxQueueSizeInMemoryBytes, maxQueueSizeInMemoryBytes);
+ }
+ }
+
+ protected QueryHistoryRepository createRepository(HiveConf conf) {
+ initializeSessionForTableCreation(conf);
+
+ try {
+ return (QueryHistoryRepository) ReflectionUtil.newInstance(
+ conf.getClassByName(conf.get(HiveConf.ConfVars.HIVE_QUERY_HISTORY_REPOSITORY_CLASS.varname)), conf);
+ } catch (ClassNotFoundException e) {
+ throw new RuntimeException(e);
+ }
+ }
+
+ private void initializeSessionForTableCreation(HiveConf conf) {
+ HiveConf.setVar(conf, HiveConf.ConfVars.HIVE_QUERY_ID, QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION);
+ SessionState ss = SessionState.get(); // if there is a SessionState in the called thread, we can use that
+ if (ss == null){
+ ss = SessionState.start(conf);
+ }
+ ss.addQueryState(QueryHistoryRepository.QUERY_ID_FOR_TABLE_CREATION,
+ new QueryState.Builder().withHiveConf(conf).build());
+ }
+
+ /**
+ * Creates a Query History record from a given DriverContext and adds it to the queue.
+ * This is done asynchronously to completely decouple from the threads that run the query to make this service work
+ * silently in the background.
+ * @param driverContext a DriverContext containing all the needed information for a record to be created
+ */
+ public void handleQuery(DriverContext driverContext) {
+ long recordCreateStart = Time.monotonicNow();
+ QueryHistoryRecord record = createRecord(driverContext);
+ long recordCreateEnd = Time.monotonicNow();
+ queryHistoryQueue.add(record);
+ LOG.info("Created query history record in {}ms, current queue size: {}",
+ recordCreateEnd - recordCreateStart, queryHistoryQueue.size());
+
+ persistRecordsIfNeeded();
+ }
+
+ private void persistRecordsIfNeeded() {
+ if (needToPersist()) {
+ persistWithBatchSize(maxBatchSize);
+ } else {
+ LOG.debug("Not persisting history records yet, current queue size: {}, maxBatchSize: {}",
+ queryHistoryQueue.size(), maxBatchSize);
+ }
+ }
+
+ private void persistWithBatchSize(int batchSize) {
+ CountDownLatch latch = new CountDownLatch(1);
+ LOG.debug("Submitting a job to persist {} history records", queryHistoryQueue.size());
+ persistExecutor.submit(() -> {
+ repository.persist(queryHistoryQueue);
+ latch.countDown();
+ });
+ if (batchSize == 0) {
+ waitForAllRecordsToBePersisted(latch);
+ }
+ }
+
+ private void waitForAllRecordsToBePersisted(CountDownLatch queueEmptyCondition) {
+ try {
+ queueEmptyCondition.await();
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new RuntimeException(e);
+ }
+ }
+
+ @VisibleForTesting
+ boolean needToPersist() {
Review Comment:
should it be synchronized or something? What if multiple query sessions call `handleQuery` and we need to persist?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939016082
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
I think it should be done something like
````
driverContext.setQueryType(
SAUtils.getQueryType(tree, sem);
````
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
I think it should be done something like
````
driverContext.setQueryType(
SAUtils.getQueryType(tree, sem);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939587218
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
I made the subclasses inherit setQueryType
e.g. ColumnStatsSemanticAnalyzer:
```
@Override
public void setQueryType(ASTNode tree) {
queryProperties.setQueryType(QueryProperties.QueryType.STATS);
}
```
isn't this sufficient?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939016082
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
I think it should be done something like
````
driverContext.setQueryType(
SAUtils.getQueryType(tree, sem);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "kasakrisz (via GitHub)" <gi...@apache.org>.
kasakrisz commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939119561
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2092,4 +2110,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
+ (queryProperties.isDML() ?
+ QueryType.DML : null));
+ // common confusion whether CTAS is DML or DDL, let's pick DDL here
+ if (queryProperties.isCTAS()) {
+ queryProperties.setQueryType(QueryType.DDL);
+ }
+ if (queryProperties.getQueryType() != null) {
+ return; //already figured out
+ }
+ queryProperties.setQueryType(rootTasks.stream().findFirst().map(t -> {
+ String type = t.getType().toString();
+ // a MAPRED stage could mean an INSERT query also
+ if (queryProperties != null && type.equalsIgnoreCase("MAPRED")) {
Review Comment:
`t.getType()` returns an enum value of `StageType`
We can use the predicate `type == StageType.MAPRED`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939016082
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
we can create a new util class just for SA and refactor like this:
````
driverContext.setQueryType(
SAUtils.getQueryType(tree, sem);
````
we can also extract `Compiler#setSchema` to `SAUtils.getSchema(sem)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2630241578
@abstractdog, what happens to the in-memory records when HS2 is shut down?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939016082
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
we can create a new util class just for SA and refactor like this:
````
driverContext.setQueryType(SAUtils.getQueryType(tree, sem);
````
we can also extract `Compiler#setSchema(sem)` to `SAUtils.getSchema(sem)`
````
driverContext.setSchema(SAUtils.getSchema(sem);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2627805416
> hi @abstractdog, there are a bunch of old comments that were skipped. Please check them, sometimes history might be collapsed.
oh, missed that, let me iterate through
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2627797148
hi @abstractdog, there are a bunch of old comments that were skipped. Please check them, sometimes history might be collapsed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930421202
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -838,6 +845,9 @@ public void storageHandlerCommit(Properties commitProperties, Operation operatio
}
HiveIcebergOutputCommitter committer = getOutputCommitter();
try {
+ if (context != null) {
+ committer.commitTask(context);
Review Comment:
can't we expose `HiveIcebergOutputCommitter` via get method?
````
OutputCommitter commiter = table.getStorageHandler().getOutputCommitter();
committer.commitTask(context);
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930421202
##########
iceberg/iceberg-handler/src/main/java/org/apache/iceberg/mr/hive/HiveIcebergStorageHandler.java:
##########
@@ -838,6 +845,9 @@ public void storageHandlerCommit(Properties commitProperties, Operation operatio
}
HiveIcebergOutputCommitter committer = getOutputCommitter();
try {
+ if (context != null) {
+ committer.commitTask(context);
Review Comment:
can't we expose `HiveIcebergOutputCommitter` via get method?
````
OutputCommitter outputCommiter = table.getStorageHandler().getOutputCommitter();
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #5613:
URL: https://github.com/apache/hive/pull/5613#issuecomment-2634259516
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613) **Quality Gate passed**
Issues
 [85 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=5613&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=5613&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_coverage&view=list)
 [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=5613&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=5613)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1941939562
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -330,6 +332,17 @@ public void analyze(ASTNode ast, Context ctx) throws SemanticException {
initCtx(ctx);
init(true);
analyzeInternal(ast);
+ postAnalyze();
+ }
+
+ /**
+ * This method is called after analyzeInternal, assuming all the semantic context is present (analyzeInternal is
+ * overridden by the subclasses of BaseSemanticAnalyzer).
+ * The reason why postAnalyze is private is that it's not supposed to be overridden.
+ */
+ private void postAnalyze() {
Review Comment:
why do we need 1 more `postAnalyze`, can't we move `setUsedTables` in `endAnalysis`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930492394
##########
itests/hive-unit/src/test/java/org/apache/hadoop/hive/ql/queryhistory/repository/TestIcebergRepository.java:
##########
@@ -0,0 +1,115 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one§
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchemaTestUtils;
+import org.apache.iceberg.mr.mapred.Container;
+import org.apache.iceberg.data.GenericRecord;
+import org.apache.iceberg.data.Record;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.metadata.Table;
+import org.apache.hadoop.hive.ql.queryhistory.QueryHistoryService;
+import org.apache.hadoop.hive.ql.queryhistory.repository.IcebergRepositoryForTest;
+import org.apache.hadoop.hive.ql.queryhistory.schema.DummyQueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.IcebergQueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.mapred.FileInputFormat;
+import org.apache.hadoop.mapred.InputFormat;
+import org.apache.hadoop.mapred.InputSplit;
+import org.apache.hadoop.mapred.JobConf;
+import org.apache.hadoop.mapred.RecordReader;
+import org.apache.hadoop.mapred.Reporter;
+import org.junit.Assert;
+import org.junit.Test;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.File;
+import java.util.Queue;
+import java.util.concurrent.LinkedBlockingQueue;
+
+public class TestIcebergRepository {
Review Comment:
maybe TestIcebergQueryHistoryRepository
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1930842192
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Queue;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.persist.QueryHistoryPersistor;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ private QueryHistoryPersistor queryHistoryPersistor;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+ private int maxBatchSize = 0;
+ private int maxQueueSizeInMemoryBytes = 0;
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService persistor thread").build());
+
+ private ServiceContext serviceContext;
+
+ public static QueryHistoryService start(HiveConf inputConf, ServiceContext serviceContext) {
+ HiveConf conf = new HiveConf(inputConf);
+ // we can only hit this codepath if the INSTANCE is already initialized while calling start(...)
+ // for convenient test compatibility, this call is allowed to be idempotent (LOG.warn instead of exception)
+ if (INSTANCE != null) {
+ LOG.warn("There is already a QueryHistoryService instance ({}), returning existing instance", INSTANCE);
+ return INSTANCE;
+ }
+
+ INSTANCE = createServiceInstance(serviceContext, conf);
+ return INSTANCE;
+ }
+
+ private static QueryHistoryService createServiceInstance(ServiceContext serviceContext, HiveConf conf) {
+ LOG.info("Starting QueryHistoryService");
+ QueryHistoryService queryHistoryService = new QueryHistoryService();
+
+ initService(serviceContext, conf, queryHistoryService);
+ return queryHistoryService;
+ }
+
+ @VisibleForTesting
+ static void initService(ServiceContext serviceContext, HiveConf conf, QueryHistoryService queryHistoryService) {
+ queryHistoryService.queryHistoryPersistor = createPersistor(conf);
+ queryHistoryService.queryHistoryPersistor.init(conf, new QueryHistorySchema());
+ queryHistoryService.maxBatchSize = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_BATCH_SIZE);
+ queryHistoryService.maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf,
+ HiveConf.ConfVars.HIVE_QUERY_HISTORY_SERVICE_PERSIST_MAX_MEMORY_BYTES);
+
+ queryHistoryService.printConfigInformation();
+ queryHistoryService.serviceContext = serviceContext;
+
+ try {
+ queryHistoryService.persistExecutor.submit(() -> {
+ // this session won't need tez, let's not mess with starting a TezSessionState
+ conf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
+
+ // localize this conf for the thread that will be running the persistor,
+ // as iceberg integration heavily relies on the session state conf
+ SessionState session = SessionState.start(conf);
Review Comment:
this is what i meant
````
private static class QueryHistoryThread extends Thread {
public QueryHistoryThread(Runnable runnable, HiveConf conf) {
super(runnable);
HiveConf jobConf = new HiveConf(conf);
// this session won't need tez, let's not mess with starting a TezSessionState
jobConf.setBoolVar(HiveConf.ConfVars.HIVE_CLI_TEZ_INITIALIZE_SESSION, false);
SessionState.start(jobConf);
}
}
public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
....
persistExecutor = Executors.newFixedThreadPool(1,
new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true)
.setThreadFactory(r -> new QueryHistoryThread(r, conf)).build());
repository = createRepository(conf);
....
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1931084089
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/schema/QueryHistorySchema.java:
##########
@@ -0,0 +1,270 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.schema;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.metastore.api.FieldSchema;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspector;
+import org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorFactory;
+import org.apache.hadoop.hive.serde2.objectinspector.primitive.PrimitiveObjectInspectorFactory;
+
+public class QueryHistorySchema {
+ public static final int CURRENT_VERSION = 1;
+
+ // how many bytes are consumed by the "short" fields in a record
+ // rough estimate (once per JVM), which is much efficient than counting all the fields
+ public static final long BASE_RECORD_SIZE_IN_MEMORY_BYTES = calculateSimpleFieldsSize();
+
+ private static int calculateSimpleFieldsSize() {
+ int baseSize = 0;
+ for (QueryHistorySchema.Field field : QueryHistorySchema.Field.values()) {
+ switch (field.getType()) {
+ case "string":
+ // consider an average 20 chars length
+ // longer fields are going to be considered again
+ baseSize += 20 * Character.BYTES;
+ break;
+ case "int":
+ baseSize += 4;
+ break;
+ case "bigint":
+ // stored as long
+ baseSize += 8;
+ break;
+ case "timestamp":
+ baseSize += 16;
+ break;
+ }
+ }
+ return baseSize;
+ }
+
+ public interface QueryHistorySchemaField {
Review Comment:
should it be public?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1937584605
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,238 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.Record;
+import org.apache.hadoop.hive.ql.queryhistory.schema.Schema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<Record> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService flushExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+ private final ScheduledExecutorService periodicFlushExecutor = Executors.newScheduledThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService periodic flush check").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int batchSize;
+ private final int maxQueueSizeInMemoryBytes;
+ private final int flushInterval;
+ private final QueryHistoryRepository repository;
+
+ public QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
+ LOG.info("Creating QueryHistoryService instance");
+
+ this.conf = new HiveConf(inputConf);
+ this.serviceContext = serviceContext;
+
+ batchSize = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_BATCH_SIZE);
+ maxQueueSizeInMemoryBytes = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_MAX_MEMORY_BYTES);
+ flushInterval = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_FLUSH_INTERVAL_SECONDS);
+ repository = createRepository(conf);
+
+ printConfigInformation();
+
+ LOG.info("QueryHistoryService created [batchSize: {}, maxQueueSizeInMemoryBytes: {}]",
+ batchSize, maxQueueSizeInMemoryBytes);
+
+ // keeping a global, static instance is usually considered an antipattern, even in case of using the Singleton
+ // design, this time the constraint was the ability to hook into the query processing (Driver) from somewhere the
+ // outer service (HiveServer2, QTestUtil, etc.), which otherwise would have needed passing a service instance
+ // through many layers, making the code and method signatures more complicated
+ INSTANCE = this;
Review Comment:
how about
````
private static Singleton instance;
private QueryHistoryService(HiveConf inputConf, ServiceContext serviceContext) {
...
}
public static QueryHistoryService newInstance(HiveConf inputConf, ServiceContext serviceContext) {
if (instance == null) {
instance = new Singleton(HiveConf inputConf, ServiceContext serviceContext);
}
return instance;
}
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1937605322
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
Review Comment:
there is:
```
flushInterval = HiveConf.getIntVar(conf, HiveConf.ConfVars.HIVE_QUERY_HISTORY_FLUSH_INTERVAL_SECONDS);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1937602469
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/QueryHistoryService.java:
##########
@@ -0,0 +1,248 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.queryhistory;
+
+import java.util.Queue;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.ExecutorService;
+import java.util.concurrent.Executors;
+import java.util.concurrent.LinkedBlockingQueue;
+
+import com.google.common.annotations.VisibleForTesting;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.DriverContext;
+import org.apache.hadoop.hive.ql.QueryState;
+import org.apache.hadoop.hive.ql.ServiceContext;
+import org.apache.hadoop.hive.ql.queryhistory.repository.QueryHistoryRepository;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+import org.apache.hadoop.hive.ql.session.SessionState;
+import org.apache.hadoop.util.Time;
+import org.apache.hive.common.util.ReflectionUtil;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public class QueryHistoryService {
+ private static final Logger LOG = LoggerFactory.getLogger(QueryHistoryService.class);
+
+ private static QueryHistoryService INSTANCE = null;
+
+ protected final Queue<QueryHistoryRecord> queryHistoryQueue = new LinkedBlockingQueue<>();
+
+ private final ExecutorService persistExecutor = Executors.newFixedThreadPool(1,
+ new ThreadFactoryBuilder().setNameFormat("QueryHistoryService repository thread").setDaemon(true).build());
+
+ private final HiveConf conf;
+ private final ServiceContext serviceContext;
+
+ private final int maxBatchSize;
+ private final int maxQueueSizeInMemoryBytes;
Review Comment:
i don't see time trigger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1937606619
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/QueryHistoryRepository.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Queue;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+
+public interface QueryHistoryRepository {
+ String QUERY_HISTORY_DB_NAME = "sys";
+ String QUERY_HISTORY_TABLE_NAME = "query_history";
+ String QUERY_HISTORY_DB_TABLE_NAME =
+ String.format("%s.%s", QUERY_HISTORY_DB_NAME, QUERY_HISTORY_TABLE_NAME);
+
+ String QUERY_HISTORY_DB_COMMENT = "Hive SYS database";
+
+ // an artificial hive query id for an artificial session for creating the iceberg query history table
+ // this is needed because the partition_transform_spec is handed over through SessionState resource which
+ // needs a QueryState instance (acquired by a query id)
+ String QUERY_ID_FOR_TABLE_CREATION = "query_history_service_query_id_for_table_creation";
+
+ void init(HiveConf conf, QueryHistorySchema schema);
+
+ void persist(Queue<QueryHistoryRecord> records);
Review Comment:
you mean the time based flushing? it's:
```
HIVE_QUERY_HISTORY_FLUSH_INTERVAL_SECONDS("hive.query.history.flush.interval.seconds",
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1937601335
##########
ql/src/java/org/apache/hadoop/hive/ql/queryhistory/repository/QueryHistoryRepository.java:
##########
@@ -0,0 +1,42 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * <p>
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * <p>
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ **/
+package org.apache.hadoop.hive.ql.queryhistory.repository;
+
+import java.util.Queue;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistoryRecord;
+import org.apache.hadoop.hive.ql.queryhistory.schema.QueryHistorySchema;
+
+public interface QueryHistoryRepository {
+ String QUERY_HISTORY_DB_NAME = "sys";
+ String QUERY_HISTORY_TABLE_NAME = "query_history";
+ String QUERY_HISTORY_DB_TABLE_NAME =
+ String.format("%s.%s", QUERY_HISTORY_DB_NAME, QUERY_HISTORY_TABLE_NAME);
+
+ String QUERY_HISTORY_DB_COMMENT = "Hive SYS database";
+
+ // an artificial hive query id for an artificial session for creating the iceberg query history table
+ // this is needed because the partition_transform_spec is handed over through SessionState resource which
+ // needs a QueryState instance (acquired by a query id)
+ String QUERY_ID_FOR_TABLE_CREATION = "query_history_service_query_id_for_table_creation";
+
+ void init(HiveConf conf, QueryHistorySchema schema);
+
+ void persist(Queue<QueryHistoryRecord> records);
Review Comment:
what about time?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1938437194
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -7136,6 +7156,11 @@ public boolean isWebUiQueryInfoCacheEnabled() {
return isWebUiEnabled() && this.getIntVar(ConfVars.HIVE_SERVER2_WEBUI_MAX_HISTORIC_QUERIES) > 0;
}
+ public boolean isQueryHistoryExplainEnabled() {
Review Comment:
okay, I'm doing this in Compiler method
```
boolean queryHistoryExplainPlanEnabled = conf.getBoolVar(
HiveConf.ConfVars.HIVE_QUERY_HISTORY_ENABLED) && conf.getBoolVar(
HiveConf.ConfVars.HIVE_QUERY_HISTORY_EXPLAIN_PLAN_ENABLED);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1938984411
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
yeah, according to the signature, but then it works on inner fields, like the instance class:
```
if (RewriteSemanticAnalyzer.class.isAssignableFrom(getClass())) {
```
root tasks:
```
List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939016082
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
I think it should be done something like
````
driverContext.setQueryType(
SAUtils.getQueryType(tree, sem);
````
we can extract `Compiler#getSchema` in `SAUtils` as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-28095: Hive Query History [hive]
Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #5613:
URL: https://github.com/apache/hive/pull/5613#discussion_r1939030773
##########
ql/src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java:
##########
@@ -2094,4 +2117,79 @@ protected String getFullTableNameForSQL(ASTNode n) throws SemanticException {
}
}
+ public QB getQB(){
+ return null;
+ }
+
+ /**
+ * Resolves the query type from the SemanticAnalyzer and the query AST, which might look magic.
+ * Magic is needed because a query can fail due to semantic exceptions, and at that time, from syntax point of view,
+ * it's already clear what's the query type, even if the SemanticAnalyzer bailed out, typical problems:
+ * 1. general semantic exception
+ * 2. transaction manager validation (throwin exception)
+ * What a user expects here is something like "QUERY", "DDL", "DML", so this magic will do its best to tell.
+ * any kind of "analyze": STATS
+ * DML operations (INSERT, UPDATE, DELETE, MERGE): DML
+ * MAPRED: QUERY, DML (depending on QueryProperties achieved in compile time)
+ * FETCH: QUERY, as a simple fetch task is a QUERY
+ * empty string if we can't determine the type of the query,
+ * e.g. when ParseException happens, we won't do further magic,
+ * even if it's obvious by reading the sql statement that user wanted to run e.g. a select query
+ * @param tree the root ASTNode of the query
+ */
+ public void setQueryType(ASTNode tree) {
+ List<Task<? extends Serializable>> rootTasks = getAllRootTasks();
+ if (queryProperties == null) {
+ return;
+ }
+ queryProperties.setQueryType(queryProperties.isAnalyze() ? QueryType.STATS :
Review Comment:
or simply move the method to Compiler, next to `setSchema`
````
driverContext.setQueryType(getQueryType(tree, sem)
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org