You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "abstractdog (via GitHub)" <gi...@apache.org> on 2026/03/02 14:39:21 UTC
[PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
abstractdog opened a new pull request, #6343:
URL: https://github.com/apache/hive/pull/6343
### What changes were proposed in this pull request?
<!--
Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue.
If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
2. If you fix some SQL features, you can provide some references of other DBMSes.
3. If there is design documentation, please add the link.
4. If there is a discussion in the mailing list, please add the link.
-->
### Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you propose a new API, clarify the use case for a new API.
2. If you fix a bug, you can clarify why it is a bug.
-->
### Does this PR introduce _any_ user-facing change?
<!--
Note that it means *any* user-facing change including all aspects such as the documentation fix.
If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
If no, write 'No'.
-->
### How was this patch tested?
<!--
If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
If tests were not added, please describe why they were not added and/or why it was difficult to add.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-3985293636
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [104 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4160639398
> Thanx Laszlo for the update, Looks good, there are several warnings from Sonar, can you check & fix the relevant ones
thanks Ayush! originally it were 100+, then only non-relevants left, I'll double-check now if I can reduce it further and let you know
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r3009280754
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ExternalSessionsRegistry.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public interface ExternalSessionsRegistry {
+ Logger LOG = LoggerFactory.getLogger(ExternalSessionsRegistry.class);
+
+ /**
+ * Returns application of id of the external session.
+ * @return application id
+ * @throws Exception in case of any exceptions
+ */
+ String getSession() throws Exception;
+
+ /**
+ * Returns external session back to registry.
+ * @param appId application id
+ */
+ void returnSession(String appId);
+
+ /**
+ * Closes the external session registry
+ */
+ void close();
+
+ Map<String, ExternalSessionsRegistry> INSTANCES = new HashMap<>();
Review Comment:
right, there is no proper closure of these so far, I'm doing the very same as we agreed in TEZ-4007
https://github.com/apache/tez/blob/fd03af686cccce04e128b0815bf30ddf5974ed66/tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java#L72-L87
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r2879871347
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ZookeeperExternalSessionsRegistryClient.java:
##########
@@ -0,0 +1,159 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ * http://www.apache.org/licenses/LICENSE-2.0
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Set;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.PathChildrenCache;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+// TODO: tez should provide this registry
+public class ZookeeperExternalSessionsRegistryClient implements ExternalSessionsRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ZookeeperExternalSessionsRegistryClient.class);
+
+ // TODO: internal only for now to start and reconnect to external session
+ private static final String ZK_PATH = "/tez_am/server";
+ private final HiveConf initConf;
+ private final Set<String> available = new HashSet<>();
+ private final Set<String> taken = new HashSet<>();
+ private final Object lock = new Object();
+ private final int maxAttempts;
+
+ private PathChildrenCache cache;
Review Comment:
PathChildrenCache is deprecated. Can you please check if we can use CuratorCache API's
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r2879774457
##########
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java:
##########
@@ -323,6 +326,8 @@ public enum AuthorizationMode{V1, V2};
*/
private final Map<String, FunctionInfo> currentFunctionsInUse = new HashMap<>();
+ private static ExternalSessionsRegistry externalSessions = null;
Review Comment:
can be made as a local variable in start() method
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4170363732
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [42 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r3005841369
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ExternalSessionsRegistry.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public interface ExternalSessionsRegistry {
+ Logger LOG = LoggerFactory.getLogger(ExternalSessionsRegistry.class);
+
+ /**
+ * Returns application of id of the external session.
+ * @return application id
+ * @throws Exception in case of any exceptions
+ */
+ String getSession() throws Exception;
+
+ /**
+ * Returns external session back to registry.
+ * @param appId application id
+ */
+ void returnSession(String appId);
+
+ /**
+ * Closes the external session registry
+ */
+ void close();
+
+ Map<String, ExternalSessionsRegistry> INSTANCES = new HashMap<>();
+
+ static ExternalSessionsRegistry getClient(final Configuration conf) {
+ ExternalSessionsRegistry registry;
+ synchronized (INSTANCES) {
+ // TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE after Tez 1.0.0 is released.
+ String namespace = conf.get("tez.am.registry.namespace");
+ // HS2 would need to know about all coordinators running on all compute groups for a given compute (namespace)
+ // Setting this config to false in client, will make registry client listen on paths under @compute instead of
+ // @compute/compute-group
+ // TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_ENABLE_COMPUTE_GROUPS after Tez 1.0.0 is released.
+ conf.setBoolean("tez.am.registry.enable.compute.groups", false);
Review Comment:
Should we do this conf.get or set inside the synchronized block? Even the logging? I believe we should keep the synchronized block as small as possible? and maybe we can avoid the double lookup using ``computeIfAbsent``
```
conf.setBoolean("tez.am.registry.enable.compute.groups", false);
String namespace = conf.get("tez.am.registry.namespace");
synchronized (INSTANCES) {
// TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE after Tez 1.0.0 is released.
// HS2 would need to know about all coordinators running on all compute groups for a given compute (namespace)
// Setting this config to false in client, will make registry client listen on paths under @compute instead of
// @compute/compute-group
// TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_ENABLE_COMPUTE_GROUPS after Tez 1.0.0 is released.
registry = INSTANCES.computeIfAbsent(namespace, ns -> {
try {
String clazz = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_REGISTRY_CLASS);
return JavaUtils.newInstance(JavaUtils.getClass(clazz, ExternalSessionsRegistry.class),
new Class<?>[] {HiveConf.class}, new Object[] {conf});
} catch (MetaException e) {
throw new RuntimeException(e);
}
});
}
LOG.info("Returning tez external AM registry ({}) for namespace '{}'", System.identityHashCode(registry),
namespace);
return registry;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-3991981447
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [53 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4169793764
@ayushtkn : fixed all the sonarqube warnings possible, also tested on cluster
current patchset from ready from my side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4173252939
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [44 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r2895172831
##########
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java:
##########
@@ -323,6 +326,8 @@ public enum AuthorizationMode{V1, V2};
*/
private final Map<String, FunctionInfo> currentFunctionsInUse = new HashMap<>();
+ private static ExternalSessionsRegistry externalSessions = null;
Review Comment:
doesn't even need a reference on this registry here, removing it altogether
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4053048504
@abstractdog I have seen this & in my bucket list. Will come back, pretty huge, like always :-)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r3009280754
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ExternalSessionsRegistry.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public interface ExternalSessionsRegistry {
+ Logger LOG = LoggerFactory.getLogger(ExternalSessionsRegistry.class);
+
+ /**
+ * Returns application of id of the external session.
+ * @return application id
+ * @throws Exception in case of any exceptions
+ */
+ String getSession() throws Exception;
+
+ /**
+ * Returns external session back to registry.
+ * @param appId application id
+ */
+ void returnSession(String appId);
+
+ /**
+ * Closes the external session registry
+ */
+ void close();
+
+ Map<String, ExternalSessionsRegistry> INSTANCES = new HashMap<>();
Review Comment:
right, there is no proper closure of these so far, I'm doing something similar to what we agreed in TEZ-4007
https://github.com/apache/tez/blob/fd03af686cccce04e128b0815bf30ddf5974ed66/tez-api/src/main/java/org/apache/tez/client/registry/zookeeper/ZkAMRegistryClient.java#L72-L87
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r3010721940
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ZookeeperExternalSessionsRegistryClient.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Set;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.CuratorCache;
+import org.apache.curator.framework.recipes.cache.CuratorCacheListener;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+// TODO: tez should provide this registry
+public class ZookeeperExternalSessionsRegistryClient implements ExternalSessionsRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ZookeeperExternalSessionsRegistryClient.class);
+
+ // TODO: internal only for now to start and reconnect to external session
+ private static final String ZK_PATH = "/tez_am/server";
Review Comment:
this is not needed at all
actually, this is the default of `tez.am.registry.namespace`: https://github.com/apache/tez/blob/bc265069fe42041aaa8640b65b77c7c51d315ed2/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java#L2364
but we're doing it wrong to append it to the already configured `/tez-external-sessions` string (configured as `hive.server2.tez.external.sessions.namespace`), because it leads to paths like:
```
2026-03-07 21:06:46,693 [INFO] [ServiceThread:DAGClientRPCServer] |zookeeper.ZkAMRegistry|: Added AMRecord to zkpath /tez-external-sessions/tez_am/server/application_1769280834537_0000
```
whatever is configured as `hive.server2.tez.external.sessions.namespace`, should be used without concatenation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4161705654
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [43 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4177781215
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [42 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4023870788
@ayushtkn : I'm kindly asking you to review this huge thing :)
please consider @Aggarwal-Raghav 's manual testing as described here https://github.com/apache/hive/pull/6343#issuecomment-4017416145
basically Hive was able to submit a DAG to an external AM (given that zookeeper and hdfs was running on that machine), which was running by the image create in the scope of [TEZ-4682](https://issues.apache.org/jira/browse/TEZ-4682)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r3009945810
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ZookeeperExternalSessionsRegistryClient.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Set;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.CuratorCache;
+import org.apache.curator.framework.recipes.cache.CuratorCacheListener;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+// TODO: tez should provide this registry
+public class ZookeeperExternalSessionsRegistryClient implements ExternalSessionsRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ZookeeperExternalSessionsRegistryClient.class);
+
+ // TODO: internal only for now to start and reconnect to external session
+ private static final String ZK_PATH = "/tez_am/server";
+ private final HiveConf initConf;
+ private final Set<String> available = new HashSet<>();
+ private final Set<String> taken = new HashSet<>();
+ private final Object lock = new Object();
+ private final int maxAttempts;
+
+ private CuratorCache cache;
+ private boolean isInitialized;
+
+
+ public ZookeeperExternalSessionsRegistryClient(final HiveConf initConf) {
+ this.initConf = initConf;
+ this.maxAttempts = HiveConf.getIntVar(initConf, ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_WAIT_MAX_ATTEMPTS);
+ }
+
+ private static String getApplicationId(final ChildData childData) {
+ return childData.getPath().substring(childData.getPath().lastIndexOf("/") + 1);
+ }
+
+ public void close() {
+ if (cache != null) {
+ cache.close();
+ }
+ }
+
+ private void init() throws Exception {
+ String zkServer = HiveConf.getVar(initConf, ConfVars.HIVE_ZOOKEEPER_QUORUM);
+ String zkNamespace = HiveConf.getVar(initConf, ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_NAMESPACE);
+ String effectivePath = zkNamespace + ZK_PATH;
+ CuratorFramework client = CuratorFrameworkFactory.newClient(zkServer, new ExponentialBackoffRetry(1000, 3));
Review Comment:
taking care of that in a shutdownhook
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4182675448
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [42 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r3010561635
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java:
##########
@@ -103,7 +103,13 @@ public interface SessionObjectFactory<SessionType> {
this.deltaRemaining = new AtomicInteger(initialSize);
- final int threadCount = HiveConf.getIntVar(initConf, ConfVars.HIVE_SERVER2_TEZ_SESSION_MAX_INIT_THREADS);
+ int threadCount = 1;
+ if (!HiveConf.getBoolVar(initConf, ConfVars.HIVE_SERVER2_TEZ_USE_EXTERNAL_SESSIONS)) {
+ // Don't use multiple threads for external sessions.
+ threadCount = Math.min(initialSize,
+ HiveConf.getIntVar(initConf, ConfVars.HIVE_SERVER2_TEZ_SESSION_MAX_INIT_THREADS));
Review Comment:
the constructor, where the executor service is initialized, won't be able to hit on initialSize==0:
```
this.initialSize = numSessionsTotal;
...
int threadCount = 1;
if (!HiveConf.getBoolVar(initConf, ConfVars.HIVE_SERVER2_TEZ_USE_EXTERNAL_SESSIONS)) {
// Don't use multiple threads for external sessions.
threadCount = Math.min(initialSize,
HiveConf.getIntVar(initConf, ConfVars.HIVE_SERVER2_TEZ_SESSION_MAX_INIT_THREADS));
}
this.executorService = MoreExecutors
```
so I would not be afraid of `initialSize == 0` and hence the TPE
this zero-check is in the `start` method, which doesn't touch thread pool executor for session initialization, so I cannot relate to that "May be resized later" comment at the moment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4177059116
> Thanx @abstractdog for the update. Changes LGTM.
>
> There is one unused import warning you need to fix https://sonarcloud.io/project/issues?sinceLeakPeriod=true&issueStatuses=OPEN%2CCONFIRMED&pullRequest=6343&id=apache_hive&open=AZ1LFubz0SKtD2DT1Ejd
>
> Feel free to merge once you fix it
done, thanks, good catch!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r3009778977
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.session.SessionState.LogHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.TezClient;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezException;
+import org.apache.tez.mapreduce.hadoop.MRHelpers;
+import org.apache.tez.serviceplugins.api.ServicePluginsDescriptor;
+
+/**
+ * A {@code TezSession} implementation that represents externally managed Tez sessions.
+ * <p>
+ * Unlike {@code TezSessionState}, these sessions are not created or owned by HiveServer2.
+ * Instead, HiveServer2 connects to an already existing Tez session.
+ * </p>
+ *
+ * <h3>Lifecycle</h3>
+ * <ol>
+ * <li>An instance of {@code TezExternalSessionState} is created.</li>
+ * <li>An artificial application ID is acquired from a registry. This does not
+ * correspond to a real YARN application, as the session is unmanaged.</li>
+ * <li>A {@code TezClient} is instantiated but not started (unlike in
+ * {@code TezSessionState}), allowing the rest of the Hive codebase to
+ * interact with it transparently.</li>
+ * </ol>
+ *
+ * <p>
+ * This abstraction enables Hive components to interact with external Tez
+ * sessions using the same interfaces as internally managed sessions.
+ * </p>
+ */
+public class TezExternalSessionState extends TezSessionState {
+ private static final Object DEFAULT_CONF_CREATE_LOCK = new Object();
+ private static TezConfiguration defaultTezConfiguration;
+
+ private String externalAppId;
+ private boolean isDestroying = false;
+ private final ExternalSessionsRegistry registry;
+
+ public TezExternalSessionState(String sessionId, HiveConf conf) {
+ super(sessionId, conf);
+ this.registry = ExternalSessionsRegistry.getClient(conf);
+ synchronized (DEFAULT_CONF_CREATE_LOCK) {
+ if (defaultTezConfiguration == null) {
+ defaultTezConfiguration = createDefaultTezConfig();
+ }
+ }
+ }
+
+ @Override
+ public void ensureLocalResources(Configuration conf, String[] newFilesNotFromConf) {
+ /*
+ * No-op implementation.
+ * External Tez sessions are not backed by a YARN application and therefore
+ * do not manage or localize resources. As a result, there are no local
+ * resources to ensure for this session type.
+ */
+ }
+
+ @Override
+ protected void openInternal(String[] additionalFilesNotFromConf,
Review Comment:
ack, introducing a guard `isOpen` and a corresponding unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r3010721940
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ZookeeperExternalSessionsRegistryClient.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Set;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.CuratorCache;
+import org.apache.curator.framework.recipes.cache.CuratorCacheListener;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+// TODO: tez should provide this registry
+public class ZookeeperExternalSessionsRegistryClient implements ExternalSessionsRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ZookeeperExternalSessionsRegistryClient.class);
+
+ // TODO: internal only for now to start and reconnect to external session
+ private static final String ZK_PATH = "/tez_am/server";
Review Comment:
this is not needed at all
actually, this is the default of `tez.am.registry.namespace`: https://github.com/apache/tez/blob/bc265069fe42041aaa8640b65b77c7c51d315ed2/tez-api/src/main/java/org/apache/tez/dag/api/TezConfiguration.java#L2364
but we're doing it wrong to append it to the already configured `/tez-external-sessions` string (configured as `hive.server2.tez.external.sessions.namespace`), because it leads to paths like:
```
2026-03-07 21:06:46,693 [INFO] [ServiceThread:DAGClientRPCServer] |zookeeper.ZkAMRegistry|: Added AMRecord to zkpath /tez-external-sessions/tez_am/server/application_1769280834537_0000
```
whatever is configured as `hive.server2.tez.external.sessions.namespace`, should be used without concatenation
double-checked, that's what we already do on Cloudera clusters:
```
<property>
<name>hive.server2.tez.external.sessions.namespace</name>
<value>/compute-1774275422-xr4r</value>
</property>
<property>
<name>tez.am.registry.namespace</name>
<value>/compute-1774859022-d5cz</value>
</property>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4011425972
@Aggarwal-Raghav : thanks for your comments so far: can you please check if this patch can work together with your current testing scenario regarding [TEZ-4682](https://issues.apache.org/jira/browse/TEZ-4682), I'm specifically interested whether [TEZ-4686](https://issues.apache.org/jira/browse/TEZ-4686) goes away if hive is configured accordingly:
```
hive.server2.use.external.sessions=true
hive.server2.use.external.sessions.namespace=...
hive.server2.tez.external.sessions.registry.class=org.apache.hadoop.hive.ql.exec.tez.ZookeeperExternalSessionsRegistryClient
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r3005855748
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.session.SessionState.LogHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.TezClient;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezException;
+import org.apache.tez.mapreduce.hadoop.MRHelpers;
+import org.apache.tez.serviceplugins.api.ServicePluginsDescriptor;
+
+/**
+ * A {@code TezSession} implementation that represents externally managed Tez sessions.
+ * <p>
+ * Unlike {@code TezSessionState}, these sessions are not created or owned by HiveServer2.
+ * Instead, HiveServer2 connects to an already existing Tez session.
+ * </p>
+ *
+ * <h3>Lifecycle</h3>
+ * <ol>
+ * <li>An instance of {@code TezExternalSessionState} is created.</li>
+ * <li>An artificial application ID is acquired from a registry. This does not
+ * correspond to a real YARN application, as the session is unmanaged.</li>
+ * <li>A {@code TezClient} is instantiated but not started (unlike in
+ * {@code TezSessionState}), allowing the rest of the Hive codebase to
+ * interact with it transparently.</li>
+ * </ol>
+ *
+ * <p>
+ * This abstraction enables Hive components to interact with external Tez
+ * sessions using the same interfaces as internally managed sessions.
+ * </p>
+ */
+public class TezExternalSessionState extends TezSessionState {
+ private static final Object DEFAULT_CONF_CREATE_LOCK = new Object();
+ private static TezConfiguration defaultTezConfiguration;
+
+ private String externalAppId;
+ private boolean isDestroying = false;
+ private final ExternalSessionsRegistry registry;
+
+ public TezExternalSessionState(String sessionId, HiveConf conf) {
+ super(sessionId, conf);
+ this.registry = ExternalSessionsRegistry.getClient(conf);
+ synchronized (DEFAULT_CONF_CREATE_LOCK) {
+ if (defaultTezConfiguration == null) {
+ defaultTezConfiguration = createDefaultTezConfig();
+ }
+ }
+ }
+
+ @Override
+ public void ensureLocalResources(Configuration conf, String[] newFilesNotFromConf) {
+ /*
+ * No-op implementation.
+ * External Tez sessions are not backed by a YARN application and therefore
+ * do not manage or localize resources. As a result, there are no local
+ * resources to ensure for this session type.
+ */
+ }
+
+ @Override
+ protected void openInternal(String[] additionalFilesNotFromConf,
+ boolean isAsync, LogHelper console, HiveResources resources)
+ throws IOException, TezException {
+ initQueueAndUser();
+
+ boolean llapMode = isLlapMode();
+
+ Map<String, String> amEnv = new HashMap<>();
+ MRHelpers.updateEnvBasedOnMRAMEnv(conf, amEnv);
Review Comment:
who uses this `amEnv`? & if we are passing empty map, then why is this method actually being called?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r3005816424
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ExternalSessionsRegistry.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public interface ExternalSessionsRegistry {
+ Logger LOG = LoggerFactory.getLogger(ExternalSessionsRegistry.class);
+
+ /**
+ * Returns application of id of the external session.
+ * @return application id
+ * @throws Exception in case of any exceptions
+ */
+ String getSession() throws Exception;
+
+ /**
+ * Returns external session back to registry.
+ * @param appId application id
+ */
+ void returnSession(String appId);
+
+ /**
+ * Closes the external session registry
+ */
+ void close();
+
+ Map<String, ExternalSessionsRegistry> INSTANCES = new HashMap<>();
+
+ static ExternalSessionsRegistry getClient(final Configuration conf) {
+ ExternalSessionsRegistry registry;
+ synchronized (INSTANCES) {
+ // TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE after Tez 1.0.0 is released.
+ String namespace = conf.get("tez.am.registry.namespace");
+ // HS2 would need to know about all coordinators running on all compute groups for a given compute (namespace)
+ // Setting this config to false in client, will make registry client listen on paths under @compute instead of
+ // @compute/compute-group
+ // TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_ENABLE_COMPUTE_GROUPS after Tez 1.0.0 is released.
+ conf.setBoolean("tez.am.registry.enable.compute.groups", false);
+ registry = INSTANCES.get(namespace);
+ String clazz = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_REGISTRY_CLASS);
Review Comment:
this should be inside the if block
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ExternalSessionsRegistry.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public interface ExternalSessionsRegistry {
+ Logger LOG = LoggerFactory.getLogger(ExternalSessionsRegistry.class);
+
+ /**
+ * Returns application of id of the external session.
+ * @return application id
+ * @throws Exception in case of any exceptions
+ */
+ String getSession() throws Exception;
+
+ /**
+ * Returns external session back to registry.
+ * @param appId application id
+ */
+ void returnSession(String appId);
+
+ /**
+ * Closes the external session registry
+ */
+ void close();
+
+ Map<String, ExternalSessionsRegistry> INSTANCES = new HashMap<>();
Review Comment:
I am not able to catch the lifecycle of this? Who closes the instances & cleans this up?
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/AbstractTriggerValidator.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.TimeUnit;
+
+import com.google.common.util.concurrent.ThreadFactoryBuilder;
+
+public abstract class AbstractTriggerValidator {
+ private ScheduledExecutorService scheduledExecutorService = null;
+ abstract Runnable getTriggerValidatorRunnable();
+
+ void startTriggerValidator(long triggerValidationIntervalMs) {
+ if (scheduledExecutorService == null) {
+ scheduledExecutorService = Executors.newSingleThreadScheduledExecutor(
+ new ThreadFactoryBuilder().setDaemon(true).setNameFormat("TriggerValidator").build());
+ Runnable triggerValidatorRunnable = getTriggerValidatorRunnable();
+ scheduledExecutorService.scheduleWithFixedDelay(triggerValidatorRunnable, triggerValidationIntervalMs,
+ triggerValidationIntervalMs, TimeUnit.MILLISECONDS);
+ TezSessionPoolSession.LOG.info("Started trigger validator with interval: {} ms", triggerValidationIntervalMs);
Review Comment:
is there a reason for using the Logger from `TezSessionPoolSession`?
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPoolManager.java:
##########
@@ -417,7 +420,13 @@ TriggerValidatorRunnable getTriggerValidatorRunnable() {
}
protected TezSessionPoolSession createSession(String sessionId, HiveConf conf) {
- return new TezSessionPoolSession(sessionId, this, expirationTracker, conf);
+ TezSessionState base;
+ if (useExternalSessions) {
+ base = new TezExternalSessionState(sessionId, conf);
+ } else {
+ base = new TezSessionState(sessionId, conf);
+ }
Review Comment:
this part seems duped with WorkloadManager, maybe we can refactor to avoid duplication
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.session.SessionState.LogHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.TezClient;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezException;
+import org.apache.tez.mapreduce.hadoop.MRHelpers;
+import org.apache.tez.serviceplugins.api.ServicePluginsDescriptor;
+
+/**
+ * A {@code TezSession} implementation that represents externally managed Tez sessions.
+ * <p>
+ * Unlike {@code TezSessionState}, these sessions are not created or owned by HiveServer2.
+ * Instead, HiveServer2 connects to an already existing Tez session.
+ * </p>
+ *
+ * <h3>Lifecycle</h3>
+ * <ol>
+ * <li>An instance of {@code TezExternalSessionState} is created.</li>
+ * <li>An artificial application ID is acquired from a registry. This does not
+ * correspond to a real YARN application, as the session is unmanaged.</li>
+ * <li>A {@code TezClient} is instantiated but not started (unlike in
+ * {@code TezSessionState}), allowing the rest of the Hive codebase to
+ * interact with it transparently.</li>
+ * </ol>
+ *
+ * <p>
+ * This abstraction enables Hive components to interact with external Tez
+ * sessions using the same interfaces as internally managed sessions.
+ * </p>
+ */
+public class TezExternalSessionState extends TezSessionState {
+ private static final Object DEFAULT_CONF_CREATE_LOCK = new Object();
+ private static TezConfiguration defaultTezConfiguration;
+
+ private String externalAppId;
+ private boolean isDestroying = false;
+ private final ExternalSessionsRegistry registry;
+
+ public TezExternalSessionState(String sessionId, HiveConf conf) {
+ super(sessionId, conf);
+ this.registry = ExternalSessionsRegistry.getClient(conf);
+ synchronized (DEFAULT_CONF_CREATE_LOCK) {
+ if (defaultTezConfiguration == null) {
+ defaultTezConfiguration = createDefaultTezConfig();
+ }
+ }
+ }
+
+ @Override
+ public void ensureLocalResources(Configuration conf, String[] newFilesNotFromConf) {
+ /*
+ * No-op implementation.
+ * External Tez sessions are not backed by a YARN application and therefore
+ * do not manage or localize resources. As a result, there are no local
+ * resources to ensure for this session type.
+ */
+ }
+
+ @Override
+ protected void openInternal(String[] additionalFilesNotFromConf,
+ boolean isAsync, LogHelper console, HiveResources resources)
+ throws IOException, TezException {
+ initQueueAndUser();
+
+ boolean llapMode = isLlapMode();
+
+ Map<String, String> amEnv = new HashMap<>();
+ MRHelpers.updateEnvBasedOnMRAMEnv(conf, amEnv);
+
+ TezConfiguration tezConfig = new TezConfiguration(defaultTezConfiguration);
+ setupSessionAcls(tezConfig, conf);
+ ServicePluginsDescriptor spd = createServicePluginDescriptor(llapMode, tezConfig);
+ Credentials llapCredentials = createLlapCredentials(llapMode, tezConfig);
+
+ final TezClient session = TezClient.newBuilder("HIVE-" + getSessionId(), tezConfig)
+ .setIsSession(true)
+ .setCredentials(llapCredentials).setServicePluginDescriptor(spd)
+ .build();
+
+ LOG.info("Opening new External Tez Session (id: {})", getSessionId());
+ TezJobMonitor.initShutdownHook();
+
+ // External sessions doesn't support async mode (getClient should be much cheaper than open,
+ // and the async mode is anyway only used for CLI).
+ if (isAsync) {
+ LOG.info("Ignoring the async argument for an external session {}", getSessionId());
+ }
+ try {
+ externalAppId = registry.getSession();
+ } catch (TezException | IOException e) {
+ throw e;
+ } catch (Exception e) {
+ throw new IOException(e);
+ }
+
+ session.getClient(ApplicationId.fromString(externalAppId));
+ LOG.info("Started an external session; client name {}, app ID {}", session.getClientName(), externalAppId);
+ setTezClient(session);
+ }
+
+ @Override
+ public void close(boolean keepDagFilesDir) throws Exception {
+ // We never close external sessions that don't have errors.
+ if (externalAppId != null) {
+ registry.returnSession(externalAppId);
Review Comment:
if this throws any `RuntimeException`, `externalAppId` won't be even nullified and would lead to resource leaks I belive
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java:
##########
@@ -103,7 +103,13 @@ public interface SessionObjectFactory<SessionType> {
this.deltaRemaining = new AtomicInteger(initialSize);
- final int threadCount = HiveConf.getIntVar(initConf, ConfVars.HIVE_SERVER2_TEZ_SESSION_MAX_INIT_THREADS);
+ int threadCount = 1;
+ if (!HiveConf.getBoolVar(initConf, ConfVars.HIVE_SERVER2_TEZ_USE_EXTERNAL_SESSIONS)) {
+ // Don't use multiple threads for external sessions.
+ threadCount = Math.min(initialSize,
+ HiveConf.getIntVar(initConf, ConfVars.HIVE_SERVER2_TEZ_SESSION_MAX_INIT_THREADS));
Review Comment:
there is a line below
```
if (initialSize == 0) return; // May be resized later.
```
which means `initialSize` can be resized, so do we need to do this `Math.min`
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.session.SessionState.LogHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.TezClient;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezException;
+import org.apache.tez.mapreduce.hadoop.MRHelpers;
+import org.apache.tez.serviceplugins.api.ServicePluginsDescriptor;
+
+/**
+ * A {@code TezSession} implementation that represents externally managed Tez sessions.
+ * <p>
+ * Unlike {@code TezSessionState}, these sessions are not created or owned by HiveServer2.
+ * Instead, HiveServer2 connects to an already existing Tez session.
+ * </p>
+ *
+ * <h3>Lifecycle</h3>
+ * <ol>
+ * <li>An instance of {@code TezExternalSessionState} is created.</li>
+ * <li>An artificial application ID is acquired from a registry. This does not
+ * correspond to a real YARN application, as the session is unmanaged.</li>
+ * <li>A {@code TezClient} is instantiated but not started (unlike in
+ * {@code TezSessionState}), allowing the rest of the Hive codebase to
+ * interact with it transparently.</li>
+ * </ol>
+ *
+ * <p>
+ * This abstraction enables Hive components to interact with external Tez
+ * sessions using the same interfaces as internally managed sessions.
+ * </p>
+ */
+public class TezExternalSessionState extends TezSessionState {
+ private static final Object DEFAULT_CONF_CREATE_LOCK = new Object();
+ private static TezConfiguration defaultTezConfiguration;
+
+ private String externalAppId;
+ private boolean isDestroying = false;
+ private final ExternalSessionsRegistry registry;
+
+ public TezExternalSessionState(String sessionId, HiveConf conf) {
+ super(sessionId, conf);
+ this.registry = ExternalSessionsRegistry.getClient(conf);
+ synchronized (DEFAULT_CONF_CREATE_LOCK) {
+ if (defaultTezConfiguration == null) {
+ defaultTezConfiguration = createDefaultTezConfig();
+ }
+ }
+ }
+
+ @Override
+ public void ensureLocalResources(Configuration conf, String[] newFilesNotFromConf) {
+ /*
+ * No-op implementation.
+ * External Tez sessions are not backed by a YARN application and therefore
+ * do not manage or localize resources. As a result, there are no local
+ * resources to ensure for this session type.
+ */
+ }
+
+ @Override
+ protected void openInternal(String[] additionalFilesNotFromConf,
+ boolean isAsync, LogHelper console, HiveResources resources)
+ throws IOException, TezException {
+ initQueueAndUser();
+
+ boolean llapMode = isLlapMode();
+
+ Map<String, String> amEnv = new HashMap<>();
+ MRHelpers.updateEnvBasedOnMRAMEnv(conf, amEnv);
Review Comment:
who uses this `amEnv`? & if we are passing empty map, this why is this method actually being called?
##########
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestTezExternalSessionsRegistryClient.java:
##########
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import static org.junit.Assert.assertTrue;
+
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.junit.Test;
+
+public class TestTezExternalSessionsRegistryClient {
+ @Test
+ public void testDummyExternalSessionsRegistry() {
+ HiveConf conf = new HiveConf();
+ conf.setVar(HiveConf.ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_REGISTRY_CLASS,
+ DummyExternalSessionsRegistry.class.getName());
+ // TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE after Tez 1.0.0 is released.
+ conf.set("tez.am.registry.namespace", "dummy");
+ ExternalSessionsRegistry externalSessionsRegistry = ExternalSessionsRegistry.getClient(conf);
+ assertTrue(externalSessionsRegistry instanceof DummyExternalSessionsRegistry);
+ }
+
+ @Test
+ public void testTezExternalSessionsRegistry() {
+ HiveConf conf = new HiveConf();
+ // TODO: change this to TezConfiguration.TEZ_AM_ZOOKEEPER_QUORUM after Tez 1.0.0 is released.
+ conf.set("tez.am.zookeeper.quorum", "test-quorum");
+ // TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE after Tez 1.0.0 is released.
+ conf.set("tez.am.registry.namespace", "tez");
+ conf.setVar(HiveConf.ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_REGISTRY_CLASS,
+ ZookeeperExternalSessionsRegistryClient.class.getName());
+ ExternalSessionsRegistry externalSessionsRegistry = ExternalSessionsRegistry.getClient(conf);
+ assertTrue(externalSessionsRegistry instanceof ZookeeperExternalSessionsRegistryClient);
Review Comment:
should we be more strict
```
assertEquals(ZookeeperExternalSessionsRegistryClient.class, externalSessionsRegistry.getClass());
```
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.session.SessionState.LogHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.TezClient;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezException;
+import org.apache.tez.mapreduce.hadoop.MRHelpers;
+import org.apache.tez.serviceplugins.api.ServicePluginsDescriptor;
+
+/**
+ * A {@code TezSession} implementation that represents externally managed Tez sessions.
+ * <p>
+ * Unlike {@code TezSessionState}, these sessions are not created or owned by HiveServer2.
+ * Instead, HiveServer2 connects to an already existing Tez session.
+ * </p>
+ *
+ * <h3>Lifecycle</h3>
+ * <ol>
+ * <li>An instance of {@code TezExternalSessionState} is created.</li>
+ * <li>An artificial application ID is acquired from a registry. This does not
+ * correspond to a real YARN application, as the session is unmanaged.</li>
+ * <li>A {@code TezClient} is instantiated but not started (unlike in
+ * {@code TezSessionState}), allowing the rest of the Hive codebase to
+ * interact with it transparently.</li>
+ * </ol>
+ *
+ * <p>
+ * This abstraction enables Hive components to interact with external Tez
+ * sessions using the same interfaces as internally managed sessions.
+ * </p>
+ */
+public class TezExternalSessionState extends TezSessionState {
+ private static final Object DEFAULT_CONF_CREATE_LOCK = new Object();
+ private static TezConfiguration defaultTezConfiguration;
+
+ private String externalAppId;
+ private boolean isDestroying = false;
+ private final ExternalSessionsRegistry registry;
+
+ public TezExternalSessionState(String sessionId, HiveConf conf) {
+ super(sessionId, conf);
+ this.registry = ExternalSessionsRegistry.getClient(conf);
+ synchronized (DEFAULT_CONF_CREATE_LOCK) {
+ if (defaultTezConfiguration == null) {
+ defaultTezConfiguration = createDefaultTezConfig();
+ }
+ }
+ }
+
+ @Override
+ public void ensureLocalResources(Configuration conf, String[] newFilesNotFromConf) {
+ /*
+ * No-op implementation.
+ * External Tez sessions are not backed by a YARN application and therefore
+ * do not manage or localize resources. As a result, there are no local
+ * resources to ensure for this session type.
+ */
+ }
+
+ @Override
+ protected void openInternal(String[] additionalFilesNotFromConf,
+ boolean isAsync, LogHelper console, HiveResources resources)
+ throws IOException, TezException {
+ initQueueAndUser();
+
+ boolean llapMode = isLlapMode();
+
+ Map<String, String> amEnv = new HashMap<>();
+ MRHelpers.updateEnvBasedOnMRAMEnv(conf, amEnv);
+
+ TezConfiguration tezConfig = new TezConfiguration(defaultTezConfiguration);
+ setupSessionAcls(tezConfig, conf);
+ ServicePluginsDescriptor spd = createServicePluginDescriptor(llapMode, tezConfig);
+ Credentials llapCredentials = createLlapCredentials(llapMode, tezConfig);
+
+ final TezClient session = TezClient.newBuilder("HIVE-" + getSessionId(), tezConfig)
Review Comment:
Can we change the name of the variable, This is bit confusing with Hive Session. maybe sessionClient or sessionTezClient?
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ExternalSessionsRegistry.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public interface ExternalSessionsRegistry {
+ Logger LOG = LoggerFactory.getLogger(ExternalSessionsRegistry.class);
+
+ /**
+ * Returns application of id of the external session.
+ * @return application id
+ * @throws Exception in case of any exceptions
+ */
+ String getSession() throws Exception;
+
+ /**
+ * Returns external session back to registry.
+ * @param appId application id
+ */
+ void returnSession(String appId);
+
+ /**
+ * Closes the external session registry
+ */
+ void close();
+
+ Map<String, ExternalSessionsRegistry> INSTANCES = new HashMap<>();
+
+ static ExternalSessionsRegistry getClient(final Configuration conf) {
+ ExternalSessionsRegistry registry;
+ synchronized (INSTANCES) {
+ // TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE after Tez 1.0.0 is released.
+ String namespace = conf.get("tez.am.registry.namespace");
+ // HS2 would need to know about all coordinators running on all compute groups for a given compute (namespace)
+ // Setting this config to false in client, will make registry client listen on paths under @compute instead of
+ // @compute/compute-group
+ // TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_ENABLE_COMPUTE_GROUPS after Tez 1.0.0 is released.
+ conf.setBoolean("tez.am.registry.enable.compute.groups", false);
Review Comment:
We are mutating the config provided by the client? Means the client config object would get changed here?
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.session.SessionState.LogHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.TezClient;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezException;
+import org.apache.tez.mapreduce.hadoop.MRHelpers;
+import org.apache.tez.serviceplugins.api.ServicePluginsDescriptor;
+
+/**
+ * A {@code TezSession} implementation that represents externally managed Tez sessions.
+ * <p>
+ * Unlike {@code TezSessionState}, these sessions are not created or owned by HiveServer2.
+ * Instead, HiveServer2 connects to an already existing Tez session.
+ * </p>
+ *
+ * <h3>Lifecycle</h3>
+ * <ol>
+ * <li>An instance of {@code TezExternalSessionState} is created.</li>
+ * <li>An artificial application ID is acquired from a registry. This does not
+ * correspond to a real YARN application, as the session is unmanaged.</li>
+ * <li>A {@code TezClient} is instantiated but not started (unlike in
+ * {@code TezSessionState}), allowing the rest of the Hive codebase to
+ * interact with it transparently.</li>
+ * </ol>
+ *
+ * <p>
+ * This abstraction enables Hive components to interact with external Tez
+ * sessions using the same interfaces as internally managed sessions.
+ * </p>
+ */
+public class TezExternalSessionState extends TezSessionState {
+ private static final Object DEFAULT_CONF_CREATE_LOCK = new Object();
+ private static TezConfiguration defaultTezConfiguration;
+
+ private String externalAppId;
+ private boolean isDestroying = false;
Review Comment:
this should also be volatile I belive
##########
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java:
##########
@@ -3998,6 +3998,15 @@ public static enum ConfVars {
"true", new StringSet("true", "false", "ignore"),
"Whether Tez session pool should allow submitting queries to custom queues. The options\n" +
"are true, false (error out), ignore (accept the query but ignore the queue setting)."),
+ HIVE_SERVER2_TEZ_USE_EXTERNAL_SESSIONS("hive.server2.use.external.sessions", false,
Review Comment:
should be ``hive.server2.tez.use.external.sessions``
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.session.SessionState.LogHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.TezClient;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezException;
+import org.apache.tez.mapreduce.hadoop.MRHelpers;
+import org.apache.tez.serviceplugins.api.ServicePluginsDescriptor;
+
+/**
+ * A {@code TezSession} implementation that represents externally managed Tez sessions.
+ * <p>
+ * Unlike {@code TezSessionState}, these sessions are not created or owned by HiveServer2.
+ * Instead, HiveServer2 connects to an already existing Tez session.
+ * </p>
+ *
+ * <h3>Lifecycle</h3>
+ * <ol>
+ * <li>An instance of {@code TezExternalSessionState} is created.</li>
+ * <li>An artificial application ID is acquired from a registry. This does not
+ * correspond to a real YARN application, as the session is unmanaged.</li>
+ * <li>A {@code TezClient} is instantiated but not started (unlike in
+ * {@code TezSessionState}), allowing the rest of the Hive codebase to
+ * interact with it transparently.</li>
+ * </ol>
+ *
+ * <p>
+ * This abstraction enables Hive components to interact with external Tez
+ * sessions using the same interfaces as internally managed sessions.
+ * </p>
+ */
+public class TezExternalSessionState extends TezSessionState {
+ private static final Object DEFAULT_CONF_CREATE_LOCK = new Object();
+ private static TezConfiguration defaultTezConfiguration;
Review Comment:
I believe this should be `volatile`,
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ExternalSessionsRegistry.java:
##########
@@ -0,0 +1,79 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.metastore.api.MetaException;
+import org.apache.hadoop.hive.metastore.utils.JavaUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public interface ExternalSessionsRegistry {
+ Logger LOG = LoggerFactory.getLogger(ExternalSessionsRegistry.class);
+
+ /**
+ * Returns application of id of the external session.
+ * @return application id
+ * @throws Exception in case of any exceptions
+ */
+ String getSession() throws Exception;
+
+ /**
+ * Returns external session back to registry.
+ * @param appId application id
+ */
+ void returnSession(String appId);
+
+ /**
+ * Closes the external session registry
+ */
+ void close();
+
+ Map<String, ExternalSessionsRegistry> INSTANCES = new HashMap<>();
+
+ static ExternalSessionsRegistry getClient(final Configuration conf) {
+ ExternalSessionsRegistry registry;
+ synchronized (INSTANCES) {
+ // TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE after Tez 1.0.0 is released.
+ String namespace = conf.get("tez.am.registry.namespace");
+ // HS2 would need to know about all coordinators running on all compute groups for a given compute (namespace)
+ // Setting this config to false in client, will make registry client listen on paths under @compute instead of
+ // @compute/compute-group
+ // TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_ENABLE_COMPUTE_GROUPS after Tez 1.0.0 is released.
+ conf.setBoolean("tez.am.registry.enable.compute.groups", false);
Review Comment:
Should we do this conf.get or set inside the synchronized block? Even the logging? I believe we should keep the synchronized block as this as possible? and maybe we can avoid the double lookup using ``computeIfAbsent``
```
conf.setBoolean("tez.am.registry.enable.compute.groups", false);
String namespace = conf.get("tez.am.registry.namespace");
synchronized (INSTANCES) {
// TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_NAMESPACE after Tez 1.0.0 is released.
// HS2 would need to know about all coordinators running on all compute groups for a given compute (namespace)
// Setting this config to false in client, will make registry client listen on paths under @compute instead of
// @compute/compute-group
// TODO: change this to TezConfiguration.TEZ_AM_REGISTRY_ENABLE_COMPUTE_GROUPS after Tez 1.0.0 is released.
registry = INSTANCES.computeIfAbsent(namespace, ns -> {
try {
String clazz = HiveConf.getVar(conf, HiveConf.ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_REGISTRY_CLASS);
return JavaUtils.newInstance(JavaUtils.getClass(clazz, ExternalSessionsRegistry.class),
new Class<?>[] {HiveConf.class}, new Object[] {conf});
} catch (MetaException e) {
throw new RuntimeException(e);
}
});
}
LOG.info("Returning tez external AM registry ({}) for namespace '{}'", System.identityHashCode(registry),
namespace);
return registry;
```
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ZookeeperExternalSessionsRegistryClient.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Set;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.CuratorCache;
+import org.apache.curator.framework.recipes.cache.CuratorCacheListener;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+// TODO: tez should provide this registry
+public class ZookeeperExternalSessionsRegistryClient implements ExternalSessionsRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ZookeeperExternalSessionsRegistryClient.class);
+
+ // TODO: internal only for now to start and reconnect to external session
+ private static final String ZK_PATH = "/tez_am/server";
+ private final HiveConf initConf;
+ private final Set<String> available = new HashSet<>();
+ private final Set<String> taken = new HashSet<>();
+ private final Object lock = new Object();
+ private final int maxAttempts;
+
+ private CuratorCache cache;
+ private boolean isInitialized;
+
+
+ public ZookeeperExternalSessionsRegistryClient(final HiveConf initConf) {
+ this.initConf = initConf;
+ this.maxAttempts = HiveConf.getIntVar(initConf, ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_WAIT_MAX_ATTEMPTS);
+ }
+
+ private static String getApplicationId(final ChildData childData) {
+ return childData.getPath().substring(childData.getPath().lastIndexOf("/") + 1);
+ }
+
+ public void close() {
+ if (cache != null) {
+ cache.close();
+ }
+ }
+
+ private void init() throws Exception {
+ String zkServer = HiveConf.getVar(initConf, ConfVars.HIVE_ZOOKEEPER_QUORUM);
+ String zkNamespace = HiveConf.getVar(initConf, ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_NAMESPACE);
+ String effectivePath = zkNamespace + ZK_PATH;
+ CuratorFramework client = CuratorFrameworkFactory.newClient(zkServer, new ExponentialBackoffRetry(1000, 3));
+ synchronized (lock) {
+ client.start();
+ this.cache = CuratorCache.build(client, effectivePath);
+ CuratorCacheListener listener = CuratorCacheListener.builder()
+ .forPathChildrenCache(effectivePath, client, new ExternalSessionsPathListener())
+ .build();
+ cache.listenable().addListener(listener);
+ cache.start();
+ cache.stream()
+ .filter(childData -> childData.getPath() != null
+ && childData.getPath().startsWith(effectivePath + "/"))
+ .forEach(childData -> available.add(getApplicationId(childData)));
+ LOG.info("Initial external sessions: {}", available);
+ isInitialized = true;
+ }
+ }
+
+ public String getSession() throws Exception {
Review Comment:
nit: add override
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ZookeeperExternalSessionsRegistryClient.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Set;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.CuratorCache;
+import org.apache.curator.framework.recipes.cache.CuratorCacheListener;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+// TODO: tez should provide this registry
+public class ZookeeperExternalSessionsRegistryClient implements ExternalSessionsRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ZookeeperExternalSessionsRegistryClient.class);
+
+ // TODO: internal only for now to start and reconnect to external session
+ private static final String ZK_PATH = "/tez_am/server";
+ private final HiveConf initConf;
+ private final Set<String> available = new HashSet<>();
+ private final Set<String> taken = new HashSet<>();
+ private final Object lock = new Object();
+ private final int maxAttempts;
+
+ private CuratorCache cache;
+ private boolean isInitialized;
+
+
+ public ZookeeperExternalSessionsRegistryClient(final HiveConf initConf) {
+ this.initConf = initConf;
+ this.maxAttempts = HiveConf.getIntVar(initConf, ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_WAIT_MAX_ATTEMPTS);
+ }
+
+ private static String getApplicationId(final ChildData childData) {
+ return childData.getPath().substring(childData.getPath().lastIndexOf("/") + 1);
+ }
+
+ public void close() {
+ if (cache != null) {
+ cache.close();
+ }
+ }
+
+ private void init() throws Exception {
+ String zkServer = HiveConf.getVar(initConf, ConfVars.HIVE_ZOOKEEPER_QUORUM);
+ String zkNamespace = HiveConf.getVar(initConf, ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_NAMESPACE);
+ String effectivePath = zkNamespace + ZK_PATH;
+ CuratorFramework client = CuratorFrameworkFactory.newClient(zkServer, new ExponentialBackoffRetry(1000, 3));
Review Comment:
who closes this client?
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.session.SessionState.LogHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.TezClient;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezException;
+import org.apache.tez.mapreduce.hadoop.MRHelpers;
+import org.apache.tez.serviceplugins.api.ServicePluginsDescriptor;
+
+/**
+ * A {@code TezSession} implementation that represents externally managed Tez sessions.
+ * <p>
+ * Unlike {@code TezSessionState}, these sessions are not created or owned by HiveServer2.
+ * Instead, HiveServer2 connects to an already existing Tez session.
+ * </p>
+ *
+ * <h3>Lifecycle</h3>
+ * <ol>
+ * <li>An instance of {@code TezExternalSessionState} is created.</li>
+ * <li>An artificial application ID is acquired from a registry. This does not
+ * correspond to a real YARN application, as the session is unmanaged.</li>
+ * <li>A {@code TezClient} is instantiated but not started (unlike in
+ * {@code TezSessionState}), allowing the rest of the Hive codebase to
+ * interact with it transparently.</li>
+ * </ol>
+ *
+ * <p>
+ * This abstraction enables Hive components to interact with external Tez
+ * sessions using the same interfaces as internally managed sessions.
+ * </p>
+ */
+public class TezExternalSessionState extends TezSessionState {
+ private static final Object DEFAULT_CONF_CREATE_LOCK = new Object();
+ private static TezConfiguration defaultTezConfiguration;
+
+ private String externalAppId;
+ private boolean isDestroying = false;
+ private final ExternalSessionsRegistry registry;
+
+ public TezExternalSessionState(String sessionId, HiveConf conf) {
+ super(sessionId, conf);
+ this.registry = ExternalSessionsRegistry.getClient(conf);
+ synchronized (DEFAULT_CONF_CREATE_LOCK) {
+ if (defaultTezConfiguration == null) {
+ defaultTezConfiguration = createDefaultTezConfig();
+ }
+ }
+ }
+
+ @Override
+ public void ensureLocalResources(Configuration conf, String[] newFilesNotFromConf) {
+ /*
+ * No-op implementation.
+ * External Tez sessions are not backed by a YARN application and therefore
+ * do not manage or localize resources. As a result, there are no local
+ * resources to ensure for this session type.
+ */
+ }
+
+ @Override
+ protected void openInternal(String[] additionalFilesNotFromConf,
Review Comment:
Curious like if someone calls open multiple times, what happens, we don't have any gaurd like isOpen() -> return or so
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/KillTriggerActionHandler.java:
##########
@@ -33,35 +32,33 @@
/**
* Handles only Kill Action.
*/
-public class KillTriggerActionHandler implements TriggerActionHandler<TezSessionState> {
+public class KillTriggerActionHandler implements TriggerActionHandler<TezSession> {
private static final Logger LOG = LoggerFactory.getLogger(KillTriggerActionHandler.class);
private final HiveConf conf;
public KillTriggerActionHandler() {
- this.conf = new HiveConf();
+ this.conf = new HiveConf();
}
@Override
- public void applyAction(final Map<TezSessionState, Trigger> queriesViolated) {
- for (Map.Entry<TezSessionState, Trigger> entry : queriesViolated.entrySet()) {
- if (entry.getValue().getAction().getType() == Action.Type.KILL_QUERY) {
- TezSessionState sessionState = entry.getKey();
- String queryId = sessionState.getWmContext().getQueryId();
- try {
- UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
- DriverUtils.setUpAndStartSessionState(conf, ugi.getShortUserName());
- KillQuery killQuery = sessionState.getKillQuery();
- // if kill query is null then session might have been released to pool or closed already
- if (killQuery != null) {
- sessionState.getKillQuery().killQuery(queryId, entry.getValue().getViolationMsg(),
- sessionState.getConf());
- }
- } catch (HiveException | IOException e) {
- LOG.warn("Unable to kill query {} for trigger violation", queryId);
- }
- } else {
- throw new RuntimeException("Unsupported action: " + entry.getValue());
+ public void applyAction(Map<TezSession, Trigger> queriesViolated) {
+ for (Map.Entry<TezSession, Trigger> entry : queriesViolated.entrySet()) {
+ if (entry.getValue().getAction().getType() == Action.Type.KILL_QUERY) {
+ TezSession sessionState = entry.getKey();
+ String queryId = sessionState.getWmContext().getQueryId();
+ try {
+ UserGroupInformation ugi = UserGroupInformation.getCurrentUser();
+ DriverUtils.setUpAndStartSessionState(conf, ugi.getShortUserName());
+ boolean wasKilled = sessionState.killQuery(entry.getValue().getViolationMsg());
+ if (!wasKilled) {
+ LOG.info("Didn't kill the query {}", queryId);
+ }
+ } catch (HiveException | IOException e) {
+ LOG.warn("Unable to kill query {} for trigger violation", queryId);
Review Comment:
we should log the exception here as well
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSession.java:
##########
@@ -0,0 +1,113 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CancellationException;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.Path;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.session.KillQuery;
+import org.apache.hadoop.hive.ql.session.SessionState.LogHelper;
+import org.apache.hadoop.hive.ql.wm.WmContext;
+import org.apache.hadoop.yarn.api.records.LocalResource;
+import org.apache.tez.client.TezClient;
+import org.apache.tez.dag.api.TezException;
+import org.apache.tez.dag.api.client.DAGStatus;
+
+/**
+ * A bogus interface that basically describes the evolved usage patterns of TezSessionStateImpl.
+ * Needed due to lack of multiple inheritance in Java; probably good to have, too - may make
+ * TezSessionState interface a little bit clearer or even encourage some future cleanup.
+ *
+ * It's implemented in two ways - core implementations (regular session, external session),
+ * and extra functionality implementation (pool session, WM session, etc.) that wraps an instance
+ * of the core implementation (i.e. use composition). With MI, each session type would just inherit
+ * from one of each.
+ */
+public interface TezSession {
+ final class HiveResources {
+ public HiveResources(Path dagResourcesDir) {
+ this.dagResourcesDir = dagResourcesDir;
+ }
+ /** A directory that will contain resources related to DAGs and specified in configs. */
+ final Path dagResourcesDir;
+ final Map<String, LocalResource> additionalFilesNotFromConf = new HashMap<>();
+ /** Localized resources of this session; both from conf and not from conf (above). */
+ final Set<LocalResource> localizedResources = new HashSet<>();
+
+ @Override
+ public String toString() {
+ return dagResourcesDir + "; " + additionalFilesNotFromConf.size() + " additional files, "
+ + localizedResources.size() + " localized resources";
+ }
+ }
+
+ // Core session operations.
+ void open() throws IOException, TezException;
+ void open(HiveResources resources) throws IOException, TezException;
+ void open(String[] additionalFilesNotFromConf) throws IOException, TezException;
+ void beginOpen(String[] additionalFiles, LogHelper console) throws IOException, TezException;
+ void endOpen() throws InterruptedException, CancellationException;
Review Comment:
nit: a bunch of child classes of this class, doesn't have override annotation for bunch of methods, if you could double check once
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ZookeeperExternalSessionsRegistryClient.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Set;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.CuratorCache;
+import org.apache.curator.framework.recipes.cache.CuratorCacheListener;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+// TODO: tez should provide this registry
+public class ZookeeperExternalSessionsRegistryClient implements ExternalSessionsRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ZookeeperExternalSessionsRegistryClient.class);
+
+ // TODO: internal only for now to start and reconnect to external session
+ private static final String ZK_PATH = "/tez_am/server";
+ private final HiveConf initConf;
+ private final Set<String> available = new HashSet<>();
+ private final Set<String> taken = new HashSet<>();
+ private final Object lock = new Object();
+ private final int maxAttempts;
+
+ private CuratorCache cache;
+ private boolean isInitialized;
+
+
+ public ZookeeperExternalSessionsRegistryClient(final HiveConf initConf) {
+ this.initConf = initConf;
+ this.maxAttempts = HiveConf.getIntVar(initConf, ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_WAIT_MAX_ATTEMPTS);
+ }
+
+ private static String getApplicationId(final ChildData childData) {
+ return childData.getPath().substring(childData.getPath().lastIndexOf("/") + 1);
+ }
+
+ public void close() {
+ if (cache != null) {
+ cache.close();
+ }
+ }
+
+ private void init() throws Exception {
+ String zkServer = HiveConf.getVar(initConf, ConfVars.HIVE_ZOOKEEPER_QUORUM);
+ String zkNamespace = HiveConf.getVar(initConf, ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_NAMESPACE);
+ String effectivePath = zkNamespace + ZK_PATH;
+ CuratorFramework client = CuratorFrameworkFactory.newClient(zkServer, new ExponentialBackoffRetry(1000, 3));
+ synchronized (lock) {
+ client.start();
+ this.cache = CuratorCache.build(client, effectivePath);
+ CuratorCacheListener listener = CuratorCacheListener.builder()
+ .forPathChildrenCache(effectivePath, client, new ExternalSessionsPathListener())
+ .build();
+ cache.listenable().addListener(listener);
+ cache.start();
+ cache.stream()
+ .filter(childData -> childData.getPath() != null
+ && childData.getPath().startsWith(effectivePath + "/"))
+ .forEach(childData -> available.add(getApplicationId(childData)));
+ LOG.info("Initial external sessions: {}", available);
+ isInitialized = true;
+ }
+ }
+
+ public String getSession() throws Exception {
+ synchronized (lock) {
+ if (!isInitialized) {
+ init();
+ }
+ long endTimeNs = System.nanoTime() + (1000000000L * maxAttempts);
+ while (available.isEmpty() && ((endTimeNs - System.nanoTime()) > 0)) {
+ lock.wait(1000L);
+ }
+ Iterator<String> iter = available.iterator();
+ if (!iter.hasNext()) {
+ throw new IOException("Cannot get a session after " + maxAttempts + " attempts");
+ }
+ String appId = iter.next();
+ iter.remove();
+ taken.add(appId);
+ return appId;
+ }
+ }
+
+ public void returnSession(String appId) {
Review Comment:
add override
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ZookeeperExternalSessionsRegistryClient.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Set;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.CuratorCache;
+import org.apache.curator.framework.recipes.cache.CuratorCacheListener;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+// TODO: tez should provide this registry
+public class ZookeeperExternalSessionsRegistryClient implements ExternalSessionsRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ZookeeperExternalSessionsRegistryClient.class);
+
+ // TODO: internal only for now to start and reconnect to external session
+ private static final String ZK_PATH = "/tez_am/server";
Review Comment:
should this be configurable?
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ZookeeperExternalSessionsRegistryClient.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Set;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.CuratorCache;
+import org.apache.curator.framework.recipes.cache.CuratorCacheListener;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+// TODO: tez should provide this registry
+public class ZookeeperExternalSessionsRegistryClient implements ExternalSessionsRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ZookeeperExternalSessionsRegistryClient.class);
+
+ // TODO: internal only for now to start and reconnect to external session
+ private static final String ZK_PATH = "/tez_am/server";
+ private final HiveConf initConf;
+ private final Set<String> available = new HashSet<>();
+ private final Set<String> taken = new HashSet<>();
+ private final Object lock = new Object();
+ private final int maxAttempts;
+
+ private CuratorCache cache;
+ private boolean isInitialized;
+
+
+ public ZookeeperExternalSessionsRegistryClient(final HiveConf initConf) {
+ this.initConf = initConf;
+ this.maxAttempts = HiveConf.getIntVar(initConf, ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_WAIT_MAX_ATTEMPTS);
+ }
+
+ private static String getApplicationId(final ChildData childData) {
+ return childData.getPath().substring(childData.getPath().lastIndexOf("/") + 1);
+ }
+
+ public void close() {
+ if (cache != null) {
+ cache.close();
+ }
+ }
+
+ private void init() throws Exception {
+ String zkServer = HiveConf.getVar(initConf, ConfVars.HIVE_ZOOKEEPER_QUORUM);
+ String zkNamespace = HiveConf.getVar(initConf, ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_NAMESPACE);
+ String effectivePath = zkNamespace + ZK_PATH;
+ CuratorFramework client = CuratorFrameworkFactory.newClient(zkServer, new ExponentialBackoffRetry(1000, 3));
+ synchronized (lock) {
+ client.start();
+ this.cache = CuratorCache.build(client, effectivePath);
+ CuratorCacheListener listener = CuratorCacheListener.builder()
+ .forPathChildrenCache(effectivePath, client, new ExternalSessionsPathListener())
+ .build();
+ cache.listenable().addListener(listener);
+ cache.start();
+ cache.stream()
+ .filter(childData -> childData.getPath() != null
+ && childData.getPath().startsWith(effectivePath + "/"))
+ .forEach(childData -> available.add(getApplicationId(childData)));
+ LOG.info("Initial external sessions: {}", available);
+ isInitialized = true;
+ }
+ }
+
+ public String getSession() throws Exception {
+ synchronized (lock) {
+ if (!isInitialized) {
+ init();
+ }
+ long endTimeNs = System.nanoTime() + (1000000000L * maxAttempts);
+ while (available.isEmpty() && ((endTimeNs - System.nanoTime()) > 0)) {
+ lock.wait(1000L);
+ }
+ Iterator<String> iter = available.iterator();
+ if (!iter.hasNext()) {
+ throw new IOException("Cannot get a session after " + maxAttempts + " attempts");
+ }
+ String appId = iter.next();
+ iter.remove();
+ taken.add(appId);
+ return appId;
+ }
+ }
+
+ public void returnSession(String appId) {
+ synchronized (lock) {
+ if (!isInitialized) {
+ throw new AssertionError("Not initialized");
Review Comment:
nit
should be
```
throw new IllegalStateException("Not initialized");
```
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/ZookeeperExternalSessionsRegistryClient.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashSet;
+import java.util.Iterator;
+import java.util.Set;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.framework.imps.CuratorFrameworkState;
+import org.apache.curator.framework.recipes.cache.ChildData;
+import org.apache.curator.framework.recipes.cache.CuratorCache;
+import org.apache.curator.framework.recipes.cache.CuratorCacheListener;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheEvent;
+import org.apache.curator.framework.recipes.cache.PathChildrenCacheListener;
+import org.apache.curator.retry.ExponentialBackoffRetry;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import com.google.common.base.Preconditions;
+
+// TODO: tez should provide this registry
+public class ZookeeperExternalSessionsRegistryClient implements ExternalSessionsRegistry {
+ private static final Logger LOG = LoggerFactory.getLogger(ZookeeperExternalSessionsRegistryClient.class);
+
+ // TODO: internal only for now to start and reconnect to external session
+ private static final String ZK_PATH = "/tez_am/server";
+ private final HiveConf initConf;
+ private final Set<String> available = new HashSet<>();
+ private final Set<String> taken = new HashSet<>();
+ private final Object lock = new Object();
+ private final int maxAttempts;
+
+ private CuratorCache cache;
+ private boolean isInitialized;
Review Comment:
should be volatile
##########
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestZookeeperExternalSessionsRegistryClient.java:
##########
@@ -0,0 +1,140 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotNull;
+import static org.junit.Assert.assertTrue;
+
+import org.apache.curator.framework.CuratorFramework;
+import org.apache.curator.framework.CuratorFrameworkFactory;
+import org.apache.curator.retry.RetryOneTime;
+import org.apache.curator.test.TestingServer;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
+import org.junit.Test;
+
+/**
+ * Tests for {@link ZookeeperExternalSessionsRegistryClient}.
+ */
+public class TestZookeeperExternalSessionsRegistryClient {
+
+ /**
+ * Integration-style unit test that ensures {@link ZookeeperExternalSessionsRegistryClient}
+ * can discover sessions from ZooKeeper and hand them out via {@link ExternalSessionsRegistry#getSession()}.
+ */
+ @Test
+ public void testGetAndReturnSession() throws Exception {
+ TestingServer server = null;
+ CuratorFramework client = null;
+ ZookeeperExternalSessionsRegistryClient registry = null;
+ try {
+ server = new TestingServer();
+ String connectString = server.getConnectString();
+
+ HiveConf conf = new HiveConf();
+ conf.setVar(ConfVars.HIVE_ZOOKEEPER_QUORUM, connectString);
+ conf.setVar(ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_NAMESPACE, "/tez_ns");
+ conf.setIntVar(ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_WAIT_MAX_ATTEMPTS, 1);
+
+ String namespace = HiveConf.getVar(conf, ConfVars.HIVE_SERVER2_TEZ_EXTERNAL_SESSIONS_NAMESPACE);
+ String effectivePath = namespace + "/tez_am/server";
Review Comment:
should we normalize?
```
String effectivePath = (zkNamespace.startsWith("/") ? zkNamespace : "/" + zkNamespace) + ZK_PATH;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r3009635431
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezExternalSessionState.java:
##########
@@ -0,0 +1,171 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.exec.tez;
+
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hive.conf.HiveConf;
+import org.apache.hadoop.hive.ql.exec.tez.monitoring.TezJobMonitor;
+import org.apache.hadoop.hive.ql.metadata.HiveException;
+import org.apache.hadoop.hive.ql.session.SessionState.LogHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.yarn.api.records.ApplicationId;
+import org.apache.tez.client.TezClient;
+import org.apache.tez.dag.api.TezConfiguration;
+import org.apache.tez.dag.api.TezException;
+import org.apache.tez.mapreduce.hadoop.MRHelpers;
+import org.apache.tez.serviceplugins.api.ServicePluginsDescriptor;
+
+/**
+ * A {@code TezSession} implementation that represents externally managed Tez sessions.
+ * <p>
+ * Unlike {@code TezSessionState}, these sessions are not created or owned by HiveServer2.
+ * Instead, HiveServer2 connects to an already existing Tez session.
+ * </p>
+ *
+ * <h3>Lifecycle</h3>
+ * <ol>
+ * <li>An instance of {@code TezExternalSessionState} is created.</li>
+ * <li>An artificial application ID is acquired from a registry. This does not
+ * correspond to a real YARN application, as the session is unmanaged.</li>
+ * <li>A {@code TezClient} is instantiated but not started (unlike in
+ * {@code TezSessionState}), allowing the rest of the Hive codebase to
+ * interact with it transparently.</li>
+ * </ol>
+ *
+ * <p>
+ * This abstraction enables Hive components to interact with external Tez
+ * sessions using the same interfaces as internally managed sessions.
+ * </p>
+ */
+public class TezExternalSessionState extends TezSessionState {
+ private static final Object DEFAULT_CONF_CREATE_LOCK = new Object();
+ private static TezConfiguration defaultTezConfiguration;
+
+ private String externalAppId;
+ private boolean isDestroying = false;
+ private final ExternalSessionsRegistry registry;
+
+ public TezExternalSessionState(String sessionId, HiveConf conf) {
+ super(sessionId, conf);
+ this.registry = ExternalSessionsRegistry.getClient(conf);
+ synchronized (DEFAULT_CONF_CREATE_LOCK) {
+ if (defaultTezConfiguration == null) {
+ defaultTezConfiguration = createDefaultTezConfig();
+ }
+ }
+ }
+
+ @Override
+ public void ensureLocalResources(Configuration conf, String[] newFilesNotFromConf) {
+ /*
+ * No-op implementation.
+ * External Tez sessions are not backed by a YARN application and therefore
+ * do not manage or localize resources. As a result, there are no local
+ * resources to ensure for this session type.
+ */
+ }
+
+ @Override
+ protected void openInternal(String[] additionalFilesNotFromConf,
+ boolean isAsync, LogHelper console, HiveResources resources)
+ throws IOException, TezException {
+ initQueueAndUser();
+
+ boolean llapMode = isLlapMode();
+
+ Map<String, String> amEnv = new HashMap<>();
+ MRHelpers.updateEnvBasedOnMRAMEnv(conf, amEnv);
Review Comment:
must be a leftover, cannot see any usage of amEnv later either
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4156786344
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [63 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4183101308
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [42 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4024497333
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [56 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4023522660
> hive.server2.tez.external.sessions.namespace
> Hi @abstractdog , i tested this patch on my local setup (hadoop, hive, TEZ-4682 branch and Zookeeper - installed via brew) and **its working**🚀. Ran a basic insert command
>
> Attaching the HS2 logs and tez_am docker logs
>
> [tez-am-working.log](https://github.com/user-attachments/files/25818516/tez-am-working.log) [working_hs2.log](https://github.com/user-attachments/files/25818518/working_hs2.log)
>
> **Few things to note:**
>
> 1. Add the following in `/etc/hosts/` file. this is necesary for docker to communicate with hdfs running on localhost
> `127.0.0.1 host.docker.internal`
> 2. tez-am docker image needs hive-exec jar i.e resorce localization. For now I added it using the plugin directory way i.e. `-v "/Users/raghav/Desktop/plugin:/opt/tez/plugins" \` and plugin directory contains only hive-exec jar otherwise .
>
> ```
> Vertex vertex_1769280834537_0000_1_01 [Reducer 2] killed/failed due to: INIT_FAILURE] ....
> vertex=vertex_1769280834537_0000_1_00 [Map 1], org.apache.tez.dag.api .TezUncheckedException: java. lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.ha
> doop.hive.ql.io. CombineHiveInputFormat not found
> ```
>
> 5. `core-site.xml`
>
> ```
> <property>
> <name>fs.defaultFS</name>
> <value>hdfs://host.docker.internal:9000</value>
> </property>
> ```
>
> 6. `hdfs-site.xml` Connecting docker AM to datanode was really problematic, took lot of time 😅
>
> ```
> <property>
> <name>dfs.datanode.use.datanode.hostname</name>
> <value>true</value>
> </property>
>
> <property>
> <name>dfs.datanode.address</name>
> <value>host.docker.internal:9866</value>
> </property>
>
> <property>
> <name>dfs.datanode.hostname</name>
> <value>host.docker.internal</value>
> </property>
> ```
>
> 7. In `docker tez-am tez-site.xml ` (Will update the PR for this as well in TEZ-4682)
>
> ```
> <property>
> <name>dfs.client.use.datanode.hostname</name>
> <value>true</value>
> </property>
> ```
>
> 8. Some `--add-opens` in tez-entrypoint.sh needs to be updated. Basically hive project --add-opens are also required. Will update the PR for TEZ-4682.
> 9. `hive-site.xml`
>
> ```
> <property>
> <name>hive.server2.use.external.sessions</name>
> <value>true</value>
> </property>
>
> <property>
> <name>hive.server2.tez.external.sessions.namespace</name>
> <value>/tez-external-sessions</value>
> </property>
>
> <property>
> <name>hive.server2.tez.external.sessions.registry.class</name>
> <value>org.apache.hadoop.hive.ql.exec.tez.ZookeeperExternalSessionsRegistryClient</value>
> </property>
>
> <property>
> <name>hive.zookeeper.quorum</name>
> <value>localhost:2181</value>
> </property>
> ```
>
> **NOTE/IMPORTANT**: There was a flaky behaviour observed with `tez-conf.pb` in tez-staging directory in hdfs. It was throwing error for `No file found`. Not sure why it was occuring but it was fixed after some time automatically and not facing it again. Will post here if I face it again.
>
> <img alt="Screenshot 2026-03-08 at 2 38 12 AM" width="762" height="387" src="https://private-user-images.githubusercontent.com/49618756/559835670-92ee985e-d1d2-4691-b5ca-53013bcbbd77.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NzMwNTMyODYsIm5iZiI6MTc3MzA1Mjk4NiwicGF0aCI6Ii80OTYxODc1Ni81NTk4MzU2NzAtOTJlZTk4NWUtZDFkMi00NjkxLWI1Y2EtNTMwMTNiY2JiZDc3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNjAzMDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjYwMzA5VDEwNDMwNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQzODk0OTZiYjNhNmQ1MWU5NzM5ZTA2NTQzZGM1OTlkM2I1NTliMGZlOTUwZTcxYjhhNmQxMTljMjJjMGY2MGQmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.UPL300XjZ8GR9-0otEjLrUteLmUs_SEIpSiNE63910w">
@Aggarwal-Raghav : this is awesome, thanks for testing! let me share the action items I can think of here to make this and TEZ-4682 happen:
1. /etc/hosts workaround: external setup steps, I wish we could get rid of them, but maybe it's crucial for the first iteration: I believe hive docker page can iterate about this: https://hive.apache.org/docs/latest/admin/setting-up-hive-with-docker/
2. tez-plugins folder: that's awesome, properly documented on TEZ-4682 side (if it's not enough, we can fully make it work out of the box in the scope of [HIVE-29419](https://issues.apache.org/jira/browse/HIVE-29419)
5/6: core-site.xml + hdfs-site.changes: these are must-have items on the hive site, however it would be better if I can have the hdfs setup and these xml configs at the same place, otherwise, the values might seem a bit vulnerable in a sense that no one knows under what circumstances `host.docker.internal:9866` and `hdfs://host.docker.internal:9000` are valid and working, this is what might be addressed by [HIVE-29493](https://issues.apache.org/jira/browse/HIVE-29493)
8. hive-related add-opens to tez config: in general, tez should not contain hive related stuff given that tez doesn't depend on hive, however, in case of add-opens args, I feel it might be beneficial to have them there as long as those don't refer to anything "hive", only java packages
9. hive-site.xml config: definitely something to be included here in this PR or a related one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4017416145
Hi @abstractdog , i tested this patch on my local setup (hadoop, hive, TEZ-4682 branch and Zookeeper - installed via brew) and **its working**🚀. Ran a basic insert command
Attaching the HS2 logs and tez_am docker logs
[tez-am-working.log](https://github.com/user-attachments/files/25818516/tez-am-working.log)
[working_hs2.log](https://github.com/user-attachments/files/25818518/working_hs2.log)
**Few things to note:**
1. Add the following in `/etc/hosts/` file. this is necesary for docker to communicate with hdfs running on localhost
```127.0.0.1 host.docker.internal```
2. tez-am docker image needs hive-exec jar i.e resorce localization. For now I added it using the plugin directory way i.e. ```-v "/Users/raghav/Desktop/plugin:/opt/tez/plugins" \``` and plugin directory contains only hive-exec jar otherwise .
```
Vertex vertex_1769280834537_0000_1_01 [Reducer 2] killed/failed due to: INIT_FAILURE] ....
vertex=vertex_1769280834537_0000_1_00 [Map 1], org.apache.tez.dag.api .TezUncheckedException: java. lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.ha
doop.hive.ql.io. CombineHiveInputFormat not found
```
5. `core-site.xml`
```
<property>
<name>fs.defaultFS</name>
<value>hdfs://host.docker.internal:9000</value>
</property>
```
6. `hdfs-site.xml` Connecting docker AM to datanode was really problematic, took lot of time 😅
```
<property>
<name>dfs.datanode.use.datanode.hostname</name>
<value>true</value>
</property>
<property>
<name>dfs.datanode.address</name>
<value>host.docker.internal:9866</value>
</property>
<property>
<name>dfs.datanode.hostname</name>
<value>host.docker.internal</value>
</property>
```
7. In `docker tez-am tez-site.xml ` (Will update the PR for this as well in TEZ-4682)
```
<property>
<name>dfs.client.use.datanode.hostname</name>
<value>true</value>
</property>
```
8. Some `--add-opens` in tez-entrypoint.sh needs to be updated. Basically hive project --add-opens are also required. Will update the PR for TEZ-4682.
**NOTE/IMPORTANT**: There was a flaky behaviour observed with `tez-conf.pb` in tez-staging directory in hdfs. It was throwing error for `No file found`. Not sure why it was occuring but it was fixed after some time automatically and not facing it again. Will post here if I face it again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4004154588
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [52 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4011671496
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [58 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "Aggarwal-Raghav (via GitHub)" <gi...@apache.org>.
Aggarwal-Raghav commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4063673798
Hi @abstractdog , **i have been able to create a fully dockerized env for this** i.e. docker image for
1. Minimal hdfs: [HIVE-29493](https://issues.apache.org/jira/browse/HIVE-29493), please check comments under this JIRA
2. tez-am: TEZ-4682
3. zookeeper
4. hive4.3.0-SNAPSHOT - using tez-1.0.0-SNAPSHOT jars and hdfs docker image.
but the 4th docker image requires lot of custom changes:
1. hive docker image is built using `BUILD_ENV=local` , the tez tarball using `mvn clean install` has different naming convention compared to the one deployed in dlcdn artifactory. **This required tez fix IMO. I already have assembly file changes ready, let me know if this requires fix**
```
docker build -t hive4:4.3.0-SNAPSHOT . \
--build-arg BUILD_ENV=local \
--build-arg HADOOP_VERSION=3.4.2 \
--build-arg HIVE_VERSION=4.3.0-SNAPSHOT \
--build-arg TEZ_VERSION=1.0.0-SNAPSHOT
```
2. The core-site.xml, hdfs-site.xml has to be mounted to point to minial hdfs docker image — Maybe we can create a new hive docker image using decoupled hdfs. Currently the tables are created on `file:///` but after mounting they are created on `hdfs:///`
3. the tez-api jar has to be manually replaced from 0.10.5 to 1.0.0-SNAPSHOT as the precedence of hive lib is greater than tez jars — this is only until tez-1.0.0 is released.
4. the hive scracth directory is shared across tez am and hs2. so a shared mount dir needs to be created.
5. `hive-site` changes can't be mouted because of customization in hive entrypoint.sh, hence manually updated configs in hive code before making docker image.
6. Created gist for docker compose.yml and logs for hs2 and tez am: https://gist.github.com/Aggarwal-Raghav/c152c8e2be4a40cc23368ad286b28645
7. the hive-exec jar is requied by tez-am to execute any query and for iceberg table even hive-iceberg jars as well. As per my understanding this will be fixed in future work — How HS2 will propogate resources to tez am docker.
Question:
1. Should docker compose go in as part of TEZ-4682, if not is it ok to keep `host.docker.internal` as zk brew installation for now.
2. Should minimal hadoop be merged first and then TEZ-4682 i..e Hadoop + zk + tez am docker image + docker compose.yml (without hive docker image)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog merged PR #6343:
URL: https://github.com/apache/hive/pull/6343
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "ayushtkn (via GitHub)" <gi...@apache.org>.
ayushtkn commented on code in PR #6343:
URL: https://github.com/apache/hive/pull/6343#discussion_r3005862621
##########
ql/src/java/org/apache/hadoop/hive/ql/exec/tez/TezSessionPool.java:
##########
@@ -103,7 +103,13 @@ public interface SessionObjectFactory<SessionType> {
this.deltaRemaining = new AtomicInteger(initialSize);
- final int threadCount = HiveConf.getIntVar(initConf, ConfVars.HIVE_SERVER2_TEZ_SESSION_MAX_INIT_THREADS);
+ int threadCount = 1;
+ if (!HiveConf.getBoolVar(initConf, ConfVars.HIVE_SERVER2_TEZ_USE_EXTERNAL_SESSIONS)) {
+ // Don't use multiple threads for external sessions.
+ threadCount = Math.min(initialSize,
+ HiveConf.getIntVar(initConf, ConfVars.HIVE_SERVER2_TEZ_SESSION_MAX_INIT_THREADS));
Review Comment:
there is a line below
```
if (initialSize == 0) return; // May be resized later.
```
which means `initialSize` can be resized, so do we need to do this `Math.min`, moreover if it hits 0, like in the if statement, I think the TPE would throw `IllegalArgumentException`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-3990088017
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [95 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org
Re: [PR] HIVE-29477: Introduce TezSession interface and a new implementation for unmanaged, external Tez sessions [hive]
Posted by "sonarqubecloud[bot] (via GitHub)" <gi...@apache.org>.
sonarqubecloud[bot] commented on PR #6343:
URL: https://github.com/apache/hive/pull/6343#issuecomment-4163600608
## [](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343) **Quality Gate passed**
Issues
 [42 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=6343&issueStatuses=ACCEPTED)
Measures
 [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=6343&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)
 [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_coverage&view=list)
 [0.3% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=6343&metric=new_duplicated_lines_density&view=list)
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=6343)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org