You are viewing a plain text version of this content. The canonical link for it is here.
Posted to gitbox@hive.apache.org by "abstractdog (via GitHub)" <gi...@apache.org> on 2023/11/20 12:41:19 UTC

[PR] [DRAFT] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon + review deleteOnExit usage [hive]

abstractdog opened a new pull request, #4882:
URL: https://github.com/apache/hive/pull/4882

   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description, screenshot and/or a reproducable example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Hive versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   
   ### Is the change a dependency upgrade?
   <!--
   If yes, please attach a file with output from mvn dependency:tree to validate a complete upgrade of dependency.
   If no, write 'No'.
   -->
   
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751766058


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {

Review Comment:
   this craziness is already part of hive come, I'm fine with changing it to any of your suggestions



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759149926


##########
common/src/java/org/apache/hadoop/hive/common/UgiFactory.java:
##########
@@ -15,8 +15,20 @@
 package org.apache.hadoop.hive.common;
 
 import java.io.IOException;
+
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
 
 public interface UgiFactory {
-  UserGroupInformation createUgi() throws IOException;
+  /*
+   * Creates a UserGroupInformation instance for a user with credentials in a scope of a specific query.
+   * Subclasses might implement a cache for taking care of reusing an existing ugi for the same query if possible.
+   */
+  UserGroupInformation createUgi(String queryIdentifier, String user, Credentials credentials) throws IOException;
+
+  /*
+   * Closes all filesystems for a specific query.
+   * In LLAP daemons for the same dag user and credentials, this call typically closes a single FileSystem instance.
+   */
+  void closeFileSystemsForQuery(String queryIdentifier);

Review Comment:
   considering factory pattern strictly, yes, in this case I think we should just rename this to something else
   AbstractLlapUgiFactory which is the primary implementor of this interface has an internal state (ugis map), which is needed when we want to close all the Fs objects that belong to a query, so the method should be called on this object
   
   so I guess all the factory hierarch should be renamed to something else, like a "manager", so from:
   ```
   UgiFactory -> AbstractLlapUgiFactory -> KerberosUgiFactory / NoopUgiFactory
   ```
   ... it might become:
   ```
   UgiManager -> AbstractLlapUgiManager -> KerberosUgiManager / NoopUgiManager
   ```
   are you fine with this @deniskuzZ ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1760045904


##########
llap-server/src/java/org/apache/hadoop/hive/llap/LlapUgiManager.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.hive.llap.daemon.impl.QueryIdentifier;
+import org.apache.hadoop.hive.llap.security.LlapUgiHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LlapUgiManager {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiManager.class);
+  private final ConcurrentHashMap<QueryIdentifier, UserGroupInformation> ugis = new ConcurrentHashMap<>();
+  private final LlapUgiFactory ugiFactory;
+
+  private LlapUgiManager(Configuration conf) {
+    try {
+      ugiFactory = LlapUgiHelper.createLlapUgiFactory(conf);
+    } catch (IOException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  public static LlapUgiManager getInstance(Configuration daemonConf) {
+    return new LlapUgiManager(daemonConf);
+  }
+
+  /**
+   * Creates (or returns) an ugi for tasks in the same query and merges the credentials.
+   * This is valid to be done once per query: no vertex-level ugi and credentials are needed, both of them
+   * are the same within the same query.
+   * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
+   * which is passed into the SignableVertexSpec.
+   * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
+   * into which dag-level credentials are passed.
+   * The most performant way would be to use a single UGI for the same user in the daemon, but that's not possible,
+   * because the credentials can theoretically change across queries.
+   */
+  public UserGroupInformation getUgi(QueryIdentifier queryIdentifier, String user, Credentials credentials) {
+    return ugis.computeIfAbsent(queryIdentifier,
+        k -> {
+          try {
+            UserGroupInformation ugi = ugiFactory.createUgi(user);
+            ugi.addCredentials(credentials);
+            LOG.info("Created ugi {} for queryIdentifier '{}', current ugis #: {}", ugi, queryIdentifier, ugis.size());

Review Comment:
   a) it gives an overall picture about currently used ugis, e.g. 4 concurrent queries means 4 ugis
   b) in the early stages of implementation I was paranoid about a leak here (ugis not removed due to any reason)
   
   both in case of a) and b) this looks useful



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759266139


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {

Review Comment:
   @abstractdog,  how about
   ````
   LlapUgiManager {
      private ConcurrentHashMap<Identifier, UserGroupInformation > fsTaskUgiMap;
      private UgiFactory fsUgiFactory;
      
     public LlapUgiManager getInstance(Configuration conf) {
       return new LlapUgiManager(conf);
     }
     private LlapUgiManager(Configuration conf){
         try {
           fsUgiFactory = LlapUgiFactoryFactory.createFsUgiFactory(daemonConf);
         } catch (IOException e) {
           throw new RuntimeException(e);
         }
      }
   
     public UserGroupInformation getUgi(QueryIdentifier queryIdentifier, String user, Credentials credentials) {
       return fsTaskUgiMap.computeIfAbsent(queryIdentifier, 
           k -> fsUgiFactory.createUgi(k, user, credentials))
     }
   
     public void closeAllForUgi(QueryIdentifier queryIdentifier) {
       ....
       fsTaskUgiMap.remove(queryIdentifier);
     }
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog merged PR #4882:
URL: https://github.com/apache/hive/pull/4882


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759107870


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();

Review Comment:
   nice catch! right, it makes the whole hacking I made useless



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-2175984857

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolutions=WONTFIX)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751745517


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {

Review Comment:
   name is crazy, could we rename to say `LlapUgiHelper`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759063702


##########
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java:
##########
@@ -292,7 +292,8 @@ public SubmitWorkResponseProto submitWork(SubmitWorkRequestProto request) throws
 
       // Lazy create conf object, as it gets expensive in this codepath.
       Supplier<Configuration> callableConf = () -> new Configuration(getConfig());
-      UserGroupInformation fsTaskUgi = fsUgiFactory == null ? null : fsUgiFactory.createUgi();
+      UserGroupInformation fsTaskUgi =
+          fsUgiFactory.createUgi(queryIdentifier.toShortString(), vertex.getUser(), credentials);

Review Comment:
   no, we create one or fail:
   ```
       UgiFactory fsUgiFactory = null;
       try {
         fsUgiFactory = LlapUgiFactoryFactory.createFsUgiFactory(daemonConf);
       } catch (IOException e) {
         throw new RuntimeException(e);
       }
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759266139


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {

Review Comment:
   @abstractdog,  how about
   ````
   LlapUgiManager {
      private ConcurrentHashMap<Identifier, UserGroupInformation > fsTaskUgiMap;
      private UgiFactory fsUgiFactory;
      
     public LlapUgiManager getInstance(Configuration conf) {
       return new LlapUgiManager(conf);
     }
     private LlapUgiManager(Configuration conf){
         try {
           fsUgiFactory = createFsUgiFactory(conf);
         } catch (IOException e) {
           throw new RuntimeException(e);
         }
      }
   
     public UserGroupInformation getOrCreateUgi(QueryIdentifier queryIdentifier, String user, Credentials credentials) {
       return fsTaskUgiMap.computeIfAbsent(queryIdentifier, 
           k -> fsUgiFactory.createUgi(k, user, credentials))
     }
   
     public void closeAllForUgi(QueryIdentifier queryIdentifier) {
       ....
       fsTaskUgiMap.remove(queryIdentifier);
     }
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759149926


##########
common/src/java/org/apache/hadoop/hive/common/UgiFactory.java:
##########
@@ -15,8 +15,20 @@
 package org.apache.hadoop.hive.common;
 
 import java.io.IOException;
+
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
 
 public interface UgiFactory {
-  UserGroupInformation createUgi() throws IOException;
+  /*
+   * Creates a UserGroupInformation instance for a user with credentials in a scope of a specific query.
+   * Subclasses might implement a cache for taking care of reusing an existing ugi for the same query if possible.
+   */
+  UserGroupInformation createUgi(String queryIdentifier, String user, Credentials credentials) throws IOException;
+
+  /*
+   * Closes all filesystems for a specific query.
+   * In LLAP daemons for the same dag user and credentials, this call typically closes a single FileSystem instance.
+   */
+  void closeFileSystemsForQuery(String queryIdentifier);

Review Comment:
   considering factory pattern strictly, I agree, in this case I think we should just rename this to something else
   
   AbstractLlapUgiFactory which is the primary implementor of this interface has an internal state (ugis map), which is needed when we want to close all the Fs objects that belong to a query, so the method should be called on this object
   
   so I guess all the factory hierarchy should be renamed to something else, like a "manager", so from:
   ```
   UgiFactory -> AbstractLlapUgiFactory -> KerberosUgiFactory / NoopUgiFactory
   ```
   ... it might become:
   ```
   UgiManager -> AbstractLlapUgiManager -> KerberosUgiManager / NoopUgiManager
   ```
   are you fine with this @deniskuzZ ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1760045422


##########
llap-server/src/java/org/apache/hadoop/hive/llap/LlapUgiManager.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.hive.llap.daemon.impl.QueryIdentifier;
+import org.apache.hadoop.hive.llap.security.LlapUgiHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LlapUgiManager {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiManager.class);
+  private final ConcurrentHashMap<QueryIdentifier, UserGroupInformation> ugis = new ConcurrentHashMap<>();

Review Comment:
   ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751769739


##########
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java:
##########
@@ -226,10 +226,6 @@ protected TaskRunner2Result callInternal() throws Exception {
 
       // TODO Consolidate this code with TezChild.
       runtimeWatch.start();
-      if (fsTaskUgi == null) {

Review Comment:
   we can, however, it won't help if it's null, but at least I can add a custom message for user convenience's sake



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751789526


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();

Review Comment:
   why do we need a map? don't we create a new Factory per query?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1760043665


##########
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java:
##########
@@ -226,10 +226,6 @@ protected TaskRunner2Result callInternal() throws Exception {
 
       // TODO Consolidate this code with TezChild.
       runtimeWatch.start();
-      if (fsTaskUgi == null) {

Review Comment:
   could we add this precondition check to the TaskRunnerCallable constructor?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751730122


##########
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java:
##########
@@ -488,6 +489,7 @@ public QueryCompleteResponseProto queryComplete(
           fragmentInfo.getFragmentIdentifierString());
         executorService.killFragment(fragmentInfo.getFragmentIdentifierString());
       }
+      fsUgiFactory.closeFileSystemsForQuery(queryIdentifier.toShortString());

Review Comment:
   `amReporter.queryComplete(queryIdentifier)` has a null check for queryIdentifier, so can we get a NullPointer here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751745517


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {

Review Comment:
   name is crazy, could we rename to say `LlapUgiHelper`/`LlapUgiUtil`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759063702


##########
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java:
##########
@@ -292,7 +292,8 @@ public SubmitWorkResponseProto submitWork(SubmitWorkRequestProto request) throws
 
       // Lazy create conf object, as it gets expensive in this codepath.
       Supplier<Configuration> callableConf = () -> new Configuration(getConfig());
-      UserGroupInformation fsTaskUgi = fsUgiFactory == null ? null : fsUgiFactory.createUgi();
+      UserGroupInformation fsTaskUgi =
+          fsUgiFactory.createUgi(queryIdentifier.toShortString(), vertex.getUser(), credentials);

Review Comment:
   no, we create one or fail:
   ```
       UgiFactory fsUgiFactory = null;
       try {
         fsUgiFactory = LlapUgiFactoryFactory.createFsUgiFactory(daemonConf);
       } catch (IOException e) {
         throw new RuntimeException(e);
       }
   ```
   
   in createFsUgiFactory:
   ```
       return hasFsKeytab ? new KerberosUgiFactory(fsKeytab, fsPrincipal) : new NoopUgiFactory();
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759266139


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {

Review Comment:
   @abstractdog,  how about
   ````
   LlapUgiManager {
      private ConcurrentHashMap<Identifier, UserGroupInformation > fsTaskUgiMap;
      private UgiFactory fsUgiFactory;
      
     public LlapUgiManager getInstance(Configuration conf) {
       return new LlapUgiManager(conf);
     }
     private LlapUgiManager(Configuration conf){
         try {
           fsUgiFactory = LlapUgiFactoryFactory.createFsUgiFactory(daemonConf);
         } catch (IOException e) {
           throw new RuntimeException(e);
         }
      }
   
     public UserGroupInformation createUgi(QueryIdentifier queryIdentifier, String user, Credentials credentials) {
       return fsTaskUgiMap.computeIfAbsent(queryIdentifier, 
           k -> fsUgiFactory.createUgi(k, user, credentials))
     }
   
     public void closeAllForUgi(QueryIdentifier queryIdentifier) {
       ....
       fsTaskUgiMap.remove(queryIdentifier);
     }
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "kokila-19 (via GitHub)" <gi...@apache.org>.
kokila-19 commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1708907598


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();
+
+    /**
+     * Creates an ugi for tasks in the same query and merges the credentials.
+     * This is valid to be done once per query: no vertex-level ugi and credentials are needed, both of them
+     * are the same within the same query.
+     * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
+     * which is passed into the SignableVertexSpec.
+     * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
+     * into which dag-level credentials are passed.
+     * The most performant way would be to use a single UGI for the same user in the daemon, but that's not possible,
+     * because the credentials can theoretically chance across queries.

Review Comment:
   **the credentials can theoretically chance across queries.** 
   Wouldn't this be change across queries?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [DRAFT] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon + review deleteOnExit usage [hive]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-1912860217

   This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
   Feel free to reach out on the dev@hive.apache.org list if the patch is in need of reviews.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [DRAFT] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon + review deleteOnExit usage [hive]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4882: [DRAFT] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon + review deleteOnExit usage
URL: https://github.com/apache/hive/pull/4882


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751802196


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();
+
+    /**
+     * Creates an ugi for tasks in the same query and merges the credentials.
+     * This is valid to be done once per query: no vertex-level ugi and credentials are needed, both of them
+     * are the same within the same query.
+     * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
+     * which is passed into the SignableVertexSpec.
+     * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
+     * into which dag-level credentials are passed.
+     * The most performant way would be to use a single UGI for the same user in the daemon, but that's not possible,
+     * because the credentials can theoretically chance across queries.
+     */
+    @Override
+    public UserGroupInformation createUgi(String queryIdentifier, String user, Credentials credentials)
+        throws IOException {
+      // non-sync fast path for 99% of the calls in case of many tasks
+      if (ugis.containsKey(queryIdentifier)) {
+        UserGroupInformation ugi = ugis.get(queryIdentifier);
+        LOG.debug("Ugi ({}) already exists for queryIdentifier '{}'", ugi, queryIdentifier);
+        return ugi;
+      }
+      synchronized (this) {
+        // double-check and return if previous thread already created ugi for this query
+        if (ugis.containsKey(queryIdentifier)) {
+          return ugis.get(queryIdentifier);
+        }
+        UserGroupInformation ugi = createNewUgiInternal(queryIdentifier, user, credentials);
+        ugis.put(queryIdentifier, ugi);
+        return ugi;
+      }
+    }
+
+    @Override
+    public void closeFileSystemsForQuery(String queryIdentifier) {
+      LOG.debug("Closing all FileSystems for queryIdentifier '{}'", queryIdentifier);
+      try {
+        FileSystem.closeAllForUGI(ugis.get(queryIdentifier));
+      } catch (IOException e) {
+        throw new RuntimeException(e);
+      }
+      ugis.remove(queryIdentifier);
+    }
+
+    protected abstract UserGroupInformation createNewUgiInternal(String queryIdentifier, String user,
+        Credentials credentials) throws IOException;
+  }
+
+  private static class KerberosUgiFactory extends AbstractLlapUgiFactory implements UgiFactory {
     private final UserGroupInformation baseUgi;
 
     public KerberosUgiFactory(String keytab, String principal) throws IOException {
       baseUgi = LlapUtil.loginWithKerberos(principal, keytab);
     }
 
     @Override
-    public UserGroupInformation createUgi() throws IOException {
+    protected UserGroupInformation createNewUgiInternal(String queryIdentifier, String user, Credentials credentials)
+        throws IOException {
       // Make sure the UGI is current.
       baseUgi.checkTGTAndReloginFromKeytab();
       // TODO: the only reason this is done this way is because we want unique Subject-s so that
       //       the FS.get gives different FS objects to different fragments.
-      // TODO: could we log in from ticket cache instead? no good method on UGI right now.
-      return SHIMS.cloneUgi(baseUgi);
+      // TODO: could we log in from ticket cache instead? no good method on UGI right now
+      UserGroupInformation ugi = SHIMS.cloneUgi(baseUgi);
+      ugi.addCredentials(credentials);

Review Comment:
   `addCredentials` could be extracted to the base class



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751809382


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();

Review Comment:
   should we use ConcurrentHashMap?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751732730


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();
+
+    /**
+     * Creates an ugi for tasks in the same query and merges the credentials.
+     * This is valid to be done once per query: no vertex-level ugi and credentials are needed, both of them
+     * are the same within the same query.
+     * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
+     * which is passed into the SignableVertexSpec.
+     * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
+     * into which dag-level credentials are passed.
+     * The most performant way would be to use a single UGI for the same user in the daemon, but that's not possible,
+     * because the credentials can theoretically chance across queries.
+     */
+    @Override
+    public UserGroupInformation createUgi(String queryIdentifier, String user, Credentials credentials)
+        throws IOException {
+      // non-sync fast path for 99% of the calls in case of many tasks
+      if (ugis.containsKey(queryIdentifier)) {
+        UserGroupInformation ugi = ugis.get(queryIdentifier);
+        LOG.debug("Ugi ({}) already exists for queryIdentifier '{}'", ugi, queryIdentifier);
+        return ugi;
+      }
+      synchronized (this) {
+        // double-check and return if previous thread already created ugi for this query
+        if (ugis.containsKey(queryIdentifier)) {
+          return ugis.get(queryIdentifier);
+        }
+        UserGroupInformation ugi = createNewUgiInternal(queryIdentifier, user, credentials);
+        ugis.put(queryIdentifier, ugi);
+        return ugi;
+      }
+    }
+
+    @Override
+    public void closeFileSystemsForQuery(String queryIdentifier) {
+      LOG.debug("Closing all FileSystems for queryIdentifier '{}'", queryIdentifier);
+      try {
+        FileSystem.closeAllForUGI(ugis.get(queryIdentifier));
+      } catch (IOException e) {
+        throw new RuntimeException(e);
+      }
+      ugis.remove(queryIdentifier);
+    }
+
+    protected abstract UserGroupInformation createNewUgiInternal(String queryIdentifier, String user,
+        Credentials credentials) throws IOException;
+  }
+
+  private static class KerberosUgiFactory extends AbstractLlapUgiFactory implements UgiFactory {
     private final UserGroupInformation baseUgi;
 
     public KerberosUgiFactory(String keytab, String principal) throws IOException {
       baseUgi = LlapUtil.loginWithKerberos(principal, keytab);
     }
 
     @Override
-    public UserGroupInformation createUgi() throws IOException {
+    protected UserGroupInformation createNewUgiInternal(String queryIdentifier, String user, Credentials credentials)
+        throws IOException {
       // Make sure the UGI is current.
       baseUgi.checkTGTAndReloginFromKeytab();
       // TODO: the only reason this is done this way is because we want unique Subject-s so that
       //       the FS.get gives different FS objects to different fragments.
-      // TODO: could we log in from ticket cache instead? no good method on UGI right now.
-      return SHIMS.cloneUgi(baseUgi);
+      // TODO: could we log in from ticket cache instead? no good method on UGI right now
+      UserGroupInformation ugi = SHIMS.cloneUgi(baseUgi);
+      ugi.addCredentials(credentials);
+      LOG.info("Created ugi in KerberosUgiFactory {} for queryIdentifier '{}', current ugis #: {}", ugi,
+          queryIdentifier, ugis.size());
+      return ugi;
     }
   }
 
-  private static class NoopUgiFactory implements UgiFactory {
+  private static class NoopUgiFactory extends AbstractLlapUgiFactory implements UgiFactory {

Review Comment:
   It looks like a dead code, I can't find any usages of it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759258344


##########
common/src/java/org/apache/hadoop/hive/common/UgiFactory.java:
##########
@@ -15,8 +15,20 @@
 package org.apache.hadoop.hive.common;
 
 import java.io.IOException;
+
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
 
 public interface UgiFactory {
-  UserGroupInformation createUgi() throws IOException;
+  /*
+   * Creates a UserGroupInformation instance for a user with credentials in a scope of a specific query.
+   * Subclasses might implement a cache for taking care of reusing an existing ugi for the same query if possible.
+   */
+  UserGroupInformation createUgi(String queryIdentifier, String user, Credentials credentials) throws IOException;
+
+  /*
+   * Closes all filesystems for a specific query.
+   * In LLAP daemons for the same dag user and credentials, this call typically closes a single FileSystem instance.
+   */
+  void closeFileSystemsForQuery(String queryIdentifier);

Review Comment:
   question, could we keep the factories and use them in UgiManager that would maintain the state?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759277457


##########
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java:
##########
@@ -292,7 +292,8 @@ public SubmitWorkResponseProto submitWork(SubmitWorkRequestProto request) throws
 
       // Lazy create conf object, as it gets expensive in this codepath.
       Supplier<Configuration> callableConf = () -> new Configuration(getConfig());
-      UserGroupInformation fsTaskUgi = fsUgiFactory == null ? null : fsUgiFactory.createUgi();
+      UserGroupInformation fsTaskUgi =
+          fsUgiFactory.createUgi(queryIdentifier.toShortString(), vertex.getUser(), credentials);

Review Comment:
   👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-2351824755

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [9 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-2350173993

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [DRAFT] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon + review deleteOnExit usage [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-2174561881

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [13 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolutions=WONTFIX)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Coverage  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/no-data-16px.png '') No data about Duplication  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751739676


##########
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java:
##########
@@ -226,10 +226,6 @@ protected TaskRunner2Result callInternal() throws Exception {
 
       // TODO Consolidate this code with TezChild.
       runtimeWatch.start();
-      if (fsTaskUgi == null) {

Review Comment:
   should we at least add `Preconditions.checkArgument(fsTaskUgi != null)`?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-2349377954

   @deniskuzZ: your comments were addressed in https://github.com/apache/hive/pull/4882/commits/c84f536c64cf6b298109473c363434d7acbf816e


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751758190


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();
+
+    /**
+     * Creates an ugi for tasks in the same query and merges the credentials.
+     * This is valid to be done once per query: no vertex-level ugi and credentials are needed, both of them
+     * are the same within the same query.
+     * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
+     * which is passed into the SignableVertexSpec.
+     * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
+     * into which dag-level credentials are passed.
+     * The most performant way would be to use a single UGI for the same user in the daemon, but that's not possible,
+     * because the credentials can theoretically chance across queries.

Review Comment:
   typo: 'change'



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759266139


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {

Review Comment:
   ````
   LlapUgiManager {
      private ConcurrentHashMap<Identifier, UserGroupInformation > fsTaskUgiMap;
      private UgiFactory fsUgiFactory;
      
     public LlapUgiManager getInstance(Configuration conf) {
       return new LlapUgiManager(conf);
     }
     private LlapUgiManager(Configuration conf){
         try {
           fsUgiFactory = LlapUgiFactoryFactory.createFsUgiFactory(daemonConf);
         } catch (IOException e) {
           throw new RuntimeException(e);
         }
      }
   
     public UserGroupInformation createUgi(QueryIdentifier queryIdentifier, String user, Credentials credentials) {
       return fsTaskUgiMap.computeIfAbsent(queryIdentifier, 
           k -> fsUgiFactory.createUgi(k, user, credentials))
     }
   
     public void closeAllForUgi(QueryIdentifier queryIdentifier) {
       ....
       fsTaskUgiMap.remove(queryIdentifier);
     }
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759266139


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {

Review Comment:
   @abstractdog,  how about
   ````
   LlapUgiManager {
      private ConcurrentHashMap<Identifier, UserGroupInformation > fsTaskUgiMap;
      private UgiFactory fsUgiFactory;
      
     public LlapUgiManager getInstance(Configuration conf) {
       return new LlapUgiManager(conf);
     }
     private LlapUgiManager(Configuration conf){
         try {
           fsUgiFactory = createFsUgiFactory(conf);
         } catch (IOException e) {
           throw new RuntimeException(e);
         }
      }
   
     public UserGroupInformation getUgi(QueryIdentifier queryIdentifier, String user, Credentials credentials) {
       return fsTaskUgiMap.computeIfAbsent(queryIdentifier, 
           k -> fsUgiFactory.createUgi(k, user, credentials))
     }
   
     public void closeAllForUgi(QueryIdentifier queryIdentifier) {
       ....
       fsTaskUgiMap.remove(queryIdentifier);
     }
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-2351592328

   > LGTM, some minor stuff
   
   thanks! addressed in https://github.com/apache/hive/pull/4882/commits/506c55d07c7e6f761c1295bfcdd6e6ec4d1c1f1e


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751789526


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();

Review Comment:
   why do we need a map? don't we create a new Factory per query?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751809382


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();

Review Comment:
   should we use ConcurrentHashMap?
   ````
   fsTaskUgiMap.computeIfAbsent(queryIdentifier, k -> createNewUgiInternal(k, user, credentials))
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751766058


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {

Review Comment:
   this craziness is already part of hive code, I'm fine with changing it to any of your suggestions



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759266139


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {

Review Comment:
   ````
   LlapUgiManager {
      private ConcurrentHashMap<Identifier, UserGroupInformation > fsTaskUgiMap;
      private UgiFactory fsUgiFactory;
      
     public LlapUgiManager getInstance(Configuration conf) {
       return new LlapUgiManager(conf);
     }
     private LlapUgiManager(Configuration conf){
         try {
           fsUgiFactory = LlapUgiFactoryFactory.createFsUgiFactory(daemonConf);
         } catch (IOException e) {
           throw new RuntimeException(e);
         }
      }
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759149926


##########
common/src/java/org/apache/hadoop/hive/common/UgiFactory.java:
##########
@@ -15,8 +15,20 @@
 package org.apache.hadoop.hive.common;
 
 import java.io.IOException;
+
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
 
 public interface UgiFactory {
-  UserGroupInformation createUgi() throws IOException;
+  /*
+   * Creates a UserGroupInformation instance for a user with credentials in a scope of a specific query.
+   * Subclasses might implement a cache for taking care of reusing an existing ugi for the same query if possible.
+   */
+  UserGroupInformation createUgi(String queryIdentifier, String user, Credentials credentials) throws IOException;
+
+  /*
+   * Closes all filesystems for a specific query.
+   * In LLAP daemons for the same dag user and credentials, this call typically closes a single FileSystem instance.
+   */
+  void closeFileSystemsForQuery(String queryIdentifier);

Review Comment:
   considering factory pattern strictly, I agree, in this case I think we should just rename this to something else
   
   AbstractLlapUgiFactory which is the primary implementor of this interface has an internal state (ugis map), which is needed when we want to close all the Fs objects that belong to a query, so the method should be called on this object
   
   so I guess all the factory hierarchy should be renamed to something else, like a "manager", so from:
   ```
   UgiFactory -> AbstractLlapUgiFactory -> KerberosUgiFactory / NoopUgiFactory
   ```
   ... it might become:
   ```
   LlapUgiManager -> AbstractLlapUgiManager -> KerberosUgiManager / NoopUgiManager
   ```
   are you fine with this @deniskuzZ ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1760043030


##########
llap-server/src/java/org/apache/hadoop/hive/llap/LlapUgiManager.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.hive.llap.daemon.impl.QueryIdentifier;
+import org.apache.hadoop.hive.llap.security.LlapUgiHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LlapUgiManager {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiManager.class);
+  private final ConcurrentHashMap<QueryIdentifier, UserGroupInformation> ugis = new ConcurrentHashMap<>();
+  private final LlapUgiFactory ugiFactory;
+
+  private LlapUgiManager(Configuration conf) {
+    try {
+      ugiFactory = LlapUgiHelper.createLlapUgiFactory(conf);
+    } catch (IOException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  public static LlapUgiManager getInstance(Configuration daemonConf) {
+    return new LlapUgiManager(daemonConf);
+  }
+
+  /**
+   * Creates (or returns) an ugi for tasks in the same query and merges the credentials.
+   * This is valid to be done once per query: no vertex-level ugi and credentials are needed, both of them
+   * are the same within the same query.
+   * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
+   * which is passed into the SignableVertexSpec.
+   * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
+   * into which dag-level credentials are passed.
+   * The most performant way would be to use a single UGI for the same user in the daemon, but that's not possible,
+   * because the credentials can theoretically change across queries.
+   */
+  public UserGroupInformation getUgi(QueryIdentifier queryIdentifier, String user, Credentials credentials) {
+    return ugis.computeIfAbsent(queryIdentifier,
+        k -> {
+          try {
+            UserGroupInformation ugi = ugiFactory.createUgi(user);
+            ugi.addCredentials(credentials);
+            LOG.info("Created ugi {} for queryIdentifier '{}', current ugis #: {}", ugi, queryIdentifier, ugis.size());

Review Comment:
   why do we need to log `ugis.size` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1760045461


##########
llap-server/src/java/org/apache/hadoop/hive/llap/LlapUgiManager.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.hive.llap.daemon.impl.QueryIdentifier;
+import org.apache.hadoop.hive.llap.security.LlapUgiHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LlapUgiManager {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiManager.class);
+  private final ConcurrentHashMap<QueryIdentifier, UserGroupInformation> ugis = new ConcurrentHashMap<>();
+  private final LlapUgiFactory ugiFactory;
+
+  private LlapUgiManager(Configuration conf) {
+    try {
+      ugiFactory = LlapUgiHelper.createLlapUgiFactory(conf);
+    } catch (IOException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  public static LlapUgiManager getInstance(Configuration daemonConf) {
+    return new LlapUgiManager(daemonConf);
+  }
+
+  /**
+   * Creates (or returns) an ugi for tasks in the same query and merges the credentials.
+   * This is valid to be done once per query: no vertex-level ugi and credentials are needed, both of them
+   * are the same within the same query.
+   * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
+   * which is passed into the SignableVertexSpec.
+   * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
+   * into which dag-level credentials are passed.
+   * The most performant way would be to use a single UGI for the same user in the daemon, but that's not possible,
+   * because the credentials can theoretically change across queries.
+   */
+  public UserGroupInformation getUgi(QueryIdentifier queryIdentifier, String user, Credentials credentials) {

Review Comment:
   ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759117586


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();
+
+    /**
+     * Creates an ugi for tasks in the same query and merges the credentials.
+     * This is valid to be done once per query: no vertex-level ugi and credentials are needed, both of them
+     * are the same within the same query.
+     * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
+     * which is passed into the SignableVertexSpec.
+     * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
+     * into which dag-level credentials are passed.
+     * The most performant way would be to use a single UGI for the same user in the daemon, but that's not possible,
+     * because the credentials can theoretically chance across queries.
+     */
+    @Override
+    public UserGroupInformation createUgi(String queryIdentifier, String user, Credentials credentials)
+        throws IOException {
+      // non-sync fast path for 99% of the calls in case of many tasks
+      if (ugis.containsKey(queryIdentifier)) {
+        UserGroupInformation ugi = ugis.get(queryIdentifier);
+        LOG.debug("Ugi ({}) already exists for queryIdentifier '{}'", ugi, queryIdentifier);
+        return ugi;
+      }
+      synchronized (this) {
+        // double-check and return if previous thread already created ugi for this query
+        if (ugis.containsKey(queryIdentifier)) {
+          return ugis.get(queryIdentifier);
+        }
+        UserGroupInformation ugi = createNewUgiInternal(queryIdentifier, user, credentials);
+        ugis.put(queryIdentifier, ugi);
+        return ugi;
+      }
+    }
+
+    @Override
+    public void closeFileSystemsForQuery(String queryIdentifier) {
+      LOG.debug("Closing all FileSystems for queryIdentifier '{}'", queryIdentifier);
+      try {
+        FileSystem.closeAllForUGI(ugis.get(queryIdentifier));
+      } catch (IOException e) {
+        throw new RuntimeException(e);
+      }
+      ugis.remove(queryIdentifier);
+    }
+
+    protected abstract UserGroupInformation createNewUgiInternal(String queryIdentifier, String user,
+        Credentials credentials) throws IOException;
+  }
+
+  private static class KerberosUgiFactory extends AbstractLlapUgiFactory implements UgiFactory {
     private final UserGroupInformation baseUgi;
 
     public KerberosUgiFactory(String keytab, String principal) throws IOException {
       baseUgi = LlapUtil.loginWithKerberos(principal, keytab);
     }
 
     @Override
-    public UserGroupInformation createUgi() throws IOException {
+    protected UserGroupInformation createNewUgiInternal(String queryIdentifier, String user, Credentials credentials)
+        throws IOException {
       // Make sure the UGI is current.
       baseUgi.checkTGTAndReloginFromKeytab();
       // TODO: the only reason this is done this way is because we want unique Subject-s so that
       //       the FS.get gives different FS objects to different fragments.
-      // TODO: could we log in from ticket cache instead? no good method on UGI right now.
-      return SHIMS.cloneUgi(baseUgi);
+      // TODO: could we log in from ticket cache instead? no good method on UGI right now
+      UserGroupInformation ugi = SHIMS.cloneUgi(baseUgi);
+      ugi.addCredentials(credentials);

Review Comment:
   ack, will do



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759122335


##########
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java:
##########
@@ -488,6 +489,7 @@ public QueryCompleteResponseProto queryComplete(
           fragmentInfo.getFragmentIdentifierString());
         executorService.killFragment(fragmentInfo.getFragmentIdentifierString());
       }
+      fsUgiFactory.closeFileSystemsForQuery(queryIdentifier.toShortString());

Review Comment:
   not anymore,because I changed this to QueryIdentifier according to your other suggestion :D



##########
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java:
##########
@@ -488,6 +489,7 @@ public QueryCompleteResponseProto queryComplete(
           fragmentInfo.getFragmentIdentifierString());
         executorService.killFragment(fragmentInfo.getFragmentIdentifierString());
       }
+      fsUgiFactory.closeFileSystemsForQuery(queryIdentifier.toShortString());

Review Comment:
   not anymore, because I changed this to QueryIdentifier according to your other suggestion :D



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-2351594490

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [10 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759080491


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();

Review Comment:
   right, the problem was that UgiFactory is in hive-common, which should not depend on hive-llap-server...however, now I double-checked, nothing else uses UgiFactory in hive repo, it's an llap specific thing, so I'm moving it and it should work



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759266139


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {

Review Comment:
   @abstractdog,  how about
   ````
   LlapUgiManager {
      private ConcurrentHashMap<Identifier, UserGroupInformation > fsTaskUgiMap;
      private UgiFactory fsUgiFactory;
      
     public LlapUgiManager getInstance(Configuration conf) {
       return new LlapUgiManager(conf);
     }
     private LlapUgiManager(Configuration conf){
         try {
           fsUgiFactory = createFsUgiFactory(daemonConf);
         } catch (IOException e) {
           throw new RuntimeException(e);
         }
      }
   
     public UserGroupInformation getUgi(QueryIdentifier queryIdentifier, String user, Credentials credentials) {
       return fsTaskUgiMap.computeIfAbsent(queryIdentifier, 
           k -> fsUgiFactory.createUgi(k, user, credentials))
     }
   
     public void closeAllForUgi(QueryIdentifier queryIdentifier) {
       ....
       fsTaskUgiMap.remove(queryIdentifier);
     }
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [DRAFT] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon + review deleteOnExit usage [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-1825835272

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-2232730418

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [5 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751766437


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();
+
+    /**
+     * Creates an ugi for tasks in the same query and merges the credentials.
+     * This is valid to be done once per query: no vertex-level ugi and credentials are needed, both of them
+     * are the same within the same query.
+     * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
+     * which is passed into the SignableVertexSpec.
+     * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
+     * into which dag-level credentials are passed.
+     * The most performant way would be to use a single UGI for the same user in the daemon, but that's not possible,
+     * because the credentials can theoretically chance across queries.

Review Comment:
   ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751718071


##########
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/ContainerRunnerImpl.java:
##########
@@ -292,7 +292,8 @@ public SubmitWorkResponseProto submitWork(SubmitWorkRequestProto request) throws
 
       // Lazy create conf object, as it gets expensive in this codepath.
       Supplier<Configuration> callableConf = () -> new Configuration(getConfig());
-      UserGroupInformation fsTaskUgi = fsUgiFactory == null ? null : fsUgiFactory.createUgi();
+      UserGroupInformation fsTaskUgi =
+          fsUgiFactory.createUgi(queryIdentifier.toShortString(), vertex.getUser(), credentials);

Review Comment:
   could we have a null fsUgiFactory?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751754010


##########
common/src/java/org/apache/hadoop/hive/common/UgiFactory.java:
##########
@@ -15,8 +15,20 @@
 package org.apache.hadoop.hive.common;
 
 import java.io.IOException;
+
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
 
 public interface UgiFactory {
-  UserGroupInformation createUgi() throws IOException;
+  /*
+   * Creates a UserGroupInformation instance for a user with credentials in a scope of a specific query.
+   * Subclasses might implement a cache for taking care of reusing an existing ugi for the same query if possible.
+   */
+  UserGroupInformation createUgi(String queryIdentifier, String user, Credentials credentials) throws IOException;

Review Comment:
   i don't think it's safe to change the common interface signature



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751732730


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();
+
+    /**
+     * Creates an ugi for tasks in the same query and merges the credentials.
+     * This is valid to be done once per query: no vertex-level ugi and credentials are needed, both of them
+     * are the same within the same query.
+     * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
+     * which is passed into the SignableVertexSpec.
+     * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
+     * into which dag-level credentials are passed.
+     * The most performant way would be to use a single UGI for the same user in the daemon, but that's not possible,
+     * because the credentials can theoretically chance across queries.
+     */
+    @Override
+    public UserGroupInformation createUgi(String queryIdentifier, String user, Credentials credentials)
+        throws IOException {
+      // non-sync fast path for 99% of the calls in case of many tasks
+      if (ugis.containsKey(queryIdentifier)) {
+        UserGroupInformation ugi = ugis.get(queryIdentifier);
+        LOG.debug("Ugi ({}) already exists for queryIdentifier '{}'", ugi, queryIdentifier);
+        return ugi;
+      }
+      synchronized (this) {
+        // double-check and return if previous thread already created ugi for this query
+        if (ugis.containsKey(queryIdentifier)) {
+          return ugis.get(queryIdentifier);
+        }
+        UserGroupInformation ugi = createNewUgiInternal(queryIdentifier, user, credentials);
+        ugis.put(queryIdentifier, ugi);
+        return ugi;
+      }
+    }
+
+    @Override
+    public void closeFileSystemsForQuery(String queryIdentifier) {
+      LOG.debug("Closing all FileSystems for queryIdentifier '{}'", queryIdentifier);
+      try {
+        FileSystem.closeAllForUGI(ugis.get(queryIdentifier));
+      } catch (IOException e) {
+        throw new RuntimeException(e);
+      }
+      ugis.remove(queryIdentifier);
+    }
+
+    protected abstract UserGroupInformation createNewUgiInternal(String queryIdentifier, String user,
+        Credentials credentials) throws IOException;
+  }
+
+  private static class KerberosUgiFactory extends AbstractLlapUgiFactory implements UgiFactory {
     private final UserGroupInformation baseUgi;
 
     public KerberosUgiFactory(String keytab, String principal) throws IOException {
       baseUgi = LlapUtil.loginWithKerberos(principal, keytab);
     }
 
     @Override
-    public UserGroupInformation createUgi() throws IOException {
+    protected UserGroupInformation createNewUgiInternal(String queryIdentifier, String user, Credentials credentials)
+        throws IOException {
       // Make sure the UGI is current.
       baseUgi.checkTGTAndReloginFromKeytab();
       // TODO: the only reason this is done this way is because we want unique Subject-s so that
       //       the FS.get gives different FS objects to different fragments.
-      // TODO: could we log in from ticket cache instead? no good method on UGI right now.
-      return SHIMS.cloneUgi(baseUgi);
+      // TODO: could we log in from ticket cache instead? no good method on UGI right now
+      UserGroupInformation ugi = SHIMS.cloneUgi(baseUgi);
+      ugi.addCredentials(credentials);
+      LOG.info("Created ugi in KerberosUgiFactory {} for queryIdentifier '{}', current ugis #: {}", ugi,
+          queryIdentifier, ugis.size());
+      return ugi;
     }
   }
 
-  private static class NoopUgiFactory implements UgiFactory {
+  private static class NoopUgiFactory extends AbstractLlapUgiFactory implements UgiFactory {

Review Comment:
   It looks like a dead code, I can't find any usages of it



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-2333993183

   > Is the credentials fix - the one Kokila pointed out going in before further merge? Otherwise looks good
   
   yes, tried to clarify this in code comments, but let me add links here for more clarity:
   ```
        * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
        * which is passed into the SignableVertexSpec.
   ```
   https://github.com/apache/hive/blob/e0bd9eac32851b27976a78cd83584cb867b83182/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java#L125
   
   https://github.com/apache/hive/blob/e0bd9eac32851b27976a78cd83584cb867b83182/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java#L919
   
   ```
        * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
        * into which dag-level credentials are passed.
   ```
   https://github.com/apache/hive/blob/e0bd9eac32851b27976a78cd83584cb867b83182/llap-tez/src/java/org/apache/hadoop/hive/llap/tezplugins/LlapTaskCommunicator.java#L918
   
   so assuming that the user is constant and credentials are DAG-level ones, we can make sure that the UserGroupInformation and the contained Credentials are not supposed to change withing a dag


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1760046750


##########
llap-server/src/java/org/apache/hadoop/hive/llap/daemon/impl/TaskRunnerCallable.java:
##########
@@ -226,10 +226,6 @@ protected TaskRunner2Result callInternal() throws Exception {
 
       // TODO Consolidate this code with TezChild.
       runtimeWatch.start();
-      if (fsTaskUgi == null) {

Review Comment:
   ack



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1760042727


##########
llap-server/src/java/org/apache/hadoop/hive/llap/LlapUgiManager.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.hive.llap.daemon.impl.QueryIdentifier;
+import org.apache.hadoop.hive.llap.security.LlapUgiHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LlapUgiManager {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiManager.class);
+  private final ConcurrentHashMap<QueryIdentifier, UserGroupInformation> ugis = new ConcurrentHashMap<>();
+  private final LlapUgiFactory ugiFactory;
+
+  private LlapUgiManager(Configuration conf) {
+    try {
+      ugiFactory = LlapUgiHelper.createLlapUgiFactory(conf);
+    } catch (IOException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  public static LlapUgiManager getInstance(Configuration daemonConf) {
+    return new LlapUgiManager(daemonConf);
+  }
+
+  /**
+   * Creates (or returns) an ugi for tasks in the same query and merges the credentials.
+   * This is valid to be done once per query: no vertex-level ugi and credentials are needed, both of them
+   * are the same within the same query.
+   * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
+   * which is passed into the SignableVertexSpec.
+   * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
+   * into which dag-level credentials are passed.
+   * The most performant way would be to use a single UGI for the same user in the daemon, but that's not possible,
+   * because the credentials can theoretically change across queries.
+   */
+  public UserGroupInformation getUgi(QueryIdentifier queryIdentifier, String user, Credentials credentials) {

Review Comment:
   minor: i think `getOrCreate`Ugi is more suitable here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1760042611


##########
llap-server/src/java/org/apache/hadoop/hive/llap/LlapUgiManager.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.hive.llap.daemon.impl.QueryIdentifier;
+import org.apache.hadoop.hive.llap.security.LlapUgiHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LlapUgiManager {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiManager.class);
+  private final ConcurrentHashMap<QueryIdentifier, UserGroupInformation> ugis = new ConcurrentHashMap<>();

Review Comment:
   minor, please use interface - `ConcurrentMap`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [DRAFT] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon + review deleteOnExit usage [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-1827510422

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [DRAFT] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon + review deleteOnExit usage [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-1819620149

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751809382


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();

Review Comment:
   should we use ConcurrentHashMap?
   ````
   userUgiMap.computeIfAbsent(queryIdentifier, k -> createNewUgiInternal(k, user, credentials))
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759107870


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();

Review Comment:
   nice catch! right, it makes the whole hacking I made unnecessary



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759266139


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {

Review Comment:
   how about
   ````
   LlapUgiManager {
      private ConcurrentHashMap<Identifier, UserGroupInformation > fsTaskUgiMap;
      private UgiFactory fsUgiFactory;
      
     public LlapUgiManager getInstance(Configuration conf) {
       return new LlapUgiManager(conf);
     }
     private LlapUgiManager(Configuration conf){
         try {
           fsUgiFactory = LlapUgiFactoryFactory.createFsUgiFactory(daemonConf);
         } catch (IOException e) {
           throw new RuntimeException(e);
         }
      }
   
     public UserGroupInformation createUgi(QueryIdentifier queryIdentifier, String user, Credentials credentials) {
       return fsTaskUgiMap.computeIfAbsent(queryIdentifier, 
           k -> fsUgiFactory.createUgi(k, user, credentials))
     }
   
     public void closeAllForUgi(QueryIdentifier queryIdentifier) {
       ....
       fsTaskUgiMap.remove(queryIdentifier);
     }
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759272759


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();

Review Comment:
   👍 :)



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] [DRAFT] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon + review deleteOnExit usage [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-1821743532

   Kudos, SonarCloud Quality Gate passed!&nbsp; &nbsp; [![Quality Gate passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/passed-16px.png 'Quality Gate passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882)
   
   [![Bug](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/bug-16px.png 'Bug')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=BUG) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=BUG) [0 Bugs](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=BUG)  
   [![Vulnerability](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/vulnerability-16px.png 'Vulnerability')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=VULNERABILITY) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=VULNERABILITY) [0 Vulnerabilities](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=VULNERABILITY)  
   [![Security Hotspot](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/security_hotspot-16px.png 'Security Hotspot')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&types=SECURITY_HOTSPOT) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&types=SECURITY_HOTSPOT) [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&resolved=false&types=SECURITY_HOTSPOT)  
   [![Code Smell](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/code_smell-16px.png 'Code Smell')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=CODE_SMELL) [![A](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/RatingBadge/A-16px.png 'A')](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=CODE_SMELL) [10 Code Smells](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&resolved=false&types=CODE_SMELL)
   
   [![No Coverage information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/CoverageChart/NoCoverageInfo-16px.png 'No Coverage information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=coverage&view=list) No Coverage information  
   [![No Duplication information](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/Duplications/NoDuplicationInfo-16px.png 'No Duplication information')](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=duplicated_lines_density&view=list) No Duplication information
   
   ![warning](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/message_warning-16px.png 'warning') The version of Java (11.0.8) you have used to run this analysis is deprecated and we will stop accepting it soon. Please update to at least Java 17.
   Read more [here](https://docs.sonarsource.com/sonarcloud/appendices/scanner-environment/)
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1745252872


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();
+
+    /**
+     * Creates an ugi for tasks in the same query and merges the credentials.
+     * This is valid to be done once per query: no vertex-level ugi and credentials are needed, both of them
+     * are the same within the same query.
+     * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
+     * which is passed into the SignableVertexSpec.
+     * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
+     * into which dag-level credentials are passed.
+     * The most performant way would be to use a single UGI for the same user in the daemon, but that's not possible,
+     * because the credentials can theoretically chance across queries.

Review Comment:
   yes! nice catch, let me fix



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #4882: HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon
URL: https://github.com/apache/hive/pull/4882


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759255413


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();

Review Comment:
   👍 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751727877


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiFactoryFactory.class);
+
   private static final HadoopShims SHIMS = ShimLoader.getHadoopShims();
 
-  private static class KerberosUgiFactory implements UgiFactory {
+  /**
+   * This class implements abstract logic for maintaining a single ugi for a specific user in a query.
+   * Subclasses implement createNewUgiInternal for creating a new ugi object when needed.
+   */
+  private static abstract class AbstractLlapUgiFactory implements UgiFactory {
+    Map<String, UserGroupInformation> ugis = new HashMap<>();

Review Comment:
   should we pass `QueryIdentifier` instead of `queryIdentifier.toShortString()`? see knownAppMasters



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751731396


##########
common/src/java/org/apache/hadoop/hive/common/UgiFactory.java:
##########
@@ -15,8 +15,20 @@
 package org.apache.hadoop.hive.common;
 
 import java.io.IOException;
+
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
 
 public interface UgiFactory {
-  UserGroupInformation createUgi() throws IOException;
+  /*
+   * Creates a UserGroupInformation instance for a user with credentials in a scope of a specific query.
+   * Subclasses might implement a cache for taking care of reusing an existing ugi for the same query if possible.
+   */
+  UserGroupInformation createUgi(String queryIdentifier, String user, Credentials credentials) throws IOException;
+
+  /*
+   * Closes all filesystems for a specific query.
+   * In LLAP daemons for the same dag user and credentials, this call typically closes a single FileSystem instance.
+   */
+  void closeFileSystemsForQuery(String queryIdentifier);

Review Comment:
   close method shouldn't be part of a factory pattern



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751754010


##########
common/src/java/org/apache/hadoop/hive/common/UgiFactory.java:
##########
@@ -15,8 +15,20 @@
 package org.apache.hadoop.hive.common;
 
 import java.io.IOException;
+
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
 
 public interface UgiFactory {
-  UserGroupInformation createUgi() throws IOException;
+  /*
+   * Creates a UserGroupInformation instance for a user with credentials in a scope of a specific query.
+   * Subclasses might implement a cache for taking care of reusing an existing ugi for the same query if possible.
+   */
+  UserGroupInformation createUgi(String queryIdentifier, String user, Credentials credentials) throws IOException;

Review Comment:
   you can add this method just in the new AbstractFactory and don't change the interface



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1751754010


##########
common/src/java/org/apache/hadoop/hive/common/UgiFactory.java:
##########
@@ -15,8 +15,20 @@
 package org.apache.hadoop.hive.common;
 
 import java.io.IOException;
+
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
 
 public interface UgiFactory {
-  UserGroupInformation createUgi() throws IOException;
+  /*
+   * Creates a UserGroupInformation instance for a user with credentials in a scope of a specific query.
+   * Subclasses might implement a cache for taking care of reusing an existing ugi for the same query if possible.
+   */
+  UserGroupInformation createUgi(String queryIdentifier, String user, Credentials credentials) throws IOException;

Review Comment:
   i don't think it's safe to change the common interface signature



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1760042727


##########
llap-server/src/java/org/apache/hadoop/hive/llap/LlapUgiManager.java:
##########
@@ -0,0 +1,85 @@
+/*
+ * Licensed under the Apache License, Version 2.0 (the "License");
+ * you may not use this file except in compliance with the License.
+ * You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.llap;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
+import org.apache.hadoop.hive.llap.daemon.impl.QueryIdentifier;
+import org.apache.hadoop.hive.llap.security.LlapUgiHelper;
+import org.apache.hadoop.security.Credentials;
+import org.apache.hadoop.security.UserGroupInformation;
+
+import java.io.IOException;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class LlapUgiManager {
+  private static final Logger LOG = LoggerFactory.getLogger(LlapUgiManager.class);
+  private final ConcurrentHashMap<QueryIdentifier, UserGroupInformation> ugis = new ConcurrentHashMap<>();
+  private final LlapUgiFactory ugiFactory;
+
+  private LlapUgiManager(Configuration conf) {
+    try {
+      ugiFactory = LlapUgiHelper.createLlapUgiFactory(conf);
+    } catch (IOException e) {
+      throw new RuntimeException(e);
+    }
+  }
+
+  public static LlapUgiManager getInstance(Configuration daemonConf) {
+    return new LlapUgiManager(daemonConf);
+  }
+
+  /**
+   * Creates (or returns) an ugi for tasks in the same query and merges the credentials.
+   * This is valid to be done once per query: no vertex-level ugi and credentials are needed, both of them
+   * are the same within the same query.
+   * Regarding vertex user: LlapTaskCommunicator has a single "user" field,
+   * which is passed into the SignableVertexSpec.
+   * Regarding credentials: LlapTaskCommunicator creates SubmitWorkRequestProto instances,
+   * into which dag-level credentials are passed.
+   * The most performant way would be to use a single UGI for the same user in the daemon, but that's not possible,
+   * because the credentials can theoretically change across queries.
+   */
+  public UserGroupInformation getUgi(QueryIdentifier queryIdentifier, String user, Credentials credentials) {

Review Comment:
   minor: i think `getOrCreate` is more suitable here



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "sonarcloud[bot] (via GitHub)" <gi...@apache.org>.
sonarcloud[bot] commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-2351711923

   ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882) **Quality Gate passed**  
   Issues  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [9 New issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=apache_hive&pullRequest=4882&issueStatuses=ACCEPTED)
   
   Measures  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=apache_hive&pullRequest=4882&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=new_coverage&view=list)  
   ![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=apache_hive&pullRequest=4882&metric=new_duplicated_lines_density&view=list)  
     
   [See analysis details on SonarCloud](https://sonarcloud.io/dashboard?id=apache_hive&pullRequest=4882)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "abstractdog (via GitHub)" <gi...@apache.org>.
abstractdog commented on PR #4882:
URL: https://github.com/apache/hive/pull/4882#issuecomment-2352300388

   ready to merge, I checked the refactored patch on cluster, everything works as expected


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org


Re: [PR] HIVE-27884: LLAP: Reuse FileSystem objects from cache across different tasks in the same LLAP daemon [hive]

Posted by "deniskuzZ (via GitHub)" <gi...@apache.org>.
deniskuzZ commented on code in PR #4882:
URL: https://github.com/apache/hive/pull/4882#discussion_r1759266139


##########
llap-server/src/java/org/apache/hadoop/hive/llap/security/LlapUgiFactoryFactory.java:
##########
@@ -15,42 +15,115 @@
 package org.apache.hadoop.hive.llap.security;
 
 import java.io.IOException;
+import java.util.Map;
+import java.util.HashMap;
 
 import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.FileSystem;
 import org.apache.hadoop.hive.common.UgiFactory;
 import org.apache.hadoop.hive.conf.HiveConf;
 import org.apache.hadoop.hive.conf.HiveConf.ConfVars;
 import org.apache.hadoop.hive.llap.LlapUtil;
 import org.apache.hadoop.hive.shims.HadoopShims;
 import org.apache.hadoop.hive.shims.ShimLoader;
+import org.apache.hadoop.security.Credentials;
 import org.apache.hadoop.security.UserGroupInformation;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /** No Java application is complete until it has a FactoryFactory. */
 public class LlapUgiFactoryFactory {

Review Comment:
   ````
   LlapUgiManager {
      private ConcurrentHashMap<Identifier, UserGroupInformation > fsTaskUgiMap;
      private UgiFactory factory;
      
     public LlapUgiManager getInstance(Configuration conf) {
       return new LlapUgiManager(conf);
     }
     private LlapUgiManager(Configuration conf){
         try {
           fsUgiFactory = LlapUgiFactoryFactory.createFsUgiFactory(daemonConf);
         } catch (IOException e) {
           throw new RuntimeException(e);
         }
      }
   }
   ````



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: gitbox-unsubscribe@hive.apache.org
For additional commands, e-mail: gitbox-help@hive.apache.org