You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/07/28 23:52:20 UTC

[GitHub] [iceberg] flyrain opened a new pull request, #5378: Hive: More distinctive cached client pool key to avoid conflict

flyrain opened a new pull request, #5378:
URL: https://github.com/apache/iceberg/pull/5378

   Hive 3+ allows users to set the default catalog name within the same HMS. A job can have two Spark catalogs configured the same uri with a different catalog name as the following example shows.
   ```
   "spark.sql.catalog.catalog2": "org.apache.iceberg.spark.SparkCatalog",
   "spark.sql.catalog.catalog2.hadoop.metastore.catalog.default": "catalog1",
   "spark.sql.catalog.catalog2.type": "hive",
   "spark.sql.catalog.catalog2.uri": "thrift://hms.com:8092",
   
   "spark.sql.catalog.catalog2": "org.apache.iceberg.spark.SparkCatalog",
   "spark.sql.catalog.catalog2.hadoop.metastore.catalog.default": "catalog2",
   "spark.sql.catalog.catalog2.type": "hive",
   "spark.sql.catalog.catalog2.uri": "thrift://hms.com:8092",
   ```
   We need to distinguish them in cached hive client pool.
   @szehon-ho @RussellSpitzer 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] szehon-ho commented on a diff in pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#discussion_r933698789


##########
hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java:
##########
@@ -56,9 +57,16 @@ public class CachedClientPool implements ClientPool<IMetaStoreClient, TException
     init();
   }
 
+  @VisibleForTesting
+  static String cacheKey(Configuration conf) {
+    return String.format(
+        "%s:%s",
+        conf.get(HiveConf.ConfVars.METASTOREURIS.varname, ""), conf.get(CATALOG_DEFAULT, ""));

Review Comment:
   Yea for Spark job it may be rare but HiveCatalog could be used beyond it, like in a long lived server, for example.   Though, true it may not be a huge problem now.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] flyrain commented on pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
flyrain commented on PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#issuecomment-1199801787

   > Also, if this is the case, then it seems we need a more generic mechanism to distinguish clients by conf, as catalog.default is probably not alone in these kind of settings that may require a new MetastoreClient
   
   what else settings are possible practically?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] szehon-ho commented on a diff in pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#discussion_r933692766


##########
hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java:
##########
@@ -56,9 +57,16 @@ public class CachedClientPool implements ClientPool<IMetaStoreClient, TException
     init();
   }
 
+  @VisibleForTesting
+  static String cacheKey(Configuration conf) {
+    return String.format(
+        "%s:%s",
+        conf.get(HiveConf.ConfVars.METASTOREURIS.varname, ""), conf.get(CATALOG_DEFAULT, ""));

Review Comment:
   Yea I was mentioning this too, and internally debating whether its overengineering.  But I wonder , is the list of config something we should expose say via HiveCatalog or just a static list?   I also dont like that if we change the timeout, it can take 5 minutes for a new client to spawn with that.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] flyrain closed pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain closed pull request #5378: Hive: More distinctive cached client pool key to avoid conflict
URL: https://github.com/apache/iceberg/pull/5378


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


Re: [PR] Hive: More distinctive cached client pool key to avoid conflict [iceberg]

Posted by "yabola (via GitHub)" <gi...@apache.org>.
yabola commented on PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#issuecomment-3305978875

   > BTW, with this change doesn't mean Spark can support multiple HMS catalos with Iceberg Catalogs. Spark still uses Hive 2.x and catalog support was added since Hive 3.
   
   Hi, as @viirya mentioned, Iceberg now still cannot support multiple HMS catalogs in Spark. 
   I have raised [another issue](https://github.com/apache/iceberg/issues/14082) to fix it. How do you think, thanks @flyrain @pvary @szehon-ho 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
pvary commented on PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#issuecomment-1199089405

   @flyrain: Why do we need to have a different HiveClientPool? Do these pools have a different configuration?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] szehon-ho commented on a diff in pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#discussion_r933698789


##########
hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java:
##########
@@ -56,9 +57,16 @@ public class CachedClientPool implements ClientPool<IMetaStoreClient, TException
     init();
   }
 
+  @VisibleForTesting
+  static String cacheKey(Configuration conf) {
+    return String.format(
+        "%s:%s",
+        conf.get(HiveConf.ConfVars.METASTOREURIS.varname, ""), conf.get(CATALOG_DEFAULT, ""));

Review Comment:
   Yea for Spark job it may be rare but HiveCatalog could be used beyond it, like in a long lived server, for example.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] flyrain commented on pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
flyrain commented on PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#issuecomment-1199767818

   > @flyrain: Why do we need to have a different HiveClientPool? Do these pools have a different configuration?
   
   Yes, it needs different config(`metastore.catalog.default`). 
   
   Once a client pool is created and cached, we don't reset its default catalog, so we can never visit the second catalog with the same uri. For example, `ClientPool1(uri1, catalog1)` is cached and always be used, but we need another one `ClientPool1(uri1, catalog2)` to visit the second catalog.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] szehon-ho commented on pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
szehon-ho commented on PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#issuecomment-1199807869

   Was looking, there seems also this way to set the metastore-uri to have all the settings encoded in the URI https://cwiki.apache.org/confluence/display/Hive/HiveServer2+Clients#HiveServer2Clients-ConnectionURLFormat. It could be a solution as well.
   
   But maybe I am missing something but still wondering how this happens:  https://github.com/apache/iceberg/blob/master/hive-metastore/src/main/java/org/apache/iceberg/hive/HiveCatalog.java#L105.  seems only one conf is passed in to CachedClientPool.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] flyrain commented on pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by "flyrain (via GitHub)" <gi...@apache.org>.
flyrain commented on PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#issuecomment-1553986022

   #6698 and #7441 have merged. We don't need this any more. Close it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on a diff in pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#discussion_r933755030


##########
hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java:
##########
@@ -56,9 +57,16 @@ public class CachedClientPool implements ClientPool<IMetaStoreClient, TException
     init();
   }
 
+  @VisibleForTesting
+  static String cacheKey(Configuration conf) {
+    return String.format(
+        "%s:%s",
+        conf.get(HiveConf.ConfVars.METASTOREURIS.varname, ""), conf.get(CATALOG_DEFAULT, ""));

Review Comment:
   I do not know of a situation where someone is currently restricted by this configuration problem, but I feel that the static list would be a small additional complexity which would show a clear path forward for anyone bumping into the situation later, even if we are not here to review it.
   
   Alternatively:
   - New configuration: `cacheKey`
   - If the `cacheKey` is provided use it for cache
   - If the `cacheKey` is not provided then use the `uri`
   
   This would solve all of our problems and it would provide the maximum flexibility for the user. The question is security, but I think that could be solved as well.
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] szehon-ho commented on a diff in pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#discussion_r933539862


##########
hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java:
##########
@@ -33,16 +33,18 @@
 
 public class CachedClientPool implements ClientPool<IMetaStoreClient, TException> {
 
+  @VisibleForTesting static final String CATALOG_DEFAULT = "metastore.catalog.default";
   private static Cache<String, HiveClientPool> clientPoolCache;
 
   private final Configuration conf;
-  private final String metastoreUri;
+  private final String clientPoolKey;
   private final int clientPoolSize;
   private final long evictionInterval;
 
   CachedClientPool(Configuration conf, Map<String, String> properties) {
     this.conf = conf;
-    this.metastoreUri = conf.get(HiveConf.ConfVars.METASTOREURIS.varname, "");
+    this.clientPoolKey =
+        conf.get(HiveConf.ConfVars.METASTOREURIS.varname, "") + conf.get(CATALOG_DEFAULT, "");

Review Comment:
   Slightly prefer to have a separate method, so its clearer where to add if there are more client settings that need to be on different pools:
   
   ```
     String cacheKey(Configuration conf) {
       return Joiner.on(":").join(
           conf.get(HiveConf.ConfVars.METASTOREURIS.varname, ""),
           conf.get(CATALOG_DEFAULT))
     }
   ```
   
   Also preferred a separator, to make debugger easier.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on a diff in pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#discussion_r933632062


##########
hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java:
##########
@@ -56,9 +57,16 @@ public class CachedClientPool implements ClientPool<IMetaStoreClient, TException
     init();
   }
 
+  @VisibleForTesting
+  static String cacheKey(Configuration conf) {
+    return String.format(
+        "%s:%s",
+        conf.get(HiveConf.ConfVars.METASTOREURIS.varname, ""), conf.get(CATALOG_DEFAULT, ""));

Review Comment:
   If I understand correctly Spark sends every configuration value to the HMS Client which starts with `spark.sql.catalog.catalog1.hadoop.` in this case.
   Choosing `metastore.catalog.default` as the only thing used in the cacheKey seems like an ad-hoc decision to me. Some interesting configs:
   - CLIENT_SOCKET_TIMEOUT
   - HMS_HANDLER_ATTEMPTS
   - HMS_HANDLER_INTERVAL
   Different users might require different values here...
   
   Would it be "overengineering" to have a static list of config keys to use in the `cacheKey`, and use a hash, or a join of the values for the cache key?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] pvary commented on a diff in pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
pvary commented on code in PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#discussion_r933632062


##########
hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java:
##########
@@ -56,9 +57,16 @@ public class CachedClientPool implements ClientPool<IMetaStoreClient, TException
     init();
   }
 
+  @VisibleForTesting
+  static String cacheKey(Configuration conf) {
+    return String.format(
+        "%s:%s",
+        conf.get(HiveConf.ConfVars.METASTOREURIS.varname, ""), conf.get(CATALOG_DEFAULT, ""));

Review Comment:
   If I understand correctly Spark sends every configuration value to the HMS Client which starts with `spark.sql.catalog.catalog1.hadoop.` in this case.
   Choosing `metastore.catalog.default` as the only thing used in the cacheKey seems like an ad-hoc decision to me. Some interesting random configs:
   - CLIENT_SOCKET_TIMEOUT
   - HMS_HANDLER_ATTEMPTS
   - HMS_HANDLER_INTERVAL
   
   Different users might require different values here...
   
   Would it be "overengineering" to have a static list of config keys to use in the `cacheKey`, and use a hash, or a join of the values for the cache key?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] RussellSpitzer commented on a diff in pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
RussellSpitzer commented on code in PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#discussion_r933549910


##########
hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java:
##########
@@ -33,16 +33,18 @@
 
 public class CachedClientPool implements ClientPool<IMetaStoreClient, TException> {
 
+  @VisibleForTesting static final String CATALOG_DEFAULT = "metastore.catalog.default";
   private static Cache<String, HiveClientPool> clientPoolCache;
 
   private final Configuration conf;
-  private final String metastoreUri;
+  private final String clientPoolKey;
   private final int clientPoolSize;
   private final long evictionInterval;
 
   CachedClientPool(Configuration conf, Map<String, String> properties) {
     this.conf = conf;
-    this.metastoreUri = conf.get(HiveConf.ConfVars.METASTOREURIS.varname, "");
+    this.clientPoolKey =
+        conf.get(HiveConf.ConfVars.METASTOREURIS.varname, "") + conf.get(CATALOG_DEFAULT, "");

Review Comment:
   I would prefer it because what about my two metastores
   
   `host1:9083 with default catalog ""`
   And my metastore
   `host1:90 with default catalog "83"`
   
   On that note are you allowed to do a metastore uri without a port?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] szehon-ho commented on pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
szehon-ho commented on PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#issuecomment-1199772975

   Yea also trying to check what is happening.  Each HiveCatalog currently uses one pool, with a set HiveConf.  Are you using CachedClientPool directly?
   
   Also, if this is the case, then it seems we need a more generic mechanism to distinguish clients by conf, as catalog.default is probably not alone in these kind of settings that may require a new MetastoreClient


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] szehon-ho commented on a diff in pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#discussion_r933698789


##########
hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java:
##########
@@ -56,9 +57,16 @@ public class CachedClientPool implements ClientPool<IMetaStoreClient, TException
     init();
   }
 
+  @VisibleForTesting
+  static String cacheKey(Configuration conf) {
+    return String.format(
+        "%s:%s",
+        conf.get(HiveConf.ConfVars.METASTOREURIS.varname, ""), conf.get(CATALOG_DEFAULT, ""));

Review Comment:
   Yea for Spark job it may be rare but multiple HiveCatalogs with different confs could be used beyond it, like in a long lived multi-tenant server, for example.   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org


[GitHub] [iceberg] flyrain commented on a diff in pull request #5378: Hive: More distinctive cached client pool key to avoid conflict

Posted by GitBox <gi...@apache.org>.
flyrain commented on code in PR #5378:
URL: https://github.com/apache/iceberg/pull/5378#discussion_r933696027


##########
hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java:
##########
@@ -56,9 +57,16 @@ public class CachedClientPool implements ClientPool<IMetaStoreClient, TException
     init();
   }
 
+  @VisibleForTesting
+  static String cacheKey(Configuration conf) {
+    return String.format(
+        "%s:%s",
+        conf.get(HiveConf.ConfVars.METASTOREURIS.varname, ""), conf.get(CATALOG_DEFAULT, ""));

Review Comment:
   > Would it be "overengineering" to have a static list of config keys to use in the cacheKey, and use a hash, or a join of the values for the cache key?
   
   That's a good question. The use case of `metastore.catalog.default` is reasonable since users move data between them, so they have to be in the same job, and to be distinguished. I'd assume it is rare to configure 2 spark catalogs within **one job** for configurations like, CLIENT_SOCKET_TIMEOUT, HMS_HANDLER_ATTEMPTS, HMS_HANDLER_INTERVAL. But I'm open to suggestion.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org