You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by GitBox <gi...@apache.org> on 2022/11/11 05:12:43 UTC
[GitHub] [iceberg] lirui-apache opened a new pull request, #6175: Hive: Add UGI to the key in CachedClientPool
lirui-apache opened a new pull request, #6175:
URL: https://github.com/apache/iceberg/pull/6175
This addresses the issue #6071 by adding current UGI to the key of `CachedClientPool.clientPoolCache`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lirui-apache commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
lirui-apache commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1314757980
@nastra Thanks for the review. PR updated accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
flyrain commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1320364877
#5378 is a similar solution. One of reasons we didn't continue is that we also think about removing the `CachedClientPool`, which might be a better solution.
The `CachedClientPool` was introduce by #2325, the idea to share the client pool for the same HMS makes sense at that time, but it seems no strong reason to do that since I think there won't be many catalogs within an app. For Spark, I never see any job uses more than two catalogs. Is it necessary to have a cache between them? Let me know if I missing something. cc @pvary.
Also the `CachedClientPool` becomes less effective when we add more and more things into the keys. Each catalog within one application will have its own HMS client pool effectively.
Considering the complex it adds to the system, we may remove it and let each catalog has its own HMS client pool. Debugging a concurrent bug is never a pleasant job. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lirui-apache commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
lirui-apache commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1321554962
I think caching can be desirable even if there's only one HMS involved. For example, if a user opens two sessions to submit queries, we might want to reuse the underlying pool. I guess this is the use case described in @pvary 's previous [comment](https://github.com/apache/iceberg/pull/6175#issuecomment-1315286360).
The problem is we cannot design an appropriate cache key that fits requirement from different engines. So maybe we can have `CachedClientPool` as the default cache (or disable caching by default), but allow engines to override it with custom implementations?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] gaborkaszab commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
gaborkaszab commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1319666617
> @lirui-apache: I am still a bit concerned that in Hive or Impala this would cause issues by not reusing the client pools. @gaborkaszab: Could you please check from Hive/Impala side? Thanks, Peter
@pvary not my area of expertise, but let me ask around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1319877477
I like that idea, to have a pluggable cache.
As I mentioned before with things like https://github.com/apache/iceberg/pull/5378 , we have users wanting to get different HMS clients based on different hive configs and unpleasantly surprised when they return same HMS client with old configs. It is kind of a large burden for Iceberg to have to make cache key for every different possible configuration.
Side note, we also tried to disable the cache before as well but it led to other errors: https://github.com/apache/iceberg/issues/5371
FYI @flyrain @RussellSpitzer , had discussed with them recently about this problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1322002592
@flyrain: The Catalog is a strange animal. We basically expect it to be a short lived and easy/cheap to drop and recreate. I had a similar discussion [1] on another PR around this. I Hive code we definitely create Catalog objects with considering the costs negligible.
I feel that we bump into the issue of closing/keeping the catalogs more and more, and the current working is kind of non-intuitive, but changing this would be a serious impact so we should consider carefully.
In the meantime I would support the idea of the configurable cache.
[1] https://github.com/apache/iceberg/pull/5166
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1318658845
@lirui-apache: I am still a bit concerned that in Hive or Impala this would cause issues by not reusing the client pools.
@gaborkaszab: Could you please check from Hive/Impala side?
Thanks,
Peter
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lirui-apache commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
lirui-apache commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1319845848
The pool is still reused as long as the same UGI instance is in use. But I agree it might be desirable to reuse the pool across queries or sessions for the same user.
Since it's difficult for iceberg to assume how UGI instances are used, I think another option is to let the engines decide what kind of cache to use. I noticed that trino overrides the FileSystem cache in their own Hadoop fork, and uses only user names in their cache key [1][2]. Maybe we can also let users specify the cache to use in `HiveCatalog`?
[1] https://github.com/trinodb/trino-hadoop-apache/blob/master/src/main/java/org/apache/hadoop/fs/FileSystemManager.java#L30
[2] https://github.com/trinodb/trino/blob/c28f676525bc426201302e71f4d9a978bb76a84b/lib/trino-hdfs/src/main/java/io/trino/hdfs/TrinoFileSystemCache.java#L254
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lirui-apache commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1484570003
Superseded by #6698
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lirui-apache commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
lirui-apache commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1314982379
@pvary Could you also have a look? Thanks~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1315286360
@lirui-apache: I am not sure how the Key/UGI equals/hashCode works, this could create funny situations.
Maybe we could add another test case where
```
UserGroupInformation foo1 = UserGroupInformation.createProxyUser("foo", current);
UserGroupInformation foo2 = UserGroupInformation.createProxyUser("foo", current);
```
would return the same pool?
Just to make sure that the cache is still working as expected, even if the UGIs are the same, but just a different instance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] nastra commented on a diff in pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
nastra commented on code in PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#discussion_r1019948958
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java:
##########
@@ -87,4 +92,50 @@ public <R> R run(Action<R, IMetaStoreClient, TException> action, boolean retry)
throws TException, InterruptedException {
return clientPool().run(action, retry);
}
+
+ @VisibleForTesting
Review Comment:
this could be simplified to:
```
@VisibleForTesting
@Value.Immutable
abstract static class Key {
abstract String metastoreUri();
abstract UserGroupInformation ugi();
static Key of(String metastoreUri) {
try {
return ImmutableKey.builder()
.metastoreUri(metastoreUri)
.ugi(UserGroupInformation.getCurrentUser())
.build();
} catch (IOException e) {
throw new UncheckedIOException("Failed to get current user", e);
}
}
}
```
You'll just need to add this compile-time dependency
```
diff --git a/build.gradle b/build.gradle
index e5ab7377b..feb2e5acc 100644
--- a/build.gradle
+++ b/build.gradle
@@ -459,6 +459,8 @@ project(':iceberg-hive-metastore') {
implementation project(':iceberg-core')
api project(':iceberg-api')
implementation project(':iceberg-common')
+ annotationProcessor "org.immutables:value"
+ compileOnly "org.immutables:value"
implementation "com.github.ben-manes.caffeine:caffeine"
```
And then you'd adjust the places where `Key` is being used to `Key.of(...`)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on a diff in pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on code in PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#discussion_r1022551766
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java:
##########
@@ -87,4 +91,23 @@ public <R> R run(Action<R, IMetaStoreClient, TException> action, boolean retry)
throws TException, InterruptedException {
return clientPool().run(action, retry);
}
+
+ @VisibleForTesting
+ @Value.Immutable
+ abstract static class Key {
Review Comment:
This is great, this is actually a common issue, a lot of values are not in the cache key but should be..
See this issue as well: https://github.com/apache/iceberg/pull/5378 for default Hive catalog, I guess we can put catalog in this as well then.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lirui-apache commented on a diff in pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
lirui-apache commented on code in PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#discussion_r1022669799
##########
hive-metastore/src/main/java/org/apache/iceberg/hive/CachedClientPool.java:
##########
@@ -87,4 +91,23 @@ public <R> R run(Action<R, IMetaStoreClient, TException> action, boolean retry)
throws TException, InterruptedException {
return clientPool().run(action, retry);
}
+
+ @VisibleForTesting
+ @Value.Immutable
+ abstract static class Key {
Review Comment:
Yeah, our use case here only needs the UGI, we can extend the cache key to take more values as needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lirui-apache closed pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache closed pull request #6175: Hive: Add UGI to the key in CachedClientPool
URL: https://github.com/apache/iceberg/pull/6175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lirui-apache commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by "lirui-apache (via GitHub)" <gi...@apache.org>.
lirui-apache commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1407978754
@szehon-ho Sorry for the delayed reply. I'll work on a PR for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] pvary commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
pvary commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1318660031
@lirui-apache: Ohh, and big thanks for the detailed check!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lirui-apache commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
lirui-apache commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1316470746
@pvary That's a good question. Unfortunately the test case you mentioned won't work. I looked into how UGI implements equals/hashCode and found the behavior is intentional [1]. Hadoop folks did mention that optimizations could be possible for proxy users, but it's not safe to do so on their end because UGI instances are mutable [2]. Instead they intend to leave that optimization to engines/services who have better knowledge about the UGI instances [3].
There were also reports about the UGI behavior causing memory issues in Hadoop FileSystem cache (which also uses UGI in the key) [4][5]. But as we have TTL on our cache entries, I guess we're fine in that regard.
What do you think?
[1] https://issues.apache.org/jira/browse/HADOOP-6670
[2] https://issues.apache.org/jira/browse/HADOOP-12529
[3] https://issues.apache.org/jira/browse/HADOOP-12529?focusedCommentId=14984176&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14984176
[4] https://issues.apache.org/jira/browse/HIVE-3098
[5] https://issues.apache.org/jira/browse/YARN-58
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] lirui-apache commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
lirui-apache commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1324633275
It seems we generally agree to support pluggable cache. How about allow users to specify ClientPool implementation class via a catalog property, and HiveCatalog can instantiate ClientPool instance accordingly. We probably need an `initialize` method in `ClientPool` so that catalogs can pass the configurations, similar to the one in `Catalog` API. Something like this:
```java
/**
* Initialize the client pool with catalog properties and an optional hadoop configuration.
*
* <p>A custom ClientPool implementation must have a no-arg constructor. A Catalog using the ClientPool will first
* use this constructor to create an instance of the pool, and then call this method to initialize the pool.
*
* @param properties catalog properties
* @param hadoopConf hadoop configuration
*/
default void initialize(Map<String, String> properties, Object hadoopConf) {}
```
What do you think?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] flyrain commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
flyrain commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1329517382
Sorry for the late reply. Thanks for the insights from @pvary @lirui-apache. Removing the cache isn't a solution since catalog object is consistently created and thrown away. We can keep the catalog objects, which is a big changer as @pvary mentioned. I'm OK with making the client pool cache pluggable. But the most important thing now is to have a reliable cache. We keep hitting the issues like cache keys are not distinguishable and other weird concurrent issue.
With that, +1 for this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
[GitHub] [iceberg] szehon-ho commented on pull request #6175: Hive: Add UGI to the key in CachedClientPool
Posted by GitBox <gi...@apache.org>.
szehon-ho commented on PR #6175:
URL: https://github.com/apache/iceberg/pull/6175#issuecomment-1363474942
@lirui-apache I think there is consensus, if you want to make a pr for that? I assume that pr will supersede this one? Ill only be able to take a look after holidays though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org