You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "MarigWeizhi (via GitHub)" <gi...@apache.org> on 2025/07/22 16:42:42 UTC
[PR] [AMORO-3677] Set clients for hive catalog if not exists [amoro]
MarigWeizhi opened a new pull request, #3678:
URL: https://github.com/apache/amoro/pull/3678
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://amoro.apache.org/how-to-contribute/
2. If the PR is related to an issue in https://github.com/apache/amoro/issues, add '[AMORO-XXXX]' in your PR title, e.g., '[AMORO-XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][AMORO-XXXX] Your PR title ...'.
-->
## Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about its use case.
2. If you fix a bug, you can clarify why it is a bug.
3. Use Fix/Resolve/Close #{ISSUE_NUMBER} to link this PR to its related issue
-->
Close #3677.
## Brief change log
<!--
Clearly describe the changes made in modules, classes, methods, etc.
-->
- Set clients for hive catalog if not exists
## How was this patch tested?
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [X] Run test locally before making a pull request
## Documentation
- Does this pull request introduce a new feature? ( no)
- If yes, how is the feature documented? ( not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients of hive catalog in one place [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on PR #3678:
URL: https://github.com/apache/amoro/pull/3678#issuecomment-3139117608
It seems this PR just rolls back the change produced by #3590.
How can we make sure the default client is 20 now?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Fill catalogProperties in creating catalog rather than building [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong merged PR #3678:
URL: https://github.com/apache/amoro/pull/3678
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients for hive catalog if not exists [amoro]
Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on PR #3678:
URL: https://github.com/apache/amoro/pull/3678#issuecomment-3131289164
@MarigWeizhi Thanks for the follow-up, IIUC, we'll merge the property when constructing the catalog [here](https://github.com/apache/amoro/blob/02a2c1fb2c940a84c310e4e336bab0e25d0b20fa/amoro-ams/src/main/java/org/apache/amoro/server/dashboard/controller/CatalogController.java#L416). Could you please help verify that it runsas expected after we remove the logic here, which will update the property.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients for hive catalog if not exists [amoro]
Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on code in PR #3678:
URL: https://github.com/apache/amoro/pull/3678#discussion_r2224845768
##########
amoro-ams/src/main/java/org/apache/amoro/server/catalog/CatalogBuilder.java:
##########
@@ -90,9 +90,11 @@ public static ServerCatalog buildServerCatalog(
case CATALOG_TYPE_HIVE:
String amsUri = getAmsURI(serverConfiguration);
catalogMeta.getCatalogProperties().put(CatalogMetaProperties.AMS_URI, amsUri);
- catalogMeta
- .getCatalogProperties()
- .put(CLIENT_POOL_SIZE, String.valueOf(CLIENT_POOL_SIZE_DEFAULT));
+ if (!catalogMeta.getCatalogProperties().containsKey(CLIENT_POOL_SIZE)) {
Review Comment:
If the logic here overrides the value user set, does that mean we can safely remove this logic here?
##########
amoro-ams/src/main/java/org/apache/amoro/server/catalog/CatalogBuilder.java:
##########
@@ -90,9 +90,11 @@ public static ServerCatalog buildServerCatalog(
case CATALOG_TYPE_HIVE:
String amsUri = getAmsURI(serverConfiguration);
catalogMeta.getCatalogProperties().put(CatalogMetaProperties.AMS_URI, amsUri);
- catalogMeta
- .getCatalogProperties()
- .put(CLIENT_POOL_SIZE, String.valueOf(CLIENT_POOL_SIZE_DEFAULT));
+ if (!catalogMeta.getCatalogProperties().containsKey(CLIENT_POOL_SIZE)) {
Review Comment:
If the logic here overrides the value the user set, does that mean we can safely remove this logic here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients for hive catalog if not exists [amoro]
Posted by "zhongqishang (via GitHub)" <gi...@apache.org>.
zhongqishang commented on code in PR #3678:
URL: https://github.com/apache/amoro/pull/3678#discussion_r2224981358
##########
amoro-ams/src/main/java/org/apache/amoro/server/catalog/CatalogBuilder.java:
##########
@@ -90,9 +90,11 @@ public static ServerCatalog buildServerCatalog(
case CATALOG_TYPE_HIVE:
String amsUri = getAmsURI(serverConfiguration);
catalogMeta.getCatalogProperties().put(CatalogMetaProperties.AMS_URI, amsUri);
- catalogMeta
- .getCatalogProperties()
- .put(CLIENT_POOL_SIZE, String.valueOf(CLIENT_POOL_SIZE_DEFAULT));
Review Comment:
Maybe just `put` -> `putIfAbsent`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients for hive catalog if not exists [amoro]
Posted by "MarigWeizhi (via GitHub)" <gi...@apache.org>.
MarigWeizhi commented on PR #3678:
URL: https://github.com/apache/amoro/pull/3678#issuecomment-3133217621
@klion26 Thank you for your reply.
I think these three lines of code `CatalogBuilder.java (lines 93–95)` is to allow users to see the default value of `clients` in the Web UI even when it is not explicitly set, making it easier to view and modify.
I tested two case after removing these three lines in `CatalogBuilder.java` (lines 93–95):
1. **When creating a catalog without configuring `clients`**:
- The catalog does **not** include the `clients` parameter.
- As a result, the `clients` field is **not visible** in the Web UI.
- Eventually, the default value of `20` is used at runtime, as defined here:
[CachedHiveClientPool.java#L54](https://github.com/apache/amoro/blob/master/amoro-common/src/main/java/org/apache/amoro/hive/CachedHiveClientPool.java#L54)

2. **When creating a catalog with `clients=15`**:
- The catalog correctly uses the configured value of `15`.
- The `clients` parameter is displayed and editable in the Web UI.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients for hive catalog if not exists [amoro]
Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on code in PR #3678:
URL: https://github.com/apache/amoro/pull/3678#discussion_r2224849183
##########
amoro-ams/src/main/java/org/apache/amoro/server/catalog/CatalogBuilder.java:
##########
@@ -90,9 +90,11 @@ public static ServerCatalog buildServerCatalog(
case CATALOG_TYPE_HIVE:
String amsUri = getAmsURI(serverConfiguration);
catalogMeta.getCatalogProperties().put(CatalogMetaProperties.AMS_URI, amsUri);
- catalogMeta
- .getCatalogProperties()
- .put(CLIENT_POOL_SIZE, String.valueOf(CLIENT_POOL_SIZE_DEFAULT));
+ if (!catalogMeta.getCatalogProperties().containsKey(CLIENT_POOL_SIZE)) {
Review Comment:
can we add some tests to cover this logic?
##########
amoro-ams/src/main/java/org/apache/amoro/server/catalog/CatalogBuilder.java:
##########
@@ -90,9 +90,11 @@ public static ServerCatalog buildServerCatalog(
case CATALOG_TYPE_HIVE:
String amsUri = getAmsURI(serverConfiguration);
catalogMeta.getCatalogProperties().put(CatalogMetaProperties.AMS_URI, amsUri);
- catalogMeta
- .getCatalogProperties()
- .put(CLIENT_POOL_SIZE, String.valueOf(CLIENT_POOL_SIZE_DEFAULT));
+ if (!catalogMeta.getCatalogProperties().containsKey(CLIENT_POOL_SIZE)) {
Review Comment:
Could we add some tests to cover this logic?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients for hive catalog if not exists [amoro]
Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on code in PR #3678:
URL: https://github.com/apache/amoro/pull/3678#discussion_r2224845768
##########
amoro-ams/src/main/java/org/apache/amoro/server/catalog/CatalogBuilder.java:
##########
@@ -90,9 +90,11 @@ public static ServerCatalog buildServerCatalog(
case CATALOG_TYPE_HIVE:
String amsUri = getAmsURI(serverConfiguration);
catalogMeta.getCatalogProperties().put(CatalogMetaProperties.AMS_URI, amsUri);
- catalogMeta
- .getCatalogProperties()
- .put(CLIENT_POOL_SIZE, String.valueOf(CLIENT_POOL_SIZE_DEFAULT));
+ if (!catalogMeta.getCatalogProperties().containsKey(CLIENT_POOL_SIZE)) {
Review Comment:
If the logic here overrides the value set, does that mean we can safely remove this logic here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients of hive catalog in one place [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on PR #3678:
URL: https://github.com/apache/amoro/pull/3678#issuecomment-3139819886
Besides, we may add the `clients` properties to the catalog meta in the creating catalog process rather than the build catalog method.
It may resolve the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients of hive catalog in one place [amoro]
Posted by "MarigWeizhi (via GitHub)" <gi...@apache.org>.
MarigWeizhi commented on PR #3678:
URL: https://github.com/apache/amoro/pull/3678#issuecomment-3139555451
> It seems this PR just rolls back the change produced by #3590.
>
> How can we make sure the default client is 20 now?
@zhoujinsong
This PR rolled back part of #3590, to avoid setting default values in two places. The default value remains 20.
If the user has not set the clients, a default value will be set at [CachedHiveClientPool.java#L54](https://github.com/apache/amoro/blob/master/amoro-common/src/main/java/org/apache/amoro/hive/CachedHiveClientPool.java#L54)
and default value is still 20 [CatalogMetaProperties.java#L67](https://github.com/apache/amoro/blob/master/amoro-common/src/main/java/org/apache/amoro/properties/CatalogMetaProperties.java#L67)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients of hive catalog in one place [amoro]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #3678:
URL: https://github.com/apache/amoro/pull/3678#issuecomment-3137065149
## [Codecov](https://app.codecov.io/gh/apache/amoro/pull/3678?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:white_check_mark: All modified and coverable lines are covered by tests.
:white_check_mark: Project coverage is 28.28%. Comparing base ([`02a2c1f`](https://app.codecov.io/gh/apache/amoro/commit/02a2c1fb2c940a84c310e4e336bab0e25d0b20fa?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`857a6b2`](https://app.codecov.io/gh/apache/amoro/commit/857a6b20571e86c95b20d5a085c738d25b23acd9?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #3678 +/- ##
============================================
- Coverage 28.35% 28.28% -0.07%
+ Complexity 3735 3732 -3
============================================
Files 616 616
Lines 49909 49859 -50
Branches 6474 6432 -42
============================================
- Hits 14150 14103 -47
+ Misses 34750 34744 -6
- Partials 1009 1012 +3
```
| [Flag](https://app.codecov.io/gh/apache/amoro/pull/3678/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [core](https://app.codecov.io/gh/apache/amoro/pull/3678/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `28.28% <ø> (-0.07%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/amoro/pull/3678?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients of hive catalog in one place [amoro]
Posted by "MarigWeizhi (via GitHub)" <gi...@apache.org>.
MarigWeizhi commented on PR #3678:
URL: https://github.com/apache/amoro/pull/3678#issuecomment-3136922072
@klion26 Thank you for your suggestion. I think it's a good idea to control the parameters in one place. I have updated the code and title of this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients for hive catalog if not exists [amoro]
Posted by "MarigWeizhi (via GitHub)" <gi...@apache.org>.
MarigWeizhi commented on PR #3678:
URL: https://github.com/apache/amoro/pull/3678#issuecomment-3109451919
@klion26 @zhongqishang Thanks for the review, I updated the code and added a test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients for hive catalog if not exists [amoro]
Posted by "klion26 (via GitHub)" <gi...@apache.org>.
klion26 commented on PR #3678:
URL: https://github.com/apache/amoro/pull/3678#issuecomment-3134557019
@MarigWeizhi Thanks for the verification. Seems the result is as expected?
1. If the property was not set, then Amoro will use the default parameter, and the value was not displayed in the ui is as expected, as the user hasn't set it, but maybe we can log the value we used in the log or somewhere
2. If a user sets the property, then Amoro will use the user-set property
I prefer to handle the configuration in a single place, so that we can maintain it easily. Please let me know what you think about this? thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Set clients of hive catalog in one place [amoro]
Posted by "zhoujinsong (via GitHub)" <gi...@apache.org>.
zhoujinsong commented on PR #3678:
URL: https://github.com/apache/amoro/pull/3678#issuecomment-3139812855
It seems only the MixedHiveCatalog and HudiHiveCatalog are dependent on [CachedHiveClientPool.java#L54](https://github.com/apache/amoro/blob/master/amoro-common/src/main/java/org/apache/amoro/hive/CachedHiveClientPool.java#L54).
I am afraid the Iceberg Hive Catalog does not work with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-3677] Fill catalogProperties in creating catalog rather than building [amoro]
Posted by "MarigWeizhi (via GitHub)" <gi...@apache.org>.
MarigWeizhi commented on PR #3678:
URL: https://github.com/apache/amoro/pull/3678#issuecomment-3143787122
> Besides, we may add the `clients` properties to the catalog meta in the creating catalog process rather than the build catalog method.
>
> It may resolve the issue.
@zhoujinsong
Thank you for your suggestion. I updated the code, I set amsUri and clients when creating catalog . Please take a look when you have time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org