You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "rhodo (via GitHub)" <gi...@apache.org> on 2025/06/04 20:58:06 UTC
[PR] Support set default tier configurations from cluster server config [pinot]
rhodo opened a new pull request, #15998:
URL: https://github.com/apache/pinot/pull/15998
### Instructions:
Sometimes we want to define system-wide defaults for tiered storage—for example, using a specific S3 client by default. To follow best practices, it’s ideal to set such defaults explicitly in the cluster config.
Currently, this isn’t feasible because tier configs require the tier name to be part of the property key, e.g.,
pinot.server.instance.tierConfigs.tierA.someKey = someValue.
This makes it difficult to apply default settings across all clusters, since each requires a named prefix.
This PR introduces a new config prefix: pinot.server.instance.defaultTierConfigs.With this change, default properties such as
pinot.server.instance.defaultTierConfigs.useSpecificS3Client=true can be applied cluster-wide, and individual tier configs/ can still override these defaults as needed.
### Test
Tested using the relevant quick start. Verified that default properties added to the cluster config are correctly picked up and applied by the appropriate components.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Support set default tier configurations from cluster server config [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince merged PR #15998:
URL: https://github.com/apache/pinot/pull/15998
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Support set default tier configurations from cluster server config [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #15998:
URL: https://github.com/apache/pinot/pull/15998#discussion_r2138769286
##########
pinot-server/src/main/java/org/apache/pinot/server/starter/helix/HelixInstanceDataManagerConfig.java:
##########
@@ -138,17 +140,28 @@ public HelixInstanceDataManagerConfig(PinotConfiguration serverConfig)
_upsertConfig = serverConfig.subset(UPSERT_CONFIG_PREFIX);
_dedupConfig = serverConfig.subset(DEDUP_CONFIG_PREFIX);
+ // Load default tier configurations
+ // Properties are prefixed with pinot.server.instance.defaultTierConfigs.
+ // e.g. pinot.server.instance.defaultTierConfigs.myKey = myValue
+ // The resulting defaultTierProperties map will be { "myKey": "myValue" }
+ Map<String, String> defaultTierProperties = new HashMap<>();
+ serverConfig.subset(DEFAULT_TIER_CONFIG_KEY_PREFIX)
+ .toMap()
+ .forEach((key, value) -> defaultTierProperties.put(key, String.valueOf(value)));
+ Map<String, String> unmodifiableDefaultTierProperties = Collections.unmodifiableMap(defaultTierProperties);
+
+ _tierConfigs = new DefaultedMap<>(unmodifiableDefaultTierProperties);
Review Comment:
can add a comment that if a tier is not found in this map, then this map of default configs will be used.
and better add some tests for this new behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Support set default tier configurations from cluster server config [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #15998:
URL: https://github.com/apache/pinot/pull/15998#issuecomment-2941659555
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15998?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 56.43%. Comparing base [(`1a476de`)](https://app.codecov.io/gh/apache/pinot/commit/1a476dea72ebf1c15e14ded28ae60c385f71b2c3?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`27b3e5a`)](https://app.codecov.io/gh/apache/pinot/commit/27b3e5a99dc4d8ed2c89116ddbd68a0df26b44a8?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 200 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (1a476de) and HEAD (27b3e5a). Click for more details.
>
> <details><summary>HEAD has 68 uploads less than BASE</summary>
>
>| Flag | BASE (1a476de) | HEAD (27b3e5a) |
>|------|------|------|
>|java-21|10|1|
>|unittests|6|1|
>|unittests1|3|1|
>|skip-bytebuffers-true|4|0|
>|temurin|15|1|
>|unittests2|3|0|
>|skip-bytebuffers-false|8|0|
>|java-11|5|0|
>|integration|9|0|
>|integration2|3|0|
>|integration1|3|0|
>|custom-integration1|3|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #15998 +/- ##
============================================
- Coverage 62.90% 56.43% -6.48%
+ Complexity 1386 690 -696
============================================
Files 2867 2272 -595
Lines 163354 124159 -39195
Branches 24952 19788 -5164
============================================
- Hits 102755 70063 -32692
+ Misses 52847 48291 -4556
+ Partials 7752 5805 -1947
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/15998/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/15998/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/15998/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/15998/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/15998/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/15998/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/15998/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.43% <ø> (-6.40%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/15998/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/15998/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/15998/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.43% <ø> (-6.48%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/15998/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.43% <ø> (-6.47%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/15998/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.43% <ø> (+0.60%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/15998/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/15998?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Support set default tier configurations from cluster server config [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #15998:
URL: https://github.com/apache/pinot/pull/15998#issuecomment-2957225040
@klsince Can you help take a look?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org