You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "krishan1390 (via GitHub)" <gi...@apache.org> on 2025/03/11 04:59:07 UTC
[PR] Improve validations for minion instance tag checks [pinot]
krishan1390 opened a new pull request, #15239:
URL: https://github.com/apache/pinot/pull/15239
https://github.com/apache/pinot/pull/14228 added validation to check if minion instance exists with configured tag during table creation / update
This is useful to check when new table / task is created or when a task's instance tag is modified.
Currently this check also runs when any other part of table config is updated. But since minions are designed to scale down, its possible minion instances don't exist at all times. So we don't need to block all table config updates. This PR makes changes to faciliate this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Improve validations for minion instance tag checks [pinot]
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on code in PR #15239:
URL: https://github.com/apache/pinot/pull/15239#discussion_r1990250404
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1881,22 +1881,47 @@ void validateTableTenantConfig(TableConfig tableConfig) {
}
}
+ /**
+ * Validates if a minion instance is configured for each task type in the table config.
+ * This is useful to verify when a new table (or task) is created to not miss out on the task execution.
+ * It is also useful when a task's minion instance type or schedule is updated.
+ * <p>
+ * However, given minion instances are designed to be scaled down when not needed,
+ * we don't require the validation when any other part of the table config is updated.
+ */
@VisibleForTesting
- void validateTableTaskMinionInstanceTagConfig(TableConfig tableConfig) {
+ void validateTableTaskMinionInstanceTagConfig(TableConfig newTableConfig) {
+
+ TableConfig currentTableConfig = getTableConfig(newTableConfig.getTableName());
List<InstanceConfig> allMinionWorkerInstanceConfigs = getAllMinionInstanceConfigs();
//extract all minionInstanceTags from allMinionWorkerInstanceConfigs
Set<String> minionInstanceTagSet = allMinionWorkerInstanceConfigs.stream().map(InstanceConfig::getTags)
.collect(HashSet::new, Set::addAll, Set::addAll);
- if (tableConfig.getTaskConfig() != null && tableConfig.getTaskConfig().getTaskTypeConfigsMap() != null) {
- tableConfig.getTaskConfig().getTaskTypeConfigsMap().forEach((taskType, taskTypeConfig) -> {
+ if (newTableConfig.getTaskConfig() != null && newTableConfig.getTaskConfig().getTaskTypeConfigsMap() != null) {
+ newTableConfig.getTaskConfig().getTaskTypeConfigsMap().forEach((taskType, taskTypeConfig) -> {
+
String taskInstanceTag = taskTypeConfig.getOrDefault(PinotTaskManager.MINION_INSTANCE_TAG_CONFIG,
CommonConstants.Helix.UNTAGGED_MINION_INSTANCE);
+ String schedule = taskTypeConfig.getOrDefault(PinotTaskManager.SCHEDULE_KEY, "");
+
+ if (currentTableConfig != null && currentTableConfig.getTaskConfig() != null
+ && currentTableConfig.getTaskConfig().isTaskTypeEnabled(taskType)) {
+ Map<String, String> currentTaskConfig = currentTableConfig.getTaskConfig().getConfigsForTaskType(taskType);
+ String currentTaskInstanceTag = currentTaskConfig.getOrDefault(PinotTaskManager.MINION_INSTANCE_TAG_CONFIG,
+ CommonConstants.Helix.UNTAGGED_MINION_INSTANCE);
+ String currentSchedule = currentTaskConfig.getOrDefault(PinotTaskManager.SCHEDULE_KEY, "");
+ if (currentTaskInstanceTag.equals(taskInstanceTag) && currentSchedule.equals(schedule)) {
Review Comment:
+1 we can just check if schedule is present in the new table config or not, if yes, then check for instance-tag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Improve validations for minion instance tag checks [pinot]
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish merged PR #15239:
URL: https://github.com/apache/pinot/pull/15239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Improve validations for minion instance tag checks [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #15239:
URL: https://github.com/apache/pinot/pull/15239#discussion_r1990594703
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1881,22 +1881,47 @@ void validateTableTenantConfig(TableConfig tableConfig) {
}
}
+ /**
+ * Validates if a minion instance is configured for each task type in the table config.
+ * This is useful to verify when a new table (or task) is created to not miss out on the task execution.
+ * It is also useful when a task's minion instance type or schedule is updated.
+ * <p>
+ * However, given minion instances are designed to be scaled down when not needed,
+ * we don't require the validation when any other part of the table config is updated.
+ */
@VisibleForTesting
- void validateTableTaskMinionInstanceTagConfig(TableConfig tableConfig) {
+ void validateTableTaskMinionInstanceTagConfig(TableConfig newTableConfig) {
+
+ TableConfig currentTableConfig = getTableConfig(newTableConfig.getTableName());
List<InstanceConfig> allMinionWorkerInstanceConfigs = getAllMinionInstanceConfigs();
//extract all minionInstanceTags from allMinionWorkerInstanceConfigs
Set<String> minionInstanceTagSet = allMinionWorkerInstanceConfigs.stream().map(InstanceConfig::getTags)
.collect(HashSet::new, Set::addAll, Set::addAll);
- if (tableConfig.getTaskConfig() != null && tableConfig.getTaskConfig().getTaskTypeConfigsMap() != null) {
- tableConfig.getTaskConfig().getTaskTypeConfigsMap().forEach((taskType, taskTypeConfig) -> {
+ if (newTableConfig.getTaskConfig() != null && newTableConfig.getTaskConfig().getTaskTypeConfigsMap() != null) {
+ newTableConfig.getTaskConfig().getTaskTypeConfigsMap().forEach((taskType, taskTypeConfig) -> {
+
String taskInstanceTag = taskTypeConfig.getOrDefault(PinotTaskManager.MINION_INSTANCE_TAG_CONFIG,
CommonConstants.Helix.UNTAGGED_MINION_INSTANCE);
+ String schedule = taskTypeConfig.getOrDefault(PinotTaskManager.SCHEDULE_KEY, "");
+
+ if (currentTableConfig != null && currentTableConfig.getTaskConfig() != null
+ && currentTableConfig.getTaskConfig().isTaskTypeEnabled(taskType)) {
+ Map<String, String> currentTaskConfig = currentTableConfig.getTaskConfig().getConfigsForTaskType(taskType);
+ String currentTaskInstanceTag = currentTaskConfig.getOrDefault(PinotTaskManager.MINION_INSTANCE_TAG_CONFIG,
+ CommonConstants.Helix.UNTAGGED_MINION_INSTANCE);
+ String currentSchedule = currentTaskConfig.getOrDefault(PinotTaskManager.SCHEDULE_KEY, "");
+ if (currentTaskInstanceTag.equals(taskInstanceTag) && currentSchedule.equals(schedule)) {
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Improve validations for minion instance tag checks [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #15239:
URL: https://github.com/apache/pinot/pull/15239#issuecomment-2713474595
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15239?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
All modified and coverable lines are covered by tests :white_check_mark:
> Project coverage is 56.03%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`134958b`)](https://app.codecov.io/gh/apache/pinot/commit/134958b0db4a6f0d6df8832142f821de1a4a4c79?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 1832 commits behind head on master.
> :exclamation: There is a different number of reports uploaded between BASE (59551e4) and HEAD (134958b). Click for more details.
>
> <details><summary>HEAD has 48 uploads less than BASE</summary>
>
>| Flag | BASE (59551e4) | HEAD (134958b) |
>|------|------|------|
>|integration|7|0|
>|integration2|3|0|
>|temurin|12|2|
>|java-21|7|2|
>|skip-bytebuffers-true|3|1|
>|skip-bytebuffers-false|7|1|
>|unittests|5|2|
>|java-11|5|0|
>|unittests2|3|0|
>|integration1|2|0|
>|custom-integration1|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #15239 +/- ##
============================================
- Coverage 61.75% 56.03% -5.72%
- Complexity 207 685 +478
============================================
Files 2436 2165 -271
Lines 133233 116809 -16424
Branches 20636 18695 -1941
============================================
- Hits 82274 65452 -16822
- Misses 44911 45981 +1070
+ Partials 6048 5376 -672
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/15239/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/15239/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/15239/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/15239/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/15239/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/15239/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/15239/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.03% <ø> (-5.60%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/15239/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.00% <ø> (-5.75%)` | :arrow_down: |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/15239/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.01% <ø> (+28.28%)` | :arrow_up: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/15239/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.03% <ø> (-5.72%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/15239/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.03% <ø> (-5.72%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/15239/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `56.03% <ø> (+9.14%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/15239/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/15239?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary>🚀 New features to boost your workflow: </summary>
- ❄ [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- 📦 [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Improve validations for minion instance tag checks [pinot]
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on code in PR #15239:
URL: https://github.com/apache/pinot/pull/15239#discussion_r1988717894
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1881,22 +1881,47 @@ void validateTableTenantConfig(TableConfig tableConfig) {
}
}
+ /**
+ * Validates if a minion instance is configured for each task type in the table config.
+ * This is useful to verify when a new table (or task) is created to not miss out on the task execution.
+ * It is also useful when a task's minion instance type or schedule is updated.
+ * <p>
+ * However, given minion instances are designed to be scaled down when not needed,
+ * we don't require the validation when any other part of the table config is updated.
+ */
@VisibleForTesting
- void validateTableTaskMinionInstanceTagConfig(TableConfig tableConfig) {
+ void validateTableTaskMinionInstanceTagConfig(TableConfig newTableConfig) {
+
+ TableConfig currentTableConfig = getTableConfig(newTableConfig.getTableName());
List<InstanceConfig> allMinionWorkerInstanceConfigs = getAllMinionInstanceConfigs();
//extract all minionInstanceTags from allMinionWorkerInstanceConfigs
Set<String> minionInstanceTagSet = allMinionWorkerInstanceConfigs.stream().map(InstanceConfig::getTags)
.collect(HashSet::new, Set::addAll, Set::addAll);
- if (tableConfig.getTaskConfig() != null && tableConfig.getTaskConfig().getTaskTypeConfigsMap() != null) {
- tableConfig.getTaskConfig().getTaskTypeConfigsMap().forEach((taskType, taskTypeConfig) -> {
+ if (newTableConfig.getTaskConfig() != null && newTableConfig.getTaskConfig().getTaskTypeConfigsMap() != null) {
+ newTableConfig.getTaskConfig().getTaskTypeConfigsMap().forEach((taskType, taskTypeConfig) -> {
+
String taskInstanceTag = taskTypeConfig.getOrDefault(PinotTaskManager.MINION_INSTANCE_TAG_CONFIG,
CommonConstants.Helix.UNTAGGED_MINION_INSTANCE);
+ String schedule = taskTypeConfig.getOrDefault(PinotTaskManager.SCHEDULE_KEY, "");
+
+ if (currentTableConfig != null && currentTableConfig.getTaskConfig() != null
+ && currentTableConfig.getTaskConfig().isTaskTypeEnabled(taskType)) {
+ Map<String, String> currentTaskConfig = currentTableConfig.getTaskConfig().getConfigsForTaskType(taskType);
+ String currentTaskInstanceTag = currentTaskConfig.getOrDefault(PinotTaskManager.MINION_INSTANCE_TAG_CONFIG,
+ CommonConstants.Helix.UNTAGGED_MINION_INSTANCE);
+ String currentSchedule = currentTaskConfig.getOrDefault(PinotTaskManager.SCHEDULE_KEY, "");
+ if (currentTaskInstanceTag.equals(taskInstanceTag) && currentSchedule.equals(schedule)) {
Review Comment:
Can we not just check if schedule param is present or not in newTableConfig ? Validate if the param is present else no-op.
We want to make sure that prior to scheduling, we need to validate. Otherwise its not required to keep minion instances around.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Improve validations for minion instance tag checks [pinot]
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on code in PR #15239:
URL: https://github.com/apache/pinot/pull/15239#discussion_r1988926177
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1881,22 +1881,47 @@ void validateTableTenantConfig(TableConfig tableConfig) {
}
}
+ /**
+ * Validates if a minion instance is configured for each task type in the table config.
+ * This is useful to verify when a new table (or task) is created to not miss out on the task execution.
+ * It is also useful when a task's minion instance type or schedule is updated.
+ * <p>
+ * However, given minion instances are designed to be scaled down when not needed,
+ * we don't require the validation when any other part of the table config is updated.
+ */
@VisibleForTesting
- void validateTableTaskMinionInstanceTagConfig(TableConfig tableConfig) {
+ void validateTableTaskMinionInstanceTagConfig(TableConfig newTableConfig) {
+
+ TableConfig currentTableConfig = getTableConfig(newTableConfig.getTableName());
List<InstanceConfig> allMinionWorkerInstanceConfigs = getAllMinionInstanceConfigs();
//extract all minionInstanceTags from allMinionWorkerInstanceConfigs
Set<String> minionInstanceTagSet = allMinionWorkerInstanceConfigs.stream().map(InstanceConfig::getTags)
.collect(HashSet::new, Set::addAll, Set::addAll);
- if (tableConfig.getTaskConfig() != null && tableConfig.getTaskConfig().getTaskTypeConfigsMap() != null) {
- tableConfig.getTaskConfig().getTaskTypeConfigsMap().forEach((taskType, taskTypeConfig) -> {
+ if (newTableConfig.getTaskConfig() != null && newTableConfig.getTaskConfig().getTaskTypeConfigsMap() != null) {
+ newTableConfig.getTaskConfig().getTaskTypeConfigsMap().forEach((taskType, taskTypeConfig) -> {
+
String taskInstanceTag = taskTypeConfig.getOrDefault(PinotTaskManager.MINION_INSTANCE_TAG_CONFIG,
CommonConstants.Helix.UNTAGGED_MINION_INSTANCE);
+ String schedule = taskTypeConfig.getOrDefault(PinotTaskManager.SCHEDULE_KEY, "");
+
+ if (currentTableConfig != null && currentTableConfig.getTaskConfig() != null
+ && currentTableConfig.getTaskConfig().isTaskTypeEnabled(taskType)) {
+ Map<String, String> currentTaskConfig = currentTableConfig.getTaskConfig().getConfigsForTaskType(taskType);
+ String currentTaskInstanceTag = currentTaskConfig.getOrDefault(PinotTaskManager.MINION_INSTANCE_TAG_CONFIG,
+ CommonConstants.Helix.UNTAGGED_MINION_INSTANCE);
+ String currentSchedule = currentTaskConfig.getOrDefault(PinotTaskManager.SCHEDULE_KEY, "");
+ if (currentTaskInstanceTag.equals(taskInstanceTag) && currentSchedule.equals(schedule)) {
Review Comment:
I think all we want to do is enforce the validation when you intend to schedule (no matter when it gets scheduled) which is when you add the schedule param. If schedule param is not present, its equivalent to disabling the schedule where we dont need the validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Improve validations for minion instance tag checks [pinot]
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on PR #15239:
URL: https://github.com/apache/pinot/pull/15239#issuecomment-2713198194
cc @tibrewalpratik17 for awareness.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Improve validations for minion instance tag checks [pinot]
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on code in PR #15239:
URL: https://github.com/apache/pinot/pull/15239#discussion_r1988828930
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1881,22 +1881,47 @@ void validateTableTenantConfig(TableConfig tableConfig) {
}
}
+ /**
+ * Validates if a minion instance is configured for each task type in the table config.
+ * This is useful to verify when a new table (or task) is created to not miss out on the task execution.
+ * It is also useful when a task's minion instance type or schedule is updated.
+ * <p>
+ * However, given minion instances are designed to be scaled down when not needed,
+ * we don't require the validation when any other part of the table config is updated.
+ */
@VisibleForTesting
- void validateTableTaskMinionInstanceTagConfig(TableConfig tableConfig) {
+ void validateTableTaskMinionInstanceTagConfig(TableConfig newTableConfig) {
+
+ TableConfig currentTableConfig = getTableConfig(newTableConfig.getTableName());
List<InstanceConfig> allMinionWorkerInstanceConfigs = getAllMinionInstanceConfigs();
//extract all minionInstanceTags from allMinionWorkerInstanceConfigs
Set<String> minionInstanceTagSet = allMinionWorkerInstanceConfigs.stream().map(InstanceConfig::getTags)
.collect(HashSet::new, Set::addAll, Set::addAll);
- if (tableConfig.getTaskConfig() != null && tableConfig.getTaskConfig().getTaskTypeConfigsMap() != null) {
- tableConfig.getTaskConfig().getTaskTypeConfigsMap().forEach((taskType, taskTypeConfig) -> {
+ if (newTableConfig.getTaskConfig() != null && newTableConfig.getTaskConfig().getTaskTypeConfigsMap() != null) {
+ newTableConfig.getTaskConfig().getTaskTypeConfigsMap().forEach((taskType, taskTypeConfig) -> {
+
String taskInstanceTag = taskTypeConfig.getOrDefault(PinotTaskManager.MINION_INSTANCE_TAG_CONFIG,
CommonConstants.Helix.UNTAGGED_MINION_INSTANCE);
+ String schedule = taskTypeConfig.getOrDefault(PinotTaskManager.SCHEDULE_KEY, "");
+
+ if (currentTableConfig != null && currentTableConfig.getTaskConfig() != null
+ && currentTableConfig.getTaskConfig().isTaskTypeEnabled(taskType)) {
+ Map<String, String> currentTaskConfig = currentTableConfig.getTaskConfig().getConfigsForTaskType(taskType);
+ String currentTaskInstanceTag = currentTaskConfig.getOrDefault(PinotTaskManager.MINION_INSTANCE_TAG_CONFIG,
+ CommonConstants.Helix.UNTAGGED_MINION_INSTANCE);
+ String currentSchedule = currentTaskConfig.getOrDefault(PinotTaskManager.SCHEDULE_KEY, "");
+ if (currentTaskInstanceTag.equals(taskInstanceTag) && currentSchedule.equals(schedule)) {
Review Comment:
When the schedule param is removed, this would trigger the validation and fail the table update right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Improve validations for minion instance tag checks [pinot]
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on PR #15239:
URL: https://github.com/apache/pinot/pull/15239#issuecomment-2718402239
@krishan1390 - Could you update the PR description as well based on the updated diff.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Improve validations for minion instance tag checks [pinot]
Posted by "krishan1390 (via GitHub)" <gi...@apache.org>.
krishan1390 commented on code in PR #15239:
URL: https://github.com/apache/pinot/pull/15239#discussion_r1988865268
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -1881,22 +1881,47 @@ void validateTableTenantConfig(TableConfig tableConfig) {
}
}
+ /**
+ * Validates if a minion instance is configured for each task type in the table config.
+ * This is useful to verify when a new table (or task) is created to not miss out on the task execution.
+ * It is also useful when a task's minion instance type or schedule is updated.
+ * <p>
+ * However, given minion instances are designed to be scaled down when not needed,
+ * we don't require the validation when any other part of the table config is updated.
+ */
@VisibleForTesting
- void validateTableTaskMinionInstanceTagConfig(TableConfig tableConfig) {
+ void validateTableTaskMinionInstanceTagConfig(TableConfig newTableConfig) {
+
+ TableConfig currentTableConfig = getTableConfig(newTableConfig.getTableName());
List<InstanceConfig> allMinionWorkerInstanceConfigs = getAllMinionInstanceConfigs();
//extract all minionInstanceTags from allMinionWorkerInstanceConfigs
Set<String> minionInstanceTagSet = allMinionWorkerInstanceConfigs.stream().map(InstanceConfig::getTags)
.collect(HashSet::new, Set::addAll, Set::addAll);
- if (tableConfig.getTaskConfig() != null && tableConfig.getTaskConfig().getTaskTypeConfigsMap() != null) {
- tableConfig.getTaskConfig().getTaskTypeConfigsMap().forEach((taskType, taskTypeConfig) -> {
+ if (newTableConfig.getTaskConfig() != null && newTableConfig.getTaskConfig().getTaskTypeConfigsMap() != null) {
+ newTableConfig.getTaskConfig().getTaskTypeConfigsMap().forEach((taskType, taskTypeConfig) -> {
+
String taskInstanceTag = taskTypeConfig.getOrDefault(PinotTaskManager.MINION_INSTANCE_TAG_CONFIG,
CommonConstants.Helix.UNTAGGED_MINION_INSTANCE);
+ String schedule = taskTypeConfig.getOrDefault(PinotTaskManager.SCHEDULE_KEY, "");
+
+ if (currentTableConfig != null && currentTableConfig.getTaskConfig() != null
+ && currentTableConfig.getTaskConfig().isTaskTypeEnabled(taskType)) {
+ Map<String, String> currentTaskConfig = currentTableConfig.getTaskConfig().getConfigsForTaskType(taskType);
+ String currentTaskInstanceTag = currentTaskConfig.getOrDefault(PinotTaskManager.MINION_INSTANCE_TAG_CONFIG,
+ CommonConstants.Helix.UNTAGGED_MINION_INSTANCE);
+ String currentSchedule = currentTaskConfig.getOrDefault(PinotTaskManager.SCHEDULE_KEY, "");
+ if (currentTaskInstanceTag.equals(taskInstanceTag) && currentSchedule.equals(schedule)) {
Review Comment:
If schedule param is configured but task isn't supposed to be scheduled right now (assume task is going to get scheduled to run once a week), do we want to disallow table config updates if minion host isn't available ?
or do you think the current changes are more appropriate given minions can scale up and down on demand ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org