You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@amoro.apache.org by "Aireed (via GitHub)" <gi...@apache.org> on 2024/10/28 06:49:37 UTC
[PR] [AMORO-1865]build a rule of relationship between table and resource group [amoro]
Aireed opened a new pull request, #3300:
URL: https://github.com/apache/amoro/pull/3300
<!--
Thanks for sending a pull request!
Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://amoro.apache.org/how-to-contribute/
2. If the PR is related to an issue in https://github.com/apache/amoro/issues, add '[AMORO-XXXX]' in your PR title, e.g., '[AMORO-XXXX] Your PR title ...'.
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][AMORO-XXXX] Your PR title ...'.
-->
## Why are the changes needed?
<!--
Please clarify why the changes are needed. For instance,
1. If you add a feature, you can talk about its use case.
2. If you fix a bug, you can clarify why it is a bug.
3. Use Fix/Resolve/Close #{ISSUE_NUMBER} to link this PR to its related issue
-->
Close #1865.
## Brief change log
<!--
Clearly describe the changes made in modules, classes, methods, etc.
-->
- Front-end:
- Configure by adding rules in the properties of optimize-group with key 'match.rules'
- Backend:
- 1. When creating a group, rule conflict detection will be performed.
And traverse the table corresponding to the catalog default value of the current optimize-group to see if it matches the current rule and assign the value.
- 2. When updating the optimize-group rules, rule conflict detection will be performed.
- 1. First, the table under the current rule will be judged. If the rule no longer matches after the rule is changed, the default group of the catalog to which the table belongs will be used.
- 2. Traverse the table corresponding to the catalog default value of the current optimize-group to see if it matches the current rule and assign the value.
- 3. when deleting the group
- 1. Configure the tables under the group to the default group of the catalog to which they belong.
When the group of the above table is changed, the tableConfigureChange responsibility chain call of the table will be triggered.
- 4. When synchronizing the external catalog, the group will be automatically matched according to regular rules.
- 5. TableRuntimeRefreshExecutor no longer pays attention to optimize-group when refreshing.
- Special note
We support manually adding the table name to the rules of a group, which is equivalent to the original manual configuration of the group. Because in our practice, there are some situations that require manual specification.
## How was this patch tested?
- [ ] Add some test cases that check the changes thoroughly including negative and positive cases if possible
- [ ] Add screenshots for manual tests if appropriate
- [x] Run test locally before making a pull request
## Documentation
- Does this pull request introduce a new feature? (yes / no)
- If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-1865]build a rule of relationship between table and resource group [amoro]
Posted by "XBaith (via GitHub)" <gi...@apache.org>.
XBaith commented on code in PR #3300:
URL: https://github.com/apache/amoro/pull/3300#discussion_r1821754451
##########
amoro-common/src/main/java/org/apache/amoro/ServerTableIdentifier.java:
##########
@@ -123,6 +123,10 @@ public String toString() {
return String.format("%s.%s.%s(tableId=%d)", catalog, database, tableName, id);
}
+ public String getTableFullName() {
Review Comment:
Okay, It not `org.apache.amoro.table.TableIdentifier`. I perfer use the following method, WDYT?
```
public org.apache.amoro.table.TableIdentifier getTableIdentifier() {
return org.apache.amoro.table.TableIdentifier.of(catalog, database, tableName);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-1865]build a rule of relationship between table and resource group [amoro]
Posted by "Aireed (via GitHub)" <gi...@apache.org>.
Aireed closed pull request #3300: [AMORO-1865]build a rule of relationship between table and resource group
URL: https://github.com/apache/amoro/pull/3300
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-1865]build a rule of relationship between table and resource group [amoro]
Posted by "XBaith (via GitHub)" <gi...@apache.org>.
XBaith commented on code in PR #3300:
URL: https://github.com/apache/amoro/pull/3300#discussion_r1820275580
##########
amoro-common/src/main/java/org/apache/amoro/ServerTableIdentifier.java:
##########
@@ -123,6 +123,10 @@ public String toString() {
return String.format("%s.%s.%s(tableId=%d)", catalog, database, tableName, id);
}
+ public String getTableFullName() {
Review Comment:
Is this method nessessary? How about invoke `getIdentifier().toString()` to get full name?
##########
amoro-common/src/main/java/org/apache/amoro/resource/ResourceGroup.java:
##########
@@ -52,6 +77,81 @@ public String getContainer() {
return container;
}
+ public String getOptimizeGroupRule() {
+ return this.properties.getOrDefault(OPTIMIZE_GROUP_RULE_MATCH_KEY, null);
+ }
+
+ public boolean match(TableIdentifier tableIdentifier) {
+ return match(this.properties, tableIdentifier);
+ }
+
+ /**
+ * if only numbers, letters, and underscores are contained, then the rule is not regexp and the
+ * rule will overwrite the regexp rule
+ */
+ private boolean noRegExp(String rule) {
+ String fmtString = rule.replaceAll(getSpaceSeparator(rule), "");
+ if (fmtString.replaceAll("[a-zA-Z0-9_]", "").length() == 0) {
+ return true;
+ }
+ return false;
+ }
+
+ public List<String> getFullMatchNameInRules() {
+ return Arrays.stream(properties.getOrDefault(OPTIMIZE_GROUP_RULE_MATCH_KEY, "").split(","))
+ .filter(item -> noRegExp(item))
+ .collect(Collectors.toList());
+ }
+
+ /**
+ * Full match means we manually assign the table to this group.
+ *
+ * @param tableIdentifier
+ * @return
+ */
+ public boolean fullMatch(TableIdentifier tableIdentifier) {
+ final String tableName =
+ String.format(
+ "%s.%s.%s",
+ tableIdentifier.getCatalog(),
+ tableIdentifier.getDatabase(),
+ tableIdentifier.getTableName());
+ return Arrays.stream(properties.getOrDefault(OPTIMIZE_GROUP_RULE_MATCH_KEY, "").split(","))
+ .filter(item -> item.contains(tableName))
+ .findAny()
+ .isPresent();
+ }
+
+ private boolean match(Map<String, String> properties, TableIdentifier tableIdentifier) {
+ String rulePropertyKey = OPTIMIZE_GROUP_RULE_MATCH_KEY;
+ if (properties != null && properties.containsKey(rulePropertyKey)) {
+ List<String> matchRules =
+ Arrays.stream(properties.getOrDefault(rulePropertyKey, "").split(","))
+ .collect(Collectors.toList());
+
+ final String tableName =
+ String.format(
Review Comment:
tableIdentifier.toString()
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-1865]build a rule of relationship between table and resource group [amoro]
Posted by "baiyangtx (via GitHub)" <gi...@apache.org>.
baiyangtx commented on PR #3300:
URL: https://github.com/apache/amoro/pull/3300#issuecomment-2451521141
This implementation method will be contrary to the current advanced issue of splitting the rest service and the scheduling service. It makes the update of the group have to occur at the master node.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-1865]build a rule of relationship between table and resource group [amoro]
Posted by "Aireed (via GitHub)" <gi...@apache.org>.
Aireed commented on code in PR #3300:
URL: https://github.com/apache/amoro/pull/3300#discussion_r1821733946
##########
amoro-common/src/main/java/org/apache/amoro/ServerTableIdentifier.java:
##########
@@ -123,6 +123,10 @@ public String toString() {
return String.format("%s.%s.%s(tableId=%d)", catalog, database, tableName, id);
}
+ public String getTableFullName() {
Review Comment:
> Is this method nessessary? How about invoke `getIdentifier().toString()` to get full name?
`getIdentifier().toString()` returns `TableIdentifier[catalog.db.table]`, So it's not suitable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-1865]build a rule of relationship between table and resource group [amoro]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #3300:
URL: https://github.com/apache/amoro/pull/3300#issuecomment-2510317813
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@amoro.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [AMORO-1865]build a rule of relationship between table and resource group [amoro]
Posted by "Aireed (via GitHub)" <gi...@apache.org>.
Aireed commented on PR #3300:
URL: https://github.com/apache/amoro/pull/3300#issuecomment-2515989066
we may reopen it after #2810 finished
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@amoro.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org