You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2023/08/20 02:50:05 UTC
[GitHub] [pinot] Jackie-Jiang opened a new pull request, #11393: Fix intermediate stage routing for multi-stage engine
Jackie-Jiang opened a new pull request, #11393:
URL: https://github.com/apache/pinot/pull/11393
Currently we use server tag to determine the worker for intermediate stage, and it has the following issues:
- Maintaining the table to tag mapping in `BrokerRoutingManager` is super expensive. It reads all table configs for each instance config change (this can result in millions of read when rolling restart all servers for a large cluster)
- It has a bug that only change from new enabled servers can be picked up. Changes to existing servers cannot be picked up
- When instance assignment config is used, server tag is ignored, so the mapping will be wrong
To address the above issues, instead of using tag to determine servers, directly picking the serving servers from the `InstanceSelector`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] xiangfu0 commented on a diff in pull request #11393: Fix intermediate stage routing for multi-stage engine
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #11393:
URL: https://github.com/apache/pinot/pull/11393#discussion_r1299368947
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/routing/WorkerManager.java:
##########
@@ -265,14 +265,42 @@ private void assignWorkersToIntermediateFragment(PlanFragment fragment, Dispatch
// TODO: actually make assignment strategy decisions for intermediate stages
List<ServerInstance> serverInstances;
Set<String> tableNames = context.getTableNames();
+ Map<String, ServerInstance> enabledServerInstanceMap = _routingManager.getEnabledServerInstanceMap();
if (tableNames.size() == 0) {
// TODO: Short circuit it when no table needs to be scanned
// This could be the case from queries that don't actually fetch values from the tables. In such cases the
// routing need not be tenant aware.
// Eg: SELECT 1 AS one FROM select_having_expression_test_test_having HAVING 1 > 2;
- serverInstances = new ArrayList<>(_routingManager.getEnabledServerInstanceMap().values());
+ serverInstances = new ArrayList<>(enabledServerInstanceMap.values());
} else {
- serverInstances = fetchServersForIntermediateStage(tableNames);
+ Set<String> servers = new HashSet<>();
+ for (String tableName : tableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableName);
+ if (tableType == null) {
+ Set<String> offlineTableServers = _routingManager.getServingInstances(
+ TableNameBuilder.forType(TableType.OFFLINE).tableNameWithType(tableName));
+ if (offlineTableServers != null) {
+ servers.addAll(offlineTableServers);
+ }
+ Set<String> realtimeTableServers = _routingManager.getServingInstances(
+ TableNameBuilder.forType(TableType.REALTIME).tableNameWithType(tableName));
+ if (realtimeTableServers != null) {
+ servers.addAll(realtimeTableServers);
+ }
+ } else {
+ Set<String> tableServers = _routingManager.getServingInstances(tableName);
+ if (tableServers != null) {
+ servers.addAll(tableServers);
+ }
+ }
+ }
+ serverInstances = new ArrayList<>(servers.size());
Review Comment:
actual serverInstances size might be smaller than `servers.size()`
##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java:
##########
@@ -328,7 +319,20 @@ void refreshSegmentStates() {
}
}
- _segmentStates = new SegmentStates(instanceCandidatesMap, unavailableSegments);
+ _segmentStates = new SegmentStates(instanceCandidatesMap, servingInstances, unavailableSegments);
+ }
+
+ private List<SegmentInstanceCandidate> getEnabledCandidates(List<SegmentInstanceCandidate> candidates,
Review Comment:
Add some comments since this method is doing more than the method name or fix the method name?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on pull request #11393: Fix intermediate stage routing for multi-stage engine
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on PR #11393:
URL: https://github.com/apache/pinot/pull/11393#issuecomment-1720329412
this is a thorough fix over https://github.com/apache/pinot/pull/11386
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11393: Fix intermediate stage routing for multi-stage engine
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11393:
URL: https://github.com/apache/pinot/pull/11393#discussion_r1299419337
##########
pinot-query-planner/src/main/java/org/apache/pinot/query/routing/WorkerManager.java:
##########
@@ -265,14 +265,42 @@ private void assignWorkersToIntermediateFragment(PlanFragment fragment, Dispatch
// TODO: actually make assignment strategy decisions for intermediate stages
List<ServerInstance> serverInstances;
Set<String> tableNames = context.getTableNames();
+ Map<String, ServerInstance> enabledServerInstanceMap = _routingManager.getEnabledServerInstanceMap();
if (tableNames.size() == 0) {
// TODO: Short circuit it when no table needs to be scanned
// This could be the case from queries that don't actually fetch values from the tables. In such cases the
// routing need not be tenant aware.
// Eg: SELECT 1 AS one FROM select_having_expression_test_test_having HAVING 1 > 2;
- serverInstances = new ArrayList<>(_routingManager.getEnabledServerInstanceMap().values());
+ serverInstances = new ArrayList<>(enabledServerInstanceMap.values());
} else {
- serverInstances = fetchServersForIntermediateStage(tableNames);
+ Set<String> servers = new HashSet<>();
+ for (String tableName : tableNames) {
+ TableType tableType = TableNameBuilder.getTableTypeFromTableName(tableName);
+ if (tableType == null) {
+ Set<String> offlineTableServers = _routingManager.getServingInstances(
+ TableNameBuilder.forType(TableType.OFFLINE).tableNameWithType(tableName));
+ if (offlineTableServers != null) {
+ servers.addAll(offlineTableServers);
+ }
+ Set<String> realtimeTableServers = _routingManager.getServingInstances(
+ TableNameBuilder.forType(TableType.REALTIME).tableNameWithType(tableName));
+ if (realtimeTableServers != null) {
+ servers.addAll(realtimeTableServers);
+ }
+ } else {
+ Set<String> tableServers = _routingManager.getServingInstances(tableName);
+ if (tableServers != null) {
+ servers.addAll(tableServers);
+ }
+ }
+ }
+ serverInstances = new ArrayList<>(servers.size());
Review Comment:
Correct, but this should give a very good estimation. It doesn't need to be exact
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #11393: Fix intermediate stage routing for multi-stage engine
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11393:
URL: https://github.com/apache/pinot/pull/11393#issuecomment-1685165711
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11393?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11393](https://app.codecov.io/gh/apache/pinot/pull/11393?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (373b649) into [master](https://app.codecov.io/gh/apache/pinot/commit/1783f2a10aae605dc424689be3a7b767556c27b9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (1783f2a) will **decrease** coverage by `46.90%`.
> Report is 2 commits behind head on master.
> The diff coverage is `32.60%`.
```diff
@@ Coverage Diff @@
## master #11393 +/- ##
=============================================
- Coverage 61.44% 14.55% -46.90%
+ Complexity 6515 495 -6020
=============================================
Files 2234 2159 -75
Lines 120174 116954 -3220
Branches 18240 17785 -455
=============================================
- Hits 73838 17018 -56820
- Misses 40911 98525 +57614
+ Partials 5425 1411 -4014
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/11393/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/11393/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/11393/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/11393/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-17](https://app.codecov.io/gh/apache/pinot/pull/11393/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-20](https://app.codecov.io/gh/apache/pinot/pull/11393/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.55% <32.60%> (-46.76%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/11393/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.55% <32.60%> (-46.90%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11393/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11393/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.55% <32.60%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11393?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...ker/routing/instanceselector/InstanceSelector.java](https://app.codecov.io/gh/apache/pinot/pull/11393?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [.../org/apache/pinot/query/routing/WorkerManager.java](https://app.codecov.io/gh/apache/pinot/pull/11393?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcm91dGluZy9Xb3JrZXJNYW5hZ2VyLmphdmE=) | `0.00% <0.00%> (-69.45%)` | :arrow_down: |
| [...che/pinot/broker/routing/BrokerRoutingManager.java](https://app.codecov.io/gh/apache/pinot/pull/11393?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9Ccm9rZXJSb3V0aW5nTWFuYWdlci5qYXZh) | `58.74% <20.00%> (-0.45%)` | :arrow_down: |
| [...broker/routing/instanceselector/SegmentStates.java](https://app.codecov.io/gh/apache/pinot/pull/11393?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL1NlZ21lbnRTdGF0ZXMuamF2YQ==) | `87.50% <50.00%> (-12.50%)` | :arrow_down: |
| [...routing/instanceselector/BaseInstanceSelector.java](https://app.codecov.io/gh/apache/pinot/pull/11393?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0Jhc2VJbnN0YW5jZVNlbGVjdG9yLmphdmE=) | `92.97% <92.85%> (-0.47%)` | :arrow_down: |
... and [1468 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11393/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] kishoreg commented on pull request #11393: Fix intermediate stage routing for multi-stage engine
Posted by "kishoreg (via GitHub)" <gi...@apache.org>.
kishoreg commented on PR #11393:
URL: https://github.com/apache/pinot/pull/11393#issuecomment-1685162471
Are we losing any functionality/feature? What about brokers as intermediate nodes.. also, how can we mark some nodes as shuffle servers in future for expensive queries
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #11393: Fix intermediate stage routing for multi-stage engine
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #11393:
URL: https://github.com/apache/pinot/pull/11393
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #11393: Fix intermediate stage routing for multi-stage engine
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #11393:
URL: https://github.com/apache/pinot/pull/11393#issuecomment-1685175693
> Are we losing any functionality/feature? What about brokers as intermediate nodes.. also, how can we mark some nodes as shuffle servers in future for expensive queries
@kishoreg We are not losing any functionality, and this should be the correct way to handle it. Without this fix, it can only support tables with the basic balanced assignment.
In the future if we want to introduce dedicated shuffle servers, we can add a new tag for it. Initially I tried to solve the problem by maintaining a tag to servers map (no table config read), but realize it won't work for more advanced assignment, e.g. one table referencing the instance partitions from another table. That won't be big change, so we can do it that way when adding the shuffle server feature.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11393: Fix intermediate stage routing for multi-stage engine
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11393:
URL: https://github.com/apache/pinot/pull/11393#discussion_r1299419565
##########
pinot-broker/src/main/java/org/apache/pinot/broker/routing/instanceselector/BaseInstanceSelector.java:
##########
@@ -328,7 +319,20 @@ void refreshSegmentStates() {
}
}
- _segmentStates = new SegmentStates(instanceCandidatesMap, unavailableSegments);
+ _segmentStates = new SegmentStates(instanceCandidatesMap, servingInstances, unavailableSegments);
+ }
+
+ private List<SegmentInstanceCandidate> getEnabledCandidates(List<SegmentInstanceCandidate> candidates,
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org