You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "walterddr (via GitHub)" <gi...@apache.org> on 2023/08/01 20:51:22 UTC
[GitHub] [pinot] walterddr opened a new pull request, #11241: [multistage] support multi-tenant lookup for v2 engine
walterddr opened a new pull request, #11241:
URL: https://github.com/apache/pinot/pull/11241
this fixes #11237
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #11241:
URL: https://github.com/apache/pinot/pull/11241#discussion_r1283521975
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -214,31 +215,19 @@ private String getMultiStageQueryResponse(String query, String queryOptions, Htt
// tenants can be completely disjoint. The leaf stages which access segments will be processed on the respective
// server tenants for each table. The intermediate stages can be processed in either or all of the server tenants
// belonging to the tables.
- brokerTenant = getCommonBrokerTenant(tableConfigList);
- if (brokerTenant == null) {
+ brokerTenants = getCommonBrokerTenants(tableConfigList);
+ if (brokerTenants.isEmpty()) {
return QueryException.getException(QueryException.BROKER_REQUEST_SEND_ERROR, new Exception(
String.format("Unable to dispatch multistage query with multiple tables : %s " + "on different tenant",
tableNames))).toString();
}
} else {
// TODO fail these queries going forward. Added this logic to take care of tautologies like BETWEEN 0 and -1.
- List<String> allBrokerList = new ArrayList<>(_pinotHelixResourceManager.getAllBrokerTenantNames());
- brokerTenant = allBrokerList.get(RANDOM.nextInt(allBrokerList.size()));
+ brokerTenants = _pinotHelixResourceManager.getAllBrokerTenantNames();
LOGGER.error("Unable to find table name from SQL {} thus dispatching to random broker.", query);
}
- List<String> instanceIds = new ArrayList<>(_pinotHelixResourceManager.getAllInstancesForBrokerTenant(brokerTenant));
-
- if (instanceIds.isEmpty()) {
- return QueryException.BROKER_RESOURCE_MISSING_ERROR.toString();
- }
-
- instanceIds.retainAll(_pinotHelixResourceManager.getOnlineInstanceList());
- if (instanceIds.isEmpty()) {
- return QueryException.BROKER_INSTANCE_MISSING_ERROR.toString();
- }
-
- // Send query to a random broker.
- String instanceId = instanceIds.get(RANDOM.nextInt(instanceIds.size()));
+ List<String> instanceIds = findCommonBrokerInstance(brokerTenants);
+ String instanceId = selectInstanceId(instanceIds);
Review Comment:
no it won't. the brokerTenant (renamed to `brokerTenantUnion` in new code) will only contain the unioned tag from table A and B.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tibrewalpratik17 commented on a diff in pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on code in PR #11241:
URL: https://github.com/apache/pinot/pull/11241#discussion_r1283316296
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -214,31 +215,19 @@ private String getMultiStageQueryResponse(String query, String queryOptions, Htt
// tenants can be completely disjoint. The leaf stages which access segments will be processed on the respective
// server tenants for each table. The intermediate stages can be processed in either or all of the server tenants
// belonging to the tables.
- brokerTenant = getCommonBrokerTenant(tableConfigList);
- if (brokerTenant == null) {
+ brokerTenants = getCommonBrokerTenants(tableConfigList);
+ if (brokerTenants.isEmpty()) {
return QueryException.getException(QueryException.BROKER_REQUEST_SEND_ERROR, new Exception(
String.format("Unable to dispatch multistage query with multiple tables : %s " + "on different tenant",
tableNames))).toString();
}
} else {
// TODO fail these queries going forward. Added this logic to take care of tautologies like BETWEEN 0 and -1.
- List<String> allBrokerList = new ArrayList<>(_pinotHelixResourceManager.getAllBrokerTenantNames());
- brokerTenant = allBrokerList.get(RANDOM.nextInt(allBrokerList.size()));
+ brokerTenants = _pinotHelixResourceManager.getAllBrokerTenantNames();
LOGGER.error("Unable to find table name from SQL {} thus dispatching to random broker.", query);
}
- List<String> instanceIds = new ArrayList<>(_pinotHelixResourceManager.getAllInstancesForBrokerTenant(brokerTenant));
-
- if (instanceIds.isEmpty()) {
- return QueryException.BROKER_RESOURCE_MISSING_ERROR.toString();
- }
-
- instanceIds.retainAll(_pinotHelixResourceManager.getOnlineInstanceList());
- if (instanceIds.isEmpty()) {
- return QueryException.BROKER_INSTANCE_MISSING_ERROR.toString();
- }
-
- // Send query to a random broker.
- String instanceId = instanceIds.get(RANDOM.nextInt(instanceIds.size()));
+ List<String> instanceIds = findCommonBrokerInstance(brokerTenants);
+ String instanceId = selectInstanceId(instanceIds);
Review Comment:
But what if my query access only table A and B here even then it would fail right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #11241:
URL: https://github.com/apache/pinot/pull/11241#discussion_r1283376252
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -214,31 +215,19 @@ private String getMultiStageQueryResponse(String query, String queryOptions, Htt
// tenants can be completely disjoint. The leaf stages which access segments will be processed on the respective
// server tenants for each table. The intermediate stages can be processed in either or all of the server tenants
// belonging to the tables.
- brokerTenant = getCommonBrokerTenant(tableConfigList);
- if (brokerTenant == null) {
+ brokerTenants = getCommonBrokerTenants(tableConfigList);
+ if (brokerTenants.isEmpty()) {
return QueryException.getException(QueryException.BROKER_REQUEST_SEND_ERROR, new Exception(
String.format("Unable to dispatch multistage query with multiple tables : %s " + "on different tenant",
tableNames))).toString();
}
} else {
// TODO fail these queries going forward. Added this logic to take care of tautologies like BETWEEN 0 and -1.
- List<String> allBrokerList = new ArrayList<>(_pinotHelixResourceManager.getAllBrokerTenantNames());
- brokerTenant = allBrokerList.get(RANDOM.nextInt(allBrokerList.size()));
+ brokerTenants = _pinotHelixResourceManager.getAllBrokerTenantNames();
Review Comment:
what would be the best solution here? i don't see a get random broker instance API in helix resource manager. should i add one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11241:
URL: https://github.com/apache/pinot/pull/11241#discussion_r1281460234
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -214,31 +215,19 @@ private String getMultiStageQueryResponse(String query, String queryOptions, Htt
// tenants can be completely disjoint. The leaf stages which access segments will be processed on the respective
// server tenants for each table. The intermediate stages can be processed in either or all of the server tenants
// belonging to the tables.
- brokerTenant = getCommonBrokerTenant(tableConfigList);
- if (brokerTenant == null) {
+ brokerTenants = getCommonBrokerTenants(tableConfigList);
+ if (brokerTenants.isEmpty()) {
return QueryException.getException(QueryException.BROKER_REQUEST_SEND_ERROR, new Exception(
String.format("Unable to dispatch multistage query with multiple tables : %s " + "on different tenant",
tableNames))).toString();
}
} else {
// TODO fail these queries going forward. Added this logic to take care of tautologies like BETWEEN 0 and -1.
- List<String> allBrokerList = new ArrayList<>(_pinotHelixResourceManager.getAllBrokerTenantNames());
- brokerTenant = allBrokerList.get(RANDOM.nextInt(allBrokerList.size()));
+ brokerTenants = _pinotHelixResourceManager.getAllBrokerTenantNames();
Review Comment:
Actually, directly picking a random broker might be even better
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #11241:
URL: https://github.com/apache/pinot/pull/11241#discussion_r1283522660
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -202,43 +203,28 @@ private String getMultiStageQueryResponse(String query, String queryOptions, Htt
QueryEnvironment queryEnvironment = new QueryEnvironment(new TypeFactory(new TypeSystem()),
CalciteSchemaBuilder.asRootSchema(new PinotCatalog(_pinotHelixResourceManager.getTableCache())), null, null);
List<String> tableNames = queryEnvironment.getTableNamesForQuery(query);
- String brokerTenant;
+ List<String> instanceIds;
if (tableNames.size() != 0) {
List<TableConfig> tableConfigList = getListTableConfigs(tableNames);
if (tableConfigList == null || tableConfigList.size() == 0) {
return QueryException.getException(QueryException.TABLE_DOES_NOT_EXIST_ERROR,
new Exception("Unable to find table in cluster, table does not exist")).toString();
}
- // When routing a query, there should be at least one common broker tenant for the table. However, the server
- // tenants can be completely disjoint. The leaf stages which access segments will be processed on the respective
- // server tenants for each table. The intermediate stages can be processed in either or all of the server tenants
- // belonging to the tables.
- brokerTenant = getCommonBrokerTenant(tableConfigList);
- if (brokerTenant == null) {
+ // find the unions of all the broker tenant tags of the queried tables.
+ Set<String> brokerTenantsUnion = getBrokerTenantTagsUnion(tableConfigList);
Review Comment:
this extracts the union of all tags used by the tables from the query only @tibrewalpratik17
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11241:
URL: https://github.com/apache/pinot/pull/11241#discussion_r1281301707
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -299,16 +277,40 @@ private List<TableConfig> getListTableConfigs(List<String> tableNames) {
return allTableConfigList;
}
+ private String selectInstanceId(List<String> instanceIds) {
+ if (instanceIds.isEmpty()) {
+ return QueryException.BROKER_RESOURCE_MISSING_ERROR.toString();
+ }
+
+ instanceIds.retainAll(_pinotHelixResourceManager.getOnlineInstanceList());
+ if (instanceIds.isEmpty()) {
+ return QueryException.BROKER_INSTANCE_MISSING_ERROR.toString();
+ }
+
+ // Send query to a random broker.
+ return instanceIds.get(RANDOM.nextInt(instanceIds.size()));
+ }
+
+ private List<String> findCommonBrokerInstance(Set<String> brokerTenants) {
+ Set<String> commonInstances = null;
+ for (String brokerTenant : brokerTenants) {
+ Set<String> instances = _pinotHelixResourceManager.getAllInstancesForBrokerTenant(brokerTenant);
+ if (commonInstances == null) {
+ commonInstances = instances;
+ } else {
+ commonInstances.retainAll(instances);
+ }
+ }
+ return commonInstances == null ? Collections.emptyList() : new ArrayList<>(commonInstances);
+ }
+
// return the brokerTenant if all table configs point to the same broker, else returns null
Review Comment:
Please update the javadoc
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -299,16 +277,40 @@ private List<TableConfig> getListTableConfigs(List<String> tableNames) {
return allTableConfigList;
}
+ private String selectInstanceId(List<String> instanceIds) {
+ if (instanceIds.isEmpty()) {
+ return QueryException.BROKER_RESOURCE_MISSING_ERROR.toString();
+ }
+
+ instanceIds.retainAll(_pinotHelixResourceManager.getOnlineInstanceList());
+ if (instanceIds.isEmpty()) {
+ return QueryException.BROKER_INSTANCE_MISSING_ERROR.toString();
+ }
+
+ // Send query to a random broker.
+ return instanceIds.get(RANDOM.nextInt(instanceIds.size()));
+ }
+
+ private List<String> findCommonBrokerInstance(Set<String> brokerTenants) {
+ Set<String> commonInstances = null;
+ for (String brokerTenant : brokerTenants) {
+ Set<String> instances = _pinotHelixResourceManager.getAllInstancesForBrokerTenant(brokerTenant);
+ if (commonInstances == null) {
+ commonInstances = instances;
+ } else {
+ commonInstances.retainAll(instances);
+ }
+ }
+ return commonInstances == null ? Collections.emptyList() : new ArrayList<>(commonInstances);
+ }
+
// return the brokerTenant if all table configs point to the same broker, else returns null
- private String getCommonBrokerTenant(List<TableConfig> tableConfigList) {
- Set<String> tableBrokers = new HashSet<>();
+ private Set<String> getCommonBrokerTenants(List<TableConfig> tableConfigList) {
Review Comment:
This is no longer the common broker tenants, but all broker tenants
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -214,31 +215,19 @@ private String getMultiStageQueryResponse(String query, String queryOptions, Htt
// tenants can be completely disjoint. The leaf stages which access segments will be processed on the respective
// server tenants for each table. The intermediate stages can be processed in either or all of the server tenants
// belonging to the tables.
- brokerTenant = getCommonBrokerTenant(tableConfigList);
- if (brokerTenant == null) {
+ brokerTenants = getCommonBrokerTenants(tableConfigList);
+ if (brokerTenants.isEmpty()) {
return QueryException.getException(QueryException.BROKER_REQUEST_SEND_ERROR, new Exception(
String.format("Unable to dispatch multistage query with multiple tables : %s " + "on different tenant",
tableNames))).toString();
}
} else {
// TODO fail these queries going forward. Added this logic to take care of tautologies like BETWEEN 0 and -1.
- List<String> allBrokerList = new ArrayList<>(_pinotHelixResourceManager.getAllBrokerTenantNames());
- brokerTenant = allBrokerList.get(RANDOM.nextInt(allBrokerList.size()));
+ brokerTenants = _pinotHelixResourceManager.getAllBrokerTenantNames();
Review Comment:
This doesn't seem correct. We should pick a random tenant
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr merged pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr merged PR #11241:
URL: https://github.com/apache/pinot/pull/11241
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tibrewalpratik17 commented on a diff in pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on code in PR #11241:
URL: https://github.com/apache/pinot/pull/11241#discussion_r1283219613
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -214,31 +215,19 @@ private String getMultiStageQueryResponse(String query, String queryOptions, Htt
// tenants can be completely disjoint. The leaf stages which access segments will be processed on the respective
// server tenants for each table. The intermediate stages can be processed in either or all of the server tenants
// belonging to the tables.
- brokerTenant = getCommonBrokerTenant(tableConfigList);
- if (brokerTenant == null) {
+ brokerTenants = getCommonBrokerTenants(tableConfigList);
+ if (brokerTenants.isEmpty()) {
return QueryException.getException(QueryException.BROKER_REQUEST_SEND_ERROR, new Exception(
String.format("Unable to dispatch multistage query with multiple tables : %s " + "on different tenant",
tableNames))).toString();
}
} else {
// TODO fail these queries going forward. Added this logic to take care of tautologies like BETWEEN 0 and -1.
- List<String> allBrokerList = new ArrayList<>(_pinotHelixResourceManager.getAllBrokerTenantNames());
- brokerTenant = allBrokerList.get(RANDOM.nextInt(allBrokerList.size()));
+ brokerTenants = _pinotHelixResourceManager.getAllBrokerTenantNames();
LOGGER.error("Unable to find table name from SQL {} thus dispatching to random broker.", query);
}
- List<String> instanceIds = new ArrayList<>(_pinotHelixResourceManager.getAllInstancesForBrokerTenant(brokerTenant));
-
- if (instanceIds.isEmpty()) {
- return QueryException.BROKER_RESOURCE_MISSING_ERROR.toString();
- }
-
- instanceIds.retainAll(_pinotHelixResourceManager.getOnlineInstanceList());
- if (instanceIds.isEmpty()) {
- return QueryException.BROKER_INSTANCE_MISSING_ERROR.toString();
- }
-
- // Send query to a random broker.
- String instanceId = instanceIds.get(RANDOM.nextInt(instanceIds.size()));
+ List<String> instanceIds = findCommonBrokerInstance(brokerTenants);
+ String instanceId = selectInstanceId(instanceIds);
Review Comment:
Umm yes but say in a scenario where there are 3 tables -- tableA, tableB, tableC on broker tenants A, B, C.
Interesection of [A, B] has 2 broker-instances common but brokertenant C as no instances common to either A or B. In this case, `findCommonBrokerInstance` will return Collections.emptyList() and `selectInstanceId` will return BROKER_RESOURCE_MISSING_ERROR.
Here `_pinotHelixResourceManager.getAllBrokerTenantNames()` doesn't include tableNames that are passed in the query but all tables in the cluster. Chances are high that there are no common instances among all these table brokerTenants.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tibrewalpratik17 commented on a diff in pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on code in PR #11241:
URL: https://github.com/apache/pinot/pull/11241#discussion_r1284243199
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -202,43 +203,28 @@ private String getMultiStageQueryResponse(String query, String queryOptions, Htt
QueryEnvironment queryEnvironment = new QueryEnvironment(new TypeFactory(new TypeSystem()),
CalciteSchemaBuilder.asRootSchema(new PinotCatalog(_pinotHelixResourceManager.getTableCache())), null, null);
List<String> tableNames = queryEnvironment.getTableNamesForQuery(query);
- String brokerTenant;
+ List<String> instanceIds;
if (tableNames.size() != 0) {
List<TableConfig> tableConfigList = getListTableConfigs(tableNames);
if (tableConfigList == null || tableConfigList.size() == 0) {
return QueryException.getException(QueryException.TABLE_DOES_NOT_EXIST_ERROR,
new Exception("Unable to find table in cluster, table does not exist")).toString();
}
- // When routing a query, there should be at least one common broker tenant for the table. However, the server
- // tenants can be completely disjoint. The leaf stages which access segments will be processed on the respective
- // server tenants for each table. The intermediate stages can be processed in either or all of the server tenants
- // belonging to the tables.
- brokerTenant = getCommonBrokerTenant(tableConfigList);
- if (brokerTenant == null) {
+ // find the unions of all the broker tenant tags of the queried tables.
+ Set<String> brokerTenantsUnion = getBrokerTenantTagsUnion(tableConfigList);
Review Comment:
Sounds good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #11241:
URL: https://github.com/apache/pinot/pull/11241#discussion_r1282623283
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -214,31 +215,19 @@ private String getMultiStageQueryResponse(String query, String queryOptions, Htt
// tenants can be completely disjoint. The leaf stages which access segments will be processed on the respective
// server tenants for each table. The intermediate stages can be processed in either or all of the server tenants
// belonging to the tables.
- brokerTenant = getCommonBrokerTenant(tableConfigList);
- if (brokerTenant == null) {
+ brokerTenants = getCommonBrokerTenants(tableConfigList);
+ if (brokerTenants.isEmpty()) {
return QueryException.getException(QueryException.BROKER_REQUEST_SEND_ERROR, new Exception(
String.format("Unable to dispatch multistage query with multiple tables : %s " + "on different tenant",
tableNames))).toString();
}
} else {
// TODO fail these queries going forward. Added this logic to take care of tautologies like BETWEEN 0 and -1.
- List<String> allBrokerList = new ArrayList<>(_pinotHelixResourceManager.getAllBrokerTenantNames());
- brokerTenant = allBrokerList.get(RANDOM.nextInt(allBrokerList.size()));
+ brokerTenants = _pinotHelixResourceManager.getAllBrokerTenantNames();
LOGGER.error("Unable to find table name from SQL {} thus dispatching to random broker.", query);
}
- List<String> instanceIds = new ArrayList<>(_pinotHelixResourceManager.getAllInstancesForBrokerTenant(brokerTenant));
-
- if (instanceIds.isEmpty()) {
- return QueryException.BROKER_RESOURCE_MISSING_ERROR.toString();
- }
-
- instanceIds.retainAll(_pinotHelixResourceManager.getOnlineInstanceList());
- if (instanceIds.isEmpty()) {
- return QueryException.BROKER_INSTANCE_MISSING_ERROR.toString();
- }
-
- // Send query to a random broker.
- String instanceId = instanceIds.get(RANDOM.nextInt(instanceIds.size()));
+ List<String> instanceIds = findCommonBrokerInstance(brokerTenants);
+ String instanceId = selectInstanceId(instanceIds);
Review Comment:
no. i think that's the problem with previous implementation. what i did here is:
1. get a UNION of all the table's broker tag from all the tables (not intersect). e.g. tableA has tenant tag A and tableB has tenant tag B then the UNION will be [A, B]
2. get a list of brokers that contains all the broker tags in the UNION --> only the broker that are tagged with both A and B will be selected.
3. select a random broker from the list of brokers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #11241:
URL: https://github.com/apache/pinot/pull/11241#discussion_r1283311073
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -214,31 +215,19 @@ private String getMultiStageQueryResponse(String query, String queryOptions, Htt
// tenants can be completely disjoint. The leaf stages which access segments will be processed on the respective
// server tenants for each table. The intermediate stages can be processed in either or all of the server tenants
// belonging to the tables.
- brokerTenant = getCommonBrokerTenant(tableConfigList);
- if (brokerTenant == null) {
+ brokerTenants = getCommonBrokerTenants(tableConfigList);
+ if (brokerTenants.isEmpty()) {
return QueryException.getException(QueryException.BROKER_REQUEST_SEND_ERROR, new Exception(
String.format("Unable to dispatch multistage query with multiple tables : %s " + "on different tenant",
tableNames))).toString();
}
} else {
// TODO fail these queries going forward. Added this logic to take care of tautologies like BETWEEN 0 and -1.
- List<String> allBrokerList = new ArrayList<>(_pinotHelixResourceManager.getAllBrokerTenantNames());
- brokerTenant = allBrokerList.get(RANDOM.nextInt(allBrokerList.size()));
+ brokerTenants = _pinotHelixResourceManager.getAllBrokerTenantNames();
LOGGER.error("Unable to find table name from SQL {} thus dispatching to random broker.", query);
}
- List<String> instanceIds = new ArrayList<>(_pinotHelixResourceManager.getAllInstancesForBrokerTenant(brokerTenant));
-
- if (instanceIds.isEmpty()) {
- return QueryException.BROKER_RESOURCE_MISSING_ERROR.toString();
- }
-
- instanceIds.retainAll(_pinotHelixResourceManager.getOnlineInstanceList());
- if (instanceIds.isEmpty()) {
- return QueryException.BROKER_INSTANCE_MISSING_ERROR.toString();
- }
-
- // Send query to a random broker.
- String instanceId = instanceIds.get(RANDOM.nextInt(instanceIds.size()));
+ List<String> instanceIds = findCommonBrokerInstance(brokerTenants);
+ String instanceId = selectInstanceId(instanceIds);
Review Comment:
well if your query access table A B and C at the same time it should do just that. b/c there's no way to run a query to join A B and C together without a broker knowing the info of all 3 tables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #11241:
URL: https://github.com/apache/pinot/pull/11241#discussion_r1283521975
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -214,31 +215,19 @@ private String getMultiStageQueryResponse(String query, String queryOptions, Htt
// tenants can be completely disjoint. The leaf stages which access segments will be processed on the respective
// server tenants for each table. The intermediate stages can be processed in either or all of the server tenants
// belonging to the tables.
- brokerTenant = getCommonBrokerTenant(tableConfigList);
- if (brokerTenant == null) {
+ brokerTenants = getCommonBrokerTenants(tableConfigList);
+ if (brokerTenants.isEmpty()) {
return QueryException.getException(QueryException.BROKER_REQUEST_SEND_ERROR, new Exception(
String.format("Unable to dispatch multistage query with multiple tables : %s " + "on different tenant",
tableNames))).toString();
}
} else {
// TODO fail these queries going forward. Added this logic to take care of tautologies like BETWEEN 0 and -1.
- List<String> allBrokerList = new ArrayList<>(_pinotHelixResourceManager.getAllBrokerTenantNames());
- brokerTenant = allBrokerList.get(RANDOM.nextInt(allBrokerList.size()));
+ brokerTenants = _pinotHelixResourceManager.getAllBrokerTenantNames();
LOGGER.error("Unable to find table name from SQL {} thus dispatching to random broker.", query);
}
- List<String> instanceIds = new ArrayList<>(_pinotHelixResourceManager.getAllInstancesForBrokerTenant(brokerTenant));
-
- if (instanceIds.isEmpty()) {
- return QueryException.BROKER_RESOURCE_MISSING_ERROR.toString();
- }
-
- instanceIds.retainAll(_pinotHelixResourceManager.getOnlineInstanceList());
- if (instanceIds.isEmpty()) {
- return QueryException.BROKER_INSTANCE_MISSING_ERROR.toString();
- }
-
- // Send query to a random broker.
- String instanceId = instanceIds.get(RANDOM.nextInt(instanceIds.size()));
+ List<String> instanceIds = findCommonBrokerInstance(brokerTenants);
+ String instanceId = selectInstanceId(instanceIds);
Review Comment:
no it won't. the brokerTenant (brokerTenantUnion in new code) will only contain the unioned tag from table A and B.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11241:
URL: https://github.com/apache/pinot/pull/11241#issuecomment-1661129631
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11241?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11241](https://app.codecov.io/gh/apache/pinot/pull/11241?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (1f65669) into [master](https://app.codecov.io/gh/apache/pinot/commit/9d7676ed8cc21914bb64126281bf9e626236e82f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (9d7676e) will **increase** coverage by `0.00%`.
> Report is 3 commits behind head on master.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #11241 +/- ##
=========================================
Coverage 0.11% 0.11%
=========================================
Files 2229 2174 -55
Lines 119803 117270 -2533
Branches 18126 17813 -313
=========================================
Hits 137 137
+ Misses 119646 117113 -2533
Partials 20 20
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1temurin11 | `?` | |
| integration1temurin20 | `?` | |
| integration2temurin11 | `?` | |
| integration2temurin17 | `?` | |
| integration2temurin20 | `?` | |
| unittests1temurin11 | `?` | |
| unittests2temurin11 | `?` | |
| unittests2temurin17 | `?` | |
| unittests2temurin20 | `0.11% <0.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files Changed](https://app.codecov.io/gh/apache/pinot/pull/11241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...t/controller/api/resources/PinotQueryResource.java](https://app.codecov.io/gh/apache/pinot/pull/11241?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90UXVlcnlSZXNvdXJjZS5qYXZh) | `0.00% <0.00%> (ø)` | |
... and [58 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11241/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11241: [multistage] support multi-tenant lookup for v2 engine
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11241:
URL: https://github.com/apache/pinot/pull/11241#discussion_r1283970208
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -299,16 +274,40 @@ private List<TableConfig> getListTableConfigs(List<String> tableNames) {
return allTableConfigList;
}
- // return the brokerTenant if all table configs point to the same broker, else returns null
- private String getCommonBrokerTenant(List<TableConfig> tableConfigList) {
- Set<String> tableBrokers = new HashSet<>();
- for (TableConfig tableConfig : tableConfigList) {
- tableBrokers.add(tableConfig.getTenantConfig().getBroker());
+ private String selectRandomInstanceId(List<String> instanceIds) {
+ if (instanceIds.isEmpty()) {
+ return QueryException.BROKER_RESOURCE_MISSING_ERROR.toString();
+ }
+
+ instanceIds.retainAll(_pinotHelixResourceManager.getOnlineInstanceList());
+ if (instanceIds.isEmpty()) {
+ return QueryException.BROKER_INSTANCE_MISSING_ERROR.toString();
}
- if (tableBrokers.size() != 1) {
- return null;
+
+ // Send query to a random broker.
+ return instanceIds.get(RANDOM.nextInt(instanceIds.size()));
+ }
+
+ private List<String> findCommonBrokerInstance(Set<String> brokerTenants) {
Review Comment:
```suggestion
private List<String> findCommonBrokerInstances(Set<String> brokerTenants) {
```
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -469,6 +469,11 @@ public List<InstanceConfig> getBrokerInstancesConfigsFor(String tableName) {
TagNameUtils.getBrokerTagForTenant(brokerTenantName));
}
+ public List<InstanceConfig> getAllBrokerInstanceConfigs() {
+ return HelixHelper.getInstanceConfigs(_helixZkManager).stream()
+ .filter(instance -> InstanceTypeUtils.isBroker(instance.getId())).collect(Collectors.toList());
+ }
Review Comment:
We can add API to get all broker instance ids which is much cheaper
```suggestion
public List<String> getAllBrokerInstances() {
return HelixHelper.getAllInstances(_helixAdmin, _clusterName).stream()
.filter(instanceId -> InstanceTypeUtils.isBroker(instanceId)).collect(Collectors.toList());
}
```
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -299,16 +274,40 @@ private List<TableConfig> getListTableConfigs(List<String> tableNames) {
return allTableConfigList;
}
- // return the brokerTenant if all table configs point to the same broker, else returns null
- private String getCommonBrokerTenant(List<TableConfig> tableConfigList) {
- Set<String> tableBrokers = new HashSet<>();
- for (TableConfig tableConfig : tableConfigList) {
- tableBrokers.add(tableConfig.getTenantConfig().getBroker());
+ private String selectRandomInstanceId(List<String> instanceIds) {
+ if (instanceIds.isEmpty()) {
+ return QueryException.BROKER_RESOURCE_MISSING_ERROR.toString();
+ }
+
+ instanceIds.retainAll(_pinotHelixResourceManager.getOnlineInstanceList());
+ if (instanceIds.isEmpty()) {
+ return QueryException.BROKER_INSTANCE_MISSING_ERROR.toString();
}
- if (tableBrokers.size() != 1) {
- return null;
+
+ // Send query to a random broker.
+ return instanceIds.get(RANDOM.nextInt(instanceIds.size()));
+ }
+
+ private List<String> findCommonBrokerInstance(Set<String> brokerTenants) {
+ Set<String> commonInstances = null;
+ for (String brokerTenant : brokerTenants) {
+ Set<String> instances = _pinotHelixResourceManager.getAllInstancesForBrokerTenant(brokerTenant);
+ if (commonInstances == null) {
+ commonInstances = instances;
+ } else {
+ commonInstances.retainAll(instances);
+ }
+ }
+ return commonInstances == null ? Collections.emptyList() : new ArrayList<>(commonInstances);
Review Comment:
`commonInstances` can never be `null` here because `brokerTenants` is not empty
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -299,16 +274,40 @@ private List<TableConfig> getListTableConfigs(List<String> tableNames) {
return allTableConfigList;
}
- // return the brokerTenant if all table configs point to the same broker, else returns null
- private String getCommonBrokerTenant(List<TableConfig> tableConfigList) {
- Set<String> tableBrokers = new HashSet<>();
- for (TableConfig tableConfig : tableConfigList) {
- tableBrokers.add(tableConfig.getTenantConfig().getBroker());
+ private String selectRandomInstanceId(List<String> instanceIds) {
+ if (instanceIds.isEmpty()) {
+ return QueryException.BROKER_RESOURCE_MISSING_ERROR.toString();
+ }
+
+ instanceIds.retainAll(_pinotHelixResourceManager.getOnlineInstanceList());
+ if (instanceIds.isEmpty()) {
+ return QueryException.BROKER_INSTANCE_MISSING_ERROR.toString();
}
- if (tableBrokers.size() != 1) {
- return null;
+
+ // Send query to a random broker.
+ return instanceIds.get(RANDOM.nextInt(instanceIds.size()));
+ }
+
+ private List<String> findCommonBrokerInstance(Set<String> brokerTenants) {
+ Set<String> commonInstances = null;
+ for (String brokerTenant : brokerTenants) {
+ Set<String> instances = _pinotHelixResourceManager.getAllInstancesForBrokerTenant(brokerTenant);
+ if (commonInstances == null) {
+ commonInstances = instances;
+ } else {
+ commonInstances.retainAll(instances);
+ }
+ }
+ return commonInstances == null ? Collections.emptyList() : new ArrayList<>(commonInstances);
+ }
+
+ // return the union of brokerTenants from the tables list.
+ private Set<String> getBrokerTenantTagsUnion(List<TableConfig> tableConfigList) {
Review Comment:
```suggestion
private Set<String> getBrokerTenantsUnion(List<TableConfig> tableConfigList) {
```
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotQueryResource.java:
##########
@@ -299,16 +274,40 @@ private List<TableConfig> getListTableConfigs(List<String> tableNames) {
return allTableConfigList;
}
- // return the brokerTenant if all table configs point to the same broker, else returns null
- private String getCommonBrokerTenant(List<TableConfig> tableConfigList) {
- Set<String> tableBrokers = new HashSet<>();
- for (TableConfig tableConfig : tableConfigList) {
- tableBrokers.add(tableConfig.getTenantConfig().getBroker());
+ private String selectRandomInstanceId(List<String> instanceIds) {
+ if (instanceIds.isEmpty()) {
+ return QueryException.BROKER_RESOURCE_MISSING_ERROR.toString();
+ }
+
+ instanceIds.retainAll(_pinotHelixResourceManager.getOnlineInstanceList());
+ if (instanceIds.isEmpty()) {
+ return QueryException.BROKER_INSTANCE_MISSING_ERROR.toString();
}
- if (tableBrokers.size() != 1) {
- return null;
+
+ // Send query to a random broker.
+ return instanceIds.get(RANDOM.nextInt(instanceIds.size()));
+ }
+
+ private List<String> findCommonBrokerInstance(Set<String> brokerTenants) {
+ Set<String> commonInstances = null;
+ for (String brokerTenant : brokerTenants) {
+ Set<String> instances = _pinotHelixResourceManager.getAllInstancesForBrokerTenant(brokerTenant);
Review Comment:
It will be much faster if we first get all instance configs, and then pass in the instance configs when getting instance ids. See the `TODO` in `PinotHelixResourceManager.getAllInstancesForBrokerTenant()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org