You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/08/12 00:07:50 UTC
[GitHub] [pinot] Jackie-Jiang opened a new pull request, #9202: Avoid parsing query multiple times
Jackie-Jiang opened a new pull request, #9202:
URL: https://github.com/apache/pinot/pull/9202
Currently we parse the same query multiple times:
1. Figure out whether the query is DQL or DML
2. (not there but needed) Figure out whether to use multi-stage engine
3. When execute the query
This PR changes the `BrokerRequestHandler` to take the parsed `SqlNodeAndOptions` so that we only need to parse the query once. It also adds support to read query options from query in the `BrokerRequestHandlerDelegate`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9202: Avoid parsing query multiple times
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9202:
URL: https://github.com/apache/pinot/pull/9202#discussion_r944624652
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java:
##########
@@ -190,27 +191,32 @@ public BrokerResponseNative handleRequest(JsonNode request, @Nullable RequesterI
if (sql == null) {
throw new BadQueryRequestException("Failed to find 'sql' in the request: " + request);
}
- return handleRequest(requestId, sql.asText(), request, requesterIdentity, requestContext);
+ String query = sql.asText();
+ requestContext.setQuery(query);
+ return handleRequest(requestId, query, sqlNodeAndOptions, request, requesterIdentity, requestContext);
}
- private BrokerResponseNative handleRequest(long requestId, String query, JsonNode request,
- @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext)
+ private BrokerResponseNative handleRequest(long requestId, String query,
+ @Nullable SqlNodeAndOptions sqlNodeAndOptions, JsonNode request, @Nullable RequesterIdentity requesterIdentity,
+ RequestContext requestContext)
throws Exception {
LOGGER.debug("SQL query for request {}: {}", requestId, query);
- requestContext.setQuery(query);
// Compile the request
- long compilationStartTimeNs = System.nanoTime();
PinotQuery pinotQuery;
try {
- pinotQuery = CalciteSqlParser.compileToPinotQuery(query);
+ if (sqlNodeAndOptions == null) {
Review Comment:
So callers like us (LinkedIn) will pass this as NULL and it should continue to work IIUC. We call handleRequest and let it parse and compile query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #9202: Avoid parsing query multiple times
Posted by GitBox <gi...@apache.org>.
walterddr commented on code in PR #9202:
URL: https://github.com/apache/pinot/pull/9202#discussion_r944033369
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java:
##########
@@ -190,26 +191,34 @@ public BrokerResponseNative handleRequest(JsonNode request, @Nullable RequesterI
if (sql == null) {
throw new BadQueryRequestException("Failed to find 'sql' in the request: " + request);
}
- return handleRequest(requestId, sql.asText(), request, requesterIdentity, requestContext);
+ String query = sql.asText();
+ requestContext.setQuery(query);
+ return handleRequest(requestId, query, sqlNodeAndOptions, compilationStartTimeNs, request, requesterIdentity,
+ requestContext);
}
- private BrokerResponseNative handleRequest(long requestId, String query, JsonNode request,
+ private BrokerResponseNative handleRequest(long requestId, String query,
+ @Nullable SqlNodeAndOptions sqlNodeAndOptions, long compilationStartTimeNs, JsonNode request,
Review Comment:
passing in compilationStartTimeNs is kinda weird here. let's just break this into parser time and still keep the `compilationStartTimeNs` inside the handleRequest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #9202: Avoid parsing query multiple times
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on PR #9202:
URL: https://github.com/apache/pinot/pull/9202#issuecomment-1212623720
# [Codecov](https://codecov.io/gh/apache/pinot/pull/9202?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#9202](https://codecov.io/gh/apache/pinot/pull/9202?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (e0281d3) into [master](https://codecov.io/gh/apache/pinot/commit/54d2813c00093e1baada9f7c8627c9360f133328?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (54d2813) will **decrease** coverage by `2.78%`.
> The diff coverage is `50.00%`.
```diff
@@ Coverage Diff @@
## master #9202 +/- ##
============================================
- Coverage 69.89% 67.11% -2.79%
+ Complexity 5007 4812 -195
============================================
Files 1852 1378 -474
Lines 98822 71810 -27012
Branches 15032 11496 -3536
============================================
- Hits 69074 48192 -20882
+ Misses 24876 20130 -4746
+ Partials 4872 3488 -1384
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `67.11% <50.00%> (-0.03%)` | :arrow_down: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/9202?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../java/org/apache/pinot/query/QueryEnvironment.java](https://codecov.io/gh/apache/pinot/pull/9202/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcGxhbm5lci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvUXVlcnlFbnZpcm9ubWVudC5qYXZh) | `82.69% <40.00%> (-4.81%)` | :arrow_down: |
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/pinot/pull/9202/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `86.41% <100.00%> (-0.64%)` | :arrow_down: |
| [...va/org/apache/pinot/core/routing/RoutingTable.java](https://codecov.io/gh/apache/pinot/pull/9202/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9yb3V0aW5nL1JvdXRpbmdUYWJsZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pinot/common/config/NettyConfig.java](https://codecov.io/gh/apache/pinot/pull/9202/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vY29uZmlnL05ldHR5Q29uZmlnLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/9202/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/9202/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/9202/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/9202/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ache/pinot/server/access/AccessControlFactory.java](https://codecov.io/gh/apache/pinot/pull/9202/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VydmVyL2FjY2Vzcy9BY2Nlc3NDb250cm9sRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/common/messages/SegmentReloadMessage.java](https://codecov.io/gh/apache/pinot/pull/9202/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWVzc2FnZXMvU2VnbWVudFJlbG9hZE1lc3NhZ2UuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [708 more](https://codecov.io/gh/apache/pinot/pull/9202/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on pull request #9202: Avoid parsing query multiple times
Posted by GitBox <gi...@apache.org>.
walterddr commented on PR #9202:
URL: https://github.com/apache/pinot/pull/9202#issuecomment-1216999438
merging and continue refining in https://github.com/apache/pinot/issues/9155
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #9202: Avoid parsing query multiple times
Posted by GitBox <gi...@apache.org>.
walterddr commented on code in PR #9202:
URL: https://github.com/apache/pinot/pull/9202#discussion_r944033369
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java:
##########
@@ -190,26 +191,34 @@ public BrokerResponseNative handleRequest(JsonNode request, @Nullable RequesterI
if (sql == null) {
throw new BadQueryRequestException("Failed to find 'sql' in the request: " + request);
}
- return handleRequest(requestId, sql.asText(), request, requesterIdentity, requestContext);
+ String query = sql.asText();
+ requestContext.setQuery(query);
+ return handleRequest(requestId, query, sqlNodeAndOptions, compilationStartTimeNs, request, requesterIdentity,
+ requestContext);
}
- private BrokerResponseNative handleRequest(long requestId, String query, JsonNode request,
+ private BrokerResponseNative handleRequest(long requestId, String query,
+ @Nullable SqlNodeAndOptions sqlNodeAndOptions, long compilationStartTimeNs, JsonNode request,
Review Comment:
passing in compilationStartTimeNs is kinda weird here. let's just break this into parser time and still keep the `compilationStartTimeNs`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9202: Avoid parsing query multiple times
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9202:
URL: https://github.com/apache/pinot/pull/9202#discussion_r944891272
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerRequestHandlerDelegate.java:
##########
@@ -71,18 +77,43 @@ public void shutDown() {
}
@Override
- public BrokerResponse handleRequest(JsonNode request, @Nullable RequesterIdentity requesterIdentity,
- RequestContext requestContext)
+ public BrokerResponseNative handleRequest(JsonNode request, @Nullable SqlNodeAndOptions sqlNodeAndOptions,
+ @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext)
throws Exception {
- if (_isMultiStageQueryEngineEnabled && _multiStageWorkerRequestHandler != null) {
- if (request.has("queryOptions")) {
- Map<String, String> queryOptionMap = BaseBrokerRequestHandler.getOptionsFromJson(request, "queryOptions");
- if (Boolean.parseBoolean(queryOptionMap.get(
- CommonConstants.Broker.Request.QueryOptionKey.USE_MULTISTAGE_ENGINE))) {
- return _multiStageWorkerRequestHandler.handleRequest(request, requesterIdentity, requestContext);
- }
+ if (sqlNodeAndOptions == null) {
+ JsonNode sql = request.get(Request.SQL);
+ if (sql == null) {
+ throw new BadQueryRequestException("Failed to find 'sql' in the request: " + request);
+ }
+ try {
+ sqlNodeAndOptions = CalciteSqlParser.compileToSqlNodeAndOptions(sql.asText());
+ } catch (Exception e) {
+ LOGGER.info("Caught exception while compiling SQL: {}, {}", sql.asText(), e.getMessage());
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.REQUEST_COMPILATION_EXCEPTIONS, 1);
+ requestContext.setErrorCode(QueryException.SQL_PARSING_ERROR_CODE);
+ return new BrokerResponseNative(QueryException.getException(QueryException.SQL_PARSING_ERROR, e));
}
}
- return _singleStageBrokerRequestHandler.handleRequest(request, requesterIdentity, requestContext);
+
+ if (_multiStageWorkerRequestHandler != null && useMultiStageEngine(request, sqlNodeAndOptions)) {
+ return _multiStageWorkerRequestHandler.handleRequest(request, sqlNodeAndOptions, requesterIdentity,
+ requestContext);
+ } else {
+ return _singleStageBrokerRequestHandler.handleRequest(request, sqlNodeAndOptions, requesterIdentity,
+ requestContext);
+ }
+ }
+
+ private boolean useMultiStageEngine(JsonNode request, SqlNodeAndOptions sqlNodeAndOptions) {
Review Comment:
+1
Thanks for this change.
This will allow us to deploy this on perf cluster and for a given prod use case run a mixed workload where JOIN / subqueries go through multi-stage engine based on the query level options and feature flag turned on and existing prod queries (non join / non subquery) continue to go through existing engine because they will not have the query level option.
We wanted to do productionization testing on our use cases this way to keep it focused on testing new queries that go down the new engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9202: Avoid parsing query multiple times
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9202:
URL: https://github.com/apache/pinot/pull/9202#discussion_r944624652
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java:
##########
@@ -190,27 +191,32 @@ public BrokerResponseNative handleRequest(JsonNode request, @Nullable RequesterI
if (sql == null) {
throw new BadQueryRequestException("Failed to find 'sql' in the request: " + request);
}
- return handleRequest(requestId, sql.asText(), request, requesterIdentity, requestContext);
+ String query = sql.asText();
+ requestContext.setQuery(query);
+ return handleRequest(requestId, query, sqlNodeAndOptions, request, requesterIdentity, requestContext);
}
- private BrokerResponseNative handleRequest(long requestId, String query, JsonNode request,
- @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext)
+ private BrokerResponseNative handleRequest(long requestId, String query,
+ @Nullable SqlNodeAndOptions sqlNodeAndOptions, JsonNode request, @Nullable RequesterIdentity requesterIdentity,
+ RequestContext requestContext)
throws Exception {
LOGGER.debug("SQL query for request {}: {}", requestId, query);
- requestContext.setQuery(query);
// Compile the request
- long compilationStartTimeNs = System.nanoTime();
PinotQuery pinotQuery;
try {
- pinotQuery = CalciteSqlParser.compileToPinotQuery(query);
+ if (sqlNodeAndOptions == null) {
Review Comment:
So callers like us (LinkedIn) will pass this as NULL and it should continue to work IIUC. We call handleRequest and let it parse and compile query
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9202: Avoid parsing query multiple times
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9202:
URL: https://github.com/apache/pinot/pull/9202#discussion_r944891272
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerRequestHandlerDelegate.java:
##########
@@ -71,18 +77,43 @@ public void shutDown() {
}
@Override
- public BrokerResponse handleRequest(JsonNode request, @Nullable RequesterIdentity requesterIdentity,
- RequestContext requestContext)
+ public BrokerResponseNative handleRequest(JsonNode request, @Nullable SqlNodeAndOptions sqlNodeAndOptions,
+ @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext)
throws Exception {
- if (_isMultiStageQueryEngineEnabled && _multiStageWorkerRequestHandler != null) {
- if (request.has("queryOptions")) {
- Map<String, String> queryOptionMap = BaseBrokerRequestHandler.getOptionsFromJson(request, "queryOptions");
- if (Boolean.parseBoolean(queryOptionMap.get(
- CommonConstants.Broker.Request.QueryOptionKey.USE_MULTISTAGE_ENGINE))) {
- return _multiStageWorkerRequestHandler.handleRequest(request, requesterIdentity, requestContext);
- }
+ if (sqlNodeAndOptions == null) {
+ JsonNode sql = request.get(Request.SQL);
+ if (sql == null) {
+ throw new BadQueryRequestException("Failed to find 'sql' in the request: " + request);
+ }
+ try {
+ sqlNodeAndOptions = CalciteSqlParser.compileToSqlNodeAndOptions(sql.asText());
+ } catch (Exception e) {
+ LOGGER.info("Caught exception while compiling SQL: {}, {}", sql.asText(), e.getMessage());
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.REQUEST_COMPILATION_EXCEPTIONS, 1);
+ requestContext.setErrorCode(QueryException.SQL_PARSING_ERROR_CODE);
+ return new BrokerResponseNative(QueryException.getException(QueryException.SQL_PARSING_ERROR, e));
}
}
- return _singleStageBrokerRequestHandler.handleRequest(request, requesterIdentity, requestContext);
+
+ if (_multiStageWorkerRequestHandler != null && useMultiStageEngine(request, sqlNodeAndOptions)) {
+ return _multiStageWorkerRequestHandler.handleRequest(request, sqlNodeAndOptions, requesterIdentity,
+ requestContext);
+ } else {
+ return _singleStageBrokerRequestHandler.handleRequest(request, sqlNodeAndOptions, requesterIdentity,
+ requestContext);
+ }
+ }
+
+ private boolean useMultiStageEngine(JsonNode request, SqlNodeAndOptions sqlNodeAndOptions) {
Review Comment:
+1 - Thanks for this change.
This will allow us to deploy this on perf cluster and for a given prod use case run a mixed workload where JOIN / subqueries go through multi-stage engine based on the query level options and feature flag turned on and existing prod queries (non join / non subquery) continue to go through existing engine because they will not have the query level option.
We wanted to do productionization testing on our use cases this way to keep it focused on finding problems with new queries that go down the new engine
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on a diff in pull request #9202: Avoid parsing query multiple times
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on code in PR #9202:
URL: https://github.com/apache/pinot/pull/9202#discussion_r944891272
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BrokerRequestHandlerDelegate.java:
##########
@@ -71,18 +77,43 @@ public void shutDown() {
}
@Override
- public BrokerResponse handleRequest(JsonNode request, @Nullable RequesterIdentity requesterIdentity,
- RequestContext requestContext)
+ public BrokerResponseNative handleRequest(JsonNode request, @Nullable SqlNodeAndOptions sqlNodeAndOptions,
+ @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext)
throws Exception {
- if (_isMultiStageQueryEngineEnabled && _multiStageWorkerRequestHandler != null) {
- if (request.has("queryOptions")) {
- Map<String, String> queryOptionMap = BaseBrokerRequestHandler.getOptionsFromJson(request, "queryOptions");
- if (Boolean.parseBoolean(queryOptionMap.get(
- CommonConstants.Broker.Request.QueryOptionKey.USE_MULTISTAGE_ENGINE))) {
- return _multiStageWorkerRequestHandler.handleRequest(request, requesterIdentity, requestContext);
- }
+ if (sqlNodeAndOptions == null) {
+ JsonNode sql = request.get(Request.SQL);
+ if (sql == null) {
+ throw new BadQueryRequestException("Failed to find 'sql' in the request: " + request);
+ }
+ try {
+ sqlNodeAndOptions = CalciteSqlParser.compileToSqlNodeAndOptions(sql.asText());
+ } catch (Exception e) {
+ LOGGER.info("Caught exception while compiling SQL: {}, {}", sql.asText(), e.getMessage());
+ _brokerMetrics.addMeteredGlobalValue(BrokerMeter.REQUEST_COMPILATION_EXCEPTIONS, 1);
+ requestContext.setErrorCode(QueryException.SQL_PARSING_ERROR_CODE);
+ return new BrokerResponseNative(QueryException.getException(QueryException.SQL_PARSING_ERROR, e));
}
}
- return _singleStageBrokerRequestHandler.handleRequest(request, requesterIdentity, requestContext);
+
+ if (_multiStageWorkerRequestHandler != null && useMultiStageEngine(request, sqlNodeAndOptions)) {
+ return _multiStageWorkerRequestHandler.handleRequest(request, sqlNodeAndOptions, requesterIdentity,
+ requestContext);
+ } else {
+ return _singleStageBrokerRequestHandler.handleRequest(request, sqlNodeAndOptions, requesterIdentity,
+ requestContext);
+ }
+ }
+
+ private boolean useMultiStageEngine(JsonNode request, SqlNodeAndOptions sqlNodeAndOptions) {
Review Comment:
+1 - Thanks for this change.
This will allow us to deploy this on perf cluster and for a given use case run a mixed workload where JOIN / subqueries go through multi-stage engine based on the query level options and feature flag and existing prod queries continue to go through existing engine because they will not have the query level option.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr merged pull request #9202: Avoid parsing query multiple times
Posted by GitBox <gi...@apache.org>.
walterddr merged PR #9202:
URL: https://github.com/apache/pinot/pull/9202
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9202: Avoid parsing query multiple times
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9202:
URL: https://github.com/apache/pinot/pull/9202#discussion_r944191022
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java:
##########
@@ -190,26 +191,34 @@ public BrokerResponseNative handleRequest(JsonNode request, @Nullable RequesterI
if (sql == null) {
throw new BadQueryRequestException("Failed to find 'sql' in the request: " + request);
}
- return handleRequest(requestId, sql.asText(), request, requesterIdentity, requestContext);
+ String query = sql.asText();
+ requestContext.setQuery(query);
+ return handleRequest(requestId, query, sqlNodeAndOptions, compilationStartTimeNs, request, requesterIdentity,
+ requestContext);
}
- private BrokerResponseNative handleRequest(long requestId, String query, JsonNode request,
+ private BrokerResponseNative handleRequest(long requestId, String query,
+ @Nullable SqlNodeAndOptions sqlNodeAndOptions, long compilationStartTimeNs, JsonNode request,
Review Comment:
Wanted to keep the existing behavior. Added `compilationStartTimeNs` into `SqlNodeAndOptions` to keep the interface clean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #9202: Avoid parsing query multiple times
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on code in PR #9202:
URL: https://github.com/apache/pinot/pull/9202#discussion_r944677811
##########
pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java:
##########
@@ -190,27 +191,32 @@ public BrokerResponseNative handleRequest(JsonNode request, @Nullable RequesterI
if (sql == null) {
throw new BadQueryRequestException("Failed to find 'sql' in the request: " + request);
}
- return handleRequest(requestId, sql.asText(), request, requesterIdentity, requestContext);
+ String query = sql.asText();
+ requestContext.setQuery(query);
+ return handleRequest(requestId, query, sqlNodeAndOptions, request, requesterIdentity, requestContext);
}
- private BrokerResponseNative handleRequest(long requestId, String query, JsonNode request,
- @Nullable RequesterIdentity requesterIdentity, RequestContext requestContext)
+ private BrokerResponseNative handleRequest(long requestId, String query,
+ @Nullable SqlNodeAndOptions sqlNodeAndOptions, JsonNode request, @Nullable RequesterIdentity requesterIdentity,
+ RequestContext requestContext)
throws Exception {
LOGGER.debug("SQL query for request {}: {}", requestId, query);
- requestContext.setQuery(query);
// Compile the request
- long compilationStartTimeNs = System.nanoTime();
PinotQuery pinotQuery;
try {
- pinotQuery = CalciteSqlParser.compileToPinotQuery(query);
+ if (sqlNodeAndOptions == null) {
Review Comment:
Yes. It is backward compatible, and keeps the current behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] siddharthteotia commented on pull request #9202: Avoid parsing query multiple times
Posted by GitBox <gi...@apache.org>.
siddharthteotia commented on PR #9202:
URL: https://github.com/apache/pinot/pull/9202#issuecomment-1213559545
Will help with https://github.com/apache/pinot/issues/9155
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org