You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "walterddr (via GitHub)" <gi...@apache.org> on 2023/05/05 21:49:41 UTC
[GitHub] [pinot] walterddr opened a new pull request, #10728: [multistage] scheduler logic fixes
walterddr opened a new pull request, #10728:
URL: https://github.com/apache/pinot/pull/10728
- making QueryServer submit but return error code upon visitor failure
- submit leaf-stage on a QueryRunnerExecutorService
- fixing tracing test b/c it has issues: see: #10399
- fixing error propagation for throwable cause on mailbox send
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #10728: [multistage] scheduler logic fixes
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10728:
URL: https://github.com/apache/pinot/pull/10728#issuecomment-1536854378
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/10728?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10728](https://app.codecov.io/gh/apache/pinot/pull/10728?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (2c68695) into [master](https://app.codecov.io/gh/apache/pinot/commit/815cd6ebb7f2236b127a4b7a7f8d59bea2c9a1cc?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (815cd6e) will **decrease** coverage by `50.55%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #10728 +/- ##
=============================================
- Coverage 64.38% 13.83% -50.55%
+ Complexity 6441 439 -6002
=============================================
Files 2068 2068
Lines 111794 111804 +10
Branches 16956 16957 +1
=============================================
- Hits 71974 15471 -56503
- Misses 34646 95070 +60424
+ Partials 5174 1263 -3911
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests1 | `?` | |
| unittests2 | `13.83% <0.00%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://app.codecov.io/gh/apache/pinot/pull/10728?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/query/mailbox/GrpcSendingMailbox.java](https://app.codecov.io/gh/apache/pinot/pull/10728?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvbWFpbGJveC9HcnBjU2VuZGluZ01haWxib3guamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...he/pinot/query/mailbox/InMemorySendingMailbox.java](https://app.codecov.io/gh/apache/pinot/pull/10728?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvbWFpbGJveC9Jbk1lbW9yeVNlbmRpbmdNYWlsYm94LmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pinot/query/runtime/QueryRunner.java](https://app.codecov.io/gh/apache/pinot/pull/10728?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvcnVudGltZS9RdWVyeVJ1bm5lci5qYXZh) | `0.00% <0.00%> (-83.20%)` | :arrow_down: |
| [...va/org/apache/pinot/query/service/QueryServer.java](https://app.codecov.io/gh/apache/pinot/pull/10728?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvc2VydmljZS9RdWVyeVNlcnZlci5qYXZh) | `0.00% <0.00%> (-75.61%)` | :arrow_down: |
| [.../pinot/query/service/dispatch/QueryDispatcher.java](https://app.codecov.io/gh/apache/pinot/pull/10728?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtcXVlcnktcnVudGltZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvcXVlcnkvc2VydmljZS9kaXNwYXRjaC9RdWVyeURpc3BhdGNoZXIuamF2YQ==) | `0.00% <0.00%> (-93.47%)` | :arrow_down: |
... and [1414 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/10728/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] xiangfu0 commented on a diff in pull request #10728: [multistage] scheduler logic fixes
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on code in PR #10728:
URL: https://github.com/apache/pinot/pull/10728#discussion_r1186750994
##########
pinot-query-runtime/src/test/resources/queries/MetadataTestQuery.json:
##########
@@ -0,0 +1,66 @@
+{
+ "metadata_test": {
+ "comments": "metadata_test is used to test Tracing and Stats metadata returns",
+ "limitation": "due to current limitation, metadata test requires all data to be located on a single partition, see: https://github.com/apache/pinot/issues/10399",
+ "tables": {
+ "tbl1" : {
+ "schema": [
+ {"name": "col1", "type": "STRING"},
+ {"name": "col2", "type": "INT"}
+ ],
+ "inputs": [
+ ["foo", 1],
+ ["bar", 2]
+ ],
+ "partitionColumns": [ ]
+ },
+ "tbl2" : {
+ "schema": [
+ {"name": "col1", "type": "STRING"},
+ {"name": "col2", "type": "INT"},
+ {"name": "col3", "type": "DOUBLE"},
+ {"name": "partitionCol1", "type": "INT"},
+ {"name": "partitionCol2", "type": "STRING"}
+ ],
+ "inputs": [
+ ["foo", 1, 3.1416, 1, "charlie"],
+ ["foo", 3, 3.1416, 1, "charlie"],
+ ["bar", 2, 2.7183, 1, "charlie"],
+ ["------"],
+ ["bar", 4, 2.7183, 1, "charlie"]
+ ],
+ "partitionColumns": [
+ "partitionCol1", "partitionCol2"
+ ]
+ }
+ },
+ "queries": [
+ {
+ "sql": "SELECT * FROM {tbl1}",
+ "comment": "only 1 segment for tbl1",
+ "expectedNumSegments": 1
+ },
+ {
+ "sql": "SELECT col1, COUNT(*) FROM {tbl2} GROUP BY col1",
+ "comment": "2 segments for tbl2",
+ "expectedNumSegments": 2
+ },
+ {
+ "sql": "SELECT {tbl1}.col1, {tbl1}.col2, {tbl2}.col3 FROM {tbl1} JOIN {tbl2} ON {tbl1}.col1 = {tbl2}.col1",
+ "expectedNumSegments": 3
+ },
+ {
+ "sql": "SELECT {tbl1}.col1, {tbl1}.col2, COUNT(*) FROM {tbl1} JOIN {tbl2} ON {tbl1}.col1 = {tbl2}.col1 GROUP BY {tbl1}.col1, {tbl1}.col2",
+ "expectedNumSegments": 3
+ },
+ {
+ "sql": "SELECT {tbl1}.col1 FROM {tbl1} WHERE {tbl1}.col2 > (SELECT 0.5 * SUM({tbl2}.col3) FROM {tbl2} WHERE {tbl1}.col2 = {tbl1}.col2 AND {tbl1}.col1 = {tbl2}.col1)",
+ "comment": "This correlated subquery test is decorrelated to 2 JOINs, so one table is scanned twice, hence expected to scan 8 segments",
+ "expectedNumSegments": 4
+ }
+ ],
+ "extraProps": {
+ "noEmptySegment": "true"
+ }
+ }
+}
Review Comment:
nit: newline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr commented on a diff in pull request #10728: [multistage] scheduler logic fixes
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr commented on code in PR #10728:
URL: https://github.com/apache/pinot/pull/10728#discussion_r1186779868
##########
pinot-query-runtime/src/test/resources/queries/MetadataTestQuery.json:
##########
@@ -0,0 +1,66 @@
+{
+ "metadata_test": {
+ "comments": "metadata_test is used to test Tracing and Stats metadata returns",
+ "limitation": "due to current limitation, metadata test requires all data to be located on a single partition, see: https://github.com/apache/pinot/issues/10399",
+ "tables": {
+ "tbl1" : {
+ "schema": [
+ {"name": "col1", "type": "STRING"},
+ {"name": "col2", "type": "INT"}
+ ],
+ "inputs": [
+ ["foo", 1],
+ ["bar", 2]
+ ],
+ "partitionColumns": [ ]
+ },
+ "tbl2" : {
+ "schema": [
+ {"name": "col1", "type": "STRING"},
+ {"name": "col2", "type": "INT"},
+ {"name": "col3", "type": "DOUBLE"},
+ {"name": "partitionCol1", "type": "INT"},
+ {"name": "partitionCol2", "type": "STRING"}
+ ],
+ "inputs": [
+ ["foo", 1, 3.1416, 1, "charlie"],
+ ["foo", 3, 3.1416, 1, "charlie"],
+ ["bar", 2, 2.7183, 1, "charlie"],
+ ["------"],
+ ["bar", 4, 2.7183, 1, "charlie"]
+ ],
+ "partitionColumns": [
+ "partitionCol1", "partitionCol2"
+ ]
+ }
+ },
+ "queries": [
+ {
+ "sql": "SELECT * FROM {tbl1}",
+ "comment": "only 1 segment for tbl1",
+ "expectedNumSegments": 1
+ },
+ {
+ "sql": "SELECT col1, COUNT(*) FROM {tbl2} GROUP BY col1",
+ "comment": "2 segments for tbl2",
+ "expectedNumSegments": 2
+ },
+ {
+ "sql": "SELECT {tbl1}.col1, {tbl1}.col2, {tbl2}.col3 FROM {tbl1} JOIN {tbl2} ON {tbl1}.col1 = {tbl2}.col1",
+ "expectedNumSegments": 3
+ },
+ {
+ "sql": "SELECT {tbl1}.col1, {tbl1}.col2, COUNT(*) FROM {tbl1} JOIN {tbl2} ON {tbl1}.col1 = {tbl2}.col1 GROUP BY {tbl1}.col1, {tbl1}.col2",
+ "expectedNumSegments": 3
+ },
+ {
+ "sql": "SELECT {tbl1}.col1 FROM {tbl1} WHERE {tbl1}.col2 > (SELECT 0.5 * SUM({tbl2}.col3) FROM {tbl2} WHERE {tbl1}.col2 = {tbl1}.col2 AND {tbl1}.col1 = {tbl2}.col1)",
+ "comment": "This correlated subquery test is decorrelated to 2 JOINs, so one table is scanned twice, hence expected to scan 8 segments",
+ "expectedNumSegments": 4
+ }
+ ],
+ "extraProps": {
+ "noEmptySegment": "true"
+ }
+ }
+}
Review Comment:
```suggestion
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] walterddr merged pull request #10728: [multistage] scheduler logic fixes
Posted by "walterddr (via GitHub)" <gi...@apache.org>.
walterddr merged PR #10728:
URL: https://github.com/apache/pinot/pull/10728
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org