You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2020/12/07 19:25:59 UTC
[GitHub] [incubator-pinot] jackjlli opened a new pull request #6329: Fix table cache logic in pinot-broker
jackjlli opened a new pull request #6329:
URL: https://github.com/apache/incubator-pinot/pull/6329
## Description
This PR fixes table cache and its related integration test in pinot-broker.
Currently the column name in queries doesn't get converted to lower cases before fetching the real column name in table cache.
And the case insensitivity setting is invalid for **pql** queries, thus changing the tests to send **sql** queries in OfflineClusterIntegrationTest class.
## Upgrade Notes
Does this PR prevent a zero down-time upgrade? (Assume upgrade order: Controller, Broker, Server, Minion)
* [ ] Yes (Please label as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
Does this PR fix a zero-downtime upgrade introduced earlier?
* [ ] Yes (Please label this as **<code>backward-incompat</code>**, and complete the section below on Release Notes)
Does this PR otherwise need attention when creating release notes? Things to consider:
- New configuration options
- Deprecation of configurations
- Signature changes to public methods/interfaces
- New plugins added or old plugins removed
* [ ] Yes (Please label this PR as **<code>release-notes</code>** and complete the section on Release Notes)
## Release Notes
If you have tagged this as either backward-incompat or release-notes,
you MUST add text here that you would like to see appear in release notes of the
next release.
If you have a series of commits adding or enabling a feature, then
add this section only in final commit that marks the feature completed.
Refer to earlier release notes to see examples of text
## Documentation
If you have introduced a new feature or configuration, please add it to the documentation as well.
See https://docs.pinot.apache.org/developers/developers-and-contributors/update-document
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6329: Fix table cache logic in pinot-broker
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6329:
URL: https://github.com/apache/incubator-pinot/pull/6329#discussion_r538844562
##########
File path: pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/SelectAstNode.java
##########
@@ -106,7 +106,7 @@ public String toString() {
public void updateBrokerRequest(BrokerRequest brokerRequest) {
// Set the query source
QuerySource querySource = new QuerySource();
- querySource.setTableName(_resourceName);
+ querySource.setTableName(_tableName);
Review comment:
I think we don't differentiate table name and resource name right now, so maybe a better fix should be in `TableNameAstNode` to remove the `resourceName` and only keep the `tableName`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] codecov-io commented on pull request #6329: Fix table cache logic in pinot-broker
Posted by GitBox <gi...@apache.org>.
codecov-io commented on pull request #6329:
URL: https://github.com/apache/incubator-pinot/pull/6329#issuecomment-742143307
# [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6329?src=pr&el=h1) Report
> Merging [#6329](https://codecov.io/gh/apache/incubator-pinot/pull/6329?src=pr&el=desc) (110af32) into [master](https://codecov.io/gh/apache/incubator-pinot/commit/1beaab59b73f26c4e35f3b9bc856b03806cddf5a?el=desc) (1beaab5) will **decrease** coverage by `1.11%`.
> The diff coverage is `45.12%`.
[](https://codecov.io/gh/apache/incubator-pinot/pull/6329?src=pr&el=tree)
```diff
@@ Coverage Diff @@
## master #6329 +/- ##
==========================================
- Coverage 66.44% 65.33% -1.12%
==========================================
Files 1075 1285 +210
Lines 54773 62034 +7261
Branches 8168 9001 +833
==========================================
+ Hits 36396 40531 +4135
- Misses 15700 18608 +2908
- Partials 2677 2895 +218
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests | `65.33% <45.12%> (?)` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/incubator-pinot/pull/6329?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [...e/pinot/broker/api/resources/PinotBrokerDebug.java](https://codecov.io/gh/apache/incubator-pinot/pull/6329/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckRlYnVnLmphdmE=) | `0.00% <0.00%> (-79.32%)` | :arrow_down: |
| [...ot/broker/broker/AllowAllAccessControlFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6329/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL0FsbG93QWxsQWNjZXNzQ29udHJvbEZhY3RvcnkuamF2YQ==) | `71.42% <ø> (-28.58%)` | :arrow_down: |
| [.../helix/BrokerUserDefinedMessageHandlerFactory.java](https://codecov.io/gh/apache/incubator-pinot/pull/6329/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYnJva2VyL2hlbGl4L0Jyb2tlclVzZXJEZWZpbmVkTWVzc2FnZUhhbmRsZXJGYWN0b3J5LmphdmE=) | `33.96% <0.00%> (-32.71%)` | :arrow_down: |
| [...ker/routing/instanceselector/InstanceSelector.java](https://codecov.io/gh/apache/incubator-pinot/pull/6329/diff?src=pr&el=tree#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcm91dGluZy9pbnN0YW5jZXNlbGVjdG9yL0luc3RhbmNlU2VsZWN0b3IuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...ava/org/apache/pinot/client/AbstractResultSet.java](https://codecov.io/gh/apache/incubator-pinot/pull/6329/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Fic3RyYWN0UmVzdWx0U2V0LmphdmE=) | `66.66% <0.00%> (+9.52%)` | :arrow_up: |
| [.../main/java/org/apache/pinot/client/Connection.java](https://codecov.io/gh/apache/incubator-pinot/pull/6329/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0Nvbm5lY3Rpb24uamF2YQ==) | `35.55% <0.00%> (-13.29%)` | :arrow_down: |
| [...inot/client/JsonAsyncHttpPinotClientTransport.java](https://codecov.io/gh/apache/incubator-pinot/pull/6329/diff?src=pr&el=tree#diff-cGlub3QtY2xpZW50cy9waW5vdC1qYXZhLWNsaWVudC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY2xpZW50L0pzb25Bc3luY0h0dHBQaW5vdENsaWVudFRyYW5zcG9ydC5qYXZh) | `10.90% <0.00%> (-51.10%)` | :arrow_down: |
| [...not/common/assignment/InstancePartitionsUtils.java](https://codecov.io/gh/apache/incubator-pinot/pull/6329/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vYXNzaWdubWVudC9JbnN0YW5jZVBhcnRpdGlvbnNVdGlscy5qYXZh) | `73.80% <ø> (+0.63%)` | :arrow_up: |
| [.../apache/pinot/common/exception/QueryException.java](https://codecov.io/gh/apache/incubator-pinot/pull/6329/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZXhjZXB0aW9uL1F1ZXJ5RXhjZXB0aW9uLmphdmE=) | `90.27% <ø> (+5.55%)` | :arrow_up: |
| [...pinot/common/function/AggregationFunctionType.java](https://codecov.io/gh/apache/incubator-pinot/pull/6329/diff?src=pr&el=tree#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vZnVuY3Rpb24vQWdncmVnYXRpb25GdW5jdGlvblR5cGUuamF2YQ==) | `100.00% <ø> (ø)` | |
| ... and [1133 more](https://codecov.io/gh/apache/incubator-pinot/pull/6329/diff?src=pr&el=tree-more) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6329?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/incubator-pinot/pull/6329?src=pr&el=footer). Last update [8ecfd4a...110af32](https://codecov.io/gh/apache/incubator-pinot/pull/6329?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6329: Fix table cache logic in pinot-broker
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6329:
URL: https://github.com/apache/incubator-pinot/pull/6329#discussion_r539555938
##########
File path: pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/SelectAstNode.java
##########
@@ -106,7 +106,7 @@ public String toString() {
public void updateBrokerRequest(BrokerRequest brokerRequest) {
// Set the query source
QuerySource querySource = new QuerySource();
- querySource.setTableName(_resourceName);
+ querySource.setTableName(_tableName);
Review comment:
I see. You might also want to use `_tableName` on line 172 for updating the pinot query
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #6329: Fix table cache logic in pinot-broker
Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #6329:
URL: https://github.com/apache/incubator-pinot/pull/6329#discussion_r538864342
##########
File path: pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/SelectAstNode.java
##########
@@ -106,7 +106,7 @@ public String toString() {
public void updateBrokerRequest(BrokerRequest brokerRequest) {
// Set the query source
QuerySource querySource = new QuerySource();
- querySource.setTableName(_resourceName);
+ querySource.setTableName(_tableName);
Review comment:
So before this PR, if the query is like `SELECT myDB.myTable FROM ...`, the table name would be parsed as `myDB` instead of `myTable`. In this case,
- For getting table name from table cache in broker, since there is no table called `myDB`, the table name won't be adjusted in `updateTableName` method of `BaseBrokerRequestHandler` class.
- In server, since there is no table called `myDB`, an empty response would be returned.
Table name with dot embedded is actually supported right now. You can refer to the calcite parser logic and the `testCaseInsensitivity` of `OfflineClusterIntegrationTest`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6329: Fix table cache logic in pinot-broker
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6329:
URL: https://github.com/apache/incubator-pinot/pull/6329#discussion_r538846006
##########
File path: pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/SelectAstNode.java
##########
@@ -106,7 +106,7 @@ public String toString() {
public void updateBrokerRequest(BrokerRequest brokerRequest) {
// Set the query source
QuerySource querySource = new QuerySource();
- querySource.setTableName(_resourceName);
+ querySource.setTableName(_tableName);
Review comment:
Also curious about how it causes the failure for PQL. I don't see any table name with dot embedded
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli merged pull request #6329: Fix table cache logic in pinot-broker
Posted by GitBox <gi...@apache.org>.
jackjlli merged pull request #6329:
URL: https://github.com/apache/incubator-pinot/pull/6329
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] Jackie-Jiang commented on a change in pull request #6329: Fix table cache logic in pinot-broker
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #6329:
URL: https://github.com/apache/incubator-pinot/pull/6329#discussion_r537810333
##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
##########
@@ -1307,7 +1308,8 @@ public void testCaseInsensitivity() {
for (String query : queries) {
try {
- postQuery(query);
+ JsonNode response = postSqlQuery(query);
+ Assert.assertTrue(response.get("numSegmentsProcessed").asLong() >= 1L, query + " failed");
Review comment:
You can directly use `assertTrue()`, same for other places
##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
##########
@@ -1307,7 +1308,8 @@ public void testCaseInsensitivity() {
for (String query : queries) {
try {
- postQuery(query);
+ JsonNode response = postSqlQuery(query);
Review comment:
Can you please check why it does not apply to pql? It should apply to both pql and sql
##########
File path: pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
##########
@@ -1002,7 +1002,7 @@ private String getActualColumnName(String rawTableName, String columnName,
columnName = splits[1];
}
if (columnNameMap != null) {
- return columnNameMap.getOrDefault(columnName, columnName);
+ return columnNameMap.getOrDefault(columnName.toLowerCase(), columnName);
Review comment:
Good catch, but how the the test pass before?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #6329: Fix table cache logic in pinot-broker
Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #6329:
URL: https://github.com/apache/incubator-pinot/pull/6329#discussion_r537842320
##########
File path: pinot-broker/src/main/java/org/apache/pinot/broker/requesthandler/BaseBrokerRequestHandler.java
##########
@@ -1002,7 +1002,7 @@ private String getActualColumnName(String rawTableName, String columnName,
columnName = splits[1];
}
if (columnNameMap != null) {
- return columnNameMap.getOrDefault(columnName, columnName);
+ return columnNameMap.getOrDefault(columnName.toLowerCase(), columnName);
Review comment:
We skipped validating the response content in `OfflineClusterIntegrationTest`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #6329: Fix table cache logic in pinot-broker
Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #6329:
URL: https://github.com/apache/incubator-pinot/pull/6329#discussion_r537859696
##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
##########
@@ -1307,7 +1308,8 @@ public void testCaseInsensitivity() {
for (String query : queries) {
try {
- postQuery(query);
+ JsonNode response = postSqlQuery(query);
Review comment:
It seems the table name is set from the resource name, which made the table cache failed to get the correct schema to compare. I've included the fix in this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #6329: Fix table cache logic in pinot-broker
Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #6329:
URL: https://github.com/apache/incubator-pinot/pull/6329#discussion_r537859045
##########
File path: pinot-integration-tests/src/test/java/org/apache/pinot/integration/tests/OfflineClusterIntegrationTest.java
##########
@@ -1307,7 +1308,8 @@ public void testCaseInsensitivity() {
for (String query : queries) {
try {
- postQuery(query);
+ JsonNode response = postSqlQuery(query);
+ Assert.assertTrue(response.get("numSegmentsProcessed").asLong() >= 1L, query + " failed");
Review comment:
Updated
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [incubator-pinot] jackjlli commented on a change in pull request #6329: Fix table cache logic in pinot-broker
Posted by GitBox <gi...@apache.org>.
jackjlli commented on a change in pull request #6329:
URL: https://github.com/apache/incubator-pinot/pull/6329#discussion_r539724816
##########
File path: pinot-common/src/main/java/org/apache/pinot/pql/parsers/pql2/ast/SelectAstNode.java
##########
@@ -106,7 +106,7 @@ public String toString() {
public void updateBrokerRequest(BrokerRequest brokerRequest) {
// Set the query source
QuerySource querySource = new QuerySource();
- querySource.setTableName(_resourceName);
+ querySource.setTableName(_tableName);
Review comment:
Thanks. Addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org